9 minute read
The Data Challenge
70 A recent NCI Agency development challenge set participants the task of improving the detection of Unmanned Aircraft Systems. Mike Bryant asks Adelica Ndoni, a junior data scientist at the NCI Agency, to explain what the judges were looking for
A particular highlight of the recent International Conference on Military Communications and Information Systems (ICMCIS), organized by the NCI Agency in collaboration with NATO’s Science and Technology Organization and held on 4-5 May, was a special session in which the winners of the Agency’s latest challenge presented their solutions to a thorny problem. This year, the participants had to focus on the application of artificial intelligence (AI) and machine learning (ML) in ways that have the potential to enable commanders to make better decisions faster, by providing more comprehensive levels of situational awareness.
Participants in the Class I Unmanned Aircraft Systems (UAS) Challenge were tasked with offering their solution to tracking, classifying and identifying Class I UAS flying within a protected zone using the sensor data made available. Class I UAS weigh less than 150kg; the classification covers most hobby drones, but that is not to say they cannot present a significant threat when flown in prohibited airspace. Those who took up the challenge could fuse together several sources of data provided by the Agency to identify and track the drones.
These sensors included radar and radio direction-finding, data from which had to be assessed to confirm the presence of a Class I UAS – as opposed to, for example, a bird or any other unidentified object flying in the test zone – then classify and identify it (based on a number of features typically found in UAS). If classified as a drone, the UAS had to be tracked and its ongoing speeds and locations recorded. It then had to be identified as a particular type of drone: for instance, a DJI Mavic Pro, DJI Phantom 4 Pro etc.
The UAS Challenge formed part of a larger research and development effort by the NCI Agency, aimed at developing effective remote sensing technologies that are suitable for detecting, tracking and identifying Class I UAS. The challenge supports the efforts that NATO and, hence, the NCI Agency is making to protect its people, facilities and missions against
the threat posed by drones, whether they are simply being misused or whether they are hostile (particularly as part of NATO’s Defence Against Terrorism Programme of Work).
The Dutch Ministry of Defence’s Counter-UAS Nucleus hosted the data-collection phase, the test range being in De Peel, the Netherlands. The NCI Agency’s Counter-Drone Alliance RDO (Research via Development to Operations) Team (CDART) led on data collection, using sensors provided by NATO nations and industry. Drones were flown into a circular area, with a radius of about 1.5km, over a three-day period in September and October last year, in order to collect the necessary sensor data to which the challenge participants were given access. The flight duration of each UAS mission was between 10 and 20 minutes.
INNOVATIVE SOLUTIONS
“We were looking for innovative solutions, ones using the most state-of-the-art in machine-learning technology,” explains Adelica Ndoni, a junior data scientist at the NCI Agency, who worked on the challenge. She and her colleague, senior scientist Cristian Coman, were heavily involved in both designing and launching the challenge.
Just as the fully virtual ICMCIS conference focused on the application of artificial intelligence (AI) and ML to the areas of military situational awareness and decision- making, so the UAS Challenge tested the participants’ use of the latest machine-learning techniques to detect and identify small unmanned vehicles. “The nature of the participants didn’t matter,” says Ndoni. “Anyone was welcome to participate in the challenge and provide their solution to our challenge’s tasks.”
A customized evaluation metric was used to compare every submission against the true data included in the log files of each of the UAS flights, which is owned by the NCI Agency for evaluation purposes and was not released to the participants. This involved the comparison of the various values provided in the applicants’ submissions (for positional data, classification and identification of the UAS) against the true data – the lower the mean error (ie the less the deviation between a submission’s values and that of the true data), the better the proposed solution.
As for how the participants’ various ideas may be deployed, Ndoni explains that this challenge has been all about “exploring more innovative technologies in machine learning for improving the performance of drone detection”. It does not form any part of an acquisition process as of now. “This project is part of the NCI Agency’s overall support to NATO,” she explains. “It was a difficult challenge, but there was great interest from participants.” Only the top four submissions saw their proposers asked to share their approach in more depth with a wider community, in a special session on counter-UAS and radio-frequency technologies at the virtual conference, where the solutions they arrived at were presented and assessed.
“A follow-on counter UAS exercise will be held in November this year,” concludes Ndoni. “We thank all those who participated in the challenge, and we are looking forward to meeting them and others in our next challenge.”
THE WINNERS
CERTH ITI VARlab proposed a tracker based on machinelearning technologies
Defence Science & Technology Laboratories
used open-source tracking and data-fusion techniques
CERTH ITI Visual Computing
Lab used techniques including the Hungarian algorithm to solve the data association problem
Horizon Lab scored high in the Mean Root Square Error parameter used in ranking 71
INDUSTRY PERSPECTIVE
Mario Hempel
Business Director NATO, Bechtle
How will the recent agreement to supply devices continue to enhance NATO capabilities?
Since 2017, Bechtle has supported the NCI Agency as a trusted partner with two IT modernization contracts – ITM WP2 and Horizon 3. We are pleased that this successful partnership continues with the current e-FIT contract. The purpose of this contract is to create a framework that enables frequent and rapid transactions of client end-user equipment and related support from our product catalogue, which includes a wide range of end-user equipment provided to the recipients by the NCI Agency. In addition to NATO as a whole, the beneficiaries also include Member States’ defence ministries.
We are advising the Agency on all technical aspects of the Agency’s future client infrastructure, including the selection and certification of all technical components and equipment for COTS and TEMPEST products.
What sort of devices are covered and how are they being integrated into the NATO network?
In principle, the contract covers the entire spectrum of client infrastructure. The product range that we offer includes mobile devices, monitors and peripherals, as well as stationary computing solutions. A particular focus of our product catalogue is IT security and associated TEMPEST products. With six TEMPEST engineers and 19 TEMPEST technicians, our partner, GBS, has a comprehensive competence team available with many years of experience, able to execute technically demanding and challenging projects in the shortest of timeframes.
GBS operates four emanations test laboratories, recognized by the German Federal Office for Information Security (BSI), and is also authorized to perform certification measurements and device-specific shortmeasurement procedures according to SDIP 27 Level A, B and C for all relevant workplace equipment covering systems and peripherals.
The integration of the systems into existing NATO networks is a central part of our contract with the Alliance. With the help of security-cleared personnel, we are able to perform onsite rollouts and connect the systems we deliver to the NATO networks. Also, the pre-installation of NATO-specific images and the pre-configuration of devices is possible in our Bechtle facilities.
How is Bechtle working with the NCI Agency and other partners on the contract?
On the NCI Agency side, we work closely with the procurement and logistics departments. They are our primary contact in managing the contract. As the contract holder, the NCI Agency acts as the link between NATO’s end customers and Bechtle as the supplier. We ensure that the product catalogue is always up to date and advise on the customers’ needs and how we can meet them, by adding new products and/or adjusting delivery schedules, for example. On our side, our main partners are Dell and GBS TEMPEST & Service GmbH. Both companies are highly respected and well established within the defence sector. Naturally, the quality of their products and delivery capabilities meet the high standards demanded by the NCI Agency. Dell hardware forms the baseline for our COTS and TEMPEST products.
TEMPEST is a highly complex IT niche within military and government organisations, and only a few companies have the technical resources, know-how, expertise and required national accreditation and certification to provide these highly secure systems in such a short time frame.
Which other security/ military organizations does Bechtle support?
Bechtle has been a partner to various armed forces and security authorities for many years, delivering hardware and services. National and international security authorities – such as the Austrian, Dutch, German and Swiss Armies – rank among our customers and benefit from our branch-specific know-how and experience.
Our commitment to the German Armed Forces deserves special mention. For many years, we have been a partner for the design, procurement, delivery and integration of IT hardware. As with the NCI Agency, our success in this is thanks to the strategic partnership with GBS TEMPEST & Service GmbH, which has been in existence for over 10 years.
Partner for NATO. Service Provider for e-FIT.
Bechtle AG.
As leading B2B IT provider, Bechtle has some 70 systems integrators in Germany, Austria and Switzerland, as well as e-commerce subsidiaries across Europe – a blend of IT sales and services that is truly unique at this scale. First established in 1983, Bechtle has been on an upwards trajectory ever since. Over 70,000 customers from diverse businesses within the industrial, service and public sectors rely on the expertise of Bechtle’s employees and the combined strength of the Bechtle Group. More than 10,000 people are committed to writing Bechtle’s success story everyday. Bechtle has been listed on the stock exchange since 2000 and is listed in the MDAX and TecDAX indexes. In 2020, the group generated revenues of around 5,8 billion euros.
e-FIT Framework Contract –Provision of COTS IT Hardware and associated support.
Since May 2021, Bechtle has been the contract holder of the NCIA framework contract „e-FIT“. Through this contract, NATO customers can purchase client end-user equipment and associated services. Our product catalog includes, among others:
� COTS IT hardware. � TEMPEST hardware acc. to NATO Standard SDIP 27 Level A, B and C. � Services (logistics, roll-out, imaging, support).
For further information please contact us by mail to „nato.order.bonn@bechtle.com“.
GBS TEMPEST & Service GmbH.
Based in Diepholz, Lower Saxony, Germany, the GBS owns and operates four inhouse emanations test laboratories recognized by the German Federal Office for Information Security (BSI). The GBS is authorized to perform certification measurements and short-measurement procedures according to the German zoning model (national) as well as to perform certification measurements and short-measurement procedures according to SDIP 27 Level A, B and C.
Furthermore, the GBS has a comprehensive competence team available which consists of highly skilled TEMPEST engineers and TEMPEST technicians with many years of experience. In addition, the GBS is maintaining a certified Quality Management System according to ISO 9001:2015.
