Feature Article:
DOI. No. 10.1109/MAES.2019.2933972
Deep Learning for Classification of Mini-UAVs Using Micro-Doppler Spectrograms in Cognitive Radar Albert Huizing, Matijs Heiligers, Bastiaan Dekker, Jacco de Wit, Radar Technology Department, TNO, The Hague, The Netherlands Lorenzo Cifola, Ronny Harmanny, Advanced Development Thales Nederland B.V., Delft, The Netherlands
INTRODUCTION Military radar can not only provide kinematic information (position, speed, and course) of land, sea, and air targets during day and night in all weather conditions, but also provides information about the type of target using microDoppler properties. The micro-Doppler properties of a target are determined by the moving parts on the target body. The number, location, and type of motion of these parts are specific for a given target type and consequently the related micro-Doppler signature can be exploited for classification. Analysis of the micro-Doppler signature may provide detailed properties of rotating parts, such as the rotation rate, number of blades, and blade length. However, for operators and/or image analysts, the interpretation and understanding of radar micro-Doppler spectrograms is much more difficult and time-consuming than the analysis of optical images because of the different nature of the radar imaging principle and target scattering mechanisms. Consequently, there is a need for automatic target recognition (ATR) in radar using micro-Doppler spectrograms. An ability to automatically recognize complex patterns in radar signals is one of the key elements of the perception–action cycle in cognitive radar as identified by Haykin et al. [1]. A cognitive radar uses the feedback from the receiver to adjust the transmitted waveforms to
Authors’ current addresses: Albert Huizing, Matijs Heiligers, Bastiaan Dekker, Jacco de Wit, Radar Technology Department, TNO, 2594 AK, The Hague, The Netherlands, (E-mail: albert.huizing@tno.nl). Lorenzo Cifola, Ronny Harmanny, Advanced Development, Thales Nederland B.V. 2628 XH, Delft, The Netherlands. Manuscript received March 27, 2019, revised July 12, 2019, and ready for publication August 6, 2019. Review handled by P. Willett. 0885-8985/19/$26.00 ß 2019 IEEE 46
operate effectively and robustly in a dynamic environment. In addition to the feedback from the receiver, a cognitive radar can also use prior knowledge about the environment and targets of interest to optimize its performance [2]. This prior knowledge can be provided by human experts in the form of heuristics, terrain databases, or computer models of the interaction between the radar and the environment, including target signatures, such as micro-Doppler spectrograms. Alternatively, prior knowledge can also be derived from previous radar measurements in the same or similar environments [3]. Examples of this are a land clutter map and micro-Doppler spectrograms of targets of interest. Figure 1 shows a high-level block diagram that incorporates two of the key elements of a cognitive radar: a perception–action cycle and priori knowledge stored in longterm memory, i.e., a knowledge base (KB). This diagram has been adapted from the cognitive radar architecture proposed by Guerci [2]. A knowledge-aided processor (KAP) uses prior knowledge in the KB to control the radar scheduler, the waveforms transmitted by the adaptive transmitter, the radio frequency filter parameters and the sampling rate in the adaptive receiver, and the parameters and configuration of the radar signal processing. The signal processing typically includes functions, such as pulse compression, Doppler filtering, detection, tracking, and classification. The KAP may decide to update the KB based on information that is extracted from the radar signal. For example, if the number of false alarms in an area is too large due to a change in the clutter distribution, the KAP may decide to update the clutter map. In conventional radars, the recognition of targets, such as mini-UAVs typically relies on the extraction of predefined target features from the received radar signal. These target features are defined and implemented by a human expert in the radar signal processing, often after a long and tedious analysis of large amounts of radar data. Instead of this hand-crafted feature engineering approach, a cognitive radar can use deep learning techniques to
IEEE A&E SYSTEMS MAGAZINE
NOVEMBER 2019
Authorized licensed use limited to: Khwaja Fareed University of Eng & IT. Downloaded on July 04,2020 at 01:10:19 UTC from IEEE Xplore. Restrictions apply.
automatically extract the relevant features from a set of radar measurements or radar simulations that are stored in the KB. The extracted features can then be used by the KAP to classify the target. Deep learning has become the preferred method for many pattern recognition applications, such as segmentation, speech recognition, and face recognition since the quantum leap in image recognition performance realized with the AlexNet convolutional neural network (CNN) in the ImageNet 2012 challenge [4]. Despite the major achievements of deep learning techniques, such as CNNs in the past few years, it is not obvious that the success of deep learning in the commercial domain can be replicated in the military domain. Large labeled datasets, which are the key to the success of most commercial applications of deep learning, are often not available in military applications. In addition, the cost of decision errors in the military domain is typically much higher than in the commercial applications. This makes it mandatory to achieve a robust performance with a low error rate. Furthermore, military ATR algorithms should not only be accurate, but their behavior should also be predictable to gain the trust of military commanders. One of the first applications of deep learning for ATR using synthetic aperture radar imagery was presented by Morgan [5]. Since then many papers on the application in radar of CNNs and other types of deep neural networks have been published, including the application of deep learning for target classification, and human gait and gesture recognition using micro-Doppler spectrograms [6]–[9]. This paper investigates the potential of deep learning techniques for the classification of mini-UAVs using microDoppler spectrograms in the context of cognitive radar. The paper also presents preliminary results on the use of deep learning in the preprocessing and training process of a classifier. This paper is organized as follows. “CLASSIFICATION OF MINI-UAVS” shows how deep neural networks can be applied for the classification of mini-UAVs using measured or simulated sets of micro-Doppler spectrograms that are stored in the long-term memory (or knowledge base) of a cognitive radar. “DETECTION OF UNKNOWN TARGET NOVEMBER 2019
CLASSES” describes how the spectrograms of target classes that are not represented in the training set, i.e., targets unknown by the cognitive radar, can be detected with deep neural networks. The detection of an unknown target can provide a trigger for the radar scheduler to collect micro-Doppler spectrograms of this target for inclusion in the training set, i.e., the knowledge base. “DENOISING OF SPECTROGRAMS” shows how a trained neural network can be used to denoise spectrograms. This denoising process may enable a cognitive radar to recognize targets at a longer range. “ADVERSARIAL TRAINING FOR SPECTROGRAM GENERATION” describes the generation of new spectrograms for training a classifier using generative adversarial networks. Finally, in “CONCLUSION,” conclusions are drawn with respect to the application of deep learning for classification of mini-UAVs with cognitive radar.
CLASSIFICATION OF MINI-UAVS The commercial availability of compact electronics and advanced open source software has led to a proliferation of mini-UAVs that can be used for many different purposes including criminal, terrorist, and military activities. Owing to their relatively slow speed and small size, they are hard to distinguish from natural targets, such as birds. Recently, radar signal processing techniques using features extracted from spectrograms and cepstrograms have been developed to discriminate birds and mini-UAVs [10], [11]. The recognition of different classes of miniUAVs is, however, a more difficult problem due to the overlapping features and rapidly changing characteristics of mini-UAVs. Therefore, an enhanced capability for radars is needed to recognize the type of mini-UAV for threat evaluation and assignment of countermeasures.
MINI-UAV MEASUREMENTS To investigate the potential of deep learning techniques for the recognition of mini-UAVs, radar measurements of five different types of mini-UAVs have been acquired
IEEE A&E SYSTEMS MAGAZINE
47
Authorized licensed use limited to: Khwaja Fareed University of Eng & IT. Downloaded on July 04,2020 at 01:10:19 UTC from IEEE Xplore. Restrictions apply.
Deep Learning for ClassiďŹ cation of Mini-UAVs Using Micro-Doppler Spectrograms in Cognitive Radar
Figure 1. High-level block diagram of a cognitive radar (adapted from Guerci [2]).
during a measurement campaign at the NLR RPAS Test Centre in Marknesse, The Netherlands. The mini-UAVs included two types of fixed wing aircraft (Robbe Air Trainer 140 and Sky Walker X8), an Align T-REX 550 helicopter, a Mikrokopter QuadroXL quadcopter, and a Mikrokopter OktoXL octocopter, see Figure 2. Measurements of the mini-UAVs were conducted with an experimental continuous wave radar that operates at a frequency of 9.5 GHz and samples the radar signal at a rate of 96 kHz. The complex radar signals are converted to micro-Doppler spectrograms using a short time fourier transform (STFT). The Doppler frequency of the mini-
UAV body is removed to center the Doppler spectrum around 0 Hz. Examples of the measured micro-Doppler spectrograms for all 5 mini-UAVs can be seen in Figure 2. The coherent integration time of the spectrograms is 10.7 ms. The modulations of the radar signal due to the moving parts can clearly be observed.
MICRO-DOPPLER SIGNAL SIMULATIONS In addition to the measurements, micro-Doppler signals have been simulated using computer models of mini-
Figure 2. Photos (1st row), measured spectrograms (2nd row), computer models (3rd row), and simulated spectrograms (4th row) of five mini-UAVs.
48
IEEE A&E SYSTEMS MAGAZINE
NOVEMBER 2019
Authorized licensed use limited to: Khwaja Fareed University of Eng & IT. Downloaded on July 04,2020 at 01:10:19 UTC from IEEE Xplore. Restrictions apply.
Huizing et al.
Figure 3. Workflow for the classification of mini-UAVs with deep neural networks using micro-Doppler spectrograms.
UAVs to test the idea that a deep neural network can be trained with simulated micro-Doppler spectra. The trained neural network can then be used to classify micro-Doppler spectra measured with a real radar system during operations. This concept would reduce the need for expensive and time-consuming radar measurements to populate the knowledge base of a cognitive radar. It also could enable the classification of mini-UAVs of which physical characteristics are available, but no measurements can be made due to security reasons. The simulation of micro-Doppler spectra of targets, such as mini-UAVs is difficult because of the complex mechanical structure of these targets and the time varying characteristic of the radar cross section (RCS) due to the moving parts. To limit the computation time for microDoppler signal simulations, an approach was chosen in which the mini-UAVs are represented with so-called principal scatterers for which the analytic expressions of their RCS exist [12]. Examples of these principal scatterers are point scatterers, cylinders, ellipsoids, flat triangular plates, and thin wires. For each sample of the radar signal, the position and attitude of each principal scatterer is determined and the associated RCS for this scatterer is computed. Finally, the reflected radar signal for the entire target is calculated by adding the contributions of all principal scatterers. This simulation approach is relatively fast, but neglects the occlusion of some scatterers by other scatterers on the body, multiple reflections, diffraction, and materials other than metal. Figure 2 shows the computer models of the five mini-UAVs considered in this paper and examples of the associated simulated micro-Doppler spectrograms. Figure 2 clearly shows a resemblance between the measured and the simulated micro-Doppler spectrograms, but also some measured characteristics that are not present in the simulated spectrograms. NOVEMBER 2019
CLASSIFICATION OF MINI-UAVS WITH DEEP NEURAL NETWORKS Figure 3 shows the workflow for the classification of miniUAVs with deep neural networks. The training phase consists of generating a training set with (a) simulated radar signals using target models or (b) real target signals measured with a radar. Micro-Doppler spectrograms are created by applying an STFT successively to overlapping sequences of the time-domain signals. The preprocessing includes removal of the speed of the mini-UAV body, and normalization of the power spectrogram. In the test phase, the trained neural network is used to classify the measured radar signals. Table 1 shows the number of micro-Doppler spectrograms that have been extracted from the measured and simulated radar signals for the training and test sets in the mini-UAV classification experiments described below.
Table 1.
Micro-Doppler Spectrograms for the Five Mini-UAVs Training
Test
Target Simulated
Measured
Simulated
Measured
Air trainer
506
570
525
499
Skywalker
475
886
545
775
T-REX 550
965
1284
1080
1092
QuadroXL
477
841
546
869
OktoXL
480
1370
555
1222
IEEE A&E SYSTEMS MAGAZINE
49
Authorized licensed use limited to: Khwaja Fareed University of Eng & IT. Downloaded on July 04,2020 at 01:10:19 UTC from IEEE Xplore. Restrictions apply.
Deep Learning for Classification of Mini-UAVs Using Micro-Doppler Spectrograms in Cognitive Radar Table 2.
Deep Neural Network Test Cases for Classification of Mini-UAVs Using Micro-Doppler Spectrograms Case
Training data
Test data
Neural network
Sampling rate
Integration time
Accuracy
A
Simulated
Simulated
CNN
96 kHz
10.7 ms
91.7%
B
Simulated
Measured
CNN
96 kHz
10.7 ms
36.7%
C
Measured
Measured
CNN
96 kHz
10.7 ms
82.8%
D
Measured
Measured
CNN
24 kHz
10.7 ms
87.1%
E
Measured
Measured
CNN
24 kHz
2.7 ms
97.7%
F
Measured
Measured
LSTM-RNN
24 kHz
2.7 ms
89.0%
CLASSIFICATION EXPERIMENTS AND RESULTS To investigate the accuracy of different types of deep neural networks for the classification of mini-UAVs, several test cases were defined, see Table 2. Test case A involves the use of a CNN with a high sampling rate (96 kHz) of the radar signal and a medium coherent integration time (10.7 ms) in the Doppler filtering and simulated training and test data. Test case B gives an idea if the simulated spectrograms can be used to classify measured spectrograms from real mini-UAVs. Test case C uses measured spectrograms for training and testing. Test case D also uses measured spectrograms for training and testing, but employs a reduced sampling rate by decimating the radar signal with a factor 4. Test case E uses a low sampling rate and a short integration time (2.7 ms). Finally, test case F uses a long short-term memory (LSTM) recurrent neural network (RNN) instead of a CNN as a classifier [14]. Table 3 shows the configuration of the CNN used in the first three test cases (A, B, and C). All convolutional and fully connected layers (except for layer 13 which is linear) use a rectified linear unit as a nonlinear activation function. The convolutional and max-pool layers of the CNNs in test cases D and E have smaller sizes than the configuration in Table 3 due to the reduced sampling rate and integration time. The CNN is trained with a categorical cross-entropy loss function and the ADAM optimizer [5]. The LSTM RNN in test case F does not use a spectrogram as an input, but operates on an input sequence of Doppler spectra. The LSTM-RNN configuration has four layers and a hidden state size of 256 for the LSTM cells. A comparison of the mean classification accuracy in Table 2 shows that good results are achieved except for the case where simulated spectrograms are used for training a CNN and measured spectrograms are used during testing. This is perhaps not surprising due to the lack of fidelity in the target models, which do not include 50
shadowing, multiple reflections, diffraction, and material properties. More advanced RCS modeling techniques using a hybrid finite element method and method of moments approach may improve the fidelity of the microDoppler spectrograms at the expense of computational cost [13]. A second conclusion is that the decimation of the sampling rate by a factor of 4 improves the mean classification accuracy for the measured spectrograms considerably. This is due to the fact that the part of the spectrogram that does not contain any relevant information is removed by
Table 3.
CNN Configuration Layer
Type
Size
Kernel
Filters
1
Input
1024 177
–
–
2
Convolution
1024 177
5 5
32
3
Convolution
1024 177
5 5
32
4
Max-pool
512 88
2 2
32
5
Convolution
512 88
5 5
64
6
Convolution
512 88
5 5
64
7
Max-pool
256 44
2 2
64
8
Convolution
256 44
5 5
64
9
Convolution
256 44
5 5
64
10
Max-pool
128 22
2 2
64
11
Flatten
1 180224
–
–
12
Fully con.
1 512
–
–
13
Fully con.
1 5
–
–
14
Soft-max
1 5
–
–
IEEE A&E SYSTEMS MAGAZINE
NOVEMBER 2019
Authorized licensed use limited to: Khwaja Fareed University of Eng & IT. Downloaded on July 04,2020 at 01:10:19 UTC from IEEE Xplore. Restrictions apply.
Huizing et al.
Figure 4. Adversarial training of an auto-encoder with the GANomaly method to detect spectrograms of unknown targets.
the decimation. This simple and effective preprocessing step is apparently difficult to learn by the CNN from the limited number of training samples in test case C. Another conclusion is that the reduction in coherent integration time in the Doppler filtering from 10.7 ms to 2.7 ms also gives a significant improvement in classification accuracy. This improvement can be explained by the fact that the modulations by the rotating propeller or rotor blades are less smeared in the Doppler spectrum due to the shorter integration time, and the changes in the Doppler spectrum become clearer. Although the use of an LSTM-RNN proved to be less accurate than the CNN in the overall classification, an important advantage of an LSTM-RNN is that it can provide good classification results already after a few coherent processing intervals, whereas the CNN only provides a classification result after an entire spectrogram has been processed [14]. Another advantage of the LSTM-RNN is that it is capable to deal with transitions in target behavior, e.g., a fixed wing mini-UAV that takes off vertically, and then, proceeds to horizontal flight.
DETECTION OF UNKNOWN TARGET CLASSES During military operations, cognitive radars will often be employed in conditions where novel target classes are observed of which there are no examples of micro-Doppler spectrograms in the training set of the deep neural network. In this case, the cognitive radar should detect the presence of this unknown target class and may decide to schedule specific radar measurements to add spectrograms of the unknown target class to the training set. The detection of unknown target classes can be achieved in several ways. The simplest approach is the NOVEMBER 2019
application of a threshold to the output of the Soft-max layer of a CNN [15]. If the maximum output of the Softmax layer does not exceed the threshold, an unknown target class is declared. This Soft-max approach is also referred to as a reject option for a classifier [16]. By varying the value of the threshold, a receiver operation characteristic (ROC) curve can be obtained, which shows the probability of detecting an unknown target class (true positive rate) as a function of the probability of falsely declaring an unknown target class (false positive rate). An alternative to the Soft-max approach called GANomaly is based on an auto-encoder that is trained in an adversarial manner [17]. This technique has been used to investigate the potential of deep learning to screen aviation luggage for anomalous items using X-ray screening. Figure 4 shows the configuration of the GANomaly network that consists of three subnetworks. The auto-encoder acts as a generator network that learns a compact latent space representation z of the input spectrograms x by trying to reconstruct the input spectrogram. The encoder compresses the reconstructed spectrogram to an estimate of the latent representation using the same architecture as the encoder in the auto-encoder. The discriminator determines, with an encoder architecture, if the input is a real spectrogram or a fake spectrogram. The detection of a spectrogram originating from an unknown target assumes that the auto-encoder is not able to reconstruct such a spectrogram accurately because the network is trained only with spectrograms from known target classes. A reconstructed spectrogram for an unknown spectrogram will also lead to discrepancies between the latent vector and its estimate. This discrepancy is used by the GANomaly method during the test phase to detect a spectrogram of an unknown target class.
IEEE A&E SYSTEMS MAGAZINE
51
Authorized licensed use limited to: Khwaja Fareed University of Eng & IT. Downloaded on July 04,2020 at 01:10:19 UTC from IEEE Xplore. Restrictions apply.
Deep Learning for Classification of Mini-UAVs Using Micro-Doppler Spectrograms in Cognitive Radar
Figure 5. ROC curve showing the true positive rate versus the false positive rate for detecting an unknown Align T-REX 550 mini-UAV with the Soft-Max and GANomaly methods.
The objective function for training the GANomaly networks consists of three loss functions. The adversarial loss function Ladv computes the L2 distance between features computed by the function f from an intermediate layer of the discriminator network for the real and generated spectrograms. This feature matching approach reduces the instability of GANs during training [18]. The context loss function Lcon penalizes the auto-encoder for reconstruction errors by measuring the L distance between the real and generated spectrograms. Finally, the encoder loss Lenc measures the L1 distance between the latent representations of the input and reconstructed spectrograms. Experiments with the Soft-max and GANomaly methods for the detection of unknown target classes have been conducted using the dataset with mini-UAV spectrograms, as described in “CLASSIFICATION OF MINI-UAVS.” Each of the mini-UAVs has been used alternatively as an unknown target class when training the neural networks on the microDoppler spectrograms from the remaining ‘“known” miniUAVs. Figure 5 shows an example of an ROC curve for a test where the T-REX 550 helicopter was used as an unknown target class. The GANomaly method performs better at low and high false positive rates, whereas the Soft-max method works better at intermediate false positive rates. Figure 6 shows the area-under-curve (AUC) for the detection of unknown target classes with the Soft-max and GANomaly methods. Each experiment is performed three times to capture the performance variations due to the random initialization of the neural networks and due to the random selection of the data samples in the test set and training set. The above results are obtained using the measured spectrograms of test case E. All spectrograms are truncated after the first 64 time samples resulting in square input data of 64 64 pixels. This input size enabled the use of network configuration and training parameters described in the paper by Akcay et al. [17] that are proven to be successful for the GANomaly method. For Soft-max, the 52
Figure 6. AUC results for the detection of unknown target classes with Softmax and GANomaly. The error bars represent variations due to the use of random seeds for the initialization of the neural networks.
network and hyperparameters are as described in “CLASSIFICATION OF MINI-UAVS.” The models for GANomaly and Soft-max are trained for, respectively, 15 and 50 epochs. Figure 6 shows that both the Soft-max and the GANomaly achieve AUC ranging from 0.5 to 0.8. Note that comparing the performance of deep learning methods is often tricky because the result can highly depend on the effort that has been put in optimizing the networks, training parameters, and data preprocessing steps. Especially adversarial training methods, such as GANomaly, are notorious for instability issues. In the experiments, it was observed that the final performance of the GANomaly method was not yet fully satisfactory, and perhaps better performances can be achieved with other GAN configurations for the detection of unknown target classes.
DENOISING OF SPECTROGRAMS The micro-Doppler spectrograms collected during the measurement campaign described in “CLASSIFICATION OF MINI-UAVS” have a relatively high signal-to-noise ratio (SNR) due to the short range between the radar and the mini-UAVs. However, the SNR will be lower at longer ranges, and therefore, the performance of a target classifier will degrade. To investigate if deep learning can be used to improve the SNR of the spectrograms in the preprocessing before classification, and hence, increase the range at which a cognitive radar can recognize a mini-UAV, an experiment was conducted with a denoising adversarial auto-encoder. Adversarial trained auto-encoders can be used to restore input data, such as images that are corrupted by noise [19]. The architecture of the denoising adversarial auto-encoder is like the architecture of the
IEEE A&E SYSTEMS MAGAZINE
NOVEMBER 2019
Authorized licensed use limited to: Khwaja Fareed University of Eng & IT. Downloaded on July 04,2020 at 01:10:19 UTC from IEEE Xplore. Restrictions apply.
Huizing et al.
Figure 7. Denoising adversarial auto-encoder applied to spectrograms corrupted with synthetic noise.
GANomaly network, as shown in Figure 4, except that the separate encoder is omitted and the objective function for training of the auto-encoder only consists of the mean square error between noise corrupted versions of the spectrograms and the uncorrupted spectrograms. Figure 7 shows two examples of measured microDoppler spectrograms of a T-REX 550 helicopter with a high SNR in the first column. The second column shows the same spectrograms to which synthetic Gaussian noise has been added to lower the SNR. The third column shows the reconstructed spectrograms using a denoising adversarial auto-encoder. Although some details of the microDoppler modulations are distorted, it is still possible to recognize the micro-Doppler contribution due to the tail rotor and, up to a certain extent, the blade tip contribution. In a second experiment, the denoising adversarial auto-encoder has been tested with measured spectrograms that are characterized by a lower SNR. As apparent from the reconstructed spectrograms shown in Figure 8, a signal to background ratio enhancement in the order of 20 dB is achieved by the denoising auto-encoder. Furthermore, information associated to tail rotor signature and blade tip rotation is partially restored. Although the SNR of the spectrograms seems to be significantly enhanced by the denoising adversarial auto-encoder, the impact of this preprocessing on the accuracy of a target classifier, such as a CNN still has to be investigated.
imbalance in the training set used in this paper (see Table 1) is approximately a factor of two. This imbalance can have a negative impact on the convergence of the classifier during training and the overall accuracy [20]. There are several ways to mitigate the effect of an imbalanced training set. The first category concerns methods that leave the training set intact and modify the training procedure or the classifier to deal with the imbalance. The second category involves changes to the training set itself by generating data for classes that are underrepresented. In this section, a generative network called InfoGAN has been investigated for the generation of realistic training data for underrepresented target classes [21]. When compared with a standard GAN, an InfoGAN has the advantage that it learns in an unsupervised way, interpretable and disentangled representations of challenging datasets. Figure 9 shows the architecture of an InfoGAN network
ADVERSARIAL TRAINING FOR SPECTROGRAM GENERATION Training sets for target classifiers based on machine learning are often characterized by an imbalance in the number of examples for different target classes. For example, the NOVEMBER 2019
Figure 8. Denoising adversarial auto-encoder applied to measured low SNR spectrograms.
IEEE A&E SYSTEMS MAGAZINE
53
Authorized licensed use limited to: Khwaja Fareed University of Eng & IT. Downloaded on July 04,2020 at 01:10:19 UTC from IEEE Xplore. Restrictions apply.
Deep Learning for ClassiďŹ cation of Mini-UAVs Using Micro-Doppler Spectrograms in Cognitive Radar
Based on an information-based regularization term within the overall training cost function, a structure is introduced in the latent variable that enables the aggregation of spectrograms characterized by similar characteristics. This is apparent in the generated spectrograms shown in Figure 10 for nine different categories associated with nine different states for the latent code (c). In addition, since clustering is based on spectrogram appearance, instances of the same class characterized by differing spectrograms will be associated to different categories. For example, both examples from category 2 and 3 belong to the T-REX 550 helicopter class, although characterized by a different Doppler bandwidth. When the specified categories are higher than the level of diversity among the dataset examples, then overlap between categories can be expected, for instance as observed for categories 5 and 9.
Figure 9. InfoGAN architecture.
that consists of a generator, a discriminator, and an auxiliary unit. The generator input vector consists of the concatenation of two parts: a noise vector (z) and a latent code (c). The generator tries to fool the discriminator by generating a realistic spectrogram, whereas the discriminator tries to distinguish the synthetic and real spectrograms. In addition to the probability that an input is a real or a fake spectrogram, the discriminator also computes a distribution Q(c j G(c,z)) that measures the mutual information between the latent code c and the generated spectrogram.
CONCLUSION The results of several experiments in this paper have demonstrated the potential of deep learning techniques for the classification of mini-UAVs with a cognitive radar. Deep neural networks, such as CNNs and RNNs can provide an accurate classification performance after being trained on a set of measured micro-Doppler spectrograms that are stored in the long-term memory of a cognitive radar. The analysis of the classification performance of different types of deep neural networks and different preprocessing parameters showed that the choice of the coherent
Figure 10. InfoGAN applied to micro-Doppler spectrograms with examples of synthesized spectrograms per category of the latent code.
54
IEEE A&E SYSTEMS MAGAZINE
NOVEMBER 2019
Authorized licensed use limited to: Khwaja Fareed University of Eng & IT. Downloaded on July 04,2020 at 01:10:19 UTC from IEEE Xplore. Restrictions apply.
Huizing et al.
integration time in the Doppler filtering had a higher impact on the classification performance than the type of the deep neural network (CNN versus LSTM-RNN) in the classifier. The use of simulated spectrograms to classify measured spectrograms requires a higher fidelity modeling of the micro-Doppler signals than the modeling approach based on principal scatterers used in this paper. When a cognitive radar measures a micro-Doppler spectrogram from a target class that is not represented in the training set, an anomaly detection algorithm could provide a trigger to the cognitive radar scheduler to collect micro-Doppler spectrograms from this unknown target. The performance of two deep learning techniques, Softmax and GANomaly, for the detection of spectrograms from unknown target classes has been compared. Both techniques had a similar performance with respect to the probability of detecting unknown classes with the GANomaly technique showing a larger performance variation due to the difficulty in training adversarial networks. Different GAN architectures for anomaly detection that are easier to train, such as a Wasserstein GAN may provide a better performance [22]. A denoising adversarial auto-encoder has been trained to remove noise from spectrograms. An improvement of 20 dB in the SNR of a measured spectrogram has been observed. If this denoising technique does not alter the micro-Doppler characteristics significantly, a considerable increase in the range at which a cognitive radar can recognize mini-UAVs could be achieved. To mitigate the effects of class imbalances in the number of training examples on the performance of a classifier, an InfoGAN can be used to generate additional spectrograms of underrepresented classes. A quantitative study concerning the benefits of this approach will be conducted in the future research. The overall conclusion of this paper is that deep learning techniques have a high potential for improving the classification of mini-UAVs using micro-Doppler spectrograms in cognitive radar. However, the selection and tuning of the appropriate neural network architectures for different processing tasks, such as classification and denoising is tedious and time consuming. A better understanding of the theory behind deep neural networks, potentially enabled by the link with compressive sensing and sparse signal representations, would not only accelerate the development process, but also improve the acceptance of these advanced techniques in military radar applications.
Dutch Radar Center of Expertise (D-RACE), a strategic alliance between Thales Nederland B.V. and TNO.
REFERENCES [1] S. Haykin, Y. Xue, and P. Setoodeh, “Cognitive radar: Step toward bridging the gap between neuroscience and engineering,” Proc. IEEE, vol. 100, no. 11, pp. 3102–3130, Nov. 2012. [2] J. R. Guerci, Cognitive Radar: The Knowledge-Aided Fully Adaptive Approach, Norwood, MA, USA: Artech House, 2010. [3] M. S. Greco, F. Gini, P. Stinco, and K. Bell, “Cognitive radars: On the road to reality: Progress thus far and possibilities for the future,” IEEE Signal Process. Mag., vol. 35, no. 4, pp. 112–125, Jul. 2018. [4] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015. [5] D. A. E. Morgan, “Deep convolutional neural networks for ATR from SAR imagery,” Proc. SPIE, vol. 9475, 2015, Art. no. 94750F. [6] Y. Kim and T. Moon, “Human detection and activity classification based on micro-Doppler signatures using deep convolutional neural networks,” IEEE Geosci. Remote Sens. Lett., vol. 13, no. 1, pp. 8–12, Jan. 2016. € [7] M. S. Seyfio glu, S. Z. G€urb€uz, A. M. Ozbayo glu, and M. Y€uksel, “Deep learning of micro-Doppler features for aided and unaided gait recognition,” in Proc. IEEE Radar Conf., 2017, pp. 1125–1130. [8] R. P. Trommel, R. I. A. Harmanny, L. Cifola, and J. N. Driessen, “Multi-target human gait classification using deep convolutional neural networks on microDoppler spectrograms,” in Proc. Eur. Radar Conf., 2016, pp. 81–84. [9] B. Dekker, S. Jacobs, A. S. Kossen, M. C. Kruithof, A. G. Huizing, and M. Geurts, “Gesture recognition with a low power FMCW radar and a deep convolutional neural network,” in Proc. Eur. Radar Conf., 2017, pp. 163–166. [10] P. Molchanov, K. Egiazarian, J. Astola, R. I. A. Harmanny, and J. J. M. de Wit, “Classification of small UAVs and birds by micro-Doppler signatures,” in Proc. Eur. Radar Conf., 2013, pp. 172–175. [11] R. Harmanny, J. J. M. de Wit, and G. Premel-Cabic, “Radar micro-Doppler mini-UAV classification using spectrograms and cepstrograms,” Int. J. Microw. Wireless Technol., vol. 7, no. Special Issue 3/4, pp. 469–477, 2015. [12] V. C. Chen, F. Li, S.-S. Ho, and H. Wechsler, “MicroDoppler effect in radar: Phenomenon, model, and simulation study,” IEEE Trans. Aerosp. Electron. Syst., vol. 42, no. 1, pp. 2–21, Jan. 2006.
ACKNOWLEDGMENT TNO has received funds for this study from the Netherlands Ministry of Defense, under Grant V1512 and Grant V1908. The study has been conducted within the scope of the
NOVEMBER 2019
[13] P. J. Speirs, A. Schr€oder, M. Renker, P. Wellig, and A. Murk, “Comparisons between simulated and measured
IEEE A&E SYSTEMS MAGAZINE
X-band signatures of quad-, hexa- and octocopters,” in Proc. 15th Eur. Radar Conf., 2018, pp. 325–328.
55
Authorized licensed use limited to: Khwaja Fareed University of Eng & IT. Downloaded on July 04,2020 at 01:10:19 UTC from IEEE Xplore. Restrictions apply.
Deep Learning for Classification of Mini-UAVs Using Micro-Doppler Spectrograms in Cognitive Radar
[14] G. Klarenbeek, R. I. A. Harmanny, and L. Cifola, “Multi-
[19] A. Creswell and A. A. Bharath, “Denoising adversarial
target human gait classification using LSTM recurrent neu-
autoencoders,” IEEE Trans. Neural Netw. Learn. Syst.,
ral networks applied to micro-Doppler,” in Proc. 14th Eur.
vol. 30, no. 4, pp. 968–984, Apr. 2019.
Radar Conf., 2017, pp. 167–170. [15] A. Bendale and T.E. Boult, “Towards open set deep
[20] M. Buda, A. Makib, and M. A. Mazurowski, “A systematic study of the class imbalance problem in convolutional
networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Rec-
neural networks,” Neural Netw., vol. 106, pp. 249–259,
ognit., Jun. 2016, pp. 1563–1572.
2018.
[16] D. M. J. Tax and R. P. W. Duin, “Growing a multi-class
[21] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever,
classifier with a reject option,” Pattern Recognit. Lett.,
and P. Abbeel, “InfoGAN: Interpretable representation
vol. 29, no. 10, pp. 1565–1570, 2008.
learning by information maximizing generative adversarial
[17] S. Akcay, A. Atapourabarghouei, and T. P. Breckon,
nets,” in Proc. 30th Int. Conf. Neural Inf. Process. Syst.,
“GANomaly: Semi-supervised anomaly detection via adversarial training,” in Proc. ACCV, Perth, Australia,
2016, pp. 2172–2180. [22] T. Schlegl, P. Seeböck, S.M Waldstein, G. Langs, and
Dec. 2018.
U. Schmidt-Erfurth, “f-anogan: Fast unsupervised anomaly
[18] T. Salimans, I. Goodfellow, W. Zaremba, W. Cheung, A. Radford, and X. Chen, “Improved techniques for train-
detection with generative adversarial networks,” Medical image analysis, vol. 54, pp. 30–44, May 2019.
ing GANs,” in Proc. 30th Int. Conf. Neural Inf. Process. Syst., 2016, pp. 2234:2242.
56
IEEE A&E SYSTEMS MAGAZINE
NOVEMBER 2019
Authorized licensed use limited to: Khwaja Fareed University of Eng & IT. Downloaded on July 04,2020 at 01:10:19 UTC from IEEE Xplore. Restrictions apply.
