An Improved Skin Lesion Matching Scheme in Total Body Photography
Abstract: Total body photography is used for early detection of malignant melanoma, primarily as a means of temporal skin surface monitoring. In prior work, we presented a scanner with a set of algorithms to map and detect changes in pigmented skin lesions, thus demonstrating that it is possible to fully automate the process of total body image acquisition and processing. The key procedure in these algorithms is skin lesion matching which determines whether two images depict the same real lesion. In this paper, we aim to improve it with respect to false positive and negative outcomes. To this end, we developed two novel methods: one based on successive rigid transformations of 3- D point clouds and one based on non-rigid coordinate plane deformations in regions of interest around the lesions. In both approaches, we applied a robust outlier rejection procedure based on progressive graph matching. Using the scanner’s images, we created a ground truth dataset tailored to diversify false positive match scenarios. The algorithms were evaluated according to their precision and recall values, and the results demonstrated the superiority of the second approach in all the tests. In the complete inter-positional matching experiment, it reached a precision and recall as
high as 99.92% and 81.65% respectively, showing a significant improvement over our original method. Existing system: The cornerstone problem that this scanner automatically solves is lesion association, or matching. This problem exists in two contexts: mapping lesions in one skin exploration and their temporal change analysis between explorations. During the image acquisition, the scanner captures a series of overlapping photographs so that every lesion is depicted at least 3–4 times from different viewing angles. Which of these photographs should be used for a temporal analysis? How to avoid analyzing the same lesion several times? In order to answer these questions, we need a method that can unambiguously match skin lesion photographs and provide a link between a real skin lesion and its various images. Similarly, during the temporal analysis, when comparing two photographs that represent the ‘before’ and ‘after’ of a lesion, we need a method that can ensure that they indeed depict the same lesion. Proposed system: This matching task is not as trivial as it may seem. Because of similarities in the appearance of different moles, or conversely, due to the absence of related visual features in the images of the same lesion, even the most experienced technicians may sometimes match them incorrectly. Moreover, when there are only two mole images without any context (e.g., surrounding skin area or part of the patient’s body), the matching problem can become virtually unsolvable. There are a handful of methods proposed to address multiple PSL matching in clinical images. All of them in some way rely on the geometrical constraints imposed by the very lesions on the skin. In the absence of a unique and reliable description of their appearance, these constraints are particularly valuable. In addition, clinical images do not always provide enough resolution to discern relevant details, especially in older imaging systems. So, representing skin lesions as constellations of points on a flat surface gives the advantage of comparing their geometrical layouts. Advantages:
The ROI size is slightly increased: from 400_400 to 600_600 pixels. This is done in order to account for distinct viewing angles at different turntable positions when detecting and matching SIFT features. It eliminates correspondence outliers among SIFT features, which were computed earlier in both ROIs, using a projective homo graphy model. If H can be fitted (there are enough SIFT feature correspondences), its validity is verified by satisfying the condition. This matching task is not as trivial as it may seem. Because of similarities in the appearance of different moles, or conversely, due to the absence of related visual features in the images of the same lesion, even the most experienced technicians may sometimes match them incorrectly. Disadvantages: The cornerstone problem that this scanner automatically solves is lesion association, or matching. This problem exists in two contexts: mapping lesions in one skin exploration and their temporal change analysis between explorations. During the image acquisition, the scanner captures a series of overlapping photographs so that every lesion is depicted at least 3–4 times from different viewing angles. Moreover, when there are only two mole images without any context (e.g., surrounding skin area or part of the patient’s body), the matching problem can become virtually unsolvable. Modules: Total body photography: Total body photography (TBP) is a technique consisting of periodically acquiring clinical photographs of patients in standardized positions. These photographs are then used by medical experts in a total body skin examination (TBSE) to document a patient’s skin state and analyse the progress of various cutaneous conditions. TBP plays a crucial role in the early detection of melanoma, the deadliest skin cancer. Using baseline photographs, it is possible to determine evolving pigmented skin lesions (PSLs) as well as the appearance of new and the disappearance of old
growths, which may indicate an early developing melanoma. In the context of this article, the terms “lesion” and “mole” are used interchangeably to refer to PSLs. In prior work, we presented a new total body scanning system capable of automatically performing a TBSE procedure: solving the lesion-to-image correspondence problem. Methodology : During the image acquisition, at each of the 24 turntable positions, overlapping photographs of the patient’s skin are captured by 21 cameras simultaneously. This is a considerable amount of visual data in which each skin lesion appears at least 3–4 times. In order to determine which photograph of a lesion is optimal for its temporal analysis (i.e., provides the most fron to-parallel view) and avoid processing it twice, we need to find all the views corresponding to that lesion. Finding these correspondences in the images acquired at the same turntable position is called intra-positional lesion matching. In the same day, inter-positional matching deals with comparing lesion images captured at different turntable positions. In this section, we describe three methods we developed to address the problem of skin lesion matching, including the method originally used in our mole mapping pipeline. Skin lesions in the images are represented by ellipses fit into maximally stable extremal regions (MSERs) . They are added to the matching algorithm after lesion detection takes place. Matching based on point cloud registration: Matching based on point cloud registration this approach was designed to improve inter-positional skin lesion matching. It uses topographical information (3-D point clouds) about the skin surface in the candidate lesions’ vicinity. That is, in order to confirm the correspondence between two moles represented by their 3-D locations, it creates and registers point clouds that precisely describe the local topography of the skin surface, i.e., its 3-D structure. Computing these point clouds is a challenging task, especially given the high requirements for precision. Nevertheless, the calibrated camera rig of the scanner has the advantage of the popular constraint. Using this constraint, we can perform fairly accurate sparse reconstructions (local with respect to the patient’s body) using SIFT points matched in stereo-pairs (entire images). Assuming only rigid deformations, the
registration of two such reconstructions created at successive turntable positions can be done by establishing correspondences between SIFT features that formed them. Thus, the inter-positional lesion matching problem, in this case, comes down to finding intra- and interposition SIFT feature correspondences. Inter-positional SIFT feature matching: Describes the procedure to match point clouds computed at sequential turntable positions. In this case, it is impossible to use the popular constraint because the images are not acquired simultaneously: involuntary movements by the patient while the turntable is rotating render the inter-positional extrinsic calibration of the camera rig unreliable. But the list of features is now limited to those previously matched in stereo-pairs at their respective positions. This reduction allows the creation of reasonably short candidate lists. Instead of popular and heuristic filtering, the procedure starts with standard feature matching based on estimating Descriptor distances (descmatches). It is followed by filtering associations to the same feature (dupfilter), which retains only those with minimal descriptor distances. Then, the candidates are clustered in the same way as in Algorithm 1, with the exception that q = 15 because of fewer available match candidates. In this case, a smaller minimum cluster size is chosen because the initial candidate count is not as big as during intra-positional feature matching. Evaluation metrics: We employ a number of statistical metrics to quantify the performance of the proposed algorithms on the ground-truth data. All these metrics are based on the concepts of true/false positive and negative test outcomes, which are commonly used to evaluate the performance of a binary classifier. In the case of lesion matching, the algorithms give a binary output for a pair of candidates: matching or non-matching lesion representations. If a matching candidate pair is identified as such by the algorithm, it is a true positive (TP) sample, while a discarded nonmatching pair produces a true negative (TN) outcome. False positive (FP) and false negative (FN) outcomes occur when a method matches or, in the latter case, discards a candidate pair erroneously. Consequently, the lesion matching methods are evaluated according to the following criteria: Precision (positive predictive value). This defines the number of true positives in all the positive outcomes: Pr =
TP=(TP + FP), where TP and FP denote the number of true and false positive samples, respectively. Recall (sensitivity or true positive rate). This shows the number of true matches identified as such by the algorithm: Re = TP=(TP + FN), where FN is the number of false negative outcomes. F1-score: a combination of the precision .