Visual inspection of the skin plays a vital role in the screening and early detection of skin cancer. Malignant lesions (skin abnormalities that undergo uncontrolled growth) typically produce non-uniform pigmentation patterns that appear violent, chaotic, and lack symmetry. On the other hand, benign lesions such as moles or nevi generally have a more balanced, symmetric appearance. Because of this difference in the appearance, visual assessment can be utilized as a first line of defense against skin cancer.
When performing skin inspection, physicians look for patterns that are known to be associated with aggressive lesion growth. Once a skin abnormality is found, the doctor’s task is to describe the observed visual clues, measure them, and, based on these observations, make the best possible diagnostic decision (for example to perform a biopsy, or surgically remove the lesion). This may seem straightforward at first, but this task presents a few fundamental and practical challenges.
To start, we need to consider that human visual perception is notoriously subjective (remember the dress? ). A pattern that appears to be “regular” to one physician may appear to be “irregular” to another. This is known as the operator bias or rater variability. Moreover, the same person may give different assessments at different times depending on some external factors (such as the day of the week, weather, or mood). An additional problem is that it is difficult to describe rarely observed patterns in natural (i.e. human) language. While the human-scale patterns (e.g. stripes, waves, branches) often have specific terms attached to them, microscopic patterns (that are typically multi-layered) are more difficult to describe. The inability to accurately capture the observed pattern in words may affect the diagnostic decision; it also presents a confound problem of transferring the accumulated knowledge and experience between physicians.
The challenge for the diagnostician is therefore to extract clinically relevant information from the images in the most consistent and reliable manner possible. This challenge has been driving the development of methods to describe and quantify observed dermoscopy patterns; over time, these methods have progressed from qualitative observation to using machine intelligence for precise pattern extraction and classification.
To improve the consistency of lesion assessment, several standardized step-by-step (algorithmic) procedures have been designed that evaluate specific visual properties of lesions. The most common assessment procedure is the “ABCD” rule, that focuses on four specific parameters: 1) lesion asymmetry (“A”), 2) irregularity of the lesion’s border (“B”), 3) the number of distinguishable colors (“C”), and 4) lesion diameter (“D”). Greater values of these parameters generally correspond to the higher likelihood of malignant behavior. Another example of algorithmic assessment is the CASH rule: 1) Color variability, 2) Architecture - ordered or chaotic, 3) degree of Symmetry, and 4) degree of apparent Heterogeneity.
The use of the algorithmic methods has improved the consistency of diagnostic decisions, and provided an easy way for the physicians to keep track of the visual clues that need to be checked . Nevertheless, due to the relative simplicity of the step-by-step procedures, there are many cases when their use may be problematic:
While the step-by-step methods provide a great baseline, it is recognized by the medical community that such methods produce an oversimplified description of the observed patterns, especially in clinically difficult cases. A more comprehensive approach has been developed to describe complex multi-layered patterns that looks at specific high-level features.
Most dermoscopy images of skin lesions have multiple dermatologic patterns present. The question "how complex is the image that I am seeing?" can be interpreted as "how many distinguishable patterns do I see in the image?". The rule of thumb is: the greater the number of distinct dermoscopy features (or structures) in the image, the higher the likelihood that the underlying physiological process is malignant. To assess the risk of malignancy, physicians dissimilate complex images into sub-regions that are easier to analyze, and search for the presence of documented high-level features that are indicative of malignant or non-malignant behavior: pigmented network, globules, streaks, branching structures, globules, etc. The diagnosis is made based on the nature of observed features and their relative spatial distribution.
This way of lesion assessment is quite involved and it suffers from several drawbacks. For example, it is easy to miss some features if they are overlapped. In addition, this type of assessment takes a longer time compared to the simple algorithmic rules described above. If the duration of physical examination is constrained, physicians with less experience may miss a clue showing that a lesion is cancerous. Another aspect to consider is that features that are not on the standardized pattern list will be ignored. Computational analysis of digital dermoscopy images has the potential to overcome this limitation.
Recent advances in computer hardware, as well as computer vision and machine learning algorithms enabled the use of computers for the analysis of high-quality, high-resolution dermoscopy images. Using algorithms that are designed to find specific patterns in images, it is possible to analyse hundreds of images per minute even on a moderately powerful computer. There are generally two approaches to developing these algorithms. The first approach is to manually implement the search for specific patterns (for example, traditional patterns used by the dermatologists). The other approach is to design a general machine learning algorithm, and allow it to learn the patterns that correlate with skin cancer. The learning can be based on the historic database of dermoscopy images that have verified diagnosis. Once the algorithm learns the relevant patterns, it can find these patterns in the images and compute a risk score.
In addition to significantly faster processing times compared to physicians, computational analysis offers other advantages:
There still exist many problems associated with the analysis methods based on machine learning. Reliability is one of those issues: even the most sophisticated computer vision algorithms sometimes produce errors that seem absurd. In healthcare, an error made by an algorithm could lead to very serious consequences. We have not reached yet a point in time where we trust the algorithms to the same degree as we trust humans. Consider this, for example: would you trust a driverless car to take you on a busy highway? (futurists predict that one day we will). Another serious challenge is the ability to use contextual information in the diagnosis. When doctors make an image-based diagnostic decision, they naturally incorporate the contextual information that is available to them (such as age, sex, ethnicity, geographical region, family history of cancer). It is quite often that the images do not contain all information required for accurate diagnosis. For example, in early stages, melanoma can look like a benign lesion. Computers are not yet capable of intelligently combining contextual information from various sources; this very issue is one of the current research topics in the field of computer science. So far, there seems to be nothing that could fundamentally prevent computers from one day reaching a human level of comprehension.
The use of imaging and computational image analysis in dermatology and medicine in general has been increasing for years, and it is expected to continue to increase. Using new algorithms and machine learning techniques has allowed us learn new information about skin cancer and other diseases, discover new pathological pathways, identify previously unnoticed patterns, and most importantly, improve the detection and treatment of disease. Rapid advancements in biomedical technology and computer science may create the impression that in the future we will rely solely on computers to perform health check-ups and treatments. However, an argument can be made that it is the combination of humans and computers that will achieve the highest diagnostic performance. How exactly this human-computer interaction will develop is yet to be seen.
There are many possible ways that such interaction could work. On large scales involving billions of people, pre-screening may be performed by computers, and suspicious cases may be forwarded to dermatologists. We may call this a “machine-forward” arrangement. Yet another form of interaction is dermatologists using the computer-based assessment as a second opinion. This kind of interaction would be especially advantageous for inexperienced physicians who have not yet seen many cases in their practice. There will also be some ethical questions to consider: would you trust pre-screening to a (Very Advanced) machine intelligence? Or would you rather trust a doctor? What about machine intelligence versus a very inexperienced physician? While the answers to these questions may be not straightforward at this point, one thing we know for sure: our ability to diagnose and treat disease will only improve in the future, with computers playing a valuable role.
 Argenziano, G.; Soyer, H P. (2001) The Lancet Oncology, vol. 2(7), p. 443-449
 Rigel, D. S.; Russak, J.; Friedman, R. (2010) CA: A Cancer Journal for Clinicians, vol. 60(5), p. 301-316
 Dolianitis C.; Kelly J.; Wolfe R.; Simpson P. (2005) Arch Dermatol. vol. 141(8), p. 1008-1014
 Argenziano, G.; Soyer, H.P.; Chimenti, S.; Talamini, R.; Corona, R. et al. (2003) Journal of the American Academy of Dermatology, vol. 48(5), p. 679-693
 Zalaudek, I.; Kreusch, J.; Giacomel, J.; Ferrara, G.; Catricalà, C. et al. (2010) Journal of the American Academy of Dermatology, vol. 63(3), p. 361-374
 Leeser M.; Ramachandran J.; Wahl T.; Yablonski D. (2012) In Workshop on Numerical Software Verification (NSV), Berkeley, CA