Anomaly Detection:
Definition, Best Practices and Use Cases

From financial fraud detection to healthcare insurance, anomaly detection is growing in significance as a technique of data analysis and alerts. Based on the assumption that similar units of data within a single dataset should be relatively homogeneous, anomaly detection allows identifying deviant entries. The latter may signal about cyber-security intrusions, attempts of fraud, insurance forgery, and other fraudulent or hazardous activities.

Thus, the method of anomaly detection is now broadly used in a variety of industries to guarantee a greater level of security. Here we explore the concept and meaning of anomaly, examine anomaly detection methods and algorithms, and review industry cases for its application.

What Is an Anomaly?

Let's start the review with the 'anomaly' definition. An anomaly is a value that deviates from the norm considerably enough to be regarded as a rare exception. Systems detecting anomalies do that based on the assumption that such outliers, or exceptions, should stand apart from the significant portion of data in the dataset. Thus, the process of detection presupposes the establishment of patterns first and then identifying the units violating those patterns.

Thus, in line with the description provided above, one may derive the 'anomalous' definition as the one that doesn't fit the norm, that is, the expectations of normal behavior or normal value typical for a specific dataset. The unit of data may be too large or too small compared to the major portion of the data, thus deviating from the average values. If the dataset is visualized, an anomalous data unit will stand far apart from the densely placed rest of the data units.

However, in practice, anomaly detection is still challenging, as in most cases, the meaning of anomalies is ambiguous. An anomaly from any perspective (a unit considerably different from the dataset) is usually termed a global anomaly. It is easily identifiable, both automatically and manually. But the problem arises with local anomalies (values different from the primary dataset insignificantly) or micro-clusters of deviant data standing quite close to the general dataset. These ambiguities suggest that a more sensitive way of detecting anomalies is via their scoring in terms of anomaly intensity rather than assigning one of the two labels – normal versus anomalous.
product recommendations using ai

Why Is it Important to Detect Anomalies?

To embrace the significance of anomaly detection in modern information systems and information security, one should answer a question, what does it mean to be an anomaly? Overall, a standard system functions within some predetermined limitations, with all incoming and outgoing data fitting a range of values typical for it. Thus, when individual deviations from the typical expectations are identified, they always serve as red flags for the security personnel and analysts.

Some areas in which anomaly detection is popular include:

  • Fraud detection (insurance, banking)
  • intrusion detection (computer networks, national surveillance)
  • medical informatics (diagnosis, disorder detection)
  • fault/damage detection (commerce, industry)

In a nutshell, detecting anomalous data within a system means that something goes wrong. For instance, if all buyers of an e-shop pay $100 on average for a pair of shoes, and some client pays $1,000 for the same purchase, it is an anomaly meaning that either there is a problem with the client's bank or some glitch occurred in the merchant's system. Similarly, if an insurer receives an average MRI check for $300-400 from patients and suddenly gets a $550 check for the same procedure, it should be an alert about a potentially fraudulent transaction requiring a closer investigation.

Another example is the significance of anomaly detection in a computer network. If some anomalous traffic patterns are identified in it, this could be a sign of sensitive data leakage from a hacked computer. Anomalies in the nervous signal transmission on the MRI scan may be a sign of some serious degenerative disease. At the same time, bizarre purchases and cashing activities with a client's credit card may be an alert of the card's theft.

Thus, as one can see, anomaly detection is helpful in many industries and fields, helping specialists identify deviations from the norm to investigate them closer and determine the cause of such deviation. The core to successful detection practices in any organization is to define 'anomalous' for their own datasets, to set specific detection signals for the analytical systems, and to feed in the feedback about correct/incorrect anomaly labeling for the system to learn.

Anomaly Detection Use Cases.

As anomalies in information systems most often suggest some security breaches or violations, anomaly detection has been applied in a variety of industries for advancing the IT safety and detect potential abuse or attacks. Here is a couple of use cases showing how anomaly detection is applied.

  • Cyber-intrusion
    Cyber-security is usually guaranteed with the help of network behavior anomaly detection (NBAD) technology. The system analyzes packet signatures to detect security threats and block incoming/outgoing data that is compromised. NBAD also conducts continuous network monitoring to detect suspicious events or trends

  • Fraud
    Graph-based anomaly detection (GBAD) is used to prevent fraud with credit cards, bank accounts, and insurance. ML systems also enable online banking fraud with the help of behavioral biometrics that also detects anomalies in consumer spending in real-time.

  • Medical anomaly detection
    Outlier identification has been applied in clinical settings in a variety of ways. For instance, the density-based clustering method can be applied to patient careflow log analysis to see whether the particular patient's careflow trace is anomalous. Anomaly detection in medical image analysis is helpful in accurate diagnostics, while treatment plan analysis may help determine potentially fatal errors in the treatment plans.

  • Industrial damage
    In the conditions of industrial automation, anomaly detection systems use data coming from numerous sensors to identify any malfunctions in the machinery, thus able to detect abnormalities early to prevent further damage or manufacturing defects.

  • Image processing
    The ability of anomaly detection systems to compare and analyze images allows accurate fraud detection in banking and insurance (when one recipient of a service submits duplicate reimbursement claims or when fraudsters try to receive reimbursement on fake claims)

  • Stock trading
    Anomaly detection algorithms deal quite well with the big masses of unstructured data in the stock exchanges, be it regular stocks or cryptocurrencies. ML systems classify the available data about price movements and sales volumes to detect anomalies and give alerts to the users about price outliers. This information may be instrumental in trading decision-making.

What Are Anomaly Detection Methods?

The choice of anomaly detection approach depends on the training data and test data measures you use. Currently, three types thereof are known:

  • Supervised detection is used with fully labeled training and test data sets. This method works well with unbalanced classes and performs data labeling based on the assumption that anomalies are well-known and already labeled. Thus, it is not applicable in cases where outliers are yet to be identified.

  • Semi-supervised detection applies training and test datasets, but training data is devoid of anomalies. This approach presupposes that a system will identify an anomaly once it learns a normal dataset and sees deviations from it.

Unsupervised anomaly detection is the most flexible of the three in terms of presenting no labels to the system and drawing no distinctions between the training and test dataset. This way, the system scores data within the dataset only based on its units' characteristics, without any predetermined normalcy values.

Introduction to Anomaly Detection Algorithms.

Speaking about supervised anomaly detection, decision trees (like C4.5) or Isolation Forest work with unbalanced data not quite productively. So, for supervised setups, Support Vector Machines and Artificial Neural Networks are more preferable. Semi-supervised anomaly detection setups work well with One-class SVMs and autoencoders. Other helpful algorithms include Gaussian Mixture Models and Kernel Density Estimation.

Isolation Forest is one of the ML algorithms used for unsupervised anomaly detection using anomaly scoring. This method is flexible in terms of not labeling units as normal/anomalous but assigning an anomaly score to them instead. As it is a tree method, it performs the outlier/non-outlier classification based on the assigned scores, visualizing the regions where the outliers fall. Other popular unsupervised algorithms include K-means, autoencoders, GMMs, PCAs, and the hypothesis tests-based analysis.

Anomaly Detection with Machine Learning.

Machine learning (ML), an area of artificial intelligence (AI), has proven highly helpful for advancing the anomaly detection accuracy and helping companies and organizations manage big data. The ability of ML systems to learn by their own experience, thus refining their analytical and predictive capacity on their own, is a valuable feature for accurate anomaly detection.

So, what is an advantage of the anomaly detection method enriched with ML technology? The first undeniable benefit is the ML system's ability to handle unlabeled and unstructured data proactively, determining what is normal and what may be regarded as a data anomaly. Second, ML systems are much more sensitive to distinguishing data anomalies from noise, allowing them to differentiate data units based on the degree of their deviation from the norm. The most common ML-based approaches to anomaly detection used today are:

1. Density-based anomaly detection.
This approach uses the k-nearest neighbor algorithm, with k-NN being a simple, non-parametric lazy learning technique for data classification. The data are categorized based on their distance from the core indicator, with Euclidean, Manhattan, Mikowski, and Hamming distance parameters applied in this analytical method. The density of data is established based on the reachability distance, and the local outlier factor is applied to label data as abnormal or normal.

2. Clustering-based anomaly detection.
Clustering is a typical approach in the area of unsupervised learning. Using it, the system clusters data points with the help of a K-means algorithm, with data distances larger than the average distance within a cluster being labeled as anomalous.

3. Support Vector Machine-Based Anomaly Detection.
A support vector machine (SVM) learn a soft boundary to cluster all data falling within that boundary as normal. Units falling beyond that cluster are labeled as abnormal.

        Enhance Your Anomaly Detection Solutions with Datrics.

        With anomaly detection methods able to give a competitive edge to any business, Datrics offers numerous setups suiting a variety of goals and dealing with different datasets. You can customize an anomaly detection product from Datrics depending on your business needs and characteristics of your data. Take advantage of ML technology to get a better understanding of your data, to enhance security protection, and to inform your anomaly-related decisions.
        The article was written by:
        Chief Data Scientist at Datrics
        Do you want to discover more about Datrics?
        Drop us a note to follow-up on any issues, support requests or enterprise plans.
        Your feedback matters!

        Read more