Data preprocessing and intelligent data analysis

doi:10.1016/S1088-467X(98)00007-9

Intelligent Data Analysis

Volume 1, Issues 1–4, 1997, Pages 3-23

Intelligent Data Ana...

https://doi.org/10.1016/S1088-467X(98)00007-9 Get rights and content

Abstract

This paper first provides an overview of data preprocessing, focusing on problems of real world data. These are primarily problems that have to be carefully understood and solved before any data analysis process can start. The paper discusses in detail two main reasons for performing data preprocessing: (i) problems with the data and (ii) preparation for data analysis. The paper continues with details of data preprocessing techniques achieving each of the above mentioned objectives. A total of 14 techniques are discussed. Two examples of data preprocessing applications from two of the most data rich domains are given at the end. The applications are related to semiconductor manufacturing and aerospace domains where large amounts of data are available, and they are fairly reliable. Future directions and some challenges are discussed at the end.

References (57)

O.E. de Noord
The Influence of Data Preprocessing on the Robustness and Parsimony of Multivariate Calibration Models
Chemometrics and Intelligent Laboratory Systems
(1994)
B.C. Falkenhainer et al.
Integrating Quantitative and Qualitative Discovery in the ABACUS System
D.B. Lenat
Learning from Observation and Discovery
B. Marangelli
Data Preprocessing for Adaptive Vector Quantization
Image and Vision Computing
(1991)
W. Meier et al.
Fuzzy Data Analysis —Methods and Industrial Applications
Fuzzy Sets and Systems
(1994)
J. Sjoberg
Regularization as a Substitute for Preprocessing of Data in Neutral Network Training
J.T.W.E. Vogels et al.
A New Method for Classification of Wines Based on Proton and Carbon-13 NMR Spectroscopy in Combination with Pattern Recognition Techniques
Chemometrics and Intelligent Laboratory Systems: Laboratory Information Management
(1993)
M.R. Azimi-Sadjadi et al.
Detection and Classification of Buried Dielectric Anomalies Using Neutral Networks —Further Results
IEEE Trans. on Instrumentations and Measurement
(1994)
J.C. Bezdek et al.
D.G. Bobrow et al.
Some principles of Memory Schemata

E. Bontrager

GAIT-ER-AID: An Expert System for Analysis of Gait with Automatic Intelligent Preprocessing of Data

R.L. Chen et al.

Statistical Data Pre-processing for Fuzzy Modelling of Semiconductor Manufacturing Process

J.J. Clark

C. Cortes et al.

Limits on Learning Machine Accuracy Imposed by Data Quality

V.G. Dabija

Learning to Learn Decision Trees

J.F. Davis et al.

Process Monitoring, Data Analysis and Data Interpretation

J. DeWitt

Adaptive Filtering Network for Associative Memory Data Preprocessing

Z. Duszak et al.

Using Principal Component Transformation in Machine Learning

W. Emde et al.

The Discovery of the Equator or Concept Driven Learning

A. Famili et al.

Intelligently Helping Human Planner in Industrial Process Planning

AIEDAM

(1991)

U. Fayyad et al.

From Data Mining to Knowledge Discovery

L. Hesselink

Research Issues in Vector and Tensor Field Visualization

S. Iwasaki

Clustering of Experimental Data and its Applications to Nuclear Data Evaluation

M. Ke et al.

MLS, A Machine Learning System for Engine Fault Diagnosis

P.M. Kelly et al.

Preprocessing Remotely-Sensed Data for Efficient Analysis and Classification, Applications of Artificial Intelligence

P. Langley et al.

Rediscovering Chemistry with the Bacon System

N. Lavrac et al.

Cost-Sensitive Feature Reduction Applied to a Hybrid Genetic Algorithm

C. Matheus

Feature Construction: An Analytic Framework and Application to Decision Trees

Cited by (304)

Physics-driven neural networks for nonlinear micromechanics
2024, International Journal of Mechanical Sciences
Micromechanical problems seek full-field solutions in response to external and/or internal thermal-mechanical loads, which have been increasingly encountered in material property and process assessments. Here we present a machine learning based strategy of solving micromechanics at finite strains. Deep neural networks (DNNs) are trained by employing the elastic energy together with other physical constraints to formulate the loss function. In particular, the crossover of material points, which can be a common issue for similar DNNs when applied to compression-dominant problems, is shown to be effectively addressed by including a kinematic penalty term that forces the DNN to maintain structural integrity and stability and avoid unphysical behavior. The generality and accuracy of the proposed physics-driven neural networks (PDNNs) are demonstrated through various micromechanical problems, including single crystal anisotropy elasticity, Eshelby's inclusion problem, buckling, and elastic homogenization of polycrystals. The effect of the choice of optimizers and hyperparameters on the PDNN training are discussed and the computational efficiency is also analyzed. It is shown that this novel PDNN framework can completely remove the need of labeled training data and exhibit improved performances as compared to the conventional physics-informed neural networks (PINNs) in terms of avoiding unphysical solutions and attaining higher computational efficiency. PDNNs can thus serve as a promising tool to solve some critical challenges associated with a wide range of nonlinear micromechanical problems.
Sub-clustering based recommendation system for stroke patient: Identification of a specific drug class for a given patient
2024, Computers in Biology and Medicine
Stroke is one of the leading causes of death worldwide. Previous studies have explored machine learning techniques for early detection of stroke patients using content-based recommendation systems. However, these models often struggle with timely detection of medications, which can be critical for patient management and decision-making regarding the prescription of new drugs. In this study, we developed a content-based recommendation model using three machine learning algorithms: Gaussian Mixture Model (GMM), Affinity Propagation (AP), and K-Nearest Neighbors (KNN), to aid Healthcare Professionals (HCP) in quickly detecting medications based on the symptoms of a patient with stroke. Our model focused on three classes of drugs: antihypertensive, anticoagulant, and fibrate. Each machine learning algorithm was used to accomplish specific tasks, thereby reducing the partial search space, computational cost, and accurately detecting a primary drug class without loss of precision and accuracy. Our proposed model, called CRGANNC (Clustering Recommendation Gaussian Affinity Nearest Neighbors Classifier), effectively addresses the sparsity and scalability issues faced by content-based recommendation models. The CRGANNC model dynamically partition clusters into sub-clusters with variable numbers based on the group, and can diagnose healthy, sick, and at-risk patients, and recommend drugs to the HCP. In addition to our analysis, we developed a semi-artificial dataset with new features such as weakness, dizziness, headache, nausea, and vomiting, using a pipeline. This dataset serves as a valuable resource for researchers in the sensitive domain of stroke, providing a starting point for building and testing models when real data is often restricted. Our work not only contributes to the development of predictive models for stroke but also establishes a framework for creating similar datasets in other sensitive domains, accelerating research efforts and improving patient care. Our experiments were conducted on our dataset consisting of 9691 patient records, with 1206 records for stroke attacks and 8485 healthy patients. The CRGANNC model achieved an average precision of 0.98, recall of 0.95 and F1-score of 0.96 across all three drugs classes. Furthermore, our model demonstrated significant improvement in computational efficiency compared to existing content-based recommendation models, reducing the processing time by 25.80% . This results indicate the effectiveness of our model in accurately detecting medications for stroke patients based on their symptoms.
Using convolutional neural networks to support examiners in duct tape physical fit comparisons
2023, Forensic Science International
This paper describes the construction and use of a machine-learning model to provide objective support for a physical fit examination of duct tapes. We present the ForensicFit package that can preprocess and database raw tape images. Using the processed tape image, we trained a convolutional neural network to compare tape edges and predict membership scores (i.e., fit or non-fit category). A dataset of nearly 2000 tapes and 4000 images was evaluated, including various quality grades: low, medium, and high, as well as two separation methods, scissor-cut and hand-torn. The model predicts medium-quality and high-quality scissor-cut tape more accurately than hand-torn, whereas for low-quality tape predicts the hand-torn tapes more accurately. These results are consistent with previous studies performed on the same datasets by analyst examinations. A method of pixel importance was also implemented to show which pixels are used to make the decision. This method can confirm some fit features that correspond with analyst-identified features, like edge morphology and backing pattern. This pilot study demonstrates the feasibility of computational algorithms to build physical fit databases and automated comparisons using deep neural networks, which can be used as a model for other materials.
Minimization of high computational cost in data preprocessing and modeling using MPI4Py
2023, Machine Learning with Applications
Data preprocessing is a fundamental stage in deep learning modeling and serves as the cornerstone of reliable data analytics. These deep learning models require significant amounts of training data to be effective, with small datasets often resulting in overfitting and poor performance on large datasets. One solution to this problem is parallelization in data modeling, which allows the model to fit the training data more effectively, leading to higher accuracy on large data sets and higher performance overall. In this research, we developed a novel approach that effectively deployed tools such as MPI and MPI4Py from parallel computing to handle data preprocessing and deep learning modeling processes. As a case study, the technique is applied to COVID-19 data from state of Tennessee, USA. Finally, the effectiveness of our approach is demonstrated by comparing it with existing methods without parallel computing concepts like MPI4Py. Our results demonstrate promising outcome for the deployment of parallel computing in modeling to minimize high computational cost.
Predictive compliance monitoring in process-aware information systems: State of the art, functionalities, research directions
2023, Information Systems
Business process compliance is a key area of business process management and aims at ensuring that processes obey to compliance constraints such as regulatory constraints or business rules imposed on them. Process compliance can be checked during process design time based on verification of process models and at runtime based on monitoring the compliance states of running process instances. For existing compliance monitoring approaches it remains unclear whether and how compliance violations can be predicted, although predictions are crucial in order to prepare and take countermeasures in time. This work, hence, analyzes existing literature from compliance monitoring as well as predictive process monitoring and provides an updated framework of compliance monitoring functionalities. Moreover, it raises the vision of a comprehensive predictive compliance monitoring system that integrates existing predicate prediction approaches with the idea of employing PPM with different prediction goals such as next activity or remaining time for prediction and subsequent mapping of the prediction results onto the given set of compliance constraints (PCM). For each compliance monitoring functionality we elicit PCM system requirements and assess their coverage by existing approaches. Based on the assessment, open challenges and research directions realizing a comprehensive PCM system are elaborated.
Beyond the Spreadsheet: Data Management for Physicians in the Era of Big Data
2023, Surgical Clinics of North America

View all citing articles on Scopus

View full text