Applies to: NeuralTools 7.x and Up

Data Preparation for Training a Neural Network in Neural Tools

1. Evaluate Data Quality

Before building any prediction or classification model, assess and clean the data:

• Duplicate records: Remove repeated entries to avoid skewing training/testing results.

• Out-of-range values: Correct or remove numerical values outside valid ranges (e.g., Age = -3).

• Invalid categorical values: Fix nonsensical categories (e.g., Marital Status = Bachelor).

• Inconsistencies: Ensure logical consistency between variables (e.g., Age = 20, Employment seniority = 25).

• Missing data: Remove variables with excessive missing values or impute missing data if manageable.

 

2. Univariate Analysis

Once data has been evaluated and cleaned, run a Univariate Analysis.

• Categorical variables. Build a bar chart or a frequency table of the variable.

If the variable only has one value, it should be removed from the analysis.

If there are categories with a frequency lower than 5%, the variable should be categorized again in order to ensure a frequency greater than 5%.

If there are only two categories and one of them has a frequency lower than 5%, it should be removed from the analysis.

3. Numerical variables

• Compute descriptive statistics
  • build a histogram and a boxplot of the variable.

If the variable is highly skewed, use the log transformation during the Neural Network training.

Outliers: Verify whether outliers are errors before deciding to exclude them.

3. Bivariate Analysis

When many independent variables exist, analyze each against the dependent variable:

• Categorical vs. Categorical: Use Chi-square test. Include the variable if p-value is low.

• Numerical vs. Numerical: Compute Spearman correlation. Include if |correlation| > 0.75.

• Numerical vs. Categorical: Use t-test (if normal) or Mann-Whitney test (if not normal). Include if p-value is low.