First, let’s wrap our heads around statistical significance and what it means in the context of AI.
What is statistical significance?
In AI and ML, statistical significance is a measure of the confidence in the results of a machine learning model or experiment. It determines whether the observed differences or patterns in the data are because of chance or if they reflect a real underlying effect.
Statistical significance is also frequently employed in AI and ML to:
- Compare models: Statistical significance helps compare the performance of different models or algorithms to determine which one is better.
- Identify important features: Statistical significance is used to identify the most important features in a dataset that contribute to the model’s performance.
- Detect anomalies: Statistical significance helps detect anomalies or outliers in the data that may indicate errors or unusual patterns.
To determine statistical significance, hypothesis testing (see this post if you want to learn about hypothesis testing) is typically used, which involves comparing the observed data with what would be expected under a null hypothesis (the assumption that there is no real effect). The commonly used measure for statistical significance is referred to as “p-value”.It represents the probability of observing the result by chance.
- ≤ 0.05 shows statistical significance (the result is unlikely to occur by chance)
- 0.05 shows no statistical significance (the result may be because of chance)
Let’s see what all this means with an example. Let’s say you are designing a solution to classify customer inquiries into different categories (e.g., billing, technical support, feedback). We’ll compare the scenarios where the AI solution reaches statistical significance and where it doesn’t.
| Scenario | Scenario 1: Reaching Statistical Significance | Scenario 2: Not Reaching Statistical Significance |
| Experiment | We train an ML classifier on a dataset of 10,000 labeled customer inquiries and evaluate its performance on a test set of 2,000 inquiries. | We train a similar classification model on a smaller dataset of 1,000 labeled customer inquiries and evaluate its performance on a test set of 200 inquiries. |
| Results | The model achieves an accuracy of 92% on the test set, with a p-value of 0.001. | The model achieves an accuracy of 85% on the test set, with a p-value of 0.12. |
| Conclusion | The model is effective in classifying customer inquiries, as shown by statistically significant results. This means that the observed accuracy is unlikely to occur by chance, and we can be confident in the model’s performance. | The lack of statistical significance prevents us from concluding that the model effectively classifies customer inquiries. The observed accuracy might be because of chance, and we need to collect more data or refine the model to improve its performance. |
| Decision-making | We can deploy the model with confidence. | We may need to refine the model or collect more data before deploying it. |
Finding the desired outcome in AI solutions often requires multiple experiments, with Scenario 2 being more prevalent than Scenario 1.
So what do you then do in order to iterate your way through to reach statistical significance?. Here are a couple of strategies you can adopt.
Revisit the problem:
- Re-examine your problem statement and objectives: Take a step back and revisit your problem statement, objectives, and key performance indicators (KPIs). Ensure that they are well-defined, measurable, and relevant to the problem you’re solving.
- Re-assess your problem’s complexity: Be honest with yourself: is the problem inherently complex, or are you tackling a problem that’s too ambitious?
- Pivot or adjust your goals: If the problem is too challenging, consider pivoting to a related problem or adjusting your goals to make them more achievable.
Reevaluate your data:
- Collect more data: Insufficient data for training and testing the models can sometimes lead to a lack of statistical significance, which can be resolved by gathering more data. Collecting more data or using data augmentation techniques can help improve the model’s performance.
- Data preprocessing and cleaning: Ensure that your data is clean, preprocessed, and free from errors or inconsistencies. This can help improve the model’s ability to learn from the data.
Tune your model(s):
- Feature engineering: Re-examine your feature set and consider adding or engineering new features that might be more informative or relevant to the problem. This can help the model learn more meaningful patterns.
- Model selection and hyperparameter tuning: Try different models or hyperparameter settings to see if they can improve performance. This might involve experimenting with different architectures, regularization techniques, or optimization algorithms.
- Model ensemble methods: Combine the predictions of multiple models to improve overall performance. This can help reduce overfitting and increase the robustness of the solution.
- Apply Early-stopping: Stop training models at the point when performance on a validation dataset starts to degrade. Implementing Early-stopping prevents overfitting and reduce the risk of over-training.
- Human-in-the-loop: Involve human experts or domain specialists to provide feedback, validate results, or even correct errors. This can help improve the model’s performance and ensure that it’s aligned with domain-specific knowledge.
Other considerations
- Re-evaluate your evaluation metrics: Consider whether your evaluation metrics are appropriate for the problem at hand. You might need to use alternative metrics that better capture the problem’s complexity.
- Consider alternative AI approaches: If you’re using a traditional machine learning approach, consider exploring alternative AI approaches, such as deep learning or reinforcement learning.
