How to Debug a Machine Learning Model: My Experience with LSTMs as an Example
<p>In machine learning, building a model is just one part of the puzzle. Some basic models are pre-constructed, but the real challenge often arises when a model doesn’t perform as expected. There might be issues with accuracy, or perhaps the model doesn’t handle unseen data well. Debugging then becomes indispensable. While I’m not an AI expert, the process of building and training models was quite daunting for me. I might not have the expertise to offer, but I can share experiences that many beginners might encounter. I’d like to share what I’ve learned from the challenges of building a simple LSTM model. The main checkpoints are:</p>
<ol>
<li>The implementation</li>
<li>Inputs and outputs during the process</li>
<li>The loss during the training</li>
<li>Regularization techniques</li>
<li>Hyperparameters tuning</li>
</ol>
<h1>0. A little background of my project</h1>
<p>I’m working on a car-following model that takes vehicle speed, speed difference, and spacing to the leading vehicle as inputs and predicts acceleration. The model is evaluated by the root mean square error (RMSE) of spacing in the trajectories from the dataset. I have successfully applied imitation models, specifically behaviour cloning and generative adversarial imitation learning, to the dataset and achieved an accuracy of around 6 meters. This means that, on average, the generated location of the car is similar to those in the dataset with a deviation of approximately 6 meters. However, my problem is that my LSTM model only achieved an accuracy of 16 meters, which is definitely not correct. According to a benchmark paper[1], the expected accuracy is around 5 meters (the root of MSE being 23.86).</p>
<p><a href="https://medium.com/pathway-to-data-analysis/how-to-debug-a-machine-learning-model-my-experience-with-lstms-as-an-example-ebb203187772">Read More</a></p>