Amazon recently announced three features that will help developers better build, test, and tune Alexa skills. A Natural Language Understanding (NLU) Evaluation Tool allows developers to batch-test utterances for a skill's interpretation against expectations. Utterance Conflict Detection allows developers to uncover utterances accidentally mapped to multiple intents. Finally, a new Get Metrics API gives developers the ability to build quality and usage reporting.
The NLU Evaluation Tool allows developers to create batches of utterances which can then be tested. It works in three uses: prevent overtraining, regression testing, and accuracy measurements. Too many sample utterances can overtrain and reduce accuracy. The batch testing feature allows developers to test what they expect users to say for a more accurate test sample. Regression tests allow developers to test newly added features to make sure that the customer experience doesn't diminish. Accuracy measurements allow developers to measure the accuracy of a skill's NLU using anonymized live utterances and then measure their impact on changes.
Utterance Conflict Detection aims to reduce inaccuracies caused by intent mismapping. The tool is automatically run on each model build. Developers can use it before publishing the skill or any particular version. By limiting such mapping, models should return fewer unintended conflicts.
The Get Metrics API allows users to analyze key metrics with third-party tools. Metrics such as unique customers, total enablements, total sessions, total utterances, and more can all be analyzed from data pulled through the API. To learn more, visit the API docs. Get Metrics is currently in beta.