Metric selection for incident detection systems

Published on July 15, 2020

Ferran Torrent

Senior Data Scientist at Aimsun

Classification problems with highly unbalanced datasets, such as incident detection, pose the trade-off between true positive rate and false negative rate. A key point for choosing the final trade-off is the selection of the metric to evaluate the performance of the system, and this decision must be made by putting oneself in the skin of the user and asking if the chosen metric and the corresponding results are representative of the concept of usefulness.

 

For example, the following two tables are examples of confusion matrices of an imbalanced dataset with 9x more negative examples than positive examples, for example, no-incident vs incident. In such a situation, accuracy is not a good metric, not even balanced accuracy. Precision might be good if you must detect as many positives as possible, even at the risk of getting lots of false positives. On the other hand, recall is more oriented to answering whether we can trust predicted positives. The F1-score is a trade-off between precision and recall. In incident detection, not only is it important to detect incidents, but also to avoid overwhelming the traffic operator with false incident detections. Therefore, the F1-score is the best choice from among these four metrics.

  • Got a question? Get in touch.

    We are here to help!

SHARE

Cite Aimsun Next

 

Aimsun Next 23

Aimsun (2023). Aimsun Next 23 User's Manual, Aimsun Next Version 23.0.0, Barcelona, Spain. Accessed on: July. 19, 2023. [Online].
Available: https://docs.aimsun.com/next/23.0.0/

 


 

Aimsun Next 20.0.5

Aimsun (2021). Aimsun Next 20.0.5 User's Manual, Aimsun Next Version 20.0.3, Barcelona, Spain. Accessed on: May. 1, 2021. [In software].
Available: qthelp://aimsun.com.aimsun.20.0/doc/UsersManual/Intro.html
 

Aimsun Next 23

@manual {​​​​​​​​AimsunManual,

title = {​​​​​​​​Aimsun Next 23 User's Manual}​​​​​,
author = {​​​​​​​​Aimsun}​​​​​​​​,
edition =  {​​​​​​​​​​​​​​​Aimsun Next 23.0.0}​​​​​​​​​​​​​​​,
address = {​​​​​​​​​​​​​​​Barcelona, Spain}​​​​​​​​​​​​​​​,
year  = {​​​​​​​​​​​​​​​2023. [Online]}​​​​​​​​​​​​​​​,
month = {​​​​​​​​​​​​​​​Accessed on: Month, Day, Year}​​​​​​​​​​​​​​​,
url = {​​​​​​​​​​​​​​​https://docs.aimsun.com/next/23.0.0/}​​​​​​​​​​​​​​​,
}​​​​​​​​​​​​​​​


Aimsun Next 20.0.5

@manual {​​​​​​​​AimsunManual,

title = {​​​​​​​​Aimsun Next 20.0.5 User's Manual}​​​​​​​​,
author = {​​​​​​​​Aimsun}​​​​​​​​,
edition =  {​​​​​​​​​​​​​​​Aimsun Next 20.0.5}​​​​​​​​​​​​​​​,
address = {​​​​​​​​​​​​​​​Barcelona, Spain}​​​​​​​​​​​​​​​,
year  = {​​​​​​​​​​​​​​​2021. [In software]}​​​​​​​​​​​​​​​,
month = {​​​​​​​​​​​​​​​Accessed on: Month, Day, Year}​​​​​​​​​​​​​​​,
url = {​​​​​​​​​​​​​​​qthelp://aimsun.com.aimsun.20.0/doc/UsersManual/Intro.html}​​​​​​​​​​​​​​​,
}​​​​​​​​​​​​​​​

Aimsun Next 23

TY  - COMP
T1  - Aimsun Next 23 User's Manual
A1  - Aimsun
ET - Aimsun Next Version 23.0.0
Y1  - 2023
Y2 - Accessed on: Month, Day, Year
CY  - Barcelona, Spain
PB  - Aimsun
UR  - [In software]. Available: https://docs.aimsun.com/next/23.0.0/


Aimsun Next 20.0.5

TY  - COMP
T1  - Aimsun Next 20.0.5 User's Manual
A1  - Aimsun
ET - Aimsun Next Version 20.0.5
Y1  - 2021
Y2 - Accessed on: Month, Day, Year
CY  - Barcelona, Spain
PB  - Aimsun
UR  - [In software]. Available: qthelp://aimsun.com.aimsun.20.0/doc/UsersManual/Intro.html