Metric selection for incident detection systems

Published on 7月 15, 2020

Ferran Torrent

Senior Data Scientist at Aimsun

Classification problems with highly unbalanced datasets, such as incident detection, pose the trade-off between true positive rate and false negative rate. A key point for choosing the final trade-off is the selection of the metric to evaluate the performance of the system, and this decision must be made by putting oneself in the skin of the user and asking if the chosen metric and the corresponding results are representative of the concept of usefulness.

 

For example, the following two tables are examples of confusion matrices of an imbalanced dataset with 9x more negative examples than positive examples, for example, no-incident vs incident. In such a situation, accuracy is not a good metric, not even balanced accuracy. Precision might be good if you must detect as many positives as possible, even at the risk of getting lots of false positives. On the other hand, recall is more oriented to answering whether we can trust predicted positives. The F1-score is a trade-off between precision and recall. In incident detection, not only is it important to detect incidents, but also to avoid overwhelming the traffic operator with false incident detections. Therefore, the F1-score is the best choice from among these four metrics.

  • 有问题吗? 请联系我们。

    我们在这里提供帮助!

  • 有问题吗? 请联系我们。

    我们在这里提供帮助!

分享

引用Aimsun Next

Aimsun Next 20

Aimsun (2021). Aimsun Next 20 User’s Manual, Aimsun Next Version 20.0.3, Barcelona, Spain. Accessed on: May. 1, 2021. [In software].
Available: qthelp://aimsun.com.aimsun.20.0/doc/UsersManual/Intro.html


Aimsun Next 8.4

Aimsun (2021). Aimsun Next 8.4 User’s Manual, Aimsun Next Version 8.4.4, Barcelona, Spain. Accessed on: May. 1, 2021. [In software]. Available: qthelp://aimsun.com.aimsun.8.4/doc/UsersManual/Intro.html

Aimsun Next 20

@manual {​​​​​​​​AimsunManual,

title = {​​​​​​​​Aimsun Next 20 User’s Manual}​​​​​​​​,

author = {​​​​​​​​Aimsun}​​​​​​​​,

edition = {​​​​​​​​​​​​​​​Aimsun Next 20.0.3}​​​​​​​​​​​​​​​,

address = {​​​​​​​​​​​​​​​Barcelona, Spain}​​​​​​​​​​​​​​​,

year = {​​​​​​​​​​​​​​​2021. [In software]}​​​​​​​​​​​​​​​,

month = {​​​​​​​​​​​​​​​Accessed on: Month, Day, Year}​​​​​​​​​​​​​​​,

url = {​​​​​​​​​​​​​​​qthelp://aimsun.com.aimsun.20.0/doc/UsersManual/Intro.html}​​​​​​​​​​​​​​​,

}​​​​​​​​​​​​​​​


Aimsun Next 8.4

@manual {​​​​​​​​AimsunManual,

title = {​​​​​​​​Aimsun Next 8.4 User’s Manual}​​​​​​​​,

author = {​​​​​​​​Aimsun}​​​​​​​​,

edition = {​​​​​​​​​​​​​​​Aimsun Next 8.4.4}​​​​​​​​​​​​​​​,

address = {​​​​​​​​​​​​​​​Barcelona, Spain}​​​​​​​​​​​​​​​,

year = {​​​​​​​​​​​​​​​2021. [In software]}​​​​​​​​​​​​​​​,

month = {​​​​​​​​​​​​​​​Accessed on: Month, Day, Year}​​​​​​​​​​​​​​​,

url = {​​​​​​​​​​​​​​​qthelp://aimsun.com.aimsun.8.4/doc/UsersManual/Intro.html}​​​​​​​​​​​​​​​,

}​​​​​​​​​​​​​​​

Aimsun Next 20

TY – COMP

T1 – Aimsun Next 20 User’s Manual

A1 – Aimsun

ET – Aimsun Next Version 20.0.3

Y1 – 2021

Y2 – Accessed on: Month, Day, Year

CY – Barcelona, Spain

PB – Aimsun

UR – [In software]. Available: qthelp://aimsun.com.aimsun.20.0/doc/UsersManual/Intro.html


Aimsun Next 8.4

TY – COMP

T1 – Aimsun Next 8.4 User’s Manual

A1 – Aimsun

ET – Aimsun Next Version 8.4.4

Y1 – 2021

Y2 – Accessed on: Month, Day, Year

CY – Barcelona, Spain

PB – Aimsun

UR – [In software]. Available: qthelp://aimsun.com.aimsun.8.4/doc/UsersManual/Intro.html