Flow: Traffic simulation joins forces with deep reinforcement learning and cloud computing

January 10th, 2019

  • Aimsun and UC Berkeley’s Institute of Transportation Studies have formed a new partnership
  • The teams have worked together to produce Flow, a tool for managing large-scale traffic systems with a mix of human-driven and autonomous vehicles
  • Flow is the first open source architecture to integrate microsimulation tools with state-of-the-art deep reinforcement learning libraries in the cloud
  • Flow provides benchmarks in the use of deep reinforcement learning to create controllers for mixed-autonomy traffic, where connected and autonomous vehicles interact with human drivers and infrastructure

Aimsun and UC Berkeley’s Institute of Transportation Studies recently created a new partnership to release Flow, a tool for managing large-scale traffic systems that feature a mix of human-driven and autonomous vehicles.

Originally launched in September 2018, Flow is now integrated with Aimsun Next mobility modeling software to become the first open source architecture to integrate microsimulation tools with state-of-the-art deep reinforcement learning libraries in the cloud.

“Working harmoniously with others, be those professional relationships or software interfaces, has always been a core value at Aimsun,” says Alex Gerodimos, President, Aimsun Inc. “Our ongoing collaboration with UC Berkeley exemplifies this philosophy perfectly at both levels: our decision to lend our direct and complete support to this project was very deliberate and our software users will be able to benefit immediately from these new machine learning libraries as a result.”

Recent developments in multi-agent deep reinforcement learning algorithms, advancements in Aimsun Next microscopic traffic simulation libraries, and improved ease of cloud computing like Amazon Web Services (AWS), together provide a powerful ecosystem for finding near-optimal decision-making rules such as AV acceleration, AV lane changing, or traffic signal splits in mixed autonomy systems.

“In mixed-autonomy traffic control, evaluating machine learning methods is challenging due to the lack of standardized benchmarks,” says Alexandre Bayen, Director, ITS Berkeley. “Systematic evaluation and comparison will not only further our understanding of the strengths of existing algorithms but also reveal their limitations and suggest directions for future research.”

The team will be presenting Flow at the Aimsun stand at the Transportation Research Board Annual Meeting 2019, Washington DC.

What is Flow?

Flow is a traffic control framework that provides a suite of pre-built traffic control scenarios, tools for designing custom traffic scenarios, and integration with deep reinforcement learning libraries such as RLlib and traffic microsimulation libraries. Engineers can use Flow to apply deep reinforcement learning breakthroughs to various cases in traffic management, which involve classical traffic infrastructure (traffic lights, metering, etc.) and mobile infrastructure (mixed autonomy traffic, in particular using connected and automated vehicles to regulate traffic).

Aimsun-Flow: Empowering microsimulation with deep reinforcement learning and cloud computing

Flow Lego-blocks: Building a library

Flow is a tool for exploring numerous case studies using a Lego-blocks approach, i.e., modular benchmark cases which can be assembled. Existing benchmarks include:

Aimsun-Flow: Empowering microsimulation with deep reinforcement learning and cloud computing

Users can build modular traffic-scenarios, which can be combined to tackle complex situations. These building-blocks help break a problem down into smaller, tractable pieces that can be composed as controllers for new scenarios. For instance, single-lane/multi-lane and merge building-blocks can be used to study stop-and-go and merging traffic behaviors along a highway.

Aimsun-Flow: Empowering microsimulation with deep reinforcement learning and cloud computing

Benchmarks

In mixed-autonomy traffic control, evaluating proposed RL methods in the literature is often difficult, largely due to the lack of a standardized testbed. Systematic evaluation and comparison will not only further our understanding of the strengths of existing algorithms but also reveal their limitations and suggest directions for future research. Flow provides benchmarks in the use of deep reinforcement learning to create controllers for mixed-autonomy traffic, where connected and autonomous vehicles interact with human drivers and infrastructure [2].

Download

https://github.com/flow-project/flow

Further information and resources

https://flow-project.github.io/index.html

References

1- C. Wu, A. Kreidieh, K. Parvate, E. Vinitsky, A. Bayen, “Flow: Architecture and Benchmarking for- Reinforcement Learning in Traffic Control,” CoRL, vol. abs/1710.05465, 2017.

2- Vinitsky, E., Kreidieh, A., Le Flem, L., Kheterpal, N., Jang, K., Wu, F., Liaw, R., Liang, E., Bayen, A. M., Benchmarks for Reinforcement Learning in Mixed-Autonomy Traffic. In Conference on Robot Learning (pp. 399-409), 2018.