Skip to main content
Springer Nature Link
Log in

TrafficDiff: diffusion model based adversarial traffic scenario controllable generation for autonomous driving robust evaluation

  • Original Article
  • Published:
Save article
View saved research
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

As foundation models (FMs) are increasingly applied in safety-critical domains such as autonomous driving, their ability to handle rare, ambiguous, or adversarial conditions becomes essential for ensuring cognitive robustness. Generative AI offers a promising path for testing such capabilities by synthesizing diverse and realistic traffic scenarios. As a prominent class of generative models, diffusion models are known for their strong diversity, yet controllable generation remains a key challenge. To address this, we propose a controllable scenario generation framework based on diffusion models. First, a dynamic spatiotemporal fusion encoding mechanism integrates contextual factors (e.g., road layout, vehicle types) to enhance realism. To enhance diversity, we introduce a global–local optimizer that guides scenario generation while preserving physical and statistical consistency. To generate safety-critical long-tail scenarios, we design an adversarial induction method that enhances scenario criticality, while a system dynamics model improves long-tail scenario generation. Finally, a mechanism-based scenario filter ensures the safety and compliance of generated scenarios by eliminating unrealistic samples. We validate our method on benchmark datasets and real-vehicle tests. Compared to existing SOTA methods, traffic scenario diversity is enhanced by 6–8 times on average. In real-vehicle evaluations, TrafficDiff increase the collision rate by 25.5% and leads much mission failure, effectively challenging system robustness. This approach provides a scalable solution for virtual scenario validation, driving advancements in autonomous driving safety assessment.Our code of TrafficDiff is available at https://github.com/Moresweet/TrafficDiff.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+
from $39.99 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

Data Availability

Data is provided within the manuscript or supplementary information files

References

  1. Rempe D, Philion J, Guibas LJ, Fidler S, Litany O (2022) Generating useful accident-prone driving scenarios via a learned traffic prior. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 17305–17315

  2. Tang C, Zhan W, Tomizuka M (2021) Exploring social posterior collapse in variational autoencoder for interaction modeling. Adv Neural Inf Process Syst 34:8481–8494

    Google Scholar 

  3. Cai J, Song S, Zhang H, Song R, Zhang B, Zheng X (2023) Satellite network traffic prediction based on LSTM and GAN. In: 2023 IEEE 3rd international conference on information technology, big data and artificial intelligence (ICIBA), vol 3. IEEE, pp 175–178

  4. Isola P, Zhu J-Y, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1125–1134

  5. Karim MM, Li Y, Qin R, Yin Z (2022) A dynamic spatial-temporal attention network for early anticipation of traffic accidents. IEEE Trans Intell Transp Syst 23(7):9590–9600. https://doi.org/10.1109/TITS.2022.3155613

    Article  Google Scholar 

  6. Lin L, Li W, Bi H, Qin L (2022) Vehicle trajectory prediction using LSTMs with spatial temporal attention mechanisms. IEEE Intell Transp Syst Mag 14(2):197–208. https://doi.org/10.1109/MITS.2021.3049404

    Article  Google Scholar 

  7. Zhong Z, Rempe D, Xu D, Chen Y, Veer S, Che T, Ray B, Pavone M (2023) Guided conditional diffusion for controllable traffic simulation. In: 2023 IEEE international conference on robotics and automation (ICRA). IEEE, pp 3560–3566

  8. Pronovost E, Wang K, Roy N (2023) Generating driving scenes with diffusion. arXiv e-prints, 2305

  9. Zheng Y, Wang J, Li K (2020) Smoothing traffic flow via control of autonomous vehicles. IEEE Internet Things J 7(5):3882–3896

    Article  Google Scholar 

  10. Fu C, Lu Z, Liu H, Wumaierjiang A (2025) Dynamic short-term crash risk prediction from traffic conflicts at signalized intersections with emerging mixed traffic flow: a novel conflict indicator. Accid Anal Prev 217:108065

  11. Lopez PA, Behrisch M, Bieker-Walz L, Erdmann J, Flötteröd Y-P, Hilbrich R, Lücken L, Rummel J, Wagner P, Wießner E (2018) Microscopic traffic simulation using sumo. In: The 21st IEEE international conference on intelligent transportation systems. IEEE. https://elib.dlr.de/124092/

  12. Fellendorf M, Vortisch P (2010) Microscopic traffic flow simulator VISSIM. Fundam Traf Simul 2010:63–93

    Article  Google Scholar 

  13. Sagaama I, Kchiche A, Trojet W, Kamoun F (2024) Energy-efficient route navigation (eco-routing) for electric vehicles in sumo itinéraires de navigation éco-énergétiques pour les véhicules électriques dans sumo. IEEE Can J Electr Comput Eng 2024:1

    Google Scholar 

  14. Liu S, Wang Y, Chen X, Fu Y, Di X (2022) Smart-eflo: an integrated sumo-gym framework for multi-agent reinforcement learning in electric fleet management problem. In: 2022 IEEE 25th international conference on intelligent transportation systems (ITSC). IEEE, pp 3026–3031

  15. Wenl L, Fu D, Mao S, Cai P, Dou M, Li Y, Qiao Y (2023) Limsim: a long-term interactive multi-scenario traffic simulator. In: 2023 IEEE 26th international conference on intelligent transportation systems (ITSC). IEEE, pp 1255–1262

  16. Jiang H, Ren Y, Zhao Y, Cui Z, Yu H (2025) Toward city-scale vehicular crowd sensing: a decentralized framework for online participant recruitment. IEEE Trans Intell Transp Syst 2025:1

  17. Jiang H, Ren Y, Fang J, Yang Y, Xu L, Yu H (2023) Ship: a state-aware hybrid incentive program for urban crowd sensing with for-hire vehicles. IEEE Trans Intell Transp Syst 25(3):3041–3053

    Article  Google Scholar 

  18. Jiang H, Ren Y, Zhao Y, Cui Z, Yu H (2025) Toward city-scale vehicular crowd sensing: a decentralized framework for online participant recruitment. IEEE Trans Intell Transp Syst 2025:1

    Google Scholar 

  19. Zhang Q, Gao Y, Zhang Y, Guo Y, Ding D, Wang Y, Sun P, Zhao D (2022) Trajgen: generating realistic and diverse trajectories with reactive and feasible agent behaviors for autonomous driving. IEEE Trans Intell Transp Syst 23(12):24474–24487

    Article  Google Scholar 

  20. Hao K, Cui W, Luo Y, Xie L, Bai Y, Yang J, Yan S, Pan Y, Yang Z (2023) Adversarial safety-critical scenario generation using naturalistic human driving priors. IEEE Trans Intel Veh. https://doi.org/10.1109/TIV.2023.3335862

    Article  Google Scholar 

  21. Cao Y, Ivanovic B, Xiao C, Pavone M (2024) Reinforcement learning with human feedback for realistic traffic simulation. In: 2024 IEEE international conference on robotics and automation (ICRA). IEEE, pp 14428–14434

  22. Feng L, Li Q, Peng Z, Tan S, Zhou B (2023) Trafficgen: learning to generate diverse and realistic traffic scenarios. In: 2023 IEEE international conference on robotics and automation (ICRA). IEEE, pp 3567–3575

  23. Sun P, Kretzschmar H, Dotiwalla X, Chouard A, Patnaik V, Tsui P, Guo J, Zhou Y, Chai Y, Caine B (2020) Scalability in perception for autonomous driving: Waymo open dataset. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2446–2454

  24. Caesar H, Bankiti V, Lang AH, Vora S, Liong VE, Xu Q, Krishnan A, Pan Y, Baldan G, Beijbom O (2020) Nuscenes: a multimodal dataset for autonomous driving. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 11621–11631

  25. Igl M, Kim D, Kuefler A, Mougin P, Shah P, Shiarlis K, Anguelov D, Palatucci M, White B, Whiteson S (2022) Symphony: learning realistic and diverse agents for autonomous driving simulation. In: 2022 international conference on robotics and automation (ICRA). IEEE, pp 2445–2451

  26. Zhang Z, Liniger A, Dai D, Yu F, Van Gool L (2023) Trafficbots: towards world models for autonomous driving simulation and motion prediction. In: 2023 IEEE international conference on robotics and automation (ICRA). IEEE, pp 1522–1529

  27. Varadarajan B, Hefny A, Srivastava A, Refaat KS, Nayakanti N, Cornman A, Chen K, Douillard B, Lam CP, Anguelov D (2022) Multipath++: efficient information fusion and trajectory aggregation for behavior prediction. In: 2022 international conference on robotics and automation (ICRA). IEEE, pp 7814–7821

  28. Li T, Hui S, Zhang S, Wang H, Zhang Y, Hui P, Jin D, Li Y (2024) Mobile user traffic generation via multi-scale hierarchical GAN. ACM Trans Knowl Disc Data 2024:1

    Google Scholar 

  29. Bergamini L, Ye Y, Scheel O, Chen L, Hu C, Del Pero L, Osiński B, Grimmett H, Ondruska P (2021) Simnet: learning reactive self-driving simulations from real-world observations. In: 2021 IEEE international conference on robotics and automation (ICRA). IEEE, pp 5119–5125

  30. Bano S, Cassará P, Valerio L (2024) Variational autoencoders for noise resistant traffic generation in B5G networks. In: 2024 IEEE international mediterranean conference on communications and networking (MeditCom). IEEE, pp 13–18

  31. Chai H, Wang H, Li T, Wang Z (2024) Generative AI-driven digital twin for mobile networks. IEEE Netw 2024:1

    Google Scholar 

  32. Zhang L, Wu J, Shen J, Chen M, Wang R, Zhou X, Xu C, Yao Q, Wu Q (2021) SATP-GAN: self-attention based generative adversarial network for traffic flow prediction. Transportmet B Transp Dyn 9(1):552–568

    Google Scholar 

  33. Li Z, Huang C, Qiu W (2024) An intrusion detection method combining variational auto-encoder and generative adversarial networks. Comput Netw 253:110724

    Article  Google Scholar 

  34. Pronovost E, Wang K, Roy N (2023) Generating driving scenes with diffusion. Preprint arXiv:2305.18452

  35. Ho J, Jain A, Abbeel P (2020) Denoising diffusion probabilistic models. Adv Neural Inf Process Syst 33:6840–6851

    Google Scholar 

  36. Kim S-S, Chung M, Kim Y-K (2020) Urban traffic prediction using congestion diffusion model. In: 2020 IEEE international conference on consumer electronics-Asia (ICCE-Asia). IEEE, pp 1–4

  37. Lin L, Li W, Bi H, Qin L (2021) Vehicle trajectory prediction using LSTMs with spatial–temporal attention mechanisms. IEEE Intell Transp Syst Mag 14(2):197–208

    Google Scholar 

  38. Arjovsky M, Chintala S, Bottou L (2017) Wasserstein GAN. In: Proceedings of the 34th international conference on machine learning (ICML), vol 70. PMLR, pp 214–223

  39. Hanselmann N, Renz K, Chitta K, Bhattacharyya A, Geiger A (2022) King: generating safety-critical driving scenarios for robust imitation via kinematics gradients. In: European conference on computer vision. Springer, pp 335–352

  40. Bai X, Dong P, Huang Y, Kumari S, Yu H, Ren Y (2024) An AR-based meta vehicle road cooperation testing systems: framework, components modeling and an implementation example. IEEE Internet Things J 2024:1

    Google Scholar 

  41. Lin L, Li W, Bi H, Qin L (2021) Vehicle trajectory prediction using LSTMs with spatial-temporal attention mechanisms. IEEE Intell Transp Syst Mag 14(2):197–208

    Google Scholar 

  42. Deo N, Trivedi MM (2018) Convolutional social pooling for vehicle trajectory prediction. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 1468–1476

  43. Liao H, Li Y, Li Z, Wang C, Cui Z, Li SE, Xu C (2024) A cognitive-based trajectory prediction approach for autonomous driving. IEEE Trans Intel Veh 2024:1

    Google Scholar 

  44. Guo K, Miao Z, Jing W, Liu W, Li W, Hao D, Pan J (2024) Lasil: learner-aware supervised imitation learning for long-term microscopic traffic simulation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 15386–15395

  45. Li Q, Peng ZM, Feng L, Liu Z, Duan C, Mo W, Zhou B (2024) Scenarionet: open-source platform for large-scale traffic scenario simulation and modeling. Adv Neural Inform Process Syst 36:1

    Google Scholar 

  46. Suo S, Regalado S, Casas S, Urtasun R (2021) Trafficsim: learning to simulate realistic multi-agent behaviors. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10400–10409

  47. Xu C, Zhao D, Sangiovanni-Vincentelli A, Li B (2023) Diffscene: diffusion-based safety-critical scenario generation for autonomous vehicles. In: The second workshop on new frontiers in adversarial machine learning

Download references

Acknowledgements

This work was supported by National Key Research and Development Project of China under Grant No. 2022YFB4300400.

Author information

Authors and Affiliations

  1. School of Transportation Science and Engineering, Beihang University, Beijing, 100191, China

    Xuesong Bai, Hongbo Li, Peng Dong, Changhang Tian, Jinchuan Zhang, Yilong Ren & Aoyong Li

  2. State Key Laboratory of Intelligent Transportation Systems, Beijing, 102206, China

    Xuesong Bai, Changhang Tian, Yilong Ren & Aoyong Li

  3. School of Vehicle and Mobility, Tsinghua University, Beijing, 100084, China

    Yang Fei

Authors
  1. Xuesong Bai
    View author publications

    Search author on:PubMed Google Scholar

  2. Hongbo Li
    View author publications

    Search author on:PubMed Google Scholar

  3. Peng Dong
    View author publications

    Search author on:PubMed Google Scholar

  4. Changhang Tian
    View author publications

    Search author on:PubMed Google Scholar

  5. Yang Fei
    View author publications

    Search author on:PubMed Google Scholar

  6. Jinchuan Zhang
    View author publications

    Search author on:PubMed Google Scholar

  7. Yilong Ren
    View author publications

    Search author on:PubMed Google Scholar

  8. Aoyong Li
    View author publications

    Search author on:PubMed Google Scholar

Contributions

Xuesong Bai: Writing-review and editing, Conceptualization, Supervision, Visualization, Data curation, Formal analysis, Project administration. Hongbo Li: Validation, Software, Formal analysis, Methodology. Changhang Tian: Visualization, Validation, Software, Formal analysis. Jinchuan Zhang: Writing-original draft, Methodology, Validation, Formal analysis, Data curation. Peng Dong: Writing-review and editing, Formal analysis, Supervision. Yang Fei: Resources, Supervision. Yilong Ren: Supervision, Resources, Project administration. Aoyong Li: Supervision, Resources, Project administration, Funding acquisition.

Corresponding author

Correspondence to Aoyong Li.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bai, X., Li, H., Dong, P. et al. TrafficDiff: diffusion model based adversarial traffic scenario controllable generation for autonomous driving robust evaluation. Pattern Anal Applic 28, 182 (2025). https://doi.org/10.1007/s10044-025-01561-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Version of record:

  • DOI: https://doi.org/10.1007/s10044-025-01561-3

Keywords

Profiles

  1. Aoyong Li View author profile