Virtual Audit of Microscale Environmental Components and Materials Using Streetscape Images with Panoptic Segmentation and Image Classification

By prof. Sungjoo Hwang
Department of Architectural & Urban Systems Engineering
PURE Research Profile
hwangsj@ewha.ac.kr

Microscale environmental components, such as street furniture, sidewalks, and green spaces, significantly enhance street quality when properly identified and managed. Traditional in-person audits are time-consuming, so virtual audits using streetscape images and computer vision have been explored as alternatives. However, these often lack a comprehensive range of microscale components and do not consider attributes like materials. This paper proposes an automatic virtual audit method that recognizes microscale component types and materials in streetscape images using panoptic segmentation and material classification of segmented images of detected components (Figure).

In this study, 33 microscale environmental components, identified through a survey of their impact on street quality and pedestrian experience, were detected using panoptic segmentation. Additionally, materials of important components, such as sidewalks, walls, fences, stairs, benches, and signboards, were identified through image classification of segmented images of detected component types. The classification achieved an accuracy of 0.932 for sidewalk pavement materials and 0.844 for architectural elements and street furniture materials. Furthermore, the overall F1 score for integrated recognition of microscale environmental component types and materials using real-world streetscape data was 0.946, indicating a significant performance improvement compared with previous studies.

The results of this study demonstrate the model’s high efficiency in accurately detecting and classifying microscale components and their materials and highlight its practical applicability in real-world streetscape audits and analysis. Moreover, this study supports microscale environmental research at a more detailed level by considering a wider range of component types and materials. Consequently, this study enables an efficient and periodic audit of the built environment across a wide area, thereby contributing to the practical evaluation and improvement of street quality and urban livability or walkability by effectively introducing and removing microscale environmental components and modifying their characteristics, such as materials. The detailed insights from the virtual audit proposed in this study can support more nuanced design and maintenance strategies for streets.

Figure. Details of components detection and material classification: (a) component detection step, (b) material classification step, and (c) a list of microscale environmental components and materials

* Related Article
Meesung Lee, Hyunsoo Kim, Sungjoo Hwang, Virtual Audit of Microscale Environmental Components and Materials Using Streetscape Images with Panoptic Segmentation and Image Classification, Automation in Construction, Volume 170, 2025.