Object Recognition in Public Spaces

Municipality of Amsterdam

The Public Space Supervision and Enforcement Department has commissioned the City of Amsterdam’s Computer Vision Team to explore how object recognition can help to ensure effective enforcement, with particular focus and priority given to vulnerable bridges and quay walls on which heavy objects are placed. 

Last change on 14th of July 2026, at 7:14 (CET) | Publication Standard 1.0

• Publication category: Impactful algorithms
• Impact assessment: DPIA, The Ethical Guide, IAMA
• Status: In use

General information

Theme

• Economy
• Space and Infrastructure

Begin date

2022-07

Contact information

[e-mail]

Responsible use

Goal and impact

The Public Space Supervision and Enforcement Directorate (THOR) has commissioned the City of Amsterdam’s Computer Vision Team (hereinafter: CVT) to investigate how object recognition can help prevent vulnerable bridges and quay walls from collapsing as a result of heavy objects being placed on them. These include containers, site huts, portable toilets and scaffolding. At present, there is no clear overview of where these objects are located in (vulnerable) parts of the city. During the pilot, tests were carried out to determine whether these objects could be successfully recognised in public spaces using a scanning vehicle fitted with a camera. Once objects in public spaces have been identified, an alert can be generated. By supplementing this alert with additional information from (municipal) sources, the work of inspectors can be carried out more effectively. This can be done, for example, by including information in a report about how urgent the alert is, based on the vulnerability of the relevant quay on which the object is situated. This allows priority to be given to urgent situations. In addition to generating an alert, staff can use a digital map showing the objects that have been detected. The pilot was successfully completed in 2024. It has been shown that objects can be successfully recognised and that the technology can be used to identify misplaced objects in a more targeted manner.

Update 2025: This technology has been put into production and is used weekly in rotating neighbourhoods across the Centrum, Zuid, Noord, West and Weesp districts.

Considerations

The bridges and quay walls in the city centre have been severely weakened by years of excessive loads. When these bridges and quays were built, there was no heavy traffic. We are currently reinforcing or replacing the bridges and quay walls. For this reason, heavy traffic and heavy objects are now prohibited in various locations.

Due to a shortage of enforcement officers within the Surveillance and Enforcement department relative to the volume of reports concerning public spaces, digitalisation is one way of meeting the legal obligation to enforce regulations on objects subject to a permit. 

Human intervention

The use of the image recognition system does not constitute automated decision-making. However, the system does automatically generate an alert. This alert is then assessed (manually) by a supervisor, after which an on-site investigation may take place. The supervisor independently assesses whether the situation is lawful or unlawful. If the latter is the case, an enforcement officer will take a decision independently. This constitutes sufficient meaningful human intervention. The ‘output’ of the algorithms does, however, contribute to the ‘decision’ as to whether or not to carry out a further investigation (on-site investigation) into the object observed in a public space. The system (and its associated algorithms) therefore does have a substantial influence. 

Risk management

Across the board, measures have been put in place to process the data securely and to resolve incidents (for example, the blurring algorithm no longer working) quickly and effectively in accordance with established procedures. In particular, the project focuses on carefully anonymising the environmental images captured and removing any unnecessary data. In addition, considerable attention is paid to the end-users of the system. They must be fully aware of how the system works (including the associated algorithms) and what the potential risks are. Regulatory authorities must always be able to make decisions independently. It is therefore important that the output can be properly interpreted and that an alert can, where appropriate, be disregarded. 

Impact assessment

• Data Protection Impact Assessment (DPIA)
• The Ethical Guide
• Human Rights and Algorithms Impact Assessment (IAMA)

Operations

Data

Training datasets:

Blur algorithm

This comprises roughly 10,000 images containing raw, i.e. non-anonymised, data. These images were required to manually train the algorithm to recognise individuals and number plates, so that these can be removed from the images. These images are only accessible to a number of developers who are training the models. The images will be retained for as long as the algorithm may need to be further developed.

Image recognition algorithm

To teach this algorithm to recognise objects accurately, roughly 1,500 images – some of which were anonymised and some of which were not – were used to train the algorithm manually. The use of non-anonymised images was necessary for this purpose, so that the context (public spaces in the municipality of Amsterdam) is kept as intact as possible. This ensures that the algorithm is better able to recognise the objects. Unlike, for example, Google Maps, the City of Amsterdam also blurs the entire figures of people in public spaces.

Production data:

The scanning system captures images containing metadata such as date, time, location and heading. These images are then all anonymised using the blurring algorithm developed by the City of Amsterdam. Immediately afterwards, the images are filtered to identify those containing containers, site huts, portable toilets and scaffolding, using the ‘objects’ image recognition algorithm. All images in which none of these objects are visible are immediately deleted. Once the above data has been obtained, this information is enriched with details from planning permissions and information on vulnerable bridges and quay walls. It is then assessed whether the object is located on a vulnerable bridge or quay wall. Based on the above data, the following information can then be generated:

· Category Orange (or Red): potentially illegal object (on a vulnerable quay)

· Distance to vulnerable quay: 25 metres

· Distance to the structure’s planning permission: 40 metres

The above data is sent to the Amsterdam Signals Information Service (SIA). SIA processes this into a ‘signal’ accompanied by a map showing the location and forwards it to CityControl, so that a supervisor can act on it. Anonymised images showing objects are reused to retrain or improve the image recognition algorithm.

Technical design

Performance

The algorithms perform well based on results from the training dataset. The planned pilot is necessary to test the algorithms using production data. Performance will be accurately measured throughout the pilot. Nevertheless, each algorithm also has a so-called margin of error. The CVT has investigated this, and these margins of error occur in the following situation:

· Image recognition algorithm

The object is too far away, meaning the algorithm fails to recognise it. However, the risk of the object being missed is low, as it is likely that, within the high-risk area, the scanning vehicle will eventually pass the object, enabling it to be recognised.  

· Blur algorithm

The blurring algorithm currently has an accuracy of roughly 95% for people standing close to the camera. For people standing further away from the camera, this figure is around 92%. Visual inspection of a random sample has shown that the individuals who are not recognised are usually unrecognisable because, for example, they are partially obscured by a tree or are too far away from the camera.  

Update 2024: Blurring as a Service (BaaS) has now been rolled out; this will be used in a potential follow-up project to anonymise the images.

Update 2025: The above has been implemented.

External provider

N/A

Link to code base

https://github.com/Computer-Vision-Team-Amsterdam/Objectherkenning-Openbare-Ruimte