Predicting the quality of performance statements

Ministry of Justice and Security

A performance statement confirms that delivered goods or services have been received in good condition and the right quality, usually before payment of the invoice. The Ministry of Justice is working on better recording and verification of performance declarations, using an algorithm to assess quality.

Last change on 23rd of December 2024, at 14:06 (CET) | Publication Standard 1.0

• Publication category: Other algorithms
• Impact assessment: Field not filled in.
• Status: In use

General information

Theme

Public finance

Begin date

01-2025

Contact information

[e-mail]

Responsible use

Goal and impact

A declaration of performance (PV) is a statement that confirms that the goods delivered were received in good condition and the correct quantity and quality, or that the services were performed satisfactorily, as agreed in the contract. Performance is declared before the invoice is paid, unless otherwise specified in the contract or the requirement for the PV.

The recording of performance certificates is done in the Leonardo financial information system. If this is not possible (e.g. due to the AVG or confidential procurement), the delivery can be explained in writing by other means, such as via the text field on the receipt entry or by an alternative receipt (type 2).

For some time, the Ministry of Justice and Security (JenV) has had a finding from the Audit Department Rijk (ADR) and the Algemene Rekenkamer (AR) on the consistent (and later qualitative) recording of attachments in financial records (Leonardo). Financial Control (FC) has taken up this finding. Because there are many payments requiring a performance statement, checking these manually is not feasible. Current sampling selects a-select items, which includes checking good performance statements. This can be improved by doing risk-based sampling.

A model has been developed to improve the capture of performance statements and quality control. This model looks at whether there is an annex where a performance statement is required. FC sets rules based on cost. The next step is to check not only whether the performance statement is present, but also whether it is of good quality. An algorithm has been created for this purpose.

Considerations

DFEZ, ADR and AR examine whether annexes, such as performance declarations, are present and of good quality. Because there are many performance declarations, checking them manually is a lot of work. Also, current spot checks do not always focus on risks, so even good performance statements are checked.

By analysing attendance and quality, we want to direct the process better. By using self-trained labels, we can automatically determine with reasonable certainty which performance declarations do not meet the requirements for a good performance declaration, as described in DFEZ's regulations.

Human intervention

The algorithm is set up so that when it gives a negative score, the performance statement has to be checked manually. This also applies when the algorithm indicates that there is no performance statement. In addition, at 10% of positive scores, or when the algorithm indicates that a performance statement is of good quality, an additional check is performed.

Risk management

No sensitive personal data will appear in the final product. The analysis is shared only with a limited group of people. Because the algorithm reads invoices and performance statements that may contain personal data, a DPIA (Data Protection Impact Assessment) has been prepared.

Legal basis

DFEZ, ADR and AR examine under the Comptabiliteitswet whether annexes, such as performance statements, are present and of good quality.

Links to legal bases

comptabiliteitswet 2016: https://wetten.overheid.nl/BWBR0039429/2023-09-26

Operations

Data

The algorithm uses performance statements and data from the Justice and Security Financial Information System, called Leonardo. The performance statements may contain personal data, and the data from Leonardo may have fields filled with free text, meaning it may also contain personal data.

Technical design

Colleagues from different departments collected labels, with support from Financial Control and DFEZ's Data Analysts. A total of 1,000 entries were labelled with a 1 (the documents accompanying this entry constitute a qualitative performance statement according to the experts) or 0 (the documents do not meet the requirements of a qualitative performance statement).

Based on these labels, different models were trained. GridSearch was used during training, which allowed the best settings for each model to be found quickly. The models were compared and the model with the best results was chosen (based on various performance indicators such as accuracy, precision, recall and F1 score).

The trained model is now applied to the previous month's financial invoices and receipts. The model compares the text in the documents to an entry with the pattern it learned from the label set with label 1. Documents that are very similar to this pattern are given a higher probability.

The model can be compared to a spam filter. By seeing many examples of spam e-mails, the filter learns to recognise a pattern and can decide for itself whether an e-mail is likely to be spam based on the words and structure in the e-mail.