Calculating Transparency Indicators with SPARQL for
the EU Public Procurement Data Space

Queries

The queries used for running the experimental evaluation are an enhanced version of the queries for indicators 1, 6, and 9 developed during the PPDS proof of concept in December 2022. We have removed unnecessary triple patterns and data fields that were retrieved in the SELECT clauses. We have removed some of the subqueries that were unnecessary. The updated queries can be found in the Gitlab of the project (link soon).

Additionally, to study the impact of querying external resources, we incorporate a new version of each query, removing the SERVICE clause that retrieved data from the authority tables of the publication office. In contrast, we upload locally the necessary RDF graph to obtain the same results. Hence, each query has two different versions (e.g., indicator1_local.rq and indicator1_service.rq).

PREFIX epo: <http://data.europa.eu/a4g/ontology#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX legal: <https://www.w3.org/ns/legal#>
PREFIX dcterms: <http://purl.org/dc/terms/>
PREFIX dc: <http://purl.org/dc/elements/1.1/>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX pocgraph: <http://data.europa.eu/a4g/graph/ppds-poc/>

SELECT DISTINCT ?countryID ?year ?amountLots ?numSingleBidders ((?indicator/100) AS ?indicator1) WHERE {
  {
    SELECT DISTINCT ?countryID ?year ?amountLots ?numSingleBidders 
                    ((xsd:float(xsd:float(?numSingleBidders)/xsd:float(?amountLots))*100) AS ?indicator) WHERE {
    {
      SELECT DISTINCT ?countryID ?year (COUNT(DISTINCT ?lot) AS ?amountLots) 
                      (SUM(if(?bidders = "single-bidder", 1, 0)) AS ?numSingleBidders) WHERE {

        ?proc a epo:Procedure .
        ?proc epo:hasProcedureType ?p .
        FILTER ( ?p != <http://publications.europa.eu/resource/authority/procurement-procedure-type/neg-wo-call>)
        ?proc epo:hasProcurementScopeDividedIntoLot ?lot .

        ?statistic a epo:SubmissionStatisticalInformation .
        ?statistic epo:concernsSubmissionsForLot ?lot .
        ?statistic epo:hasReceivedTenders ?c .
        BIND(IF(?c = 1,"single-bidder","multiple-bidders") AS ?bidders) .

        ?resultnotice a epo:ResultNotice .
        ?resultnotice epo:refersToProcedure ?proc .
        ?resultnotice epo:hasDispatchDate ?ddate .
        BIND(year(xsd:date(?ddate)) AS ?year) .

        ?resultnotice epo:refersToRole ?buyerrole .

        {
          SELECT DISTINCT ?buyerrole ?countryID WHERE {
            ?buyerrole epo:playedBy ?org .
            ?org legal:registeredAddress ?orgaddress .
            ?orgaddress epo:hasCountryCode ?countrycode  .
            ?countrycode dc:identifier ?countryID .

            ?org epo:hasBuyerType ?buytype .
            FILTER (?buytype != <http://publications.europa.eu/resource/authority/buyer-legal-type/eu-int-org> )
            }    
          }
        } GROUP BY ?countryID ?year ORDER BY ?countryID ?year  }
      }
    }
  }

Engines

We use the free version of GraphDB (v10.2.1), the open-source Jena Fuseki (v4.8.0), and the community version of Virtuoso (v7.2.1). All triplestores are configured to store the graph in physical memory. We performed previous experiments over Jena Fuseki in its in-memory configuration, but the performance is highly impacted, so we decided to not take into account. Each engine has different function implementations that are used for some of the queries used in the benchmark. For example, for calculating the difference between two dates, GraphDB uses the function ofn:days-from-duration, Jena Fuseki is able to parse fn:days-from-duration, while in Virtuoso a simple rest operation works ((?date1-?date2)/(3600*24)).

We tried to incorporate the qEndpoint engine, used by the Kohesio platform, but we found that the engine has important performance bugs for aggregated queries, see the discussion. So at this moment, we do not recommend its use for the PPDS.

Metrics and Setup

We measure query execution time as the total time from query execution until the complete answer is generated. We also report the geometric mean of all queries that generate graphs in the same metadata representation. The geometric mean reports the central tendency of all execution times for a set of queries, reducing the effect of outliers. With this metric, we provide a general measurement of the performance of each triplestore when generating graphs in each representation. We run every experiment 5 times with a timeout of 24h, measure the time, and calculate the median. We run all experiments over an Ubuntu 20.04 server with 32 cores Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz 102400 Mb RAM RDIMM, 3200 MT/s and 100 Gb HD SSD 6 Gb/s.

Results

Figure 1 reports the geometric mean of all queries over each dataset (including queries with local and federated access). We observe that Virtuoso obtains better results in terms of execution time for all datasets. This means that, in general, Virtuoso outperforms Fuseki and GraphDB for all queries and dataset configurations. We also observe that the increase in the dataset size impacts the performance of the engines. For example, Virtuoso outputs 0,114 seconds as the geometric mean for the 2018 dataset, while the time is increased to 2,221 seconds for the 2018-2022 dataset. GraphDB, and mostly Fuseki, are highly impacted by the increasing of the dataset performance. Fuseki results in 64,007 seconds as the geometric mean for the 2018-2022 dataset. GraphDB reports better results, but in general, the performance is not comparable to the results obtained by Virtuoso.

For a more fine-grained analysis of the results, we show the results of Virtuoso for each query in Figure 2. The median of the 5 executions is reported. The first observation is that the dependency on external resources such as the Publication Office SPARQL endpoint, in this case, has a clear impact on all queries. For all configurations, the query version that depends on external resources (i.e. qX_opt, where X is the indicator number) reports always worse performance than the ones that retrieve these answers from a local graph (i.e. qX_opt_local, where X is the indicator number).