Invitation to Tender: provision of a secure data platform (‘Smart Data Sandbox’) for participants in a Smart Data Challenge Prize
Deadline for submissions: 17:00pm BST, Friday 10 May 2024
1. Background to Nesta and Challenge Works
We are Nesta, the UK's innovation agency for social good. We design, test and scale solutions to society's biggest problems Our three missions are to give every child a fair start, help people live healthy lives, and create a sustainable future where the economy works for both people and the planet For over 20 years, we have worked to support, encourage and inspire innovation. We work in three roles: as an innovation partner working with frontline organisations to design and test new solutions, as a venture builder supporting new and early stage businesses, and as a system shaper creating the conditions for innovation.
Challenge Works, a social enterprise within Nesta, is a global leader in design and delivery of open innovation challenges that mobilise diverse, innovative thinkers to solve pressing problems and unlock change. Challenge prizes are a unique approach to funding innovation, offering a series of incentives, with a final prize given to whoever can first or most effectively meet a defined goal They are public, open competitions which lower barriers to entry to attract the broadest possible community of innovators Challenge Works has delivered over 80 challenges to date and distributed more than £124 million to winning innovators.
More information can be found at www.nesta.org.uk and at www.challengeworks.org
2. Background to the project
Smart Data is the secure sharing of customer data, upon the customer’s request, with authorised third-party providers (ATPs) who often enhance the customer data with broader, contextual ‘business’ data. These ATPs provide the customer with innovative services to improve decision making and engagement in a market The UK open banking framework is an example of Smart Data in action.
The Smart Data Challenge Prize is designed to showcase the potential opportunities
Smart Data could bring to the UK economy, and to help inform the development of Smart Data schemes beyond open banking The Prize will take place between summer 2024 and March 2025. Participants in the Prize will benefit from a share of up to £750,000 in grant funding and access to a bespoke Data Sandbox enabling them
to prototype and test a range of cross-sector Smart Data use cases This ITT seeks a supplier to provide this Data Sandbox.
The Smart Data Challenge Prize is delivered in partnership between the Department for Business & Trade (DBT), Challenge Works (a Nesta enterprise), the Open Data Institute (ODI) and Smart Data Foundry (SDF)
The Smart Data Challenge Prize follows the Smart Data Discovery Challenge, where innovators were encouraged to share their innovative cross-sector use case ideas, and four winners were awarded £5,000. Find out more at https://smartdata challenges org/
3. Indicative timetable
Set out below is the proposed procurement timetable This is intended as a guide only.
DATE TIMETABLE
17 April 2024
26 April 2024
10 May 2024, 17:00pm BST
17 May 2024
17-24 May 2024
28 May 2024
Tender issued
Deadline for submission of clarification questions by potential bidders, by email to smartdata@challenges.org
Contact Nesta by email at smartdata@challenges org by 26
April to obtain copies of the resources referenced in Annex 3
Deadline for receipt of tenders
DBT project team reviews submissions and evaluates against Award Criteria
Shortlist of submissions identified by DBT Project Team
Clarification questions to/ interviews of shortlisted suppliers
Further evaluation of submissions against published Award Criteria
Bidders notified of tender result
28 May-
14 June 2024
17 June 2024
Contracting
Commencement of work
4. What is being commissioned through this Invitation to Tender
This tender has been prepared by Nesta for the purpose of inviting proposals from suppliers for the provision of a secure data platform that will serve as a Smart Data Sandbox, enabling participants in the Smart Data Challenge Prize to prototype and test a range of cross-sector Smart Data use cases.
The primary function of the Smart Data Sandbox is that it will provide access to synthetic data that replicates identified industry sectors This synthetic data will be generated by the DBT project team and provided to the Sandbox supplier.
Functional requirements
1. Direct data access and prototyping workspace
a A workspace for innovators to directly access and analyse data sets to enable them to develop their solutions.
b The workspace will provide access to the synthetic data generated to replicate a range of industry sectors eg, financial services, retail, transport, energy & property.
c Innovators should be provided with a private workspace per team, where they can manipulate the data without impacting other innovators
d. Innovators will be able to upload, create or request the creation of new tables/temporary tables in their workspace
e. Innovators should be able to directly access data tables using a computational notebook supporting the specified software (see Appendix D).
f Innovators should be able to directly connect to the data using common data connections to Microsoft Excel or PowerBI
g Innovators are able to develop and deploy prototype applications and solutions.
2. API access & development
a. Sandbox to provide APIs emulating, for example, a bank, utility company or similar in a Smart Data landscape These are not intended to play any formal role in future Smart Data API scheme definition, but to emulate access for prototyping See Annex 3 for details about the APIs required.
b APIs (see Annex 3) will provide simple data retrieval (read only) covering customer, company, entity and transaction information as relevant for each sector
c. Provision (development and deployment) of a suite of simple APIs to provide basic access to the synthetic data from different sectors within the Sandbox as specified in Annex 3
d. Authentication required to validate authorised Sandbox users.
e APIs to be externally addressable but non-discoverable
f. APIs to be clearly documented (eg, via Swagger).
g Access to be provided to the specified public or govt APIs - see Annex 4.
3. Innovator permissions
a Innovators should only be able to access the Sandbox after having been provided with unique credentials (DBT project team will determine the approved innovators).
b. User access should be via Username and Password which will support all authentication in the Sandbox (Direct Data Access Workspace and APIs)
c. At the request of DBT project team access should be granted or revoked for specific users
d. Innovator permissions can be set at the workspace level, ensuring teams can protect any intellectual property
4. Innovator support
a. The Sandbox will provide the ability for the DBT project team to share support documentation regarding synthetic data and APIs
b It will provide all the information Innovators require to be operational in the Sandbox. This should include the ability to store training videos, how to guides, ‘hello world’ examples, etc.
c The DBT project team must have a specific Data Access Workspace and accounts for support activities such as testing new and existing data, code snippets and replicating/analysing user issues
5. Project team access
a. The DBT project team comprises representatives from the Department for Business and Trade, Smart Data Foundry, the Open Data Institute (ODI) and Nesta; and includes synthetic data experts, Smart Data experts, industry experts and Regulators who will all be providing support to innovators while these have access to the Sandbox.
b. The DBT project team should be able to observe and review the approach and use of the Sandbox to provide support to Innovators
Non-functional requirements
6. Synthetic data
a. DBT project team will provide synthetic data covering multiple industry sectors
b. synthetic data is anticipated to be approximately 100 tables and 220m rows of data. and 100GB of data.
c Ability for the project team to upload new or updated data sets throughout the duration of the Sandbox.
7. Expected performance
a APIs should provide response within three seconds for >95% of all requests.
b The data workspaces should be sized with adequate storage and compute power to provide similarly prompt and robust performance for innovators performing analyses, data manipulation and prototyping solutions.
c. Complexity and concurrent demand is expected to scale in the later phases of the project.
8. Security
a. Innovators must not have the ability to remove data in bulk from the Data Access Workspace.
b API access must limit data retrieval to one customer/entity account and supporting transactions at a time.
c Innovators must not be able to access other innovators solutions or workspaces.
d Innovator activity should be audited, see Annex 5
e. All IP belonging to the Innovators or DBT project team remains the property of the relevant party and the Sandbox provider has no claim or right to use without explicit consent.
9. Service expectations
a. Sandbox should be able to support the expected user base in line with the expected performance defined
b. Estimated user base of 10 Innovator teams with approx. 3 to 5 users per team and DBT project team of approx 15 users
c. Sandbox should be available between 8.00am & 18:00pm weekdays and available on best endeavours outside of core hours
d. Sandbox should be provisioned in the cloud in line with standard government practice
10. Reporting and analytics
a. Service analytics provided to the project team covering Sandbox usage and performance and user analytics to identify insights on most used datasets and APIs, Sandbox features, etc.
b. See Annex 5 for required reporting.
11. Sandbox maintenance
a Computational Notebook including Python & R libraries and external data connections/drivers to be updated to requested versions within 10 working days of request of DBT project team
b. DBT project team may provide updates to synthetic data sets with corrections or additional data that will require to be updated in the Sandbox within two working days of request.
c APIs to be maintained through the duration of the Sandbox phase including corrective maintenance or extensions/additional APIs to support new synthetic data provided.
12. Extended access
a As interest in Smart Data ecosystems and use cases gains momentum, DBT can envisage running the Sandbox beyond the initial Smart Data Challenge Prize timeline above, potentially for a period of up to three years. Any extension would be procured by DBT and contracted separately, but this potential to extend should be considered in the response.
Value-add capabilities
13. Solution value-add capabilities
a. Please list any value add capabilities related to your Sandbox offering as part of this proposal
14. Organisation value-add capabilities
a. Please list any value-add capabilities your organisation offers as part of this proposal
5. Milestones and budget
The following are the key firm deadlines for delivery of the Sandbox:
● Sandbox to be provisioned by Friday 2 August 2024.
● Sandbox to be populated with synthetic data and APIs by Friday 23 August 2024.
● Sandbox acceptance signoff by DBT project team by Friday 30 August 2024.
● Sandbox to be open for use by Challenge Prize participants from Monday 2 September 2024 until Friday 28 February 2025.
Payments will be released in stages alongside agreed milestones, providing these have been met The final milestones shall be clearly documented in any contract
entered into as a result of this tender, and in accordance with the terms and conditions of this tender.
The budget available for the above services is up to a maximum of £220,000 including all expenses and travel costs (excluding VAT). Bids above £220,000 will not be considered
6. How to apply
Your entire proposal including all supporting documentation should be e-mailed, as a single document, to smartdata@challenges.org by no later than 17:00pm BST on Friday 10 May.
If you are interested in submitting a proposal you are invited to notify Nesta by email at smartdata@challenges org by Friday 26 April 2024 to express your interest We will then share further details of the APIs required (see Annex 3).
Proposals should be submitted:
● in English
● as a Microsoft Word file (or similar), single-spaced in 12-point font
Your proposal must not exceed 15 sides of A4, not including CVs of key team members, case studies and Additional Information (see below).
In preparing a proposal in response to this tender, suppliers should be guided by the Award Criteria that will apply to proposals, as set out in Annex 1 of this tender (Evaluation of Proposals) Where Annex 1 states that bidders must supply specified information, then your proposal must include that information.
Annex 2 sets out the Terms and Conditions which will govern your submission of a proposal to Nesta.
If you have any specific questions or want any clarification in relation to the tender, please email smartdata@challenges.org by Friday 26 April 2024. Responses will be shared by Tuesday 30 April 2024
Annex 1
Evaluation of Proposals
1. Evaluation Process
Nesta will run this competitive procurement process in a manner which is open and transparent and ensures equal treatment of participants in the process Decisions will be made on the basis of the Award Criteria below. The tender that scores the most marks against the criteria below is the preferred supplier
1.1 Contract Award Criteria
Your proposal should describe your methodology for meeting the Functional Requirements, consistent with the firm deadlines, as set out in this tender
Your proposal must identify any high impact, high probability risks you have identified to delivery of the Functional Requirements and your approach to mitigating these
Your proposal should describe your approach to meeting the Non Functional Requirements, consistent with the firm deadlines, as set out in this tender
Your proposal must identify any high impact, high probability risks you have identified to delivery of the Non Functional Requirements and your approach to mitigating these
Your proposal should describe any value add capabilities that you will bring to delivery of the requirements set out in this ITT.
Your proposal should detail your organisation’s relevant experience of delivering similar services. You must provide at least one, and up to three, relevant case studies Each case study should include a) a description of what you delivered, b) the timescale on which you delivered, c) the approximate value of the work, and d) for each case study the name and contact details of a referee who will be able to comment on your work.
If you are bidding as a consortium, you must provide the proposed management structure and details of the partner organisations. Please identify a lead organisation that will be responsible for completing this form on behalf of all consortium members.
You must identify the key team members who will be involved in delivering the requirements, including the Senior Responsible Owner (or similar) for the project
Please describe the relevant experience of the key team members. You may include CVs for key team members as appendices to your proposal
You must provide a breakdown of your proposed fees and pricing structure including the number of days, or estimated percentage of time, each key team member is expected to spend on the project. Please also indicate which team member will be responsible for delivery of key workstreams
You must show how contingency resource for additional API development (11c) and additional data support (6c) throughout the Challenge Prize is reflected in your pricing
As part of your proposal you must also provide the following Additional Information, which is not scored and is not included in the specified page limit:
Additional information: contractual terms
Please provide your standard contractual terms for our consideration. Please note that selection of the successful tenderer by Nesta does not indicate acceptance by Nesta of the tenderer's standard contractual terms. Any contract entered into as a result of this ITT will be subject to negotiation and approval by Nesta
Bidders should note that the scope of work under this ITT is part of a wider project under which Nesta has contracted with a funder Some of the terms of that contract may need to be flowed down into the contractual arrangements between Nesta and the successful tenderer. You will be advised of such flow down terms in due course
Additional information: conflicts of interest
Please confirm if you are aware of any potential or actual conflicts of interest which may prevent you from working with Nesta and/or from providing the services described in this ITT. If you consider a conflict may exist, please provide details in your response
Where conflicts of interest are identified Nesta reserves the right to exclude the supplier from further participation in this tender process
1.3 The following scoring methodology will be applied to responses:
Score Commentary
1-3 Response provides Nesta with a poor-weak level of confidence in the supplier’s ability to meet its requirements
4-6 Response provides Nesta with a moderate-good level of confidence in the supplier’s ability to meet its requirements
7-8 Response provides Nesta with a very good level of confidence in the supplier’s ability to meet its requirements
9-10 Response provides Nesta with a high level of confidence in the supplier's ability to meet its requirements
Annex 2
Terms and Conditions
This appendix provides the terms and conditions which will govern your submission of a proposal to Nesta. You are required to read this appendix carefully together with the Invitation to Tender (ITT) brief (including all its annexes) before submitting a proposal. By submitting a proposal, you accept these terms and conditions and you agree to abide by them. If you do not agree to these terms and conditions please do not submit a proposal
1. Submitting a proposal
1 1 By submitting a proposal you confirm that:
1.1.1 you have no actual or potential conflict of interest with Nesta (including Nesta’s Trustees, officers or employees) If at any time during the ITT process you discover an actual or potential conflict of interest, please inform Nesta promptly;
1.1.2 you have legal capacity to submit a proposal in response to this ITT and are acting lawfully, ethically and in good faith in your dealings with Nesta;
1.1.3 you have sufficient time, skill, experience and resources to carry out the services to the highest professional standards expected of a competent supplier of services identical or similar and are able to obtain all necessary rights, licences, consents, waivers, approvals,
permissions, permits, certificates and insurances necessary to provide the services to Nesta;
1.1.4 your tender will remain open for acceptance for a period of 90 days. A tender valid for a shorter period may be rejected;
1.1.5 all information contained in your proposal is true, accurate and not misleading; and
1.1.6 Nesta may share your proposal with any third party as Nesta in its absolute discretion deems necessary for the purpose of evaluation
2 Compliance
2.1 Nesta reserves the right to reject or disqualify your tender where:
2 1 1 you fail to comply with the requirements of this ITT (including but not limited to these terms and conditions), your tender is incomplete, or you are guilty of a serious misrepresentation in supplying information in response to this ITT;
2 1 2 your tender is received after the deadline set out in this ITT Nesta will not consider any requests for an extension of the time or date fixed for the submission of responses;
2 1 3 there is a change in your identity, control, financial standing or any other factor impacting on the selection and/or evaluation of your tender;
2 1 4 you are or become insolvent or have a petition issued against you;
2.1.5 you do not have the economic and financial standing and/ or the technical and professional ability to carry out the services;
2.1.6 you are suspected either directly or indirectly of behaving in a collusive, canvassing, or anti-competitive manner or you offer or accept an inducement or reward in order to gain a commercial, contractual, regulatory or personal advantage; and/or
2.1.7 you (or if you are a commercial entity, a director or person who has the power of control or power to make representations or decisions on your behalf) have committed any offence relating to conspiracy, corruption, bribery, fraud, money laundering or any other criminal offence related to your course of business or profession
2.2 Nesta reserves the right in its absolute discretion to:
2 2 1
refuse any tender submitted;
2 2 2 disqualify any potential tenderer who has been convicted of any of the offences listed at Regulation 23(1) of the Public Contracts Regulations 2006;
2 2 3 extend the time or date for fixed submission In such circumstances Nesta will endeavour to notify all tenderers of any change;
2 2 4 amend any aspect of this ITT (including but not limited to the evaluation criteria and the timeline) or cease the process at any time;
2 2 5 negotiate the award of additional services which are a repetition of the services advertised in this Invitation to Tender to the successful bidder;
2.2.6 limit the number of proposals invited to participate in any follow-up activity (for example, interviews) or to dispense with any follow-up altogether Meetings will take place in London at our offices; and/ or
2.2.7 require that you clarify your tender in writing and/ or provide additional information and/or adequate references to Nesta’s satisfaction. A failure to respond adequately may result in you not being selected
3.1 You shall be responsible for all of your own costs, expenses and losses which may be incurred in relation to the preparation of your tender, attendance at interviews, site visits or similar
3.2 You shall at all times treat the contents of Nesta documentation as confidential, as well as any information regarding Nesta imparted to you by any other means, and only disclose such information as may be necessary for the preparation of a compliant response At Nesta’s request you shall return or destroy all documents, other materials, working papers relating to this ITT and all copies thereof including all electronic copies. When completed you shall confirm such to Nesta in writing.
3 3 You shall not before the date and time specified within the ITT documentation disclose to any person the amount of your proposal except where the disclosure in confidence is necessary to obtain insurance premiums or guarantees required as part of any proposal to Nesta
3.4 Any contract entered into as a result of this ITT will be subject to negotiation and approval by Nesta
4 Intellectual Property
4 1 All intellectual property rights in this ITT and all materials provided by Nesta or any third party acting on its behalf shall remain the property of Nesta.
4 2 Any intellectual property arising out of the provision of the services shall belong absolutely and exclusively to Nesta, unless otherwise agreed
5 Warnings/Disclaimers
5 1 Nothing contained in this ITT or any other communication made in respect of it between Nesta or its representatives and any party will constitute an agreement, contract or representation between Nesta and any other party. For the avoidance of doubt, receipt by you of this ITT does not imply the existence of a contract or commitment by or with Nesta for any purpose
5.2 The information contained in this ITT does not purport to contain all the information which you may require While Nesta has taken all reasonable steps to ensure, as at the date of this ITT that the facts contained in it are true and accurate in all material respects, Nesta does not make any representation or warranty as to the accuracy or completeness or otherwise of this ITT
5.3 Nesta accepts no liability to you whatsoever and however arising and whether resulting from the use of this ITT, or any omissions from or deficiencies in it
6 Waiver
Failure or neglect by Nesta to enforce at any time any of the provisions of these terms and conditions shall not be construed nor shall it be deemed to be, a waiver of our respective rights hereunder, nor in any way affect the validity of the whole or any part of this agreement, nor prejudice our respective rights to take subsequent action
7 Jurisdiction
This ITT shall be governed by and interpreted in accordance with the laws of England and Wales and subject to the exclusive jurisdiction of the English courts.
Annex 3
API Specification
The following read only APIs providing access to the Sandbox data are to be developed by the successful supplier to represent the actors across the sectors
Contact Nesta by email at smartdata@challenges.org by 26 April 2024 to obtain copies of the resources referenced below
CitizenID (mock DigitalID)
Get citizen information
See citizenID.xslx for details - will be provided on request
CompanyID (mock DigitalID)
Get company information
Get director information
See companyID xslx for details - will be provided on request
Finance
Open Banking APIs (2 Retail Banks)
Get Account
Get Transactions
Lending (2 Retail Banks)
Get Account
Savings & Investments (2 Investment firms)
DBT project team to provide developed APIs for provider to run
Insurance (2 Insurers)
Get vehicle policy information
Get property insurance information
Get claim information
See retail bank xslx, investment firm xslx, home insurer xslx, car insurer xslx for details - will be provided on request
Energy
Electricity (2 Utility companies)
Get Customer and Account Information
Get Meter Readings
Gas (2 Utility companies)
Get Customer and Account Information
Get Meter Readings
See gas xslx and electric xslx for details - will be provided on request
Retail (2 Supermarket Retailers)
Get Customer and Loyalty Account Information
Get Basket Information
See supermarket xslx for details - will be provided on request
Property/Home Buying
Land Registry (1 Land Registry)
Get property details
Get Property Charges
Conveyancing (2 Estate Agents)
Get property details
See land reg.xslx and conveyancing.xslx for details - will be provided on request
Transport (2 Fleet tracking providers)
Get vehicle details
Get trip details
Guidance spreadsheet to be provided once scope finalised
Annex 4
Public Data API Access APIs
External sources of data to be brought into the Sandbox via API calls Required public data APIs are:
1 Bus Open Data Service
2. National Public Transport Access Nodes (NaPTAN) and National Public Transport Gazetteer (NPTG) API
3. TfL API
4 Met Office Weather APIs
5. Google Solar API
6 Ordnance Survey APIs
7. Property Data Trust Framework APIs
8. Octopus Energy APIs.
Datasets
There is also a requirement to provide participants with external datasets, which will include:
1. UKPN: Commercial electric vehicles trial
2. ONS: Median estimated C02 emissions, England and Wales
3. Use land and property data
4. HBSG: Property Data Trust Framework
5. Our World in Data: Greenhouse gas emissions per kilogram of food product.
Annex 5 Reports
As a minimum the following reports to be provided to the DBT project team available on a weekly basis
Management information
● Top 10 users by login.
● Top 10 users by access duration
● Top 10 API calls by volume.
● Top 10 Data Table access by volume
Audit information
● User Access – General information on users registered, login sessions, failed logins.
● Data Table Access – Date, time and user for each database query.
● API Access – Date, time and user for each API call
● Data tables created – by innovator workspace.
System performance
● Data table query performance
o Average - max, min, median, 95th percentile
o Top 5 longest query tables.
o Top 5 longest query users
● API performance – max, min, median, 95th percentile.
Annex 6
Computational Notebook
Technical requirements
1. Volume of the data
a. The data is expected to have a size of 100GB unzipped so the systems for processing data and the database engine need to be robust enough to handle large files.
b Limitations on the volume of data the notebook can handle must be stated clearly.
2. RAM
a The notebook needs to support a minimum of 32GB per user and should have the ability to scale according to demand.
b. Limitations on the memory of the notebook must be stated clearly.
3. Languages
a The minimum requirement for the notebook is Python but the Sandbox should also provide other languages.
i Desirable languages are R and SQL
b. Users should have the ability to install other packages, if needed, to process the data
c. The minimum package starting list for Python and R contains:
Python:
▪ pandas (2 0 with arrow back end)
▪ ydata-profiling
▪ dask
▪ numpy
▪ polars
▪ matplotlib
▪ seaborn
▪ plotly
▪ scikit-learn.
R:
▪ tidyverse
▪ arrow
▪ tidymodels
▪ data.table
▪ janitor
▪ shiny
▪ rmarkdown
▪ quarto
▪ devtools
▪ caret.
4. User privileges
a. Users should have appropriate access and must not be able to delete or interfere with the data provided
5. External data access
a. Users should be able to access samples outside of the Sandbox, if required
i. APIs created to control/monitor the volume of data that can exit the Sandbox, with a limitation on call size
6. Data ingress
a. The project team should be made aware of any processes of data manipulation on the synthetic data once in the Sandbox.
b. The project team should have access to the Sandbox for verification and testing.