AI & Digital Healthcare Technologies Framework
1.0 Digital Implementation
| L1 - ‘I am aware of a variety of digital enablers and can distinguish the main features of each (e.g. the web)’; ‘I am aware of the digital technologies available to me in my role, and how to access training to become competent to use these technologies’ | ExR is one of many digital enablers that can support the development of digital competencies. |
| L2 - ‘I have an awareness of accessibility issues among different patient/user populations and how to mitigate against the risk of widening inequalities as a result of introducing digital technologies and can provide or recommend suitable alternatives if required or requested’; ‘I develop my professional practice to work with other experts and teams to support digital implementation, both within my professional community and wider (e.g. contributing lessons learnt with others’ | As ExR trainings are available to all across the NHS, trainings can contribute to sharing lessons and professional practices with other professional communities. |
| L3 - ‘I can investigate, record and communicate socio-technical factors that may impact the adoption of digital technology implementation in my organisation’; ‘I am aware of the potential environmental implications of digital transformation (e.g. carbon footprint of AI) and attempt to offset this impact by improving efficiency once systems are deployed’ | ExR provides a robust outline of their environmental impacts. Using ExR technology has a lower digital footprint than completing trainings in person contributing to NHS Net Zero goals. |
| L4 - ‘I can capture and model existing information /data pathways in my organisation in order to understand the current workflows and to prepare for digital transformation projects’; ‘I am able to assess and implement the required infrastructural changes to physical environments to support digital transformation (e.g. digitally integrated operating room)’ | ExR requires no infrastructural changes to physical environments as trainings can be completed on a mobile phone or laptop. Trainings can also be completed with VR headsets should they be available. |
2.0 Digital Health for Patients and the Public
| L1 - ‘I am aware of a variety of digital NHS resources available that are relevant to my role or my team (e.g. apps/websites) and can give guidance to patients to access and use them appropriately and provide support (e.g. with care and/or increasing knowledge about their health)’; ‘I am able to identify and find alternative solutions to maintaining emergency care where digital systems fail or are identified as failing’; ‘I am proficient at recording and accessing clinical observations accurately and securely in an electronic record system for clinical/health care’; ‘I can recognise situations where patients encounter ‘digital exclusion’ and proactively ensure alternative access to care when identified’; ‘I can explain digital health information to patients to support them in making data-driven decisions about their health (e.g. health risk scores/metrics)’; ‘I respect patients’ preferences around the use of digital technology and empower and support them to make informed choices about technology used in their care’; ‘I can evaluate and recognise the pros and cons of automated systems (e.g. triage, symptom checkers) used in the NHS and can help signpost to inform and guide patients and users when access to such systems is appropriate to their care’ | N/A |
| L2 - ‘I can confidently explain to patients what AI systems in my specialism/area of practice are doing in a broad sense and can tailor my explanation accordingly with awareness that patients have different levels of digital literacy’; ‘I can initiate conversations with patients to improve their understanding of data collection, use and data security’ | N/A |
| L3 - ‘I can clearly explain to patients how their data will be collected, stored and safeguarded by systems recommended for their care/research and their respective rights’; ‘I can critically appraise resources aimed at improving digital literacy for health and support members of the public to find and assess appropriate data or health information’; ‘I actively promote digital literacy by signposting them to appropriate resources to assist them with technology we intend to interact with’ | N/A |
| L4 - ‘I can recommend digital resources through policies and guidelines to patients and the public for various health conditions based on the Digital Technology Assessment Criteria (e.g. apps through the NHS website)’; ‘I can provide training and education to healthcare professionals that train patients and the public to improve digital literacy skills using formalised pedagogical approaches (e.g. using techniques such as ‘teach-back’ and ‘chunk and check’)’ | N/A |
2.1 Providing Direct Care with Digital Technologies
| L1 - ‘I am aware that a variety of sensor devices are available to the general public and that these devices are not necessarily approved for clinical purposes and that as such the quality of the data collected may not be of sufficient quality/reliability for use in remote patient monitoring and care based decision making’; ‘ I can work in adapted clinical environments integrated with digital technology safely and effectively (e.g. digitally integrated operating rooms’ | Completing trainings with ExR helps individuals learn to integrate with digital technology by familiarising them with the idea of technology in the workplace. |
| L2 - ‘I am technically proficient in the use of common computational devices (e.g. smartphones, tablets) to facilitate clinical interactions and view/edit records using software relevant to my discipline (e.g. health information systems)’ | Though ExR technology does not allow one to view or edit patient records, completing ExR trainings encourages individuals to develop technical proficiency in the use of common computational devices. |
| L3 - ‘I am aware of and can define digital/eHealth and related sub-categories (e.g. uHealth, mHealth)’; ‘I am aware of and can describe Extended Reality (XR) and its subdomains (e.g. virtual reality)’; ‘I can define the term ‘digital twin’ (a digital representation of a real world object, service or process) and can give examples of their use in healthcare (e.g. bed/hospital management)’; ‘I am able to customize and configure eHealth solutions for use in clinical practice and can troubleshoot common issues arising from such solutions’; | ExR trainings encourage individuals to develop digital competencies and XR and VR skill sets. Trainings utilise digital twins to aid individuals in developing practices that can be utilised in real-life. |
| L4 - ‘I create opportunities to engage patients and the public in the design of digital solutions that reflect their needs and priorities (e.g. co-design, user centred design)’; ‘When applying Artificial Intelligence (AI) processes to direct patient care, I am able to second-check decisions made by automated processes to ensure they meet the required standards of safety and quality and know how and where to report concerns if they fail to meet these standards’ | TBC |
2.2 Remote Consultation and Monitoring
| L1 - ‘I recognise the benefits and risks of using different messaging platforms for patient communication and always abide by the recommendations of my organisation regarding preferred tools to achieve this safely and securely’; ‘I recognise the role of remote patient monitoring for care (e.g. post-surgical intervention) and how this can support monitoring and safety of patients at increased risk (e.g. in remote/rural areas)’; ‘I always use workplace recommended systems to facilitate secure messaging and file transfer (e.g. cryptographically secure tools)’; ‘I can carry out patient consultations, facilitate secure real time audio/visual communication tools, interpret data collected with these tools and make decisions based on these data’ | N/A |
| L2 - ‘I am aware of the stages involved in remote care pathways from consultation through to remote monitoring and applying digital interventions in my specialty/area’; ‘I can advise members of the public on preparing for a consultation online and support a shared decision making discussion in line with the National Institute for Health and Care Excellence (NICE) shared decision making guidelines’ | N/A |
| L3 - ‘I am confident in the use and troubleshooting of digital tools to facilitate secure real time audio/visual communication (e.g. patient consultations, internal/external meetings with other healthcare providers and groups) and can adapt my consultation approach to remote consultations for the majority of my patients’; ‘I am aware of different digital remote monitoring options that are available to patients in my area/speciality (e.g. mobile apps, implantable/wearable devices) and can communicate the basic operation of these options to others’; ‘I can provide user support to help patients setup remote monitoring technology and collect data through biosensors’; ‘I am able to use, interpret and critically appraise data provided by individual patients (e.g. collected by commercial wearables) and use this to promote patient wellbeing, health and the prevention of disease’ | N/A |
| L4 - ‘I can explain to patients and the public how wearables related to my area of practice work in terms of what data they collect, how the data is collected from the wearer and any available data sharing options’; ‘I can recommend (through accessing policies/guidelines) different remote monitoring options that are available to patients in my area/specialty (e.g. mobile apps); ‘I can interpret large quantities of remotely collected telemetry data with appropriate tools to monitor patients and use insights from this process for decision making in patient care’ | N/A |
3.0 Ethical, Legal and Regulatory Considerations
| L1 - ‘I champion a culture of ethical responsibility around the use of Artificial Intelligence (AI) and digital technology to ensure that systems and processes are fair, transparent, equitable and non-discriminatory to patients, staff and the wider public; espousing the principles of beneficence, non-maleficence and autonomy’; ‘I know how to access up to date guidelines and knowledge about digital systems at the point of care and only use such systems I am trained and competent to use, recognising my own limitations and competence’; ‘I always ensure that appropriate consent for data sharing has been granted prior to processing, storing or sharing data’; ‘I abide by legal requirements and directives regarding the access to and processing of data (e.g. GDPR)’; ‘I am aware of the potential litigious consequences of a data breach to my organisation (e.g. fines, damage to reputation) and actively take steps to prevent this by abiding by my organisations standards and guidelines’ | ExR abides by all relevant data protection and privacy laws and ensures that consent is given by participants in all of their trainings and videos. In addition, ExR trainings always begin with requiring the participant to gain the consent of the patient, ensuring that ethical, legal and regulatory considerations are embedded into trainings from the beginning. |
| L2 - ‘I am aware of ethical considerations around information governance for digital health technologies including remote consultations/monitoring (e.g. consent)’; ‘I am aware of the ethical issues of automating clinical decision making or relying on algorithms and digital technology that are not fully transparent/understandable’; ‘I am aware of the legal, ethical and accountability implications for my organisation of using Artificial Intelligence (AI) for augmented clinical decision making’; ‘I am aware of the legal, ethical and accountability implications for myself and my organisation when following or overruling AI based recommendations in augmented clinical decision making’; ‘I am aware that legislation may not keep pace with technological innovation. Where clear regulations do not exist I aim to apply the ethical principles of beneficence, non-maleficence, autonomy and justice as a guide to using digital technology (e.g. promoting principles of privacy)’; ‘I am familiar with the governance requirements (including UK Conformity Assessed (UKCA/UKNI), CE marking) for medical devices including software and AI concerning regulation, quality management, testing and ongoing monitoring’ | N/A |
| L3 - ‘I am aware that data collection tools and systems should be designed to account for patient/user diversity so that the output of data processing/analysis will not cause or increase health inequalities’; ‘I am aware of the need for citizen consent and am fluent in the reasons and actions taken to protect and enshrine public trust’; ‘I am aware of clinical evaluation frameworks (e.g. International Medical Device Regulators Forum (IMDRF) recommendations, National Institute for Health and Care Excellence (NICE) evidence standards framework for digital health technologies) that can be used to clinically evaluate software and AI as a medical device’; ‘I am aware of how to determine if a given automated system has been appropriately validated for use in its intended clinical context and has undergone a clear, transparent approval process’; ‘I am aware of issues around consent and medical negligence (e.g. Bolam, Bolitho, Montgomery v Lanarkshire) and the challenges of applying these to AI and digital technologies’; ‘I understand regulatory approval does not guarantee the performance of a medical device or algorithm in any given context’ | ExR collects the minimum required data from individuals |
| L4 - ‘I am aware of the different wearable remote monitoring devices and ensure that when recommended to patients they have appropriate regulatory approval (e.g. CE, UKNI or UKCA mark)’ | N/A |
4.0 Human Factors
| L1 - ‘I acknowledge the need for multidisciplinary collaboration to ensure the safe and effective use of digital health and Artificial Intelligence (AI) technology’; ‘I am aware of specialists roles and teams involved in AI and digital healthcare (e.g. informaticians, software/data engineers)’ | ExR is built by healthcare and technology specialists who ensure that ExR trainings are safe and effective through their multidisciplinary collaborations. |
| L2 - ‘I am able to assess the potential impact of embedding digital technologies and AI into clinical workflows through engagement with affected teams and individuals’ | ExR trainings encourage individuals to be more aware of the ways in which technology can influence their workflows and improve their practice. |
| L3 - ‘I am aware of the level of my organisation’s digital maturity and its strategy for digital transformation’; ‘I promote and initiate effective communication between key stakeholders during the implementation of a digital technology/AI system’; ‘I work effectively with technology industry partners to procure digital services and tools (e.g. data storage) and recognise the advantages and disadvantages of out-sourcing, on-premises and co-creation options’ | ExR has a history of partnering with NHS trusts across the UK to partner effectively on developing digital trainings and tools. |
| L4 - ‘I practice patient/user centred design. I work with patients and users to gather and produce requirements (e.g. user stories, personas), set priorities and evaluate and integrate technologies’; ‘I can clearly explain the strengths and weaknesses of a variety of digital technologies to non-technical stakeholders to facilitate effective collaboration’; ‘I can discuss the challenges associated with my organisation’s digital transformation strategy and discuss potential technological solutions that may address them, including consideration of the impact of these technologies on patient outcomes and wider organisational goals’ | ExR trainings are built in partnership with NHS trusts and can be tailored to patient/user centred metrics as needed. |
4.1 Management and Leadership Planning
| L1 - ‘I can effectively use information derived from data driven approaches for the purposes of management planning (e.g. data summaries via digital dashboards)’; ‘I am confident applying formal change management methods and implementation science to digital transformation projects’; ‘I actively promote appropriate confidence for healthcare professionals regarding the use of digital technology in healthcare’; ‘I lead and enable others to support the development and use of digital technologies, supporting national guidelines and agendas’ | ExR trainings can be used to meet CPD requirements and as such can be built into effective management and leadership planning. |
| L2 - ‘I am aware of the need to ensure retention of core skills when introducing digital technology. I ensure back-up pathways are in place to be used in situations when digital technology fails’; ‘I appreciate that responsibly designed and implemented digital technologies and Artificial Intelligence (AI) systems has potential to increase efficiencies, reduce costs and support healthcare staff to deliver better care’; ‘I am able to recognise an adverse effect (e.g. failure of technology) of an automated or data enhanced pathway and can take steps to rectify this within my organisation’; ‘I am proficient in carrying out cost-benefit analysis of the impact of new technologies on healthcare systems’; ‘I recognise the need for protected time and space for professionals to access appropriate learning resources related to digital technologies. I support a learning culture and emphasise the importance of continuing professional development in digital skills and capabilities’; ‘I can recognise different kinds of expertise in staff interacting with Artificial Intelligence (AI) and digital technology and can effectively bring together and leverage this expertise to implement these technologies successfully’ | ExR trainings can be used to meet CPD requirements and are effective at ensuring that core skill sets remain strong. |
| L3 - ‘I am aware of software and project management methodologies (e.g. Agile, Lean) and the pros and cons of the different methods for different types of projects’; ‘Following AI/digital technology implementation I can effectively monitor and evaluate AI systems and digital technology projects to ensure they align with financial, clinical and organisational objectives and ensure ongoing effectiveness’; ‘I am able to carry out a review of existing technologies and consider how these can be utilised to meet the strategic goals of my organisation’; ‘I actively identify key problems in clinical areas and have an awareness of how digital technologies and AI can be used to address identified problems’; ‘I am familiar with digital clinical safety and have implemented an effective contingency/mitigation plan’ | N/A |
| L4 - ‘I know how to identify the quality assurance (verification and validation) of AI systems and digital technologies in my locality based on local policies and guidelines’; ‘I create protocols for the safe use of AI and digital technology in my organisation/area based on evidence (e.g. risk assessment). I define the situations in which the technology will be used and by whom’; ‘I am able to evaluate the most appropriate technical situation to a problem including AI/digital technology. This includes reviewing financial, technical and logistical considerations’; ‘I lead a digital transformation strategy for my organisation which considers both short and long-term goals and determines metrics to measure success’; ‘I actively sponsor, recognise and promote intellectuals who demonstrate excellence and maturity in the capabilities found within this document’ | N/A |
5.0 Health Data Management
5.1 Data Management and Processing
5.1.1 Data Collection and Context
| L1 - ‘I am proficient at interpreting information presented in a variety of commonly used visualisations (e.g. bar charts)’; ‘I can use interactive data dashboards to view summaries of data in my domain of expertise’ | N/A |
| L2 - ‘I am proficient at interpreting and summarising data from dashboards and other tools relating to my area of expertise’ | N/A |
| L3 - ‘I am capable of identifying suitable methods of visualisation for different data types’ | N/A |
| L4 - ‘I can evaluate the quality of data to ensure it is fit for purpose to be used for reporting/communicating findings, and only report on data that meets this standard’; ‘I am proficient at interpreting information presented in a variety of specialised visualisations relevant to my field of practice (e.g. Kaplan-Meier curves)’; ‘I can create data dashboards to summarise, visualise and present data to a variety of stakeholders’; ‘I can create a range of visualisations for a variety of audiences (e.g. technical and lay audiences) to present data visually for data exploration and reporting using appropriate visualisation design theories’ | N/A |
5.1.2 Data Storage
| L1 - ‘I am aware that data can be stored locally (local storage on servers) or remotely (e.g. cloud storage requiring an internet connection)’; ‘I am aware of different systems used to store health and clinical data (e.g. Electronic Health/Medical Records) and the impact of this on subsequent data analysis and access (e.g. retrieving data from different database systems)’ | N/A |
| L2 - ‘I am aware of the data storage requirements for my organisation and adhere to these requirements (e.g. where to store data)’; ‘I am proactive in the maintenance of data to ensure its integrity, including the backing up/archiving of data and the options available in my workplace’ | N/A |
| L3 - ‘I am aware that data can be stored in different types of database (e.g. relational databases)’ | N/A |
| L4 - ‘I can design systems and infrastructure for data storage with a focus on accessibility, privacy and security’ | N/A |
5.1.3 Data Visualisation and Reporting
| L1 - ‘I am aware that data can be stored locally (local storage on servers) or remotely (e.g. cloud storage requiring an internet connection)’; ‘I am aware of different systems used to store health and clinical data (e.g. Electronic Health/Medical Records) and the impact of this on subsequent data analysis and access (e.g. retrieving data from different database systems)’ | N/A |
| L2 - ‘I am aware of the data storage requirements for my organisation and adhere to these requirements (e.g. where to store data)’; ‘I am proactive in the maintenance of data to ensure its integrity, including the backing up/archiving of data and the options available in my workplace’ | N/A |
| L3 - ‘I am aware that data can be stored in different types of database (e.g. relational databases)’ | N/A |
| L4 - ‘I can design systems and infrastructure for data storage with a focus on accessibility, privacy and security’ | N/A |
5.1.4 Data Processing and Analytics
| L1 - ‘I am aware of the importance of data provenance, data transparency and audit’; ‘I have an awareness of information governance processes and procedures when dealing with organisations external to the NHS and adhere to local guidance when dealing with external entities’ | N/A |
| L2 - ‘I am aware of health data sets available to me in my area and the types of clinical questions that could be answered with these data. I am aware that public datasets may have been processed and that this may make them unsuitable for certain applications’; ‘I am able to critically analyse a health dataset in terms of clinical questions it could answer and develop a data analysis strategy (e.g. a plan of how the data could be analysed to answer these questions) | N/A |
| L3 - ‘I am aware of and know how to access secure and trusted IT resources for tasks like high performance computing and cloud based data storage’; ‘I recognise and promote the use of common data standards where appropriate to store and share data’; ‘I can query healthcare databases applying analytical tools to analyse large datasets for audit and research purposes’; ‘I promote the use of data provenance for data transparency and audit, and how this can impact on subsequent decisions made using data’; ‘As someone working with data for research, monitoring or process improvement purposes, I know how to effectively deidentify/pseudonymise data’ | N/A |
| L4 - ‘I am able to filter data to derive a subset of interest for further processing/analysis using statistical/programming tools (e.g. SPSS, Excel)’; ‘I am confident at applying data linkage and integration into composite data sets and record steps taken (e.g. metadata audit trail)’; ‘I know where to look to find relevant anonymised datasets and trusted analytics services’; ‘I understand fundamental statistical principles and can use statistical and software tools (e.g. Python, R) to analyse data’; ‘I understand the pros and cons of different statistical methods for different analytical tasks’; ‘I can take account of data provenance issues when reporting the results of analysis (e.g. which populations were included/excluded)’; ‘I am confident in the use of Learning Health Systems in daily practice for continual improvement of care and can factor in generated knowledge to adapt my practice and improve processes accordingly making decisions based on data’ | N/A |
5.2 Data/Cyber Security
| L1 - ‘I abide by the organisational regulations and guidelines aimed to prevent data loss and theft, including when data is being stored and transferred’; ‘I understand and comply with cybersecurity standards in my organisation by keeping my training record up to date’; ‘I am able to access and act on requirements communicated to me about new security threats (e.g. emails from IT personnel asking me to carry out a certain action or be aware of a particular threat)’ | N/A |
| L2 - ‘I understand the need for data encryption to protect data and use this where appropriate to store and transfer data’ | N/A |
| L3 - ‘I regularly update systems (e.g. installing routine software updates when prompted) to protect systems/equipment from cyber-attacks’; ‘As someone who procures technology (e.g. mobile technology), I ensure that such technology is secure for the purposes of storing and transmitting data’; ‘I am proactive in learning about and implementing security protocols and applying recommended data/cyber security standards’ | N/A |
| L4 - ‘I am capable of championing the education of other staff/users in the security issues related to health technology and promote best practices in maintaining the security of health systems’ | N/A |
5.2.1 Data Privacy and Confidentiality
| L1 - ‘I am aware of the need to maintain confidentiality and privacy of health and social care data at all times respecting the data subjects right to privacy’; ‘I always ensure that the data subject is informed of the reasons why data is collected, stored and who will have access to their data and for what purposes’ | N/A |
| L2 - ‘When sharing data, I always ensure that only those with a legitimate reason to view or access the data are included in the data sharing to prevent the leaking of sensitive information to those without permission or need to access/view data’ | N/A |
| L3 - ‘I am aware of the risks of re-identification of pseudonymised data (e.g. using data matching) and can assess and document if further steps are required to further de-identify data’; ‘I am aware of and can apply de-identification (anonymisation) to data and recognise its importance for maintaining confidentiality of data subjects and sources’; ‘I am familiar with the architecture and reasoning behind creating Trusted Research Environments (TREs) and apply/use them wherever appropriate’ | N/A |
| L4 - ‘I practice the application of analysis methods to reduce the risk of re-identification of pseudonymised data’; ‘I maintain a closed loop consent process with data providers including the citizens and model openness, transparency and accountability for its use’ | N/A |
6.0 Artificial Intelligence
| L1 - ‘I understand that Artificial Intelligence (AI) is an umbrella term used to define digital technologies capable of performing tasks commonly thought to require human intelligence. I am aware AI is common in modern technology and can list uses of AI outside healthcare (e.g. voice recognition)’; ‘I can provide examples of AI systems used in healthcare and understand their potential benefits and risks (e.g. imaging diagnostics)’; ‘I am aware that “machine learning” is a sub-set of AI and is an umbrella term used to refer to techniques that allow computers to learn from examples/data without being explicitly programmed with step-by-step instructions’ | N/A |
| L2 - ‘I am aware that all AI applications in healthcare are defined as ‘narrow’ AI that are trained to perform a particular and specific task’; ‘I can identify the contribution that AI could make to my healthcare processes in my area of practice and how it has the potential to benefit the organization, workforce and patient’; ‘I can articulate the risks and limitations of AI relevant to my professional area and consider them in my use of AI’ | N/A |
| L3 - ‘I can explain intellectual property issues pertaining to AI models and how this impacts on AI algorithms co-developed between the NHS and commercial providers’; ‘I can define the sub-fields of AI and machine learning and their key applications (e.g. computer vision)’ | N/A |
| L4 - ‘I can describe the main types of bias that could affect AI systems (e.g. reporting)’; ‘I can take steps to identify and mitigate bias in AI systems, such as designing models inclusively (human centred design approaches), training with representative data and testing for bias’; ‘I understand the importance of and promote transparency of AI models used within my area of practice. For example, identifying the type of model used, training data, methods and potential model limitations and weaknesses’; ‘I understand the benefits and limitations of AI explainability. I keep abreast of research and developments in this area and am aware of the potential impact on confidence in clinical decision making’ | N/A |
6.1 Machine Learning and Natural Language Processing
| L1 - ‘I understand that machine learning algorithms require large quantities of data to learn from, and must be trained and evaluated using independent sub-sets of the available data’; ‘I am aware of some of the common uses for Natural Language Processing (NLP) methods and text mining within and outside of healthcare (e.g. chatbots, speech/virtual assistants)’ | N/A |
| L2 - ‘I am aware of the use of virtual assistants (e.g. Amazon Alexa) in healthcare to improve accessibility for patients (e.g. patients with disabilities) to access health information and can recommend their use to patients where appropriate’; ‘I understand the differences between Artificial Intelligence (AI) for prediction (prospective) and AI for explanation of existing data (retrospective). I am aware of the risk of conflation of these use cases’ | N/A |
| L3 - ‘I am familiar with core concepts and methodologies used in the field of Machine Learning (ML) (e.g. data science)’; ‘I am aware of different learning methods and their suitability for a clinical task based on the available dataset (e.g. supervised, unsupervised, reinforcement learning)’ | N/A |
| L4 - ‘I am familiar with model evaluation metrics (e.g. sensitivity, precision) presented in academic papers and can use these to evaluate machine learning algorithms’; ‘I am confident in critically appraising literature regarding performance and validation of an AI solution for my area of expertise, comparing it to alternatives and to the current standard of care’; ‘I can critically assess the model development process for a machine learning solution (e.g. feature labelling/extraction)’ | N/A |
6.2 Using and Implementing AI Systems
| L1 - ‘I am able to use Artificial Intelligence (AI) systems confidently to assist me to improve task efficiency while maintaining quality and safety’; ‘I know how to respond if an AI system fails or is inaccessible and can initiate an alternative process to maintain effective service provision’; ‘I understand the importance of sharing learning following failures of an AI system, to improve systems and practice’ | N/A |
| L2 - ‘I am aware of the limitations of AI systems and how to respond when AI derived information contradicts my clinical/professional intuition. I retain a ‘critical eye’ and am aware of how AI may influence my decision making’; ‘I can justify the use of AI in specific clinical scenarios and know when it is and isn’t appropriate to implement an AI solution based on desired outcomes, potential risks and organisational goals’; ‘I actively maintain my clinical knowledge and skills to ensure that my clinical performance is not adversely affected by de-skilling resulting from using AI’ | N/A |
| L3 - ‘I am aware of the various stages of implementing an AI system, including risk assessment, interoperability, workflow integration, validation and verification, user training, on-going monitoring and model iteration’; ‘I can set thresholds for monitoring patients using AI enabled decision support systems for chronic health conditions to generate alerts to initiate appropriate action (e.g. call patients in for review)’ | N/A |
| L4 - ‘I am confident at selecting suitable AI methods for given use cases’; ‘I understand the potential benefits and limitations of data augmentation, including data simulation and synthesis techniques when there is little available data for training AI systems. I can design testing using real-world data to ensure robustness of AI models trained this way.’; ‘I can effectively cost AI-driven solutions taking into account factors such as initial set up, workforce, maintenance and other running costs (e.g. cloud storage costs), balancing those against potential efficiency savings.’ | N/A |
6.3 Evaluating AI Systems
| L1 - ‘I can explain the difference between internal validation, external validation, local evaluation and prospective clinical evaluation of Artificial Intelligence (AI) technology and their relevance to clinical performance. I am aware of recommended standards for validation of different types of AI technologies used in healthcare (such as the National Institute for Health and Care Excellence (NICE) evidence standards framework for the digital health technologies),’; ‘When evaluating an AI system for use in my professional workflow, I can compare its performance against the expected standards of my professional area of practice’ | N/A |
| L2 - ‘I am aware of the challenges of bias and generalisability for AI algorithms (the ability to perform well in a different demographic group or clinical context to that used for evaluation). I am able to evaluate the publicly available evidence supporting an AI tool and identify the need for further evaluation in my local setting’; ‘I am aware of the potential ways in which use of an AI solution may affect human decision makers (e.g. human cognitive biases around the use of AI and AI derived information, such as automation, anchoring and confirmation biases’ | N/A |
| L3 - ‘When commissioning AI technologies, I can discuss the requirements for safety testing (e.g. user acceptance testing, quality assurance testing), in addition to algorithm performance evaluation’; ‘To improve performance of AI systems - I am aware that ‘optimisation’ can be used to discover a sub-set of potential ‘best choices’ of model once some AI analysis has been carried out’; ‘I can discuss the clinical validation standards and approval phases for healthcare AI models and understand the importance of continual post deployment surveillance’ | N/A |
| L4 - ‘I am aware of recommendations that for evaluating and reporting of AI interventions in clinical trials (e.g. SPIRIT-AI, CONSORT-AI) and can apply these where appropriate’; ‘I can communicate potential benefits (improved consistency, availability, speed, efficiency) and challenges (e.g. model explainability, biases, model under/over fitting) of using AI systems in healthcare to various stakeholders’; ‘I am able to carry out post market surveillance and ongoing clinical monitoring to determine if the system is still meeting required needs and to identify model decay (e.g. the tendency for AI model performance to drop over time as data and patient characteristics change, requiring models to be constantly updated’ | N/A |
6.4 Robotics
| L1 - ‘I am aware of the use of robotic technology for healthcare and can cite examples of the use of robots for health, medical and social care (e.g. social companion robots, surgical robots, room disinfectant robots)’ | N/A |
| L2 - ‘I am aware that robotics comprises an intersection of most areas with Artificial Intelligence (AI) (e.g. cognitive modelling, computer vision)’; ‘I am aware of the uses of telepresence robots to carry out basic procedures (e.g. temperature monitoring) in highly infectious patients and for use with elderly to support independence or for remote care and assessment’ | N/A |
| L3 - ‘I know where and how to access training (local or manufacturer provided) on robotic technology that I am expected to use’; ‘In my domain, I can work confidently with robotic technology and recognise the limits of such technology and when to override or desist use (e.g. for safety reasons, malfunction)’; ‘I am responsible for demystifying the use of robotics and democratising its use and integration within appropriate pathways’ | N/A |
| L4 - ‘I am confident in critically appraising literature regarding robotics for my area of expertise in terms of their applications, limitations, required resources and suitability’; ‘As someone who instructs others to use robot-assisted surgery - I am able to identify and agree clear training goals with those I instruct’; ‘As someone involved in procurement of robotic technology/solutions, I can effectively cost a solution taking into account factors such as initial set up, maintenance and other running costs’ | N/A |