This page outlines a potential proof of concept demo being explored by ACTS Collaborative Learning Community participants to illustrate how making evidence and guidance more computable and standards-based could enhance the development and updating of 'living' CDS interventions and eCQMs for COVID-19 and beyond.

• The figure in section A. is an overview of potential enhanced information flow around the Evidence/Quality Ecosystem Cycle - i.e., driven by more computable and standards-based evidence and guidance
• The table in section B. outlines ecosystem enhancement needs and opportunities, as well as the notes on a potential concept demo toolkit (in the 4th table column) for addressing those needs 
• The diagram in section C. illustrates where and how a concept demo toolkit that makes evidence and guidance more computable and standards-based could enhance the Evidence/Quality Ecosystem (and Learning Health System) Cycle
• The outline in section D. provides more details on a potential concept demo toolkit
• The email excerpt in section E. provides considerations related to using computable/standards-based evidence descriptions to make developing and updating computable clinical recommendations more efficient and effective. 

A. Evidence Ecosystem Enhancement - Overview Diagram (Enhanced Ecosystem Concept Demo Opportunity 8.26.20.pptx)

[DOC search hyperlink]

B. Ecosystem Needs, Enhancement Opportunities and Potential Concept Demo Outline

¹By those doing the work - e.g., EPCs, VA/UMN/Health Centers, Agile KE teams, NACHC/ACEP/EvidenceCare, many others

²More information about 'building blocks' for creating components of this 'fantasy' (e.g., standards, sources for inputs/outputs, tools/methods platforms) is on the Community of Practice webpages (see navigation bar left side of this page), and in this emerging catalog from the COVID-END project

C. Overview Diagram for Proof of Concept Demo Toolkit (computable evidence slide.pptx)

D. More Details on Knowledge Supply Chain Enhancements 

D.1: UNVETTED DRAFT Notes on a Near-term Approach for Propagating Down the Knowledge Supply Chain Notifications about Impactful Updates  

Goal:

Illustrate how for a sample target (anticoagulation and/or COVID-19 testing/triage):

• we can signal to care teams (through 'living' CDS interventions) when a change in the evidence-review-guidance supply chain content for this target indicates a change in recommended care. Or this change indicates that the strength of evidence/guidance supporting a recommendation has changed. (The latter is important so this new information can be factored into patient-clinician shared decision making accordingly.)

Approach:

• VA and UMN (for the most part) manage all facets of this supply chain and also consume its results via CDS interventions that support care for your respective patient populations. This suggests that these 2 organizations could each be logical homes/centerpieces for efforts focused on demonstrating the tighter link outlined above between new guidance/evidence and updating the corresponding CDS interventions (e.g., focused on anticoagulation and testing/triage [here is the ACEP/EvidenceCare COVID-19 Severity Classification/Triage/Disposition tool]).

• This would mean having the teams that are responsible in each of these organizations for evidence surveillance/synthesis, guidance development/updating, and CDS development/updating/deployment collaborate among themselves and with other organizations in the Collaborative on this 'update notification' process and tooling.
• We could explore ways to leverage related efforts for content and approaches to apply in the notification system. For example, consider mutually beneficial ways to apply SRDR/COKA, C19HCC Digital Guideline WG, COVID-NMA, COVID-END, AU Living Guidelines, and related efforts to produce a a triggering system to that  suggests to living CDS owners/implementers that updates should be considered.
• For example, below is a sampling of external evidence/guidance sources that could potentially be leveraged to enhance current VA/UMN processes in developing the 'update notification' system. We could start by documenting how evidence/guidance changes are detected and addressed in current VA/UMN processes, and exploring enhancing these approaches to include a scalable notification function that propagates supply chain updates to all pertinent stakeholders throughout the chain, including those responsible for developing/maintaining CDS interventions.  

• We could begin by synthesizing a largely manual update detection and notification mechanism that runs throughout the supply chain, then semi-automated approaches (e.g., that detect changes on target web pages), and ultimately fully automated approaches that leverage computable, standards-based, interoperable information throughout the supply chain.
• Important foundational work for this has been laid in SRDR/COKA collaborations, and we should explore how to fully leverage and build on this in developing an 'update notification' approach. Especially since SRDR is an important component of AHRQ's current digital knowledge platform.

• Note from Jens Jap, SRDR Team: "my team is interested in ... the development of automated literature searches to assist in SR updates or at least signal an opportunity for one. A preliminary step to this effort was the development of a RCT classifier. I think this is similar to what you previously referred to as COKA enhanced tagging tool, at least in nature. Using these kinds of machine learning assisted tools can bring us a step closer to more automation and living SRs. " Response from David Tovey: "In terms of an RCT Classifier, you may be interested to know that a tool with exactly this name has been developed by James Thomas and his team at UCL in London. It is currently in use within Cochrane but it might be useful to reach out to James if you are interested to explore this. ... The tool is capable of assessing large bundles of citation and abstracts very quickly with an accuracy level that is at least as good as could be achieved manually."

Sampling of External Sources to Check for Updated Evidence/Guidance on Targets

• NIH anticoagulation adaptive clinical trials announced 9/20
• COVID-NMA (search for heparin)
• VA COVID Reviews (search for triage, testing, anticoagulation)
• L*VE Epistimonikos
• COVID END Best Evidence Synthesis (see testing/triage here, and others/anticoagulants here)
• NIH Guideline on Antithrombotic therapy
• CDC Phone Triage Guidance
• ACEP/EvidenceCare computable guidance on ED triage (based on this guide)
• AU Living Guidelines - see 'definition of disease severity' and 'VTE' 
• IDSA Guideline on Serologic Testing (frequently updated)
• [other resources listed on Collaborative's evidence/guidance processing CoP page]

D.2: Notes on a More Comprehensive Proof of Concept Software Toolset 

The 4 proof of concept tools and related repository outlined below can be placed on an open source developmental website for public dissemination. Content development for these tools will be driven by Collaborative participant current efforts, e.g., focused on COVID-19 testing/triage in ambulatory and ED settings and anticoagulation in inpatient settings.

Tool 1: Create/Store/Access Computable Study Results Representation   

• Develop tools that enable users (e.g., via web interfaces) and systems (e.g., via APIs) to input to/output from a proof of concept repository. The repository could be an AHRQ development website created for purposes of the concept demo. This tool will enter/store/retrieve standards-based, computable data about a study related to a COVID-19 target, e.g., triage/testing, anticoagulation. Input sources can include medRxiv, journals, and article repositories, e.g., CORD-19. The data input tool user could be a study author/publisher, or someone else (e.g., in an EPC) for studies already published. Data output tool use could include systematic review developers and other individuals and applications requesting the study results. These proof of concept capabilities could be designed to align with SRDR+ functionality enhancements under consideration/development. 
  • This proof of concept tool demonstrates an open mechanism for capturing and presenting (e.g., via web-based viewer or API) summary results of individual studies in a manner that is re-usable by many different systems. This reduces redundant work currently required to extract key study variables that aren’t standardized and therefore aren’t interoperable or re-usable. 
  • The tool could also demonstrate linkages with clinicaltrials.gov, e.g., for visibility into what trials on the topic of interest (COVID-19 testing/triage) are underway and completed.
  • It also demonstrates a ‘notification feature’ that notifies ‘subscribers’ to the repository (e.g., systematic reviewers) that a new study pertinent to the topic of interest (COVID-19 ambulatory triage/testing) is entered. 

Tool 2: Create/Store/Access Computable Systematic Review Representation

• Develop tools that enable users (e.g., via web interfaces) and systems (e.g., via APIs) to input to/output from a repository (e.g., an AHRQ development website created for proof of concept purposes) systematic review information related to COVID-19 ambulatory triage/testing in a computable, standardized format.
• Includes notification tool/function to notify ‘subscribers’ (e.g., guideline developers) that there’s a new systematic review on the topic of interest.

Tool 3: Create/Store/Access Computable Rationale for Guidance

• Develop a proof of concept web interface and an API that use the FHIR standard for data exchange (i.e., EvidenceReport and PlanDefinition Resources maintained by the COKA/EBM on FHIR/CPG on FHIR HL7 projects) to encode a recommendation's evidence basis, i.e., pertinent studies and systematic reviews encoded by tools 1 (computable study results) and 2 (computable systematic reviews) above. This computable guidance rationale tool will also encode the rationale and confidence for the recommendation.  
• This enhanced, automated evidence processing for recommendations could be demonstrated to support data entry into the 'Evidence Supporting Recommendation' section of the C19HCC knowledge elicitation tool (see excerpt in diagram below). [Note - there are other templates for producing guidance, e.g., this one from GRADEpro, WHO "Digital Accelerator Kits", and others]
• Two functions that could be demonstrated are support for: 1) gathering and synthesizing the evolving evidence base pertinent to a recommendation and 2) assigning standard codes for the rationale behind and confidence in the recommendation.

Excerpt from Knowledge DRAFT Elicitation Tool:

[intervening portions omitted]

• Likewise, the proof concept tool could also demonstrate web-based connection to other guideline development/presentation software to facilitate use of standards-based, computable evidence in guideline development. For example, MAGICapp, used in the Australia National COVID-19 Living Guidelines initiative - see disease severity, section 4, pertinent to triage.
• The tool could contain a feature that notifies CDS and guideline developers/implementers that new systematic reviews on the topic of interest (testing/triage) is available.
• This tool uses EBM on FHIR and CPG on FHIR: leads for these efforts are closely engaged in the ACTS COVID Collaborative and could be involved in developing this toolkit
• Outputs from the use of the C19HCC knowledge elicitation tool are provided to the C19HCC Agile Knowledge Engineering teams, who create L3 guideline representations (i.e., in CQL and BPMN) and CDS/eCQMs/eCaseReports that are deployed in clinical settings.
• The resulting CDS interventions could be placed in CDS Connect; consider synergies between living computable evidence/ guidance as demonstrated in the proof of concept toolkit and the CDS Connect Authoring Tool. 

Tool 4: Identify/Store/Access Terminology for Computable Recommendation Definition

• Develop a proof of concept tool that enables guideline and CDS developers to identify/gather (e.g., via web interfaces, APIs) appropriate standard coded concepts and/or value sets for the required data elements needed for guideline execution. 
• For example, exposures, symptoms, high risk conditions, and special circumstances listed in the CDC Phone Line Advice triage algorithm (see page 6) that underlies part of the NACHC CDS intervention. 
• Proof of concept tool addresses challenges that knowledge engineers currently face in identifying code sets and value sets needed to make concepts and data elements in clinical guidelines computable. In a leading effort to put the CPG on FHIR implementation guide for computable guidance into practice, members of the C19HCC Digital Guideline WG are using a knowledge elicitation tool (see figure below) to elicit these concepts/terms from SMEs and then manually searching code sets/value sets to obtain codes needed to make recommendations computable. This part of the concept demo toolkit provides an automated lookup function that maps concepts/terms to potentially matching items from pertinent value/code sets. These codes/value are needed to access pertinent data from EHRs to execute the recommendation. 
• For example, the new tool could leverage the data dictionary being developed under the NACHC project to help computable guidance developers search code set repositories (e.g., from the COVID-19 Interoperability Alliance and others) to express guideline-related clinical data elements (e.g., body temperature) using specific, standard code sets (e.g., LOINC code, ICD-10, SNOMED, etc.).

Excerpt from Knowledge DRAFT Elicitation Tool: 

E.  Excerpts from 9/4/20 email exchange about using computable/standards-based evidence descriptions to make developing and updating computable, evidence-based clinical recommendations more efficient/effective.

Adapted version of note from Jerry Osheroff:

Below is a small excerpt from the HL7 CPG on FHIR Draft Computable Guideline L2 Template (Recommendation tab) being used by the C19HCC Digital Guidelines WG. The question the Learning Community is exploring is whether/how the trajectory of SRDR/COKA efforts to provide computable, standards-based input and out from SRDR could at some point and in some way lead to auto-populating/updating this type of information in some future version of this template:

Response from Brian Alper (lead of EBM on FHIR and COKA) - [the rest of the back and forth below deals with an important issue that's downstream from the issue of computable evidence]:

The format for human expression can look very different from the format for computable expression.   But if we can agree to a standard for computable expression we can support a near-infinite set of patterns of human expression.

Some thoughts below to inform a computable expression of a “Recommendation” and I end with a link to a first draft for it.

As a recap to some of the concepts to clarify recommendation vs. CDS artifact:

One of the challenges in defining L2/L3 and recommendation/CDS may be recognized by 2 different factors (Recommendation/Decision Rule, Digital/Computable):

• A Recommendation can be an expression of what should be done – Flu vaccine is recommended for people who have not received a flu vaccine this season.
• A Decision Rule can be an expression of the logic to be applied – If a person does not have a record of receiving a flu vaccine this season, then offer/provide a flu vaccine.
• The Recommendation and Decision Rule can be applied in clinical practice completely using Print expressions.
• The Recommendation and Decision Rule can be applied in clinical practice completely using Digital expressions.   This sentence is shared with you now in a Digital expression in this email but is not a Computable expression of the concepts.
• The Recommendation can be converted to a Computable expression – an L3 artifact that provides “Flu vaccine” as a codeable concept, “people who have not received a flu vaccine this season” as a codeable concept, and “is recommended for”
  as a codeable concept.   This L3 artifact can be considered a CDS artifact but it is not yet sufficient for immediate functional use in a specific CDS system.
• The Decision Rule can be converted to a Computable expression – in addition to the codeable concepts in the Recommendation, additional codeable concepts to express include “have a record of”, “offer/provide”, and the “if…then” logic.  This L3 artifact would also be considered a CDS artifact.

The original goal of the EBMonFHIR project was to provide for computable expressions of evidence and recommendations.   With the CDC ACQ Informatics Value Stream effort focused on converting guidelines to CDS artifacts, a companion CPGonFHIR project developed.   It appears the CPGonFHIR expresses Decision Rules in computable form and the PlanDefinition Resource expresses the action(s) in computable form.   We have been discussing shared use of Group and EvidenceVariable Resources that can express parts of the “when recommended” concepts in computable form. However, there is not yet a specific resource for the Recommendation in computable form which can be used prior to creating the Decision Rule derived from that Recommendation.

Working off of what we have learned from the evidence-related Resources and the PlanDefinition Resource, I have created a first draft of a Recommendation Resource to bridge this gap.

Response from Davide Sottara:

Brian, I generally agree with your distinctions on Decision vs Recommendation, and Digital vs Computable.

Yet, I would like to understand how far do you envision a "L3 Recommendation".

A "computable, structured" Recommendation would enable a CDS system to reason over the Recommendation,
and e.g. allow to distinguish the different actions that are recommended, evaluate applicability conditions, and afford for a more
contextual delivery. 
Yet, I believe that the current PlanDefinition may be able to express this notion. It's already so polymorphic in nature that
adding 'mood' extensions and profiles for confidence, certainty, and strength may be enough(@Bryn?) 

Yet - from a formal knowledge representation & reasoning perspective - there are more aspects to explore. 
As 'Recommendations' convey an agent's proposal to close a gap between a current state and a perceived goal state, they can be considered plan fragments,
with "mood(al)", deontic and speech act aspects, and can have elements of belief, confidence and evidence (explanation).

These aspects, which allow to reason with formal Recommendations, require capabilities beyond 'inferences' and 'ECA rules'. 

That is, the more we increase the expressivity with additional resources, the more we need to provide tools and guidance on how to correctly
use them - and not only to exchange information. 

So - (1) do we need/want a new Resource, or a new Profile, (2) what are the computational implications of the new resource, 
(3) can we standardize the pragmatics of reasoning with the resource or leave it to the implementers?

Response from Bryn Rhodes:

I don't quite agree with the opening statement of the document. I would say there _is_ a resource that represents a computable recommendation, it's PlanDefinition, and there is even a profile in CPG-on-FHIR called cpg-recommendationdefinition, that's exactly what we're trying to capture there. If there are gaps between what's there and what you are looking for, I'd like to understand what those are.

In short, I don't think we need a new resource here, or at least I don't see what the gaps are that would require it.

Response from Brian Alper:

I don’t necessarily want to create a new Recommendation Resource if a Recommendation Profile (of PlanDefinition Resource) or other form meets the need.

I have been getting asked by more people to extend the EBMonFHIR efforts to provide computable expressions of the “Recommendation” concept on the “L2 to L3 path” for Guideline-to-CDS translation and was not sure what is being requested vs. what is already covered in CPGonFHIR efforts.

My quick take was to distinguish Recommendation from Decision Rule and see what is missing for the computable expression (precise, unambiguous, machine-interpretable expression) of the Recommendation component.   On a quick view of PlanDefinition Resource the “action(s)” appear really well specified so it appears the “whenToAct” specification is what is not easily translated from the guideline recommendation statement to the later expression.  If there is an easy path for doing that let’s use it.  If there are adjustments to make the path easy, let’s do it.  If not then perhaps a Recommendation Profile is a way to make this easier.

Response [Excerpt] from Matt Burton (Lead for C19HCC Digital Guideline WG):

The HL7 CPG IG is not first and foremost CDS, its very much intended to be a computer-interpretable expression of the guideline itself; then with: 1) the means to derive highly related artifacts such as ECA-Rule-based CDS, patient-specific, practice-level metrics that *may* be rolled up into Quality Measures, eCaseReports to provide all the detail of guideline ‘execution’ at the patient-level (when they met criteria for a recommendation, when CDS notified the clinician of the proposed action, when/if clinicians took said action [order/request], when said proposal or request was fulfilled [whether evidence of request or not], any desired metrics that were captured and how they may have evolved over time [e.g. “on/off path”, “on with a history of off”], and any provider Impressions that may qualify any of the afore mentioned); and 2) lots of human consumable narrative on how to implement an HL7 CPG across its entire lifecycle (from working with Guideline Developers & the Evidence Ecosystem to working with local Practices and their informatics and EHR teams, and numerous other ecosystem touchpoints between- not the least of which is getting the data semantics as applied in point of care clinical information systems nailed- not just a bunch of terms hurled at ehr’s).  Not sure if it is the name, the specification, how we have presented it to date, or some other factor, but that the HL7 Clinical Practice Guideline Implementation Guide describes the process and patterns that afford the means to computationally express the intent of the guideline is a point oft missed.

F.  Notes from 9/18/20 Weekly Call

Pawan Goyal: ACEP has a blog where more than 4,000 Emergency Physicians participate across the globe to share their experiences. ACEP meets with CDC, CMS, NIH, and FDA on regular basis. Our Clinical Policies Committee and Clinical Practices Committee collect evidence/knowledge and review/approve them on periodic basis.

Matt Burton:  Citation analysis can give some linkage... EMB-on-FHIR and CPG-on-FHIR retain provenance (incl to citations)...  linkage analysis can ASSIST, but still need human in loop at some point there are some "semantic linkages" that can be leveraged, but just used to associate...  Sensitivity/ Specificity issue for "relevance", Motive, Stanson, and other CDS vendors live and breathe these types of approaches. Potentially with citation analysis Preston.

Sivaram Arbandi: Information can be open-world, but when it comes to knowledge it will need to be closed-world.

David Tovey:  A resource that has not been mentioned on this call is the L*VE Epistemonikos platform, which tracks all new studies and reviews and includes a very strong search facility that could be used to track new studies looking at clinical predition rules or anticoagulation interventions for example. https://app.iloveevidence.com/loves/5e6fdb9669c00e4ac072701d?utm=ile

Maria Michaels: Link to all the checklists: http://build.fhir.org/ig/HL7/cqf-recommendations/checklists.html. Link to the L4 checklist: http://build.fhir.org/ig/HL7/cqf-recommendations/L4Checklist.html.

Brian Alper: If the data to transfer for updating is put into a standard structure, then a computer can manage a "subscription service" to notify people when the data changes per some parameters.  A FHIR subscription service could be used if the data is put in FHIR on a FHIR server.  If you select a specific type of data for piloting we can explore FHIR tooling for this purpose.

Sandra Zelman Lewis: DOC Search (https://covid-search.doctorevidence.com/) is also a way to search and monoitor PubMed (31.6 million), ClinicalTrials.gov (351,984), COVID-19 Open Research Dataset (197,184), DailyMed (128,431), EPAR (1,502), WHO-ICTRP (648,473), and RSS feeds (870,546 from 491 feeds). You can set up alerts if matches to your search terms.

David Tovey: Indeed Sandy, and also DOC Analytics for automating meta-analysis.  Emails:  daviditovey@gmail.com and Gabriel Rada: radagabriel@gmail.com