What to know/do (and why) Overview

('nouns and verbs' - what goes in rows below in each table are the 'lists' associated with this the bullets in this row)

include: what tools to use under what circumstance. Information about the item so people can match it to their need.

--------------------

Regarding Value Sets recommend the following process (applies to anticoagulation, testing/triage, vaccines, steroids, and any other topic):

1. Take inventory of what value sets you’ll need (be specific)
2. Check Alliance website or Value Set Authority Center (VSAC), we might have already published some of them among the 600+ value sets
3. If certain value sets are not present or you are unsure, contact Victor Lee (Victor_Lee@ClinicalArchitecture.com) to inquire or to request development

Clinical Architecture and other Alliance collaborators are donating our efforts, and Clinical Architecture is leveraging its software tooling, hosting infrastructure, and manual labor to donate all of the COVID-19-related value set content to the public domain (i.e., completely free, no strings attached). That being said, we all have day jobs, so we appreciate having as much lead time as possible to fulfill requests. Happy to take questions.

COVID specific systematic and rapid reviews

• L*VE/Epistemonikos
• COVID-NMA
• COVID-END inventory
• ESP COVID-19 Evidence Reviews

COVID specific mixed (reviews and trials)

• NIPH COVID-19 Living Evidence Map
• EBSCO COVID-19 Information Portal
•  University of Michigan COVID-19 Best Evidence Front Door

COVID specific Trials

• Cochrane COVID-19 study register
• ClinicalTrials.gov/COVID-19
• DoctorEvidence CORD-19 Dataset Search
• EBSCO COVID-19 Information Portal

• NEWS2 predicts severity/mortality at admission
• Annals IM, hospitalized patient trajectories
• Sensitivity, Specificity Predictive Values of Tests for COVID-19 in ED
• PREDI-CO Risk assessment tool

• For steroids systematic meta-review, 8 sources have been identified these are (MEDLINE, CORD-19, L-OVE/Episteminokos, NIH iSearch COVID-19, EuropePMC, WHO COVID-19 Database, EMBASE + Prospero) 

• Librarian Reserve Corps
  • COVID-Evidence Project - building a database to gather all the evidence around a drug (University of Basil - currently focused on hydroxychloroquine, but model could be adaptable to other targets)
  • Identification of sources to identify primary studies - validation study of specialized COVID-19 databases (systematic reviewers are going to different sources - this effort is to identify best practices)
  • Advocate for librarian representation in searches and reviews - leverage skillsets/best practices in this work
    [SLMC]: close gaps between needs that clinicians are seeing on the front line and topics covered in reviews guidelines
• COKA Evidence/ Tools WG [Project Google Drive](work in process)

• Framework slides: Applying Standards to the Evidence Domain (from the COKA Evidence Ecosystem Liaison WG)

• includes evaluation of quality of evidence 
• also needs to address processing of real world evidence
• [weave in ICER - search ISER opioids]

• For people creating evidence syntheses
  • See if work on planned topic has been done already - search L*VE/Epistemonikos (2K reviews - cover many questions - only a fraction registered in PROSPERO; most reviews are out of date - only 3% have all the trials they should), PROSPERO for what's in the pipeline. Make sure that any existing reviews look at outcomes that are pertinent to your needs [leverage COMET - outcome measures], etc. that of interest for the need at hand. Very hard for user to identify a review that's up to date. COVID-END repository of Best Evidence Syntheses can be helpful here.
  • Need to have access to raw data that fed into the review so it's more re-usable.
  • important to understand end-user needs so that the review can be aimed at addressing them
  • AHRQ has methods guides, Cochrane has them. Are there tools for this on COVID - e.g., Resources and tools for researchers considering and conducting
    COVID-19 evidence syntheses (From COVID-END Synthesizing WG)
• For organizations consuming evidence syntheses
  • For selecting/using synthesized evidence: Is there a GRADE evidence profile? Do they make quality assessments? How comprehensive are they? Search date

--------------------

Regarding Value Sets recommend the following process (applies to anticoagulation, testing/triage, vaccines, steroids, and any other topic):

1. Take inventory of what value sets you’ll need (be specific)
2. Check Alliance website or Value Set Authority Center (VSAC), we might have already published some of them among the 600+ value sets
3. If certain value sets are not present or you are unsure, contact Victor Lee (Victor_Lee@ClinicalArchitecture.com) to inquire or to request development

Clinical Architecture and other Alliance collaborators are donating our efforts, and Clinical Architecture is leveraging its software tooling, hosting infrastructure, and manual labor to donate all of the COVID-19-related value set content to the public domain (i.e., completely free, no strings attached). That being said, we all have day jobs, so we appreciate having as much lead time as possible to fulfill requests. Happy to take questions.

• See Sources/Repositories from Identify Studies table
• ClinicalTrials.gov/COVID-19; New pubmed interface; DoctorEvidence CORD-19 Dataset Search
• PROSPERO

• Underlying justification and evidence tables (critically ill and acutely ill) from ASH Anticoagulation Living Guidelines
• L*VE/Epistemonikos search on antithrombotics; can send email when new study/review/guideline is published.
• Rapid Cochrane review on anticoagulation in COVID-19 patients

• BMJ EBM rapid review on COVID tests

• SRDR+
• COVID NMA Project
• COVID-END Best Evidence Syntheses Inventory ('best evidence syntheses' on clinical management)
  • reviews that are living, up to date, high quality and where there is a GRADE evidence profile. 
• LIVING Project - living systematic review of COVID-19 interventions;
• L*VE/Epistemonikos, 
• VA COVID-19 Evidence Reviews;
• GIN COVID-19 Evidence Resources
• EvidenceAid
• Usher Network for COVID-19 Evidence Reviews

• IOM Report: Finding What Works in Health Care:  Standards for Systematic Reviews
• EBM on FHIR/COKA (standards include: Evidence, EvidenceReport, EvidenceVariable, Group, Citation
• Vocabulary Map for Evidence-related resources[ Project Google Drive])
• AGREE II for rating the quality of guidelines
• [Add AMSTAR and all study quality rating tools here]

• COKA Steroids for COVID-19 Systematic Meta-Review Protocol; slides about this effort

• Resources and tools for researchers considering and conducting
  COVID-19 evidence syntheses (From COVID-END Synthesizing WG)
• GIN COVID-19 Evidence Resources
• SRDR+, Covidence, Distiller SR, RevMan, JBI Sumari
• Doctor Evidence Analytics Tool
• AHRQ EPC Methods Guides
• COMET (Core Outcome Measures in Effectiveness Trials
• Equator Network: Enhancing the QUAlity and Transparency Of health Research
• STARR Decision Tool (for rapid review approaches)

• COVID-NMA has initiated communication with all trialists to try to ensure consistent approaches e.g. selection of outcomes, reduction of risk of bias, and to invite them to contribute missing data. 
• Framework slides: Applying Standards to the Evidence Domain (from the COKA Evidence Ecosystem Liaison WG)

For people using or consuming guidance:

• COVID-END Inventory of Resources to Support Decisionmakers for evidence on public health measures, clinical management, health systems, and social and economic issues
• GIN Library of Guidelines – to search for completed, planned, or in-development guidelines
• ECRI COVID-19 Guideline Resources and ECRI Library for the full repository of guidelines
• AGREE II for rating the quality of guidelines

For people producing guidance:

• make sure that recommendations include the most specific information possible on who/what/when/where/how (and why); bring guideline consumer stakeholders to the table very early in the process of developing guidelines to ensure that the recommendations will be helpful in meeting clinical needs - and be consumable in a computable fashion. (e.g., - have a knowledge engineer in the loop - see 'make guidance computable section below for details.)
• Coding is important to support downstream computability efforts - (Need to lay out exactly what guidance producers need to know and do to make this - integrate guidance development with efforts to make computable)
• For maintenance, i.e., 'living guidance' -
  • Monitor literature (studies and systematic reviews) that relate to the focus of the guideline. Grade material is it becomes available. Don't start from scratch- leverage other efforts to process the upstream evidence. (include links in this guidance section back to pertinent tools in the 'upstream' supply chain tables above)
  • Modify the guideline according.
• GIN Library to register guidelines in process
• PICO(TTS) paradigm
• GRADEpro GDT
• WHO-INTEGRATE evidence to decision framework
• EtD Framework
• mDelphi surveys to achieve consensus
• GRADE for grading recs
• GIN-McMaster Guideline Development Checklist
• WHO Handbook for Guideline Development
• Many medical professional societies develop guidelines and each has their own manual, available on the society’s website and/or published in their journal

--------------------

Regarding Value Sets recommend the following process (applies to anticoagulation, testing/triage, vaccines, steroids, and any other topic):

1. Take inventory of what value sets you’ll need (be specific)
2. Check Alliance website or Value Set Authority Center (VSAC), we might have already published some of them among the 600+ value sets
3. If certain value sets are not present or you are unsure, contact Victor Lee (Victor_Lee@ClinicalArchitecture.com) to inquire or to request development

Clinical Architecture and other Alliance collaborators are donating our efforts, and Clinical Architecture is leveraging its software tooling, hosting infrastructure, and manual labor to donate all of the COVID-19-related value set content to the public domain (i.e., completely free, no strings attached). That being said, we all have day jobs, so we appreciate having as much lead time as possible to fulfill requests. Happy to take questions.

• See Output Repositories from Synthesize Evidence table

• ASH Living Guideline on COVID anticoagulation and underlying justification and evidence tables (critically ill and acutely ill)
• NIH Guideline
• AU living Guideline
• NEJM vignette with expert opinion

• BMJ/Remote Triage in Ambulatory Care

• Living Map of COVID-19 recommendations (Canadian Institutes of Health Research)
• ECRI COVID-19 Guideline Resources
• GIN will publish (has published?) a registry of guidelines in development (COVID and beyond)

IOM Standards: Clinical Practice Guidelines We Can Trust

• GIN COVID-19 Guidance Resources
• MAGICapp; 
• COVID-END Rapid Response Model and Tips/Tools
• GRADE
• International Patient Decision Aid Standards
• 9/15/20 Annals IM article on COVID guidelines. 
• AGREE II for assessing guideline quality and reporting;
• Africa Centre for Evidence Living Hub of COVID-19 Knowledge Hubs

• COVID-END Recommending WG guidance on producing COVID-19 Guidelines [to be released soon]
• 2011 NAM (IOM) Standards for Guidelines
• Guideline development resources: GIN/McMaster Checklist, Australia Guidelines for Guidelines Handbook
• Methodologies for the Development of CHEST Guidelines and Expert Panel Reports 

• [CPG on FHIR and BPM+ IGs do this separately - need to be combined and simplified (e.g., with helpful graphics); e.g., leverage L1-L4]
• [Approach section from CPG on FHIR IG might be helpful in framing scope]

There are threads of work in HL7 and BPM+ community. Results of this work should be available in the coming weeks/months.

By end of Nov HL7 should publish details of how the DGWG got the ED use case guidance to L3.

• Knowledge Elicitation: This goes through types of input that are needed, what needs to be made explicit and computable. Interactions with people developing the guideline. Getting information from guidance developer tools/artifacts into a DGWG tool (based on McMaster work) for making guidance computable. The DGWG template will be made available as part of this tool.
• Terminology Management: reach out to terminology vendors for mappings about how terms are expressed in working systems (e.g., EHRs) and connect this to how guidance developers described terms. Narrative guidelines have clinicians as an audience (e.g., understand what terms like 'patients with diabetes' mean); for guidelines to be computerized, these terms must be expressed based on computable code sets. Fleshing out these terms/definitions is the lion's share of the work in making guidelines computable.
• Execution Model: describes target representation that models the execution semantics that are necessary for any specific implementation (Sivaram/Matt to provide example). Has 'pragmatics' that consistently conveys guidance intent. Translates what SMEs, knowledge engineers, etc. know into system behavior. There are tools to facilitate this work (e.g., OMG BPM+ tools, OWL). 'Case' describes patient details, 'Plan' describes what is (or should be) done for specific patients.

The approach above is intended as a paradigm shift in the approach to developing guidance. Historically it has started with developing a text representation for the guidance that is directly applied by clinicians to enhance decisions and actions. The shift here is developing a computable representation of the guidance that serves as the 'source of truth' for subsequent implementation and modifications. When changes to the guidance model are made, these changes can then flow more seamlessly to CDS interventions, quality measures, eCase Reports, etc. This is more efficient than having a non-computable, text-based representation of the guidance as the 'source of truth,' since the former requires extensive adaptation (which introduces error, ambiguity, time delays, etc.) for implementation and modification.

(Sivaram/Matt to flesh this out) Use shared ontologies to ensure consistent guidelines, reduce rework. These get pulled into authoring tools...

--------------------

Regarding Value Sets recommend the following process (applies to anticoagulation, testing/triage, vaccines, steroids, and any other topic):

1. Take inventory of what value sets you’ll need (be specific)
2. Check Alliance website or Value Set Authority Center (VSAC), we might have already published some of them among the 600+ value sets
3. If certain value sets are not present or you are unsure, contact Victor Lee (Victor_Lee@ClinicalArchitecture.com) to inquire or to request development

Clinical Architecture and other Alliance collaborators are donating our efforts, and Clinical Architecture is leveraging its software tooling, hosting infrastructure, and manual labor to donate all of the COVID-19-related value set content to the public domain (i.e., completely free, no strings attached). That being said, we all have day jobs, so we appreciate having as much lead time as possible to fulfill requests. Happy to take questions.

(VA has library of 100 knowledge artifacts - used HL7 KNART spec to make XML rendering. Using clinical content from these to represent as FHIR questionnaires. This other HL7 work outlined above resonates, and potentially could be leveraged to support VA efforts. There are only a handful of guidelines that VA pushes out from national - other CDS interventions are developed more locally by VA facilities using tools available in the HIT infrastructure. Question: How does the above process reconcile when there are competing recommendations on a particular topic. Answer: The HL7 CPG on FHIR IG addresses localization - can related to workflows, but can also reflect how organizations combine different external recommendations to make essentially a local guideline that differs from the external guidelines. There are 10 workflows and visio diagrams: Robert Lario and Linda Wedemeyer are expressing these in BPM+ as an experiment.)

------------------

[Evidence synthesis teams would like to have something that summarizes for a COVID resource database, where are they pulling information from, what are their inclusion/exclusion criteria, why you might use one source vs. another]

[CPG on FHIR team would like to incorporate insight we generate here - including BPM+ synergies, back into that resource. The 'Integrated Process' about how to develop narrative and computable guidelines in parallel - will be published in about a month.]

Robert Lario - co-chairs OMG BPM+ activities. 3 languages - process modeling (BPMN), decision modeling (DMN), case/event (CMMN) modeling. VA using these to express clinical practice guidelines - sometimes just instructive, other times executable. All have execution models. BPM+ has its own ecosystem. Gaps and hard to do some things with BPM +. Started working on 3 other modeling languages. Situational data - how do you represent structure of data, etc. Provenance who owns/controls and access. and Pedigree: what produces what. Knowledge Package: Many languages/constructs used in a guideline (sequencing). How do you bundle these up into a CPG. How do you surface models, discuss dependencies. Focusing on how do you express knowledge in a clear and unambiguous way, and how to you create artifacts? CPG on FHIR speaks more to methodology - complements BPM+ which doesn't get into deep detail on this. Also not looking at curation and management of models.

Blackford: DGWG ran through effort to implement guidance based on CPG on FHIR. Would like to use a resource like this table to know which tools to use to make guidance computable in different circumstances. How do you implement this at scale.

Address dissemination and marketplaces. (HL7 Marketplaces spec)

• See output repositories under Produce Guidance

• CDS Connect Repository

• CPG on FHIR (working to coordinate with BPM+, and don't yet incorporate value sets); value could be enhanced by better connection through connecting what's in CPG on FHIR to links before and after it in the ecosystem cycle. 
• BPM+
• Logica-led Care Management Implementation Guide [standards and guidance] to cultivate synergies between CPG on FHIR and BPM+ [in development - e.g., see info on Care Management Track at 2020-09 Connectathon]) 
• CDS Hooks;
• SMART on FHIR
• COVID-19 Interoperability Alliance Value Sets
• Logica COVID-19 FHIR Profile Library IG
• [under development] MCBK Standards WG Metadata work

• CDS Connect Authoring Tool

From C19HCC Digital Guideline WG:

• Using CPG-on-FHIR standard for representing/ expressing the full intent of the Guidance in computer-interpretable artifacts (part of HL7 CPG-IG)
• Using the Agile Approach to CPG Development (inclusive of Integrated Process) to concurrently Produce Guidance and Make Guidance Computable (part of HL7 CPG-IG)
• Use Agile Knowledge Engineering methods, principles, and tools
  • Cross-functional Integrated team (Agile CPG Team)
  • Leverage composite nature of CPGs (e.g. can develop logic for inferences on patient information- CPG_CaseFeatures) to build incrementally and iteratively with rapid feedback
  • Pull knowledge engineers into Content design/reviews; pull domain SMEs into knowledge representation design/reviews
• Leverage CPG-on-FHIR as a faithful expression of Guidance and its ability to create computationally derived CDS and Cognitive Support, patient-specific, practice-level digital Quality Measures/Metrics, eCaseReports, etc. to create computable artifacts used downstream in the Learning Health System and to provide closed-loop feedback/feedforward.
• Leverage established tools and capabilities (e.g. BPM+ process and tooling, Clinical Ontology) to author computable Guidance and Open Source tooling to translated into HL7 CPG-on-FHIR to leverage derivative and native compute
• tips:
  • Use established standards and work with standards community (to understand and evolve as needed)
  • Engage consumers/ users early and often
  • Engage downstream vendors (e.g Terminology vendor USED in the EHRs) early
  • Just because everyone everyone is using the same terminology systems doesn't mean they're agreeing how to use the actual terms- this needs to be considered and addressed to make ecosystem/supply chain work properly (feedforward from Evidence, but also feedback of data semantics back into evidence)
  • Learn from related communities of practice (e.g. Agile Software Engineering)

• Implement Guidance includes integrating the computable guidance into organizational information system infrastructure, and deploying the intervention to users, and maintaining the interventions over time. Includes looking at 'leading' indicators (e.g., process changes) regarding intervention use and results. (as opposed to the next "Analyze Care Results" table below that addresses 'lagging indicators' (e.g., clinical outcomes))

--------------------

Regarding Value Sets recommend the following process (applies to anticoagulation, testing/triage, vaccines, steroids, and any other topic):

1. Take inventory of what value sets you’ll need (be specific)
2. Check Alliance website or Value Set Authority Center (VSAC), we might have already published some of them among the 600+ value sets
3. If certain value sets are not present or you are unsure, contact Victor Lee (Victor_Lee@ClinicalArchitecture.com) to inquire or to request development

Clinical Architecture and other Alliance collaborators are donating our efforts, and Clinical Architecture is leveraging its software tooling, hosting infrastructure, and manual labor to donate all of the COVID-19-related value set content to the public domain (i.e., completely free, no strings attached). That being said, we all have day jobs, so we appreciate having as much lead time as possible to fulfill requests. Happy to take questions.

• See output repositories for Make Guidance Computable

• C19 Digital Guidelines WG developing and implementation guide for COVID-19 interventions

• HITEQ Center Guide to Improving Care Processes and Outcomes
• HIMSS Improving Outcomes with CDS

• Includes gathering the data that will be analyzed
• 'Results' include 'lagging indicators' such as clinical outcomes. Analyzing/addressing 'leading indicators' (e.g., process changes) is addressed under "Implement Guidance"

• Curate data into a structured, centralized resource - applying specific instructions about how to obtain and document the data (historically a manual process to locate information but becoming more automated)
  • Need semantic interoperability so that results can be aggregated/compared across CDOs; should be based on widely used, open data and standards
• Put information into registry (institution-specific or cross-institution)
• Generate report - to get feedback about effectiveness, safety, etc. of various interventions

--------------------

Regarding Value Sets recommend the following process (applies to anticoagulation, testing/triage, vaccines, steroids, and any other topic):

1. Take inventory of what value sets you’ll need (be specific)
2. Check Alliance website or Value Set Authority Center (VSAC), we might have already published some of them among the 600+ value sets
3. If certain value sets are not present or you are unsure, contact Victor Lee (Victor_Lee@ClinicalArchitecture.com) to inquire or to request development

Clinical Architecture and other Alliance collaborators are donating our efforts, and Clinical Architecture is leveraging its software tooling, hosting infrastructure, and manual labor to donate all of the COVID-19-related value set content to the public domain (i.e., completely free, no strings attached). That being said, we all have day jobs, so we appreciate having as much lead time as possible to fulfill requests. Happy to take questions.

• MedMorph IG draft

• Registries, e.g., from specialty societies. (discuss with Frank Opelka, ACS)

• [under development] MCBK Standards WG Metadata work
• standards that do (or could) underpin registries

• MedMorph, DEQM FHIR IG

• eCQI Resource Center
• UMich/MCBK Knowledge Grid efforts
• MCBK UK ‘knowledge hackathons’ to implement NICE guidelines
• AHRQ-funded toolkit for ePROs in patient care
• NQF project to create a measurement framework linking quality of care delivered by telehealth, healthcare system readiness, and health outcomes in a disaster

• Use reports (see analyze care results)
• Review reports (see analyze care results) with key stakeholders

--------------------

Regarding Value Sets recommend the following process (applies to anticoagulation, testing/triage, vaccines, steroids, and any other topic):

1. Take inventory of what value sets you’ll need (be specific)
2. Check Alliance website or Value Set Authority Center (VSAC), we might have already published some of them among the 600+ value sets
3. If certain value sets are not present or you are unsure, contact Victor Lee (Victor_Lee@ClinicalArchitecture.com) to inquire or to request development

Clinical Architecture and other Alliance collaborators are donating our efforts, and Clinical Architecture is leveraging its software tooling, hosting infrastructure, and manual labor to donate all of the COVID-19-related value set content to the public domain (i.e., completely free, no strings attached). That being said, we all have day jobs, so we appreciate having as much lead time as possible to fulfill requests. Happy to take questions.

• EBM on FHIR/COKA (standards include: Evidence, EvidenceReport, EvidenceVariable, Group, Citation)
• Vocabulary Map for Evidence-related resources [Project Google Drive]
• DEQM FHIR IG
• FHIR Electronic Case Reporting

• MedMorph