Increased focus on health API standardization and interoperability of clinical and health data—most notably via FHIR—has spurred tremendous innovation in high-level clinical application development. Yet, informatics professionals providing underlying data models, algorithms, libraries, and support services have no widely established vehicle or trusted process facilitating market adoption and life cycle management for this array of computable knowledge assets. Those who produce or consume the informatics resources underlying those channels have yet to achieve the interoperability necessary for broader classes of clinical knowledge, tooling, and knowledge assets to flourish under the rapid deployment and feedback cycles typically favored by app developers. These are also required for rapid responses to evolving medical knowledge, or emergent situations such as SARS pandemics.
This appendix is intended to seed discussion and planning for next-generation national computable health ecosystem with many functioning marketplaces within it and has been refined by the ACTS Marketplace Workgroup in collaboration with the broader ACTS Stakeholder Community (52). The remainder of this document establishes a vision for a sustainable ecosystem, gaps from current works, and participation needed to achieve it in Production HIT systems directly enabling transactional care. Resultant work has been taken into consideration for inclusion within the ACTS Roadmap deliverable.
General-purpose computing has faced similar issues for years. Google and Apple control the governance and distribution for almost all software targeted at their respective mobile platforms. Conflicts arise when 3rd-party developers publish software/content competitive with a native capability, commissions are imposed for licensing, subscription, or usage, and alternative channels are either banned or made implausible. Antitrust debates continue worldwide.
As clinical knowledge and information assets become increasingly computable, stewardship of distribution channels for standards-based content will become increasingly important to deciding what is approved for publication. For EHR and other services vendors, control is strategically critical to maximizing usage of metered services, maintaining a competitive edge, and directing how their platforms are used. For CDS implementers and providers, ready access to curated knowledge assets is important to keep the corpus of knowledge up to date, and to respond to emergent situations (e.g., new CDS for COVID outbreak).
Prior AHRQ-funded research demonstrated the feasibility of and challenges concerning the sharing of knowledge assets (e.g., the CDS Consortium (317) (318)). Funding for the CDS Connect (319) and separate PCCDS-LN programs ended with FY19, and both have established "sustainability" workgroups. This AHRQ ACTS initiative (31) has spun up multiple working groups to address the challenges, as has the MCBK community (71). Recently, MCBK also announced a journal utilizing a peer reviewed, academic-publication-like approach to computable knowledge publication in partial overlap with these recommendations. Logica Health (formerly Healthcare Services Platform Consortium or "HSPC") has driven standardization of two HL7 specifications: a core Marketplace API (320) and pluggable product metadata and packaging specification. Commercial entities have driven the adoption of HL7 CQL, and the Clinical Practice Guidelines on FHIR efforts. Taken together, all stakeholders aim to operationalize a reference architecture and various marketplace and services implementations for a large-scale public implementation effort (non-commercial), for which AHRQ is suggested to be a driving agency. As proprietary vendor implementations already have operational marketplaces and mechanisms for knowledge authoring, distribution, and execution, all providers and agencies will benefit from increased collaboration and standardization in this space.
Such efforts help to address the need to rapidly translate new knowledge into computable biomedical knowledge and deliver it to points of care. There exists today a disjoint ecosystem of various knowledge creators and implementers who largely operate in silos with little coordination with other participants in the knowledge supply chain to streamline, standardize, and simplify the creation and implementation of computable knowledge in the form of provider- and consumer-facing decision aids. Coordination of the stakeholders in this ecosystem and the development of best-practices and standards will reduce the friction in knowledge translation and specification into computable biomedical knowledge and decision aids for patients and providers alike. We turn now to describe the various building blocks of this knowledge ecosystem starting with the overarching concept of knowledge repositories and marketplaces.
Repositories (e.g., CDS Connect, GitHub/BitBucket, PubMed Central, likely most AHRQ collections, and revision control systems in general) are designed to serve authors and curators in both the active development and governing processes leading up to official releases of a product being made generally available. From an engineering perspective, they provide collaborative functions in the production of expert content or software but tend to be limited in what is offered beyond point of general release. Collaborative workflow, version management, issue triage, and historical tracking are all typically considered core "repository" functions, usually with a general-purpose UI for search/indexing. Many also have close tie-ins with automated services for QA, builds, deployment/syndication, and other business functions that reduce production efforts. These issues may be particularly problematic for knowledge artifacts composed of multiple individual components that may be sourced from different source systems.
It is rare, however, to see a "repository" that offers sophisticated product licensing functions (beyond offering raw downloads) or customer support functions; most are very clear that comments and issues should be addressed directly between the consumer and upstream maintainer and do not take detailed positions on "fitness for purpose." As repository systems are often selected based on the needs of producers or developers, they are often customized for process- and content-specific needs.
Marketplaces (e.g., Logica and/or Graphite implementation, AWS Marketplace, Apple App Store, Google Play, Apervita Marketplace, etc.) are designed to serve the needs of consumers in finding, licensing, and using products that provide a capability within the constraints of their available resources. In contrast to repositories, they typically require a deep understanding of the user's runtime capabilities to show only appropriate results, and usually favor automated, platform-specific product deployment, installation, and consumption over download links. This is due to the fact that consumption of a digital artifact is often highly dependent on local technology, and most platforms are highly proprietary to a specific vendor's ecosystem. Also, unlike repositories, marketplaces focus on assets at the "shrink wrapped" level; they tend to have few code-level collaboration functions, if any, with little interest in the upstream development process. Most customizations are for marketing, sales, licensing, and security purposes. Marketplaces operating for global downstream audiences may further allow for region-specific pricing, metadata, legal documents, analytics, descriptive language, product metadata, provisioning, customer billing, vendor payment, customer service, and feedback management. These business functions are far outside the scope of most repository systems.
At the highest level, marketplaces differ from repositories in terms of their (1) primary stakeholder audiences and (2) core business purposes and functions. The subject content may be the same, but they are very different in their roles within the supply chain. The discussion thus carries highly ambiguous terms, some in overloaded or incompatible ways. For our purposes, we adopt the following understanding in discerning between a repository and marketplace, recognizing our general categorizations are imperfect.
Existing projects that create repository-like content hubs typically differ in that repositories focus on aggregating the computable knowledge assets and play a more passive role in the specifics of how artifacts are consumed. Marketplace-like systems tend to focus more on the commercial activities of knowledge delivery and support, presume the consumer holds certain capabilities, may initiate local functions, aid in acquisition of licensing rights ("entitlements"), and generally facilitate a meaningful degree of interaction and commercial relationship between a knowledge consumer with the various custodians and upstream knowledge producers in the asset supply chain.
Table D-1. Repository & Marketplace Differentiators
Physically holds knowledge asset
Moderates consumer/producer communication
Supports orchestrated deployment
Tolerates non-conventionalized artifacts
Facilitates updates/decommissioning processes
Permits "direct" publishing to consumers (non-reviewed governance)
Brokers entitlement (user license) acquisition
Establishes curation processes
Explicitly supports commercial processes
These are by no means agreed upon differentiators, but this outline nevertheless highlights a difference of intent amongst operators of such systems. An "app store," in the general colloquial sense, is more aligned with the notion of a marketplace than a repository, in the expectation that a marketplace system:
The term "marketplace" implies a broad notion of product types to different audiences. While some implementations will naturally specialize based on specific business needs, the following examples are all implied to be supported by the specification.
Table D-2. Artifact Classes, Standard Types & Exclusions
APIs, CDS Hooks, FHIR resource server
Desktop/Mobile native app, SMART UI (web)
When restricted by proprietary platform or EULA
CQL, BPM+ Activity script, expression logic, CQL, eCQMs
General-purpose computer language libs
Clinical Data Models
CIMI/CEM, Argonaut FHIR Profiles, HL7 ANF, QDM, QICore, USCDI data elements, etc.
Knowledge Documents (e.g., ECA rules, documentation templates, order sets)
KNARTs, FHIR Clinical Reasoning] Clinical Practice Guidelines on FHIR
PDFs, ARDEN, Level 1-2 (source documents should be referenced by submissions, but not validated by a marketplace)
Terminology and Ontology Packages
SNOMED RF2, RxNorm, LOINC, ANF, SOLOR
Anything absent a conventionalized packaging mechanism
Workflow / Care Processes
BPM+, FHIR PlanDefinition
While significant progress has been made with standardizing measure expressions in CQL in the CMS eCQI Resource Center (321) and in the NCQA Healthcare Effectiveness Data and Information Set (HEDIS) (322), in terms of knowledge-ecosystem-level standardization, interoperability efforts are limited except for Logica/Graphite and HL7 Marketplace projects, driven by their respective constituents. The January 2019 Informative Marketplace ballot generally received praise as being a rationally thought-out approach to marketplace interoperability, and the September 2019 Marketplace 2 STU1 ballot filled gap areas based on conceptual feedback, implementation experience, and publication format changes.
Outside of these efforts, other marketplace systems only exist in a narrowly scoped form. These may usually be classified as either:
Additionally, core concepts that enable rapid artifact exchange in content-heavy domains and across disparate clinical systems are foreign to clinical consumers except in a few cases (323). Healthcare research culture often assumes a degree of openness wherein viral disbursement of content and software artifacts is commonly viewed as positive and encouraged. Ethics and policy tend to support this practice. By contrast, expectations for technologically similar artifacts in fields such as entertainment—movies, music, digital gaming etc.—is viewed in the opposite light. Any fiscally sustainable ecosystem must address the reality that creators of computable artifacts have diverse financial business models and licensing strategies.
Creating a mutually beneficial knowledge ecosystem requires that upstream parties – ISVs and content publishers – have a sensible pathway towards monetizing their work, and all parties realize net positive value. While this balance falls outside of the technical HL7 specification, understanding the more common business models is critical in assuring implementations provide more benefit than burden. A shortlist of common licensing and support models include:
Table D-3. Artifact Business Models, Revenue Centers & Examples
Professional Services & Support (e.g., for publicly supported offerings)
Consultative/implementation services, Support tickets, custom development
Consultant organizations, Red Hat, MITRE
SaaS/PaaS vendors, MS Office, Netflix
Advanced features, plugins, SLAs
GitHub, Dropbox, Skype
1-time charge per major version
Kindle books, Physical/Immutable goods, mobile apps
Per utility unit (e.g., CPU-hour, transaction, user, object etc.)
SaaS/PaaS vendors (e.g., Apervita), Amazon, Google, Azure, Oracle
Third-party ad views, pay-per-click
Google search, "free" games
Interoperable, computable clinical content is enabled by common underlying terminology systems and information models. Presence of required dependencies unfortunately does not imply the consumer holds the right to use those dependencies.
This is further complicated by a jungle of terminology-level licensing models. As a notable example, NLM's UMLS license defines a tiered class system of licensing terms based on the nature of the work, and is used by RxNorm, VSAC, and many others. US users may only acquire SNOMED CT through UMLS license. No means of organization-level licensing is provided, nor a means of redistributing UMLS content: not even for Government organizations. It is woefully inadequate and presents significant headaches for content producers wanting to publish computable content with "plug-and-play" ease. Unfortunately, most informaticians seem to ignore these types of licensing issues. Violations are commonplace.
The opportunity is to provide NLM with these compelling, real-world cases of how the UMLS license can significantly impede content adoption when the license is actually followed. Changes to existing national policy will need to come from the top down.
The environmental risks and duplicative community efforts warrant a coordinated approach to nurturing a functional knowledge ecosystem and marketplaces that encourage development and effective distribution of needed computable information and tools to support care processes, while recognizing that making non-computable assets FAIR is also a goal. In pursuit of a competitive, open ecosystem, and the Quintuple Aim, we envision the establishment of a national platform, policies, standards, and governance/regulation to support trustworthy, standards-based, vendor-neutral exchanges for executable clinical content and services. This computable health ecosystem and public and private marketplaces aim to provide:
These desiderata will be pursued in the AHRQ-funded Computable Health Marketplace pilot program described further below, with the goal that public and private marketplace efforts can align on a standard reference architecture and approach for common components.
The trajectory of the current Logica/Graphite non-profit marketplace initiative and related Government-driven programs may be leveraged as the collaborative public/private basis for a computable health ecosystem, along with collaboration and contributions from existing vendors and other stakeholders in the ecosystem. Logica/Graphite can provide technical infrastructure, central identity and access management, operational expertise, direct standards engagement, and access to clinical stakeholder membership for the AHRQ Computable Health Marketplace. The direction of the Marketplace itself, however, should not be driven by a single entity but a consolidated partnership of workgroups with informative principles heavily informed by external groups such as the MCBK Infrastructure, Policy, and Standards workgroups, and vendors deeply invested in commercial marketplaces. Continuing efforts of other grant programs (e.g., CDS Connect) could be integrated as available, including those of vendors, university interests, and Government departments (e.g., VA). The CDS Connect repository—in which VHA has vested conceptual interest in long-term sustainability—could potentially merge content into this system. Individual tools, including validation frameworks and authoring environments, would be integrated with the Computable Health Marketplace in a manner targeted to the relevant standards when deemed valuable by the workgroup and based on technical requirements, demand, and developer availability.
The HL7 Marketplace specification achieved STU1 status in September 2019. As the specification does not address governance mechanisms or detailed process for financial transactions, these must be addressed as operational matters by the consolidated workgroups with heavy input from HHS agencies, vendors, and arms-length communities including HL7, the PCCDS-LN, and the MCBK movement. Existing sandbox systems can be leveraged to provide maximal value for consumers of published artifacts.
The notion of a jointly operated program distributes cost burdens but does not alleviate them. Agencies including VHA and AHRQ will submit existing artifacts and provide expertise in repository governance and focused community engagement. Also, software development is needed that provider organizations (notably VHA) may not be able to provide. Software artifacts funded with public funds will be made generally available as open source (IP management described further below at "Product Rights").
Solicitation should be made for provider software development contributions for reasons similar to EHR vendors: direct knowledge of and influence over the direction of the computable health ecosystem is immensely valuable towards creating credibility and democratizing access with the providers to whom services/content are marketed. Further, policy in this area is critical to the nature of a fair ecosystem: a single entity cannot realistically represent such diverse viewpoints simultaneously. The scope of such engagements will remain at the discretion of the respective entities, and the door will be open for groups, in whole or in part, to join a united effort.
Successful ecosystems balance the incentives of all critical stakeholder groups. Understanding the motivations of detractors across these complicated relationships is key to enticing symbiotic participation. The table below builds upon the PC-CDS Learning Network Trust Framework exploration of "key actors" to include their incentives and concerns likely to affect participation in a marketplace ecosystem.
Table D-4. Key Stakeholders, Incentives & Concerns
Medical professionals who care for patients (physicians, nurses, etc.).
Ability to quickly discover available assets, gauge suitability for purpose, understand supporting evidence, and estimate costs/risks; evaluate performance in sandbox context using local data
Quality of artifacts; safety for stated purpose; level of support; clinical localization; technical bindings; fitness for purpose; costs
Commercial entities that provide health-related technology solutions (EHR vendors, CDS vendors, etc.).
Clear testing/certification criteria; access to validation tooling & data sets; easy licensing and distribution mechanism; access to captive audience
Submission costs; sales commission; duplicating efforts for different marketplace systems
Professionals such as domain experts and professional societies who write guidelines or other materials that provide clinical evidence to users in unstructured format (narrative text, image files, etc.).
Targeted packaging format per specification type; access to runtime testing tools
Restricted to supported standards/representations
Professionals who maintain knowledge artifact libraries to ensure evidence is trustworthy (accurate, reliable, timely, etc.).
Oversight of publication process from initial submission through end of life; ability to apply domain endorsements
Overwhelming amount of content beyond what is reviewable; life cycle management beyond publication; access to testing/certification tooling
Knowledge Publishers / Distributors
Professional organizations that package, market, or sell knowledge artifacts as private organizations or in PPPs.
Control policies, licensing, and rights management; standardized API for submission activities and downstream consumer systems; ability to sustain an ecosystem independent of specific EHR platform
Complex ecosystem and operational requirements; many actors required to be successful; no common metadata structures across classes of products
Professionals who translate clinical guidelines into artifacts in semi-structured human readable form (L2), a computer interpretable form (L3), and machine- executable formats (L4).
Targeted packaging format per specification type; access to runtime testing tools
Restricted to supported standards/representations
Organizational Governance Bodies
A governance body that reviews and approves CDS to be used in an organization or across networks.
Trusted source of rapidly deployable clinical knowledge modules and services.
Persons who are the ultimate decision-makers in their healthcare and managing their health.
Alternate end user to clinicians; PHR tooling/services; direct value in addition to care activities; transparency into care; trust in quality
Difficult to understand; different from general-purpose computing marketplaces.
Organizations that pay clinicians or patients for health-related activities.
Fairness; Common vehicle for distribution of payer services and tools (e.g. HL7 DaVinci)
IP protections; leaking of proprietary/confidential information; reverse engineering of risk models
Persons who develop legal or policy guidance that guide care or payment.
Operational target regarding governance, culture, and technology
Population Health End Users
Professionals who support clinicians and clinical teams by monitoring population health trends and recommending actions.
Trusted source of rapidly deployable clinical knowledge modules and services.
Professionals who measure the impact of implemented CDS within health IT.
Multiple vectors—Direct integration w/marketplaces themselves for certification purposes and monitoring of usage, feedback, response, licensing etc; distribution vehicle for CQMs and other formalized computable quality measures
The Marketplace ecosystem architecture will adopt several IP rights management concepts friendly to both open access and open source-licensed products as well as commercial products. While DRM considerations are debatably optional in the former cases, commercial ISVs and content vendors argue that they are required for reasonable, fair protections surrounding their work.
To provide concrete guidance for these use cases, the Marketplace specification defines several models related to product-level DRM that form the basis of operator enforcement activities typically absent from "repository" systems.
Almost all our individual daily interactions with digital content are licensed. Many exceptions exist—such as for Government works and public domain materials outside of copyright—but the average person interacts primarily with copyrighted, licensed works. "Open source" software under Apache 2.0, MIT, BSD license etc. is still licensed for use, even though it is "free." In order words, lack of financial transaction does not imply lack of binding legal obligations.
In the world of "closed source" digital content, license rights tend to be temporary. For example, renting a movie for online play may only allow for playback within 24 hours from the start of play, 1 week from the time of rental, or other complex terms. Materials for trial or demonstration use may have similar restrictions, while access to software applications is enforced in more diverse ways.
When a user acquires a license to a product, that record constitutes an "entitlement." The entitlement is bound to the consumer acquiring the product and carries metadata on specific constraints imposed by the license.
All marketplace operators need to decide if using entitlement management applies to their use cases. For national-level resources, the recommendation is to support the most diverse set of licensing models as possible for both F/OSS and commercial works.
Marketplaces implementing the entitlement portion of the API carry several functions for both (1) local clients/platforms and (2) products themselves to check for valid entitlement. This authorization mechanic is referred to as claiming an entitlement. Marketplaces may provide simple verification that the entitlement is valid, or complex enhancements for directly handling financial transactions and enforcing usage limits if needed.
Marketplaces that support diverse usage-based or metered business monetization models (as described in the previous sections) may use the entitlement and claim capabilities to implement technical solutions corresponding to the discussed common business models. For example, to:
This simple mechanism aims to provide the core capabilities for F/OSS and commercial products to co-exist under the same set of operational principles.
The distinction is not as simple as "F/OSS vs commercial," with exception cases and a broad grey area requiring support. For example, consider the less common licensing cases:
Early feedback on the concept of an ecosystem revealed strong reservations for incorporating any form of strict payment processing into a marketplace specification, as not all operators are capable of providing (let alone interested in) "checkout" processes often present in commercial platforms.
Technical Note: For this reason, the core Marketplace API allows but does not require an ordering process. For products requiring an explicit ordering process for entitlement, end users are redirected to an external ordering URL provided by the ISV. When the user completes checkout, the ISV creates an entitlement via marketplace API prior to redirecting them back to the marketplace.
Externalization of payment processing allows for operators to avoid the complexities, administrative, and legal headaches of being a middleman in a resale arrangement if they so choose. Focus may then remain on curation and validation activities. Alternatively, operators may do the complete opposite: elect to fully manage the ordering process on behalf of ISVs, as is common practice in other domains.
A main value proposition to providers, outside of acquisition and technical automation capabilities, is the assembly of expert curation of all published informatics products. In the future state, the majority of support staff are engaged in domain- and/or standards-specific review activities. Ranging from regression testing to hands-on review, this process is central to all submissions. Moderated back-and-forth dialog between reviews and submitters, and resultant improved quality, is expected to form the central portion of the value proposition for providers. Without this, the value of an interoperable marketplace ecosystem becomes tied to technological capabilities more than clinical outcomes.
In alignment with AHRQ's mission "...to produce evidence to make health care safer, higher quality, more accessible, equitable, and affordable…," AHRQ has a substantial interest in every area in the ecosystem.
Across all recommended actions, the Marketplaces Workgroup suggests a 3-year interrelated commitment covering the following distinct areas of focus. Each may be subcontracted or granted individually, though cross-cutting administration and steering resources should remain stable throughout the duration. 3-year budget: $5 million allocated across the following projects.
The Computable Health Marketplace pilot will be a jointly operated program run as a cooperative under Logica/Graphite or similar collaborative body in the public interest. A targeted Notice of Funding Opportunity (NOFO) is suggested to solicit for a more detailed breakdown of work. AHRQ's operational role will be to:
As a joint program, a stakeholder agreement will be written and adopted by the major stakeholder groups based on the operational vision of mutual expectations, and to resolve any concerns regarding conflict of interest in governance. To assure representation and participation across stakeholder groups, stakeholders will include multiple entities in each of government, provider, vendor, and subject-matter specific classes. For the duration of the chartered work, operation will run under a consortium-type 501(c)(3), 501(c)(4), or equivalent organization. Deliver: Meaningful, executed agreement under a neutral operational entity.
The 1st-year deliverable will be the launch of a Production system with an initial operating capacity (IOC) that is scope-limited to value-rich areas, designed to incrementally expand business capabilities of subsequent years. Due to the interrelated nature across sub-programs, focus on advanced automation use cases and complex curation functions will evolve iteratively. Deliver: Live Production system with limited IOC and incremental improvements.
A key scalability strategy in seeding a national ecosystem is in the reduction of variable costs per submitted product and sourcing of review activities. AHRQ will work towards externalizing the content review activities from the CDS Connect program into a reformed initiative expressly dedicated to product curation. This will provide:
AHRQ's ongoing investigations into the trust relationships between parties should be interpreted and applied to the marketplace pilot system as design time guidance at minimum, and more substantially in the creation of balanced operational policies (324). (Specific patient safety and quality measurement criteria for services/tools/content artifacts cannot be fully addressed at the API level as scoped the technical specification.) AHRQ is well positioned to lead the drafting and revision of policies and standard operating procedures governing (a) the marketplace pilot and (b) future external implementations. Relying too heavily on vendor momentum in this area presents risk of skewing operational policies to an unbalanced degree.
The computable health ecosystem should be positioned such that it can be repurposed for general academic publishing; that is, it can positively disrupt the ability of researchers to reduce the implementation time of findings rapidly and safely when canonically published in computable form.
VHA has further gained substantial knowledge in scaling authoring and publication processes through direct engagement with AHRQ's CDS Connect program, as have select academic institutions and vendors. In the future, those lessons should be aggregated with AHRQ policy inputs. This network will tie into marketplace submission review queues based on clinical domain area and level of technical awareness. This is similar to the model used in academic publishing, but with a focus on moving towards automated solutions whenever plausible.
The marketplace pilot system is intended to seed a much broader network of compatible systems adapted to local needs. Expanding beyond the pilot will require direct action to:
The pilot will use high-value projects as initial inputs, such as MITRE software services including HL7 DaVinci (payer) projects, CDS Connect artifacts, and select successful vendor use cases. It will also capture successful/unsuccessful metrics as case series data for release as future publication.
The ultimate technical goal of the marketplace pilot is to support fully automated ingestion of clinical content and services into HIT systems across heterogeneous care delivery environments. Long term, a successful pilot will spur the need for plug-and-play interoperability at all levels with minimal IT involvement. Example use cases include:
Ecosystems of complex interactions typically do not instantly materialize. Certain classes of stakeholders are necessary from day zero; others will find value further along the adoption curve. As a producer of evidence, AHRQ should "eat its own dog food" by using itself as a submitter of services/content for review by the Curation Network.
Tracking actual marketplace pilot usage should yield a meaningful data set for upstream analysis and decision making. Feedback will flow in all directions such that monitoring systems may be implemented and improvements may be more rapidly adopted.