Background

Increased focus on health API standardization and interoperability of clinical and health data—most notably via FHIR—has spurred tremendous innovation in high-level clinical application development. Yet, informatics professionals providing underlying data models, algorithms, libraries, and support services have no widely established vehicle or trusted process facilitating market adoption and life cycle management for this array of computable knowledge assets. Those who produce or consume the informatics resources underlying those channels have yet to achieve the interoperability necessary for broader classes of clinical knowledge, tooling, and knowledge assets to flourish under the rapid deployment and feedback cycles typically favored by app developers. These are also required for rapid responses to evolving medical knowledge, or emergent situations such as SARS pandemics.

This appendix is intended to seed discussion and planning for next-generation national computable health ecosystem with many functioning marketplaces within it and has been refined by the ACTS Marketplace Workgroup in collaboration with the broader ACTS Stakeholder Community (52). The remainder of this document establishes a vision for a sustainable ecosystem, gaps from current works, and participation needed to achieve it in Production HIT systems directly enabling transactional care. Resultant work has been taken into consideration for inclusion within the ACTS Roadmap deliverable.

Industry Lessons

General-purpose computing has faced similar issues for years. Google and Apple control the governance and distribution for almost all software targeted at their respective mobile platforms. Conflicts arise when 3rd-party developers publish software/content competitive with a native capability, commissions are imposed for licensing, subscription, or usage, and alternative channels are either banned or made implausible. Antitrust debates continue worldwide.
As clinical knowledge and information assets become increasingly computable, stewardship of distribution channels for standards-based content will become increasingly important to deciding what is approved for publication. For EHR and other services vendors, control is strategically critical to maximizing usage of metered services, maintaining a competitive edge, and directing how their platforms are used. For CDS implementers and providers, ready access to curated knowledge assets is important to keep the corpus of knowledge up to date, and to respond to emergent situations (e.g., new CDS for COVID outbreak).
Prior AHRQ-funded research demonstrated the feasibility of and challenges concerning the sharing of knowledge assets (e.g., the CDS Consortium (317) (318)). Funding for the CDS Connect (319) and separate PCCDS-LN programs ended with FY19, and both have established "sustainability" workgroups. This AHRQ ACTS initiative (31) has spun up multiple working groups to address the challenges, as has the MCBK community (71). Recently, MCBK also announced a journal utilizing a peer reviewed, academic-publication-like approach to computable knowledge publication in partial overlap with these recommendations. Logica Health (formerly Healthcare Services Platform Consortium or "HSPC") has driven standardization of two HL7 specifications: a core Marketplace API (320) and pluggable product metadata and packaging specification. Commercial entities have driven the adoption of HL7 CQL, and the Clinical Practice Guidelines on FHIR efforts. Taken together, all stakeholders aim to operationalize a reference architecture and various marketplace and services implementations for a large-scale public implementation effort (non-commercial), for which AHRQ is suggested to be a driving agency. As proprietary vendor implementations already have operational marketplaces and mechanisms for knowledge authoring, distribution, and execution, all providers and agencies will benefit from increased collaboration and standardization in this space.
Such efforts help to address the need to rapidly translate new knowledge into computable biomedical knowledge and deliver it to points of care. There exists today a disjoint ecosystem of various knowledge creators and implementers who largely operate in silos with little coordination with other participants in the knowledge supply chain to streamline, standardize, and simplify the creation and implementation of computable knowledge in the form of provider- and consumer-facing decision aids. Coordination of the stakeholders in this ecosystem and the development of best-practices and standards will reduce the friction in knowledge translation and specification into computable biomedical knowledge and decision aids for patients and providers alike. We turn now to describe the various building blocks of this knowledge ecosystem starting with the overarching concept of knowledge repositories and marketplaces.

Repositories vs. Marketplaces

Repositories (e.g., CDS Connect, GitHub/BitBucket, PubMed Central, likely most AHRQ collections, and revision control systems in general) are designed to serve authors and curators in both the active development and governing processes leading up to official releases of a product being made generally available. From an engineering perspective, they provide collaborative functions in the production of expert content or software but tend to be limited in what is offered beyond point of general release. Collaborative workflow, version management, issue triage, and historical tracking are all typically considered core "repository" functions, usually with a general-purpose UI for search/indexing. Many also have close tie-ins with automated services for QA, builds, deployment/syndication, and other business functions that reduce production efforts. These issues may be particularly problematic for knowledge artifacts composed of multiple individual components that may be sourced from different source systems.
It is rare, however, to see a "repository" that offers sophisticated product licensing functions (beyond offering raw downloads) or customer support functions; most are very clear that comments and issues should be addressed directly between the consumer and upstream maintainer and do not take detailed positions on "fitness for purpose." As repository systems are often selected based on the needs of producers or developers, they are often customized for process- and content-specific needs.
Marketplaces (e.g., Logica and/or Graphite implementation, AWS Marketplace, Apple App Store, Google Play, Apervita Marketplace, etc.) are designed to serve the needs of consumers in finding, licensing, and using products that provide a capability within the constraints of their available resources. In contrast to repositories, they typically require a deep understanding of the user's runtime capabilities to show only appropriate results, and usually favor automated, platform-specific product deployment, installation, and consumption over download links. This is due to the fact that consumption of a digital artifact is often highly dependent on local technology, and most platforms are highly proprietary to a specific vendor's ecosystem. Also, unlike repositories, marketplaces focus on assets at the "shrink wrapped" level; they tend to have few code-level collaboration functions, if any, with little interest in the upstream development process. Most customizations are for marketing, sales, licensing, and security purposes. Marketplaces operating for global downstream audiences may further allow for region-specific pricing, metadata, legal documents, analytics, descriptive language, product metadata, provisioning, customer billing, vendor payment, customer service, and feedback management. These business functions are far outside the scope of most repository systems.
At the highest level, marketplaces differ from repositories in terms of their (1) primary stakeholder audiences and (2) core business purposes and functions. The subject content may be the same, but they are very different in their roles within the supply chain. The discussion thus carries highly ambiguous terms, some in overloaded or incompatible ways. For our purposes, we adopt the following understanding in discerning between a repository and marketplace, recognizing our general categorizations are imperfect.
Existing projects that create repository-like content hubs typically differ in that repositories focus on aggregating the computable knowledge assets and play a more passive role in the specifics of how artifacts are consumed. Marketplace-like systems tend to focus more on the commercial activities of knowledge delivery and support, presume the consumer holds certain capabilities, may initiate local functions, aid in acquisition of licensing rights ("entitlements"), and generally facilitate a meaningful degree of interaction and commercial relationship between a knowledge consumer with the various custodians and upstream knowledge producers in the asset supply chain.

Table D-1. Repository & Marketplace Differentiators

These are by no means agreed upon differentiators, but this outline nevertheless highlights a difference of intent amongst operators of such systems. An "app store," in the general colloquial sense, is more aligned with the notion of a marketplace than a repository, in the expectation that a marketplace system:

1. Actively participates in the consumer's acquisition of an entitlement to products. That is, a marketplace carries certain responsibility in the:
   1. Creation and management of an executable knowledge asset at any level (terminology, ontology, value sets, logic, etc.)
   2. Management of end-user entitlements according to machine-friendly terms, not just presenting a blanket product license and/or EULA agreement.
   3. Enforcement of high-level rights via passage of entitlements during automated activities (digital rights management)
   4. Support of commercialization (seller/buyer transactions) for products
2. Directly triggers or otherwise aids in deployment of the product
3. Moderates consumer and performance feedback to knowledge authors or source systems to improve quality and monitor for red flags

Asset Classes

The term "marketplace" implies a broad notion of product types to different audiences. While some implementations will naturally specialize based on specific business needs, the following examples are all implied to be supported by the specification.

Table D-2. Artifact Classes, Standard Types & Exclusions

Challenges and Opportunities

While significant progress has been made with standardizing measure expressions in CQL in the CMS eCQI Resource Center (321) and in the NCQA Healthcare Effectiveness Data and Information Set (HEDIS) (322), in terms of knowledge-ecosystem-level standardization, interoperability efforts are limited except for Logica/Graphite and HL7 Marketplace projects, driven by their respective constituents. The January 2019 Informative Marketplace ballot generally received praise as being a rationally thought-out approach to marketplace interoperability, and the September 2019 Marketplace 2 STU1 ballot filled gap areas based on conceptual feedback, implementation experience, and publication format changes.
Outside of these efforts, other marketplace systems only exist in a narrowly scoped form. These may usually be classified as either:

1. Specialized proprietary distribution channels operated by the vendor of a specific underlying platform (e.g., EHR vendors, cloud PaaS, integration/analytics shops)
2. General-purpose computing marketplaces that have variable awareness of prevailing HIT standards

Additionally, core concepts that enable rapid artifact exchange in content-heavy domains and across disparate clinical systems are foreign to clinical consumers except in a few cases (323). Healthcare research culture often assumes a degree of openness wherein viral disbursement of content and software artifacts is commonly viewed as positive and encouraged. Ethics and policy tend to support this practice. By contrast, expectations for technologically similar artifacts in fields such as entertainment—movies, music, digital gaming etc.—is viewed in the opposite light. Any fiscally sustainable ecosystem must address the reality that creators of computable artifacts have diverse financial business models and licensing strategies.

Artifact Monetization

Creating a mutually beneficial knowledge ecosystem requires that upstream parties – ISVs and content publishers – have a sensible pathway towards monetizing their work, and all parties realize net positive value. While this balance falls outside of the technical HL7 specification, understanding the more common business models is critical in assuring implementations provide more benefit than burden. A shortlist of common licensing and support models include:

Table D-3. Artifact Business Models, Revenue Centers & Examples

Dependencies and Sublicensure

Interoperable, computable clinical content is enabled by common underlying terminology systems and information models. Presence of required dependencies unfortunately does not imply the consumer holds the right to use those dependencies.
This is further complicated by a jungle of terminology-level licensing models. As a notable example, NLM's UMLS license defines a tiered class system of licensing terms based on the nature of the work, and is used by RxNorm, VSAC, and many others. US users may only acquire SNOMED CT through UMLS license. No means of organization-level licensing is provided, nor a means of redistributing UMLS content: not even for Government organizations. It is woefully inadequate and presents significant headaches for content producers wanting to publish computable content with "plug-and-play" ease. Unfortunately, most informaticians seem to ignore these types of licensing issues. Violations are commonplace.
The opportunity is to provide NLM with these compelling, real-world cases of how the UMLS license can significantly impede content adoption when the license is actually followed. Changes to existing national policy will need to come from the top down.

Future State Vision

The environmental risks and duplicative community efforts warrant a coordinated approach to nurturing a functional knowledge ecosystem and marketplaces that encourage development and effective distribution of needed computable information and tools to support care processes, while recognizing that making non-computable assets FAIR is also a goal. In pursuit of a competitive, open ecosystem, and the Quintuple Aim, we envision the establishment of a national platform, policies, standards, and governance/regulation to support trustworthy, standards-based, vendor-neutral exchanges for executable clinical content and services. This computable health ecosystem and public and private marketplaces aim to provide:

1. Tooling for the automated validation and deployment of standards-conformant interfaces and capabilities.
2. Robustness of business processes balancing the interests of all parties in the supply chain of clinical content and software development through deployment, maintenance, and end-of-life.
3. Underlying sustainability at scale through shared burdens and appeal to specific clinical interests.
4. Semantic review of artifacts from clinical domain experts most qualified and vested in their use.
5. Dialog across trust boundaries via simple, understandable standards-based certification processes and user feedback.

These desiderata will be pursued in the AHRQ-funded Computable Health Marketplace pilot program described further below, with the goal that public and private marketplace efforts can align on a standard reference architecture and approach for common components.

Proposed Long-Term Direction

The trajectory of the current Logica/Graphite non-profit marketplace initiative and related Government-driven programs may be leveraged as the collaborative public/private basis for a computable health ecosystem, along with collaboration and contributions from existing vendors and other stakeholders in the ecosystem. Logica/Graphite can provide technical infrastructure, central identity and access management, operational expertise, direct standards engagement, and access to clinical stakeholder membership for the AHRQ Computable Health Marketplace. The direction of the Marketplace itself, however, should not be driven by a single entity but a consolidated partnership of workgroups with informative principles heavily informed by external groups such as the MCBK Infrastructure, Policy, and Standards workgroups, and vendors deeply invested in commercial marketplaces. Continuing efforts of other grant programs (e.g., CDS Connect) could be integrated as available, including those of vendors, university interests, and Government departments (e.g., VA). The CDS Connect repository—in which VHA has vested conceptual interest in long-term sustainability—could potentially merge content into this system. Individual tools, including validation frameworks and authoring environments, would be integrated with the Computable Health Marketplace in a manner targeted to the relevant standards when deemed valuable by the workgroup and based on technical requirements, demand, and developer availability.
The HL7 Marketplace specification achieved STU1 status in September 2019. As the specification does not address governance mechanisms or detailed process for financial transactions, these must be addressed as operational matters by the consolidated workgroups with heavy input from HHS agencies, vendors, and arms-length communities including HL7, the PCCDS-LN, and the MCBK movement. Existing sandbox systems can be leveraged to provide maximal value for consumers of published artifacts.
The notion of a jointly operated program distributes cost burdens but does not alleviate them. Agencies including VHA and AHRQ will submit existing artifacts and provide expertise in repository governance and focused community engagement. Also, software development is needed that provider organizations (notably VHA) may not be able to provide. Software artifacts funded with public funds will be made generally available as open source (IP management described further below at "Product Rights").
Solicitation should be made for provider software development contributions for reasons similar to EHR vendors: direct knowledge of and influence over the direction of the computable health ecosystem is immensely valuable towards creating credibility and democratizing access with the providers to whom services/content are marketed. Further, policy in this area is critical to the nature of a fair ecosystem: a single entity cannot realistically represent such diverse viewpoints simultaneously. The scope of such engagements will remain at the discretion of the respective entities, and the door will be open for groups, in whole or in part, to join a united effort.

Stakeholder Value Propositions

Successful ecosystems balance the incentives of all critical stakeholder groups. Understanding the motivations of detractors across these complicated relationships is key to enticing symbiotic participation. The table below builds upon the PC-CDS Learning Network Trust Framework exploration of "key actors" to include their incentives and concerns likely to affect participation in a marketplace ecosystem.

Table D-4. Key Stakeholders, Incentives & Concerns

Product Rights

The Marketplace ecosystem architecture will adopt several IP rights management concepts friendly to both open access and open source-licensed products as well as commercial products. While DRM considerations are debatably optional in the former cases, commercial ISVs and content vendors argue that they are required for reasonable, fair protections surrounding their work.
To provide concrete guidance for these use cases, the Marketplace specification defines several models related to product-level DRM that form the basis of operator enforcement activities typically absent from "repository" systems.

Entitlements

Almost all our individual daily interactions with digital content are licensed. Many exceptions exist—such as for Government works and public domain materials outside of copyright—but the average person interacts primarily with copyrighted, licensed works. "Open source" software under Apache 2.0, MIT, BSD license etc. is still licensed for use, even though it is "free." In order words, lack of financial transaction does not imply lack of binding legal obligations.
In the world of "closed source" digital content, license rights tend to be temporary. For example, renting a movie for online play may only allow for playback within 24 hours from the start of play, 1 week from the time of rental, or other complex terms. Materials for trial or demonstration use may have similar restrictions, while access to software applications is enforced in more diverse ways.
When a user acquires a license to a product, that record constitutes an "entitlement." The entitlement is bound to the consumer acquiring the product and carries metadata on specific constraints imposed by the license.
All marketplace operators need to decide if using entitlement management applies to their use cases. For national-level resources, the recommendation is to support the most diverse set of licensing models as possible for both F/OSS and commercial works.

Dynamic Authorization

Marketplaces implementing the entitlement portion of the API carry several functions for both (1) local clients/platforms and (2) products themselves to check for valid entitlement. This authorization mechanic is referred to as claiming an entitlement. Marketplaces may provide simple verification that the entitlement is valid, or complex enhancements for directly handling financial transactions and enforcing usage limits if needed.
Marketplaces that support diverse usage-based or metered business monetization models (as described in the previous sections) may use the entitlement and claim capabilities to implement technical solutions corresponding to the discussed common business models. For example, to:

• Inject raw object data when products are deployed.
• Create time-based constraints.
• Meter for certain parameters periodically, such as local user count, local CPU count etc.
• Monitor for possible abuse, such as account sharing or fraudulent claims.

This simple mechanism aims to provide the core capabilities for F/OSS and commercial products to co-exist under the same set of operational principles.
The distinction is not as simple as "F/OSS vs commercial," with exception cases and a broad grey area requiring support. For example, consider the less common licensing cases:

• F/OSS works that require upstream reporting. This may require entitlement management even though the upstream vendor does not require a financial transaction to have taken place.
• Entitlements to commercial works that cannot be purchased, and must be granted by the vendor. This may occur when a vendor:
  • releases updates to a legacy product no longer distributed.
  • offers a special product offered only via voucher (e.g. gamification).
  • incentives such as marketing signup (e.g. timeshare presentations).

External Transactions

Early feedback on the concept of an ecosystem revealed strong reservations for incorporating any form of strict payment processing into a marketplace specification, as not all operators are capable of providing (let alone interested in) "checkout" processes often present in commercial platforms.
Technical Note: For this reason, the core Marketplace API allows but does not require an ordering process. For products requiring an explicit ordering process for entitlement, end users are redirected to an external ordering URL provided by the ISV. When the user completes checkout, the ISV creates an entitlement via marketplace API prior to redirecting them back to the marketplace.
Externalization of payment processing allows for operators to avoid the complexities, administrative, and legal headaches of being a middleman in a resale arrangement if they so choose. Focus may then remain on curation and validation activities. Alternatively, operators may do the complete opposite: elect to fully manage the ordering process on behalf of ISVs, as is common practice in other domains.

Trusted Quality Assurance

A main value proposition to providers, outside of acquisition and technical automation capabilities, is the assembly of expert curation of all published informatics products. In the future state, the majority of support staff are engaged in domain- and/or standards-specific review activities. Ranging from regression testing to hands-on review, this process is central to all submissions. Moderated back-and-forth dialog between reviews and submitters, and resultant improved quality, is expected to form the central portion of the value proposition for providers. Without this, the value of an interoperable marketplace ecosystem becomes tied to technological capabilities more than clinical outcomes.

Enacting the Vision

In alignment with AHRQ's mission "...to produce evidence to make health care safer, higher quality, more accessible, equitable, and affordable…," AHRQ has a substantial interest in every area in the ecosystem.
Across all recommended actions, the Marketplaces Workgroup suggests a 3-year interrelated commitment covering the following distinct areas of focus. Each may be subcontracted or granted individually, though cross-cutting administration and steering resources should remain stable throughout the duration. 3-year budget: $5 million allocated across the following projects.

Action: Establish AHRQ Computable Health Marketplace Pilot Program

The Computable Health Marketplace pilot will be a jointly operated program run as a cooperative under Logica/Graphite or similar collaborative body in the public interest. A targeted Notice of Funding Opportunity (NOFO) is suggested to solicit for a more detailed breakdown of work. AHRQ's operational role will be to:

1. Lead codification of governance processes and policies into the standards-based, open-source reference software systems, as defined by the curation network leads
2. Support the direct ingestion and release of standards-based, computable Government (AHRQ, CMS, ONC, VHA, DOD, etc.) knowledge artifacts or products as they become available
3. Provide outreach activities that amplify the reach and impact of disseminated standards and best practices
4. Directly engage with and mediate the needs of the vendor community with those of provider, payer, and .gov organizations to seek coordination and alignment of public-sector activities with private-sector (i.e., commercial) activities

As a joint program, a stakeholder agreement will be written and adopted by the major stakeholder groups based on the operational vision of mutual expectations, and to resolve any concerns regarding conflict of interest in governance. To assure representation and participation across stakeholder groups, stakeholders will include multiple entities in each of government, provider, vendor, and subject-matter specific classes. For the duration of the chartered work, operation will run under a consortium-type 501(c)(3), 501(c)(4), or equivalent organization. Deliver: Meaningful, executed agreement under a neutral operational entity.
The 1st-year deliverable will be the launch of a Production system with an initial operating capacity (IOC) that is scope-limited to value-rich areas, designed to incrementally expand business capabilities of subsequent years. Due to the interrelated nature across sub-programs, focus on advanced automation use cases and complex curation functions will evolve iteratively. Deliver: Live Production system with limited IOC and incremental improvements.

Action: Establish Trusted Curation Network

A key scalability strategy in seeding a national ecosystem is in the reduction of variable costs per submitted product and sourcing of review activities. AHRQ will work towards externalizing the content review activities from the CDS Connect program into a reformed initiative expressly dedicated to product curation. This will provide:

1. Recruitment to, and management of, organizational and individual expert reviewers, as well as the delegation of "endorsement" authority used to communicate trust and/or confidence to consumers
2. Identification of key components or knowledge sources to be maintained within the Network (e.g., value sets in VSAC, sources of terminologies and ontologies used, libraries of knowledge components (CQL 'defines' for example), etc.)
3. Streamlined clinical and technical reviews, supporting editorial process, and submitter communication, with direct authority to publish, unpublish, reject, or modify product listings with the marketplace pilot system
4. Specified standards-based product formats (e.g., CQL, KNARTs, OMG, etc.) at L3+ level
5. Integration and/or development of standard-specific automated validation tools to be triggered by the Marketplace pilot system (e.g., automated regression testing, packaging compatibility, syntax-level checks etc. Definition of minimal testing requirements, performance thresholds, and test data sets)
6. Simple, clear policies, standard operating procedures, and documentation for all stakeholder groups

AHRQ's ongoing investigations into the trust relationships between parties should be interpreted and applied to the marketplace pilot system as design time guidance at minimum, and more substantially in the creation of balanced operational policies (324). (Specific patient safety and quality measurement criteria for services/tools/content artifacts cannot be fully addressed at the API level as scoped the technical specification.) AHRQ is well positioned to lead the drafting and revision of policies and standard operating procedures governing (a) the marketplace pilot and (b) future external implementations. Relying too heavily on vendor momentum in this area presents risk of skewing operational policies to an unbalanced degree.
The computable health ecosystem should be positioned such that it can be repurposed for general academic publishing; that is, it can positively disrupt the ability of researchers to reduce the implementation time of findings rapidly and safely when canonically published in computable form.
VHA has further gained substantial knowledge in scaling authoring and publication processes through direct engagement with AHRQ's CDS Connect program, as have select academic institutions and vendors. In the future, those lessons should be aggregated with AHRQ policy inputs. This network will tie into marketplace submission review queues based on clinical domain area and level of technical awareness. This is similar to the model used in academic publishing, but with a focus on moving towards automated solutions whenever plausible.

Action: Amplify the Supply

The marketplace pilot system is intended to seed a much broader network of compatible systems adapted to local needs. Expanding beyond the pilot will require direct action to:

1. Seed the "supply" side by direct partnership with strategic content and software partners, and the knowledge sources described above
2. Educate and evangelize through connectathons (HL7, IHE), conferences (HIMSS, AMIA) etc.
3. Continue standardization track within HL7 and Logica/Graphite. STU is Sept 2019, with a normative goal in 1–2 years.
4. Onboard key .gov, .edu, provider, payer, and vendor commercial partners to maximize impact.

The pilot will use high-value projects as initial inputs, such as MITRE software services including HL7 DaVinci (payer) projects, CDS Connect artifacts, and select successful vendor use cases. It will also capture successful/unsuccessful metrics as case series data for release as future publication.

Action: Demonstrate Automated Consumption (Intermediate Term)

The ultimate technical goal of the marketplace pilot is to support fully automated ingestion of clinical content and services into HIT systems across heterogeneous care delivery environments. Long term, a successful pilot will spur the need for plug-and-play interoperability at all levels with minimal IT involvement. Example use cases include:

• Implementation of an externally developed CDS service into an EHR to be effective immediately.
• Access and use of externalized CDS services via standardized APIs into transactional systems (EHR, PHR, consumer-facing apps etc.).
• Rapid update of clinical model and terminology systems and deployment to transactional systems or CDS service implementations.
• Import of a new care process developed by an external provider organization for purposes of immediate implementation.

Action: Study the Value

Ecosystems of complex interactions typically do not instantly materialize. Certain classes of stakeholders are necessary from day zero; others will find value further along the adoption curve. As a producer of evidence, AHRQ should "eat its own dog food" by using itself as a submitter of services/content for review by the Curation Network.
Tracking actual marketplace pilot usage should yield a meaningful data set for upstream analysis and decision making. Feedback will flow in all directions such that monitoring systems may be implemented and improvements may be more rapidly adopted.
Deliver:

• Core observations and evidence-based analysis of the usage and impact of all other program activities
• Assessment of and recommendations concerning the scalability, security, and non-function qualities of operational systems
• Suggestions for areas that may be repurposed or adapted to provide value in unintended areas
• Long-term economic evaluation of sustainability based on empirical data