The original ACMG/AMP guidelines can be translated into an algorithm with only modest adaptations
Two adaptations of the original guidelines had to be introduced to facilitate formal algorithmic reasoning. First, several metarules were introduced to capture implicit “common sense” rules practitioners were expected to apply without explicit instruction. For example, if there are two strong and three moderate-strength evidence tags in favor of pathogenicity, then the variant can be pathogenic (≥2 strong evidence tag) or likely pathogenic (≥3 moderate-strength evidence tags). In such a case the metarules specify that the variant is pathogenic. Conversely, evidence tags that can be combined to yield both Likely Benign and Benign are scored as Benign. The Calculator distinguishes between uncertain significance that arises due to insufficient evidence from uncertain significance due to conflicting evidence.
Second, the guidelines allow the practitioners to change the strength level originally assigned to certain tags based on perceived strength of supporting data for the tag (“To provide critical flexibility to variant classification, some criteria listed as one weight can be moved to another weight using professional judgment, depending on the evidence collected,” from [5]). To accommodate this flexibility within the formal algorithmic reasoning system of the Calculator, multiple new tags were defined for each original tag where the new tags are identical except that their evidence strength was formally redefined. For example, the original “PVS1” tag constitutes “Very Strong” evidence for pathogenicity while the newly defined “PVS1-Strong” tag is defined as the same type of evidence except that it constitutes only “Strong” evidence for pathogenicity.
The calculator reduces human error when applying variant assessment guidelines
The ACMG/AMP guidelines and the Calculator were tested in the context of a multi-site study [1] involving nine molecular diagnostic laboratories involved in the Clinical Sequencing Exploratory Research (CSER) consortium and 99 variants that were evaluated by multiple laboratories who applied the ACMG/AMP rules. The Calculator helped identify human error when these groups applied the guidelines. The Calculator identified that for 16 of 353 (5%) of the variant assessments, the ACMG/AMP evidence codes listed by the participating laboratory did not support the conclusion provided by the laboratory; of these, nine (2.5%) of the 353 were errors. The remaining seven were cases when the labs used professional judgment to override the ACMG/AMP rules.
A user may initiate a session by selecting a previously created assessment from a list or by creating a new assessment (user interfaces not shown). A new assessment is initiated by providing an HGVS allele identifier (for example, NM_000169.2:c.639 + 919G > A in the alpha galactosidase gene, GLA) and condition name and mode of inheritance (Fig. 2a). Using the services of the ClinGen Allele Registry, the Calculator identifies the allele by a canonical identifier (“CA021883” in the top panel) and, if the allele is not present in the registry, provides an option to register the new allele within seconds. Upon registration, all future assessments of the same allele become linked via the canonical identifier, thus enabling the sharing of assessments among users and between the users and any automated assessment pipelines. Shared assessments appear under different tabs in the second panel (Fig. 1) where one may also initiate a new assessment by creating a new blank evidence document (assessment record).
The user can then add evidence tags using one of two available interfaces. A click on an individual cell in Panel 4 activates a new panel where one or more tags that belong to that cell may be applied (Fig. 2b). A user may enter an optional free-text summary and may also associate evidence tags with multiple links to supporting data (Fig. 2c). Figure 2d shows a snapshot of a panel to manage tags using an alternative interface. This panel lists all possible tags for a selected type of variant interpretation data (e.g. population, functional, etc.). This interface is useful when the user does not know the exact location of a tag within the evidence grid (fourth panel in Fig. 1). Immediately upon addition or deletion of a tag (using any of the two interfaces), conclusions in the third panel (Fig. 1) are updated. The conclusions reached based on current tag selection appear highlighted at the top of this panel. A click on the icon to the right of this conclusion highlights the columns in the evidence grid (Additional file 1: Figure S1) that contain supporting evidence for the conclusion. In addition, potential conclusions that are close to being reached are listed, together with the total number of tags that need to be turned on for a conclusion to be reached. A click on the icon to the right of such conclusions highlights the columns in the evidence grid where the missing evidence tags reside (Additional file 1: Figure S1).
Advanced users (application developers) can access the functionalities of the Calculator via Genboree REST-APIs. Using the services, users can upload evidence tags for several variants at a time. The uploaded data become accessible through the interface as well as through REST-APIs. In the future, the bulk upload functionality will be made accessible even for regular users without any background in computer programming.
The calculator facilitates critical evaluation of pathogenicity assertions
Sharing of pathogenicity assessments of pathogenicity of variants by clinical laboratories via databases such as ClinVar aids in the practice of clinical genome interpretation. However, currently, the format of information by which pathogenicity for a given variant was determined by multiple submitters to ClinVar often appears in a free text unstructured format, if it is provided at all. It would be useful if this knowledge sharing was further extended to include the reasoning and evidence codes that supported conclusions about pathogenicity in a standardized format. For example, evidential support for an early assessment that a variant is pathogenic may not have addressed more recent information about the high frequency of the variant in a specific population, thus explaining discordance with a newer independent assessment that includes this recent information. With the evidential support available for inspection, a user of the database may benefit from being able to rapidly and critically evaluate both assessments based on the evidence provided in a structured way. Toward this goal, the Calculator facilitates the sharing of the reasoning and the evidential support behind every assessment. Access to this information is provided via the interfaces (Fig. 1, Panel 3) and also via a single comprehensive report that is also accessible via a URL and is printable as a PDF (Fig. 3). The report (as PDF) may be included in submissions to databases such as ClinVar or with Electronic Health Record (EHR) systems, thus enabling critical evaluation of conclusions about pathogenicity. Although the tool and underlying services are built in anticipation of integration with EHR and other clinical information systems by trained personnel, projects of this type have not yet been initiated.
The calculator helps detect and resolve discordant conclusions
Recent studies indicate that the initial application of the ACMG/AMP guidelines did not increase concordance in laboratory assessment [1]; however, it did help facilitate resolution of the inter-laboratory discordances by providing a framework for organizing and communicating evidential support behind independent assessments of the same variant. To facilitate this process, the ability of the Calculator to track the reasoning and evidence discussed above is necessary but not sufficient. Two more features are essential. First, independent assessments of the same allele must be standardized, as different labs/groups may be using different transcripts or genome builds. This feature is accomplished through the services of ClinGen Allele registry (top panel, Fig. 1). Second, public or group-based sharing of assessments is critical, which is accomplished by using group access management features of the Genboree system. As a result, for each assessment for a given allele that is either owned by a user or available to him/her for public or group-based sharing, the user sees separate tabs in the second panel (Fig. 1) of the Calculator. By clicking on the tabs, the user may access complete information about other assessments (in panels 3 and 4, Fig. 1, Additional file 1: Figure S1 and Additional file 3: Figure S3). This group-based comparison of variant interpretation can also be used for proficiency testing challenges where different sites can be asked to interpret the same variant. The interpretation at the end of a challenge can then be compared to evaluate the degree of overall concordance and identify specific discordances.
The Calculator does not share information about pathogenicity of variant publicly without explicit user request. The information entered is protected using a login and password. It provides optional free-text fields summarizing tag assignments and links to supporting evidence. The Calculator does not require or have fields for providing identifying information about patients. Public sharing per user request will be enabled in future releases.
The calculator guidelines are configurable without programming to accommodate guideline evolution and customizations for specific conditions
Although ACMG/AMP guidelines provide a general framework for Mendelian sequence variant interpretation, it was always recognized that this framework would evolve and be customized to meet requirements for specific genes and disease conditions [2, 16–19]. To address these anticipated needs, the Calculator is designed to support guideline evolution and customization without programming. By using a lower-level GenboreeKB interface (not shown), an “admin” user may fully configure the Calculator by editing the following three structured documents: (1) guideline rules and metarules; (2) evidence codes and their assignments to the columns and rows of the evidence grid (fourth panel in Fig. 1); and (3) the rows and columns of the evidence grid itself. These three documents completely define a formal “guideline” used by the Calculator. Considering this configurability, it is possible to customize the Calculator for specific purposes (e.g. to implement guidelines for somatic variant interpretation and/or multifactorial inheritance).
While each assessment refers to a specific guideline, different guidelines may exist within the same system. For example, for the purpose of assessing variants within genes causing inborn errors of metabolism, metabolite levels could be used as an evidence type in pathogenicity assessments. Because the original ACMG/AMP guidelines do not specify this type of evidence or potentially others that will be added in the future, we have designed flexibility into the Calculator to allow for addition of new evidence types (new rows in the evidence grid) and corresponding evidence tags and of new rules without programming.
