Comparison of methods to identify aberrant expression patterns in individual patients: augmenting our toolkit for precision medicine
- Daniel Bottomly1, 2Email author,
- Peter A Ryabinin1, 2,
- Jeffrey W Tyner1, 3,
- Bill H Chang1, 4,
- Marc M Loriaux1, 5,
- Brian J Druker1, 6, 7,
- Shannon K McWeeney†1, 2, 8, 9 and
- Beth Wilmot†1, 2, 8
© Bottomly et al.; licensee BioMed Central Ltd. 2013
Received: 5 July 2013
Accepted: 9 October 2013
Published: 29 November 2013
Patient-specific aberrant expression patterns in conjunction with functional screening assays can guide elucidation of the cancer genome architecture and identification of therapeutic targets. Since most statistical methods for expression analysis are focused on differences between experimental groups, the performance of approaches for patient-specific expression analyses are currently less well characterized. A comparison of methods for the identification of genes that are dysregulated relative to a single sample in a given set of experimental samples, to our knowledge, has not been performed.
We systematically evaluated several methods including variations on the nearest neighbor based outlying degree method, as well as the Zscore and a robust variant for their suitability to detect patient-specific events. The methods were assessed using both simulations and expression data from a cohort of pediatric acute B lymphoblastic leukemia patients.
We first assessed power and false discovery rates using simulations and found that even under optimal conditions, high effect sizes (>4 unit differences) were necessary to have acceptable power for any method (>0.9) though high false discovery rates (>0.1) were pervasive across simulation conditions. Next we introduced a technical factor into the simulation and found that performance was reduced for all methods and that using weights with the outlying degree could provide performance gains depending on the number of samples and genes affected by the technical factor. In our use case that highlights the integration of functional assays and aberrant expression in a patient cohort (the identification of gene dysregulation events associated with the targets from a siRNA screen), we demonstrated that both the outlying degree and the Zscore can successfully identify genes dysregulated in one patient sample. However, only the outlying degree can identify genes dysregulated across several patient samples.
Our results show that outlying degree methods may be a useful alternative to the Zscore or Rscore in a personalized medicine context especially in small to medium sized (between 10 and 50 samples) expression datasets with moderate to high sample-to-sample variability. From these results we provide guidelines for detection of aberrant expression in a precision medicine context.
The use of functional assays such as the interrogation of patient-derived cancer cells against panels of small interfering RNA (siRNA) duplexes or small molecule inhibitors allows patients who are part of the same disease subgroup to be further stratified based on an assessment of the effect of gene down-regulation on cancer cell viability [1, 2]. The advent of precision medicine represents a methodological paradigm shift from traditional detection of differences between experimental groups towards identification of individual events or outliers (for example, individual expression patterns and patient-specific siRNA/drug sensitivities). Although some work has been done characterizing patient-specific dysregulation of pathways [3–6], univariate patient-specific analysis of gene expression has not been thoroughly explored.
Arguably the most common type of analysis procedure applied to mRNA expression experiments is the determination of putative differential expression [7–9]. However, even within specific subgroups of patients with cancer, the same genes are not always dysregulated in the same manner in every specimen. Individual expression patterns can reflect underlying mutation, chromosomal rearrangement and copy number events. This shifts the focus to a different type of analysis procedure: identification of a single sample or small subgroups that have divergent expression from the rest of the group (for example, the detection of candidate oncogenic chromosomal aberrations on the basis of outlier gene expression in prostate cancer ). Many procedures have been devised to detect the latter situation with earliest efforts, cancer outlier profile analysis (COPA)  and the outlier sum (OS) , focused on prioritization after a robust standardization procedure. Others have extended this to robust t or F tests [12–16] or similar procedures [17–20]. Additionally, the problem has also been viewed as one of ‘population’ or proportional differences between two groups [21–23]. Recently, the anti-profile method was developed to look for genes with high variability across samples and used to discriminate colon cancer cases from controls . A limitation of these procedures is that they assume both a control as well as an experimental group though several, including OS, COPA and the very recently described mCOPA , will work with only one group. Others have focused on the observation that, in the presence of outlying subgroups of patients for a given gene, the distribution would become bi- or multimodal [26–28]. Effective parameter estimation for such mixture models would require substantial sample sizes thereby limiting these approaches to large, well-defined cohorts. Additionally, general methods originally devised in other fields such as the outlying degree (OD) [29, 30] or the gene tissue index  can be used in a gene-wise univariate context for finding outlying subgroups. However all of these methods, with the exception of the OD method, provide a ranking of genes for a given cohort, not for a specific sample within the cohort. Searching for outliers or ‘hits’ for a given sample is a common procedure for some types of experiments, such as genome-wide siRNA screens. Two procedures used for these experiments are a Z-transformation (Zscore) or robust Zscore (Rscore) along with a cutoff dictating outlier or hit status . Both methods have been applied to microarray analysis as well. For instance, the Zscore approach was first applied to microarray datasets a decade ago  and still is used for sample-specific analyses as implemented in the cBio web portal . We also note that the Rscore is the first step of the COPA and OS methods with outlier status determined empirically [10, 11].
Complicating the search for outlying subgroups is the fact that microarrays as well as other high throughput assays can be sensitive to many technical factors [35–39]. In addition, expression differences between samples can be caused by many potentially confounding factors regarding clinical samples such as gender, ethnicity and age as well as differences in tissue or cell preparation. A concrete example of such an effect leading to expression differences among two subgroups is the non-coding RNA XIST, which is highly expressed in females but has almost negligible levels in males . Although effective methods exist to correct both known  and unknown  factors, it may still be important to consider overall sample dissimilarities when the expression values of single genes are compared between samples and/or groups. This will become an even more important issue as we move towards a precision medicine clinical paradigm where it is likely that sample processing would immediately follow acquisition rather than forming balanced batches (in terms of relevant covariates) that can be randomized.
In this paper, we consider the question of how to detect genes that exhibit aberrant expression for a subset of patients focusing on the situation where the subset contains only a single patient sample. We perform simulations testing the effectiveness of multiple approaches including the widely used Zscore and Rscore as well as weighted and unweighted variants of the OD method. We first evaluate these approaches, simulating a wide variety of conditions, and show that the OD methods have advantages over the other two methods in terms of performance. In addition to the simulations, we examine gene ranking results across methods for exon array leukemia expression data in the context of corresponding functional assay results (siRNA hits performed on samples from the same patients). We show that the OD methods provide more flexibility than the Zscore and Rscore and further show that the OD method performs similarly or better than the Zscore for two analytical use cases relating the expression data to the siRNA results.
Where I() is the indicator function, S is the number of true positive genes (100 in all the simulations), T is a vector of length n containing the absolute value of the statistics from a given method, and O is an ordered version of T such that its elements are decreasing. The set G 1 is limited to a single integer in (1) whereas G is a set of size S in (2) representing the position in T or O respectively with the true outlier gene(s) for a given sample j. Note that the FDR in this case assumes 100 true positives, which is meant to simulate an activated pathway. The sample assessed for these statistics was always the one with simulated outlier gene expression. Each different combination of parameters was run 10,000 times. Power was computed as the proportion of the 10,000 iterations which were significant at the .05 level. The FDR was reported as the average FDR observed over the 10,000 iterations for each simulation. Simulations and calculation of the P-values and FDR were carried out in parallel on a Beowulf-style cluster using Rmpi  with parallel random number generation using L’Ecuyer’s method  via the rlecuyer package  using R-2.15.1 . Plotting and summaries for the simulations were performed using ggplot2  on R-3.0.1 . Example code for performing these simulations is provided in Additional file 1.
RNA sample preparation and array processing
All research was performed according to the guidelines of the Helsinki Declaration. Specifically, both oral and written informed consent was obtained from the patient or parent/legal guardian for inclusion in the study. Assent was also obtained for patients between ages 7 and 17 years. The study was reviewed and approved by the Institutional Review Board of Oregon Health & Science University.
Mononuclear cells from blood or bone marrow of patients with hematologic malignancies were isolated using a Ficoll gradient. Cell pellets were snap-frozen in liquid nitrogen and cryopreserved at -80°C for subsequent batch extraction of RNA. RNA was extracted using Qiagen RNeasy kits (Qiagen, Valencia, CA) according to the manufacturer instructions, including performance of the optional on-column DNase treatment step. Samples were amplified and labeled using the Ambion WT Expression/Affymetrix WT Terminal Labeling protocol (Affymetrix, Santa Clara, CA). Amplified and labeled cDNA target samples were each hybridized to a Human Exon 1.0 ST array (Affymetrix, Santa Clara, CA). Image processing was performed using Affymetrix Command Console (AGCC) v.3.1.1 software. The expression data have been deposited in the Gene Expression Omnibus database under the identifier [GEO:GSE42731]. Twelve samples had acceptable RNA integrity number scores (>8) and similar overall intensity distributions and were analyzed further. These samples were processed using the oligo  Bioconductor package . Background correction and normalization performed via the RMA method  using the core metaprobesets as well as probesets. Note that probeset-level summarizations were used for the visualizations whereas metaprobeset-level summarizations were used for the methods comparisons. Here we consider metaprobesets to denote single transcripts or genes. Ensembl annotations and coordinates were retrieved from Ensembl Build 69  and used in conjunction with the GenomicFeatures package  to provide gene contexts for the (meta)probesets. All of the applied analysis was performed using R-3.0.1  with plotting again performed using ggplot2  as well as GenomeGraphs . The reshape2  and biomaRt  packages were also utilized.
Expression prioritization approaches
where the mad function is the median absolute deviation.
For all the OD methods, k was set to nine or six for the simulated and real data respectively, based on the simulations in Figures S3 and S4 in Additional file 2. An implementation of these methods is provided in Additional file 1 and will be provided as part of an R package ‘pod’ at .
Results and discussion
Methods and parameters
The Zscore as defined (see Methods) is a simple approach to assessing whether an outlier exists in a moderately sized dataset . However, its use of the difference from the mean as the numerator (as well as the standard deviation in the denominator) means that it potentially could be influenced by outliers itself. This is a well-known property of related procedures based on means and many alternatives exist to reduce the influence of outliers, such as the use of trimmed means or medians. The median-based robust analogue of the Zscore utilizes the difference from the median divided by the median absolute deviation (Rscore) as has been suggested in some of the initial work in looking for genomic outliers . The OD, as implemented (see Methods), is a measure of how different the expression value for a given sample is from the expression values from the k nearest samples for a given gene. The choice of the k parameter in this respect is important as it may impact sensitivity and specificity. The k parameter can take integer values between 1 and m −1 (assuming m >1) with the case of k = 1 equivalent to the absolute difference between the given sample and the most similar of the remaining samples for a given gene. For the case of finding genes containing single sample outliers, we carried out several simulations examining both power and FDR for a wide range of k values. For our simulation size of 20 samples, we found that k = 9 seemed to provide good performance over a range of effect sizes (Figures S1 and S2 in Additional file 2) with relatively little additional performance gains above nine. In general a k value set to a value near m/2 seemed to provide adequate performance for cohort sizes >10 (Figures S3 and S4 in Additional file 2). Note that this assumes that the conditions of the simulation roughly approximate that of the dataset in question and that one is mainly interested in finding single sample outliers. This is likely to be the case for the simulations as they were carried out using similar parameters. Utilizing a different k value may influence power and FDR estimates for a given simulation, though from these simulations it appears that decreases in performance would mainly occur when utilizing a substantially lower k value. Although performance is difficult to assess using experimental data, we argue that for detection of single sample outliers it is similar enough due to the RMA preprocessing, which makes the overall expression distributions more comparable to each other as well as having a range of expression values similar to the simulations.
Evaluation using simulations
Several aspects of the OD method could be improved based on an examination of actual array experiments. First, overall dissimilarities between samples could inappropriately increase the score for a given gene, making it desirable to down-weight sample-sample differences based on a measure of overall dissimilarity. An example of this would be an array that had a subset of genes with dissimilar hybridization characteristics but not to the extent that it would be removed for quality control purposes. Also, this would be important in a precision medicine context as we would expect samples to vary in similarity based on technical and biological factors. A straightforward adaptation of the OD method would be to incorporate weights that would decrease the influence on sample-sample comparisons for a given gene if the samples themselves were highly dissimilar. Based on previous work in the field of spatial statistics , we implemented several variants of the weighted OD, the only difference being whether the weighting was taken into account before (WODb) or after (WODa) the nearest neighbors were computed.
We first compared all methods using a straightforward power simulation where a single gene had a single sample outlier with a true effect size ranging from three to five units, and where data were either generated from a re-centered normal or t-distribution to capture the range of sample-sample variability to which actual samples might belong (Figure 1A). Weighting the OD method based on overall sample dissimilarity in this context had no benefit over the basic OD approach as all samples would be overall very similar as a product of the simulation approach. However, the OD methods had significantly higher power than either the Zscore or Rscore in all six simulations (Figure 1B). Even for the normal distribution simulation, large effect sizes of four or five were necessary to reach high power (>0.9) for all methods whereas only the OD method achieved adequate power (>0.8) at the lowest evaluated effect size. For the t-distribution, no method was able to achieve adequate power even at the highest effect size. An analogous simulation addressing the FDR was also carried out, which demonstrated that the OD method overall had lower FDR values (all six were significantly lower than Zscore or Rscore; Figure 1C). For both distribution types, the FDR was high especially for an effect size of three. The OD method was the only one to achieve acceptable FDR (<0.10) at an effect size of five for the normal distribution. Together, this indicates that reasonable power should be achievable for experimental samples at the expense of higher false discovery for effect sizes greater than four. As discussed below, effect sizes of this magnitude are observable in expression datasets.
Evaluations using experimental data
As an initial applied analysis, we examined the results of the OD and Zscore in the context of the patient sample T119, which had an siRNA hit for ROR1 (Table S2 in Additional file 4). We chose patient sample T119 as it had only a single siRNA hit and therefore we could expect some dysregulated genes that were unique to the sample, demonstrating the arguably most common use case for the Zscore. Overexpression of ROR1 in acute lymphoblastic leukemia samples with the t(1;19) translocation has been previously characterized  and it was hypothesized that the resulting fusion of the genes E2A (TCF3) and PBX1 halt the development of the progenitor B cells and continue the expression of ROR1 along with the preBCR complex. ROR1 and the preBCR complex contribute to proliferation and survival through the PI3K, AKT and MEK/ERK pathways. Examining the expression of E2A and PBX1 in our dataset, we found that E2A was highly expressed across all samples while PBX1 was highly expressed in sample T119 with moderate or low expression in the other samples. As a result, PBX1 was ranked first and second for the Zscore and OD methods respectively for sample T119. It has also been previously suggested by their joint down-regulation following siRNA treatment of the fusion product  that EB-1 (ANKS1B) and one isoform of WNT16 were also up-regulated as a consequence of the E2A-PBX1 fusion. The ANKS1B gene was ranked first by the OD method and ninth by the Zscore method, while the WNT16 gene was ranked 11th by both methods. ROR1 itself was ranked 28th and 30th for the Zscore and OD methods respectively. This demonstrates that both the OD and Zscore methods are effective at pulling out potential gene expression signatures related to a specific patient’s disease characteristics when divergent from the rest of the cohort.
Another use case is the identification of commonalities among patient samples that share one or more gene targets. In our dataset, the siRNA hit for samples 09206 and 08419 was shared (TNK1) so a natural question is whether there are genes that have shared expression dysregulation between the pairs (Table S2 in Additional file 4). As they differ by gender, we first took the step of removing expression differences due to gender. This was done by first fitting a linear model contrasting the genders for all genes and using the resulting matrix of residuals from the model fit as the expression matrix. We then reran the OD and Zscore methods on the matrix of residuals and compared the ranks of the top 20 of each sample using both methods. For the OD method, the TMPRSS15 gene was shared between the top 10 genes in the two patient samples ranked second and third for 09206 and 08419 respectively. By comparison, the Zscore method applied to both samples did not share any genes in the top 10, in fact the lowest ranked gene for 08419 that was in the top 10 of 09206 ranked 10,424th. This demonstrates that the OD method can, in addition to finding the divergently expressed genes for a single sample, identify and prioritize those genes with shared dysregulation between samples with similar functional or clinical phenotypes.
We have addressed the motivating problem of how to detect patient-specific expression dysregulation events, as well as providing guidelines and considerations for these types of analyses. Our emphasis was on the situation where one sample was an outlier relative to the rest and on small to moderate cohort sizes, which was representative of our cohort of patient samples also analyzed using an siRNA sensitivity screen. We benchmarked several methods, Zscore and Rscore as well as several variants of the OD method, under a variety of conditions including different effect sizes and the introduction of technical noise. We determined that the OD method performed equally well or better than the others in the majority of our simulations in terms of power and FDR. The weighted variants of the OD methods had greater performance when a large amount of technical noise was introduced into the simulations.
When these methods were applied to a set of 12 expression arrays from acute B lymphoblastic leukemia samples, we showed that the OD method ranked the majority of high effect size genes higher or equivalent to Zscore or Rscore. Focusing on the Zscore and OD comparison, we found that the Zscore ranked higher those genes that had low sample-sample variation outside of a single outlier, whereas the OD method was more tolerant of sample-sample variation depending on the choice of k. It was further shown that the results of an OD run with k = 1 were more similar to Zscore than OD runs with higher k values. When examining the expression data in the context of the siRNA hits, we noted that the pattern of hits derived from the siRNA screen could either be unique to a cohort or be similar among multiple members. This implies that related gene expression outliers should either be unique or shared. The OD was able to robustly prioritize such unique and shared genes whereas the Zscore was only effective at finding the former. We note that there are other similar contexts in which these methods may be successfully applied outside of finding genes related to siRNA screens. For instance, one could find genes related to adverse clinical outcomes that affect only one or two subjects in a given small to medium sized cohort.
Here, we focused on the detection of genes containing sample expression values that were up-regulated relative to the remaining samples. The OD method can also be applied for the detection of down-regulated genes, by determining the sign of the difference from the sample in question and the mean or median of the samples for a given gene.
One of the difficulties of focusing on the detection of outliers for a given set of samples is that it is much more difficult to control for potential confounders, because any number of technical or biological factors can impact a given sample in a high throughput expression experiment. One way to address known confounders would be the application of these methods to the residuals from a least squares fit or robust alternative, as we demonstrated through the correction of gender effects. Protecting against unknown confounders as in the surrogate variable analysis method  would seem a natural extension to this idea though further research would be necessary.
For our simulations, we assumed that the overall distributions between the samples were highly similar. This assumption is likely to be valid for Affymetrix arrays when RMA  preprocessing and summarization is applied due to the default use of quantile normalization . Because RMA requires the arrays to be preprocessed together, it is desirable to have the expression distributions as comparable as possible to ensure the expression estimates are accurate. As the B-ALL dataset described here was processed in a single batch and each sample analyzed relative to other members of the batch, the RMA procedure was utilized. If multiple batches or even single arrays are analyzed together, a variant of RMA, frozen RMA , is an alternative.
This work represents a step towards the analysis of patient samples in a personalized or precision medicine context. We found that the OD method was more efficient at the task of prioritizing gene expression outliers than other alternatives. Also, by being able to take into account overall sample dissimilarities, it is better suited to address the issues inherent in such a clinical paradigm where analysis should not ideally wait for sufficient sample accrual before processing and analysis. The OD method provides the user with the ability to potentially detect gene expression dysregulation events shared between several samples. It can be used in relatively small cohorts and has high power in that scenario to detect outlier samples if there is a high effect size and relatively little sample-sample variability. We note that these requirements appear to be satisfied in the dataset examined here. Because of this, the OD can perform well in many situations and provides a robust analytical approach for the detection of patient-specific events.
The following additional data are available with the online version of this paper.
cancer outlier profile analysis
false discovery rate
frozen robust multi-array analysis
robust multi-array analysis
robust Z score
small interfering RNA
weighted outlying degree with weighting performed after nearest neighbor computations
weighted outlying degree with weighting performed before nearest neighbor computations.
This work was supported in part by grants from the Leukemia and Lymphoma Society, the National Center for Advancing Translational Sciences (5UL1RR024140) and the National Cancer Institute (5P30CA069533-13) (both to DB, BJD, BW and SKM), and the St. Baldricks Foundation (to BHC), and by grants from the V Foundation for Cancer Research, Gabrielle’s Angel Foundation for Cancer Research, the William Lawrence and Blanche Hughes Fund, and the National Cancer Institute (4 R00CA151457-03) (all to JWT). BJD is an investigator of the Howard Hughes Medical Institute.
- Tyner JW, Deininger MW, Loriaux MM, Chang BH, Gotlib JR, Willis SG, Erickson H, Kovacsovics T, O'Hare T, Heinrich MC, Druker BJ: RNAi screen for rapid therapeutic target identification in leukemia patients. Proc Natl Acad Sci. 2009, 106: 8695-8700.View ArticlePubMedPubMed CentralGoogle Scholar
- Maxson JE, Gotlib J, Pollyea DA, Fleischman AG, Agarwal A, Eide CA, Bottomly D, Wilmot B, McWeeney SK, Tognon CE, Pond JB, Collins RH, Goueli B, Oh ST, Deininger MW, Chang BH, Loriaux MM, Druker BJ, Tyner JW: Oncogenic CSF3R mutations in chronic neutrophilic leukemia and atypical CML. N Engl J Med. 2013, 368: 1781-1790.View ArticlePubMedPubMed CentralGoogle Scholar
- Vaske CJ, Benz SC, Sanborn JZ, Earl D, Szeto C, Zhu J, Haussler D, Stuart JM: Inference of patient-specific pathway activities from multi-dimensional cancer genomics data using PARADIGM. Bioinformatics. 2010, 26: i237-i245.View ArticlePubMedPubMed CentralGoogle Scholar
- Gundem G, Lopez-Bigas N: Sample level enrichment analysis unravels shared stress phenotypes among multiple cancer types. Genome Med. 2012, 4: 28-View ArticlePubMedPubMed CentralGoogle Scholar
- Yi M, Stephens RM: SLEPR: A sample-level enrichment-based pathway ranking method—seeking biological themes through pathway-level consistency. PLoS One. 2008, 3: e3288-View ArticlePubMedPubMed CentralGoogle Scholar
- Bottomly D, Wilmot B, Tyner JW, Eide CA, Loriaux MM, Druker BJ, McWeeney SK: HitWalker: variant prioritization for personalized functional cancer genomics. Bioinformatics. 2013, 29: 509-510.View ArticlePubMedPubMed CentralGoogle Scholar
- Dudoit S, Yang YH, Callow MJ, Speed TP: Statistical methods for identifying differentially expressed genes in replicated cDNA microarray experiments. Statistica Sinica. 2002, 12: 111-140.Google Scholar
- Tusher VG, Tibshirani R, Chu G: Significance analysis of microarrays applied to the ionizing radiation response. Proc Natl Acad Sci. 2001, 98: 5116-5121.View ArticlePubMedPubMed CentralGoogle Scholar
- Smyth GK: Linear models and empirical Bayes methods for assessing differential expression in microarray experiments. Stat Appl Genet Mol Biol. 2004, 3: 3-Google Scholar
- Tomlins SA, Rhodes DR, Perner S, Dhanasekaran SM, Mehra R, Sun X, Varambally S, Cao X, Tchinda J, Kuefer R, Lee C, Montie JE, Shah RB, Pienta KJ, Rubin MA, Chinnaiyan AM: Recurrent fusion of TMPRSS2 and ETS transcription factor genes in prostate cancer. Science. 2005, 310: 644-648.View ArticlePubMedGoogle Scholar
- Tibshirani R, Hastie T: Outlier sums for differential gene expression analysis. Biostatistics. 2007, 8: 2-8.View ArticlePubMedGoogle Scholar
- Wu B: Cancer outlier differential gene expression detection. Biostatistics. 2007, 8: 566-575.View ArticlePubMedGoogle Scholar
- Liu F, Wu B: Multi-group cancer outlier differential gene expression detection. Comput Biol Chem. 2007, 31: 65-71.View ArticlePubMedGoogle Scholar
- Gleiss A, Sanchez-Cabo F, Perco P, Tong D, Heinze G: Adaptive trimmed t‒statistics for identifying predominantly high expression in a microarray experiment. Stat Med. 2011, 30: 52-61.View ArticlePubMedGoogle Scholar
- Wang Y, Rekaya R: LSOSS: detection of cancer outlier differential gene expression. Biomarker Insights. 2010, 5: 69-View ArticlePubMedPubMed CentralGoogle Scholar
- Lian H: MOST: detecting cancer differential gene expression. Biostatistics. 2008, 9: 411-418.View ArticlePubMedGoogle Scholar
- Wang Y, Wu C, Ji Z, Wang B, Liang Y: Non-parametric change-point method for differential gene expression detection. PloS One. 2011, 6: e20060-View ArticlePubMedPubMed CentralGoogle Scholar
- Pinese M, Scarlett CJ, Kench JG, Colvin EK, Segara D, Henshall SM, Sutherland RL, Biankin AV: Messina: a novel analysis tool to identify biologically relevant molecules in disease. PloS One. 2009, 4: e5337-View ArticlePubMedPubMed CentralGoogle Scholar
- Wang Y, Sun G, Ji Z, Xing C, Liang Y: Weighted change-point method for detecting differential gene expression in breast cancer microarray data. PloS One. 2012, 7: e29860-View ArticlePubMedPubMed CentralGoogle Scholar
- Hu J: Cancer outlier detection based on likelihood ratio test. Bioinformatics. 2008, 24: 2193-2199.View ArticlePubMedPubMed CentralGoogle Scholar
- Gadgil M: A population proportion approach for ranking differentially expressed genes. BMC Bioinformatics. 2008, 9: 380-View ArticlePubMedPubMed CentralGoogle Scholar
- Ghosh D: Discrete nonparametric algorithms for outlier detection with genomic data. J Biopharm Stat. 2010, 20: 193-208.View ArticlePubMedPubMed CentralGoogle Scholar
- Ghosh D, Chinnaiyan AM: Genomic outlier profile analysis: mixture models, null hypotheses, and nonparametric estimation. Biostatistics. 2009, 10: 60-69.View ArticlePubMedPubMed CentralGoogle Scholar
- Bravo HC, Pihur V, McCall M, Irizarry R, Leek J: Gene expression anti-profiles as a basis for accurate universal cancer signatures. BMC Bioinformatics. 2012, 13: 272-View ArticlePubMedGoogle Scholar
- Wang C, Taciroglu A, Maetschke SR, Nelson CC, Ragan MA, Davis MJ: mCOPA: analysis of heterogeneous features in cancer expression data. J Clin Bioinforma. 2012, 2: 22-22.View ArticlePubMedPubMed CentralGoogle Scholar
- Karrila S, Lee JHE, Tucker-Kellogg G: A comparison of methods for data-driven cancer outlier discovery, and an application scheme to semisupervised predictive biomarker discovery. Cancer Inform. 2011, 10: 109-PubMedPubMed CentralGoogle Scholar
- Ji Z, Wang Y, Wu C, Wu X, Xing C, Liang Y: Mean, median and tri-mean based statistical detection methods for differential gene expression in microarray data. Image Signal Processing (CISP). 2010, 7: 3142-3rd International Congress on 2010View ArticleGoogle Scholar
- Wang J, Wen S, Symmans WF, Pusztai L, Coombes KR: The bimodality index: a criterion for discovering and ranking bimodal signatures from cancer gene expression profiling data. Cancer Inform. 2009, 7: 199-PubMedPubMed CentralGoogle Scholar
- Angiulli F, Pizzuti C: Fast outlier detection in high dimensional spaces. Principles Data Mining Knowledge Discovery. 2002, 2431: 15-27. 10.1007/3-540-45681-3_2.View ArticleGoogle Scholar
- Zhang J, Wang H: Detecting outlying subspaces for high-dimensional data: the new task, algorithms, and performance. Knowledge Information Systems. 2006, 10: 333-355. 10.1007/s10115-006-0020-z.View ArticleGoogle Scholar
- Mpindi JP, Sara H, Haapa-Paananen S, Kilpinen S, Pisto T, Bucher E, Ojala K, Iljin K, Vainio P, Björkman M: GTI: a novel algorithm for identifying outlier gene expression profiles from integrated microarray datasets. PloS One. 2011, 6: e17259-View ArticlePubMedPubMed CentralGoogle Scholar
- Chung N, Zhang XD, Kreamer A, Locco L, Kuan P, Bartz S, Linsley PS, Ferrer M, Strulovici B: Median absolute deviation to improve hit selection for genome-scale RNAi screens. J Biomol Screen. 2008, 13: 149-158.View ArticlePubMedGoogle Scholar
- Cheadle C, Vawter MP, Freed WJ, Becker KG: Analysis of microarray data using Z score transformation. J Mol Diagn. 2003, 5: 73-81.View ArticlePubMedPubMed CentralGoogle Scholar
- Cerami E, Gao J, Dogrusoz U, Gross BE, Sumer SO, Aksoy BA, Jacobsen A, Byrne CJ, Heuer ML, Larsson E: The cBio cancer genomics portal: an open platform for exploring multidimensional cancer genomics data. Cancer Discov. 2012, 2: 401-404.View ArticlePubMedGoogle Scholar
- Yang YH, Dudoit S, Luu P, Lin DM, Peng V, Ngai J, Speed TP: Normalization for cDNA microarray data: a robust composite method addressing single and multiple slide systematic variation. Nucleic Acids Res. 2002, 30: e15-e15.View ArticlePubMedPubMed CentralGoogle Scholar
- Simon R, Radmacher MD, Dobbin K: Design of studies using DNA microarrays. Genet Epidemiol. 2002, 23: 21-36.View ArticlePubMedGoogle Scholar
- Leek JT, Scharpf RB, Bravo HC, Simcha D, Langmead B, Johnson WE, Geman D, Baggerly K, Irizarry RA: Tackling the widespread and critical impact of batch effects in high-throughput data. Nat Rev Genet. 2010, 11: 733-739.View ArticlePubMedGoogle Scholar
- Xu J, Wong C: Hunting for robust gene signature from cancer profiling data: sources of variability, different interpretations, and recent methodological developments. Cancer Lett. 2010, 296: 9-16.View ArticlePubMedGoogle Scholar
- Lazar C, Meganck S, Taminau J, Steenhoff D, Coletta A, Molter C, Weiss-Solís DY, Duque R, Bersini H, Nowé A: Batch effect removal methods for microarray gene expression data integration: a survey. Brief Bioinform. 2013, 14: 469-490.View ArticlePubMedGoogle Scholar
- Kay GF, Barton SC, Surani MA, Rastan S: Imprinting and X chromosome counting mechanisms determine Xist expression in early mouse development. Cell. 1994, 77: 639-650.View ArticlePubMedGoogle Scholar
- Johnson WE, Li C, Rabinovic A: Adjusting batch effects in microarray expression data using empirical Bayes methods. Biostatistics. 2007, 8: 118-127.View ArticlePubMedGoogle Scholar
- Leek JT, Storey JD: Capturing heterogeneity in gene expression studies by surrogate variable analysis. PLoS Genet. 2007, 3: e161-10.1371/journal.pgen.0030161.View ArticlePubMed CentralGoogle Scholar
- Emerson SC, Emerson SS: The importance of statistical theory in outlier detection. UW Biostatistics Working Paper Series. Working Paper 381. 2011,http://biostats.bepress.com/uwbiostat/paper381Google Scholar
- Yu H: Rmpi: Interface (Wrapper) to MPI (Message-Passing Interface).2011http://www.stats.uwo.ca/faculty/yu/RmpiGoogle Scholar
- L'Ecuyer P, Simard R, Chen EJ, Kelton WD: An object-oriented random-number package with many long streams and substreams. Oper Res. 2002, 50: 1073-1075. 10.1287/opre.50.6.1073.358.View ArticleGoogle Scholar
- Sevcikova H, Rossini T: rlecuyer: R interface to RNG with multiple streams.2011http://cran.r-project.org/web/packages/rlecuyer/Google Scholar
- Team RC: R: A Language and Environment for Statistical Computing. 2013, R Foundation for Statistical Computing: Vienna, AustriaGoogle Scholar
- Wickham H: ggplot2: Elegant Graphics for Data Analysis. 2009, New York: SpringerView ArticleGoogle Scholar
- Carvalho BS, Irizarry RA: A framework for oligonucleotide microarray preprocessing. Bioinformatics. 2010, 26: 2363-2367.View ArticlePubMedPubMed CentralGoogle Scholar
- Gentleman RC, Carey VJ, Bates, others DM: Bioconductor: open software development for computational biology and bioinformatics. Genome Biol. 2004, 5: R80-View ArticlePubMedPubMed CentralGoogle Scholar
- Irizarry RA, Hobbs B, Collin F, Beazer-Barclay YD, Antonellis KJ, Scherf U, Speed TP: Exploration, normalization, and summaries of high density oligonucleotide array probe level data. Biostat. 2003, 4: 249-264. 10.1093/biostatistics/4.2.249.View ArticleGoogle Scholar
- Flicek P, Amode MR, Barrell D, Beal K, Brent S, Carvalho-Silva D, Clapham P, Coates G, Fairley S, Fitzgerald S: Ensembl 2012. Nucleic Acids Res. 2012, 40: D84-D90.View ArticlePubMedPubMed CentralGoogle Scholar
- Lawrence M, Wolfgang H, Pagès H, Aboyoun P, Carlson M, Gentleman R, Morgan MT, Carey VJ: Software for computing and annotating genomic ranges. PLoS Comput Biol. 2013, 9: e1003118-View ArticlePubMedPubMed CentralGoogle Scholar
- Durinck S, Bullard J, Spellman PT, Dudoit S: GenomeGraphs: integrated genomic data visualization with R. BMC Bioinformatics. 2009, 10: 2-View ArticlePubMedPubMed CentralGoogle Scholar
- Wickham H: Reshaping data with the reshape Package. J Stat Software. 2007, 21: 1-20.View ArticleGoogle Scholar
- Durinck S, Moreau Y, Kasprzyk A, Davis S, De Moor B, Brazma A, Huber W: BioMart and Bioconductor: a powerful link between biological databases and microarray data analysis. Bioinformatics. 2005, 21: 3439-3440.View ArticlePubMedGoogle Scholar
- Bottomly D: Tools for patient-specific outlier detection.1988https://github.com/dbottomly/podGoogle Scholar
- Shiffler RE: Maximum Z scores and outliers. Am Statistician. 1988, 42: 79-80.Google Scholar
- Kou Y, Lu C, Chen D: Spatial weighted outlier detection. Proceedings of SIAM Conference on Data Mining: April 20-22 2006; Bethesda. 2006, 614-618. Bethesda: SIAMView ArticleGoogle Scholar
- Kaur H, Burden-Gulley SM, Phillips-Mason PJ, Basilion JP, Sloan AE, Brady-Kalnay SM: Protein tyrosine phosphatase mu regulates glioblastoma cell growth and survival in vivo. Neurooncology. 2012, 14: 561-573.Google Scholar
- Bicocca VT, Chang BH, Masouleh BK, Muschen M, Loriaux MM, Druker BJ, Tyner JW: Crosstalk between ROR1 and the Pre-B cell receptor promotes survival of t (1; 19) acute lymphoblastic leukemia. Cancer Cell. 2012, 22: 656-667.View ArticlePubMedPubMed CentralGoogle Scholar
- Casagrande G, te Kronnie G, Basso G: The effects of siRNA-mediated inhibition of E2A-PBX1 on EB-1 and Wnt16b expression in the 697 pre-B leukemia cell line. Haematologica. 2006, 91: 765-771.PubMedGoogle Scholar
- Bolstad BM, Irizarry RA, Astrand M, Speed TP: A comparison of normalization methods for high density oligonucleotide array data based on variance and bias. Bioinformatics. 2003, 19: 185-193.View ArticlePubMedGoogle Scholar
- McCall MN, Bolstad BM, Irizarry RA: Frozen robust multiarray analysis (fRMA). Biostatistics. 2010, 11: 242-253.View ArticlePubMedPubMed CentralGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.