Hail Format
Release files#
The results of this analysis are released in two main files on Google Cloud Storage (file format compatible with Hail >= 0.2.42):
- Summary statistics MatrixTable:
gs://ukb-diverse-pops-public/sumstats_release/results_full.mt(12.78 T) - Meta-analysis MatrixTables (see the detailed descriptions here)
- "High-quality" meta-analyses:
gs://ukb-diverse-pops-public/sumstats_release/meta_analysis.h2_qc.mt(1.5 T) - All ancestries (no QC) meta-analyses:
gs://ukb-diverse-pops-public/sumstats_release/meta_analysis.raw.mt(12.5 T)
- "High-quality" meta-analyses:
These are also available on Amazon S3:
- Summary statistics MatrixTable:
s3://pan-ukb-us-east-1/sumstats_release/results_full.mt(12.78 T) - Meta-analysis MatrixTables (see the detailed descriptions here)
- "High-quality" meta-analyses:
s3://pan-ukb-us-east-1/sumstats_release/meta_analysis.h2_qc.mt(1.5 T) - All ancestries (no QC) meta-analyses:
s3://pan-ukb-us-east-1/sumstats_release/meta_analysis.raw.mt(12.5 T)
- "High-quality" meta-analyses:
In addition, in-sample full LD matrices and scores are available on Amazon S3:
- LD BlockMatrix
s3://pan-ukb-us-east-1/ld_release/UKBB.{pop}.ldadj.bm(43.3 T in total)- Size by population: AFR: 12.0 T, AMR: 3.3 T, CSA: 6.4T, EAS: 2.6T, EUR: 14.1T, MID: 4.9T
- Variant index Hail Table
s3://pan-ukb-us-east-1/ld_release/UKBB.{pop}.ldadj.variant.ht(1.7 G in total) - LD score Hail Table
s3://pan-ukb-us-east-1/ld_release/UKBB.{pop}.ldscore.ht(4.0 G in total)
where {pop} represents one of the population abbreviations (i.e., AFR, AMR, CSA, EAS, EUR, or MID).
Requester pays#
Note that the files in the Google Cloud Storage bucket are "requester pays." In order to compute over these files or download them, you will need to specify a project which may be billed for access and download costs. The data are stored in a US multi-region bucket: thus, access to the dataset is free for use for Compute Engine instances started within US regions, as well as for full downloads within the US and Canada. When performing large analyses on the dataset, we suggest "bringing the compute to the data" and starting a VM or Dataproc cluster in a US region. You can browse the directory structure in a requester pays bucket with the -u flag (and note the hl.init call below to access the data using Hail):
Using the libraries and files#
note
Please note that p-values are now stored as -log10 p-values to avoid underflow.
The files on Google Cloud Platform can be accessed by cloning the ukbb_pan_ancestry and the ukb_common repos and accessing them programmatically. We recommend using these functions, as they allow for automatic application of our QC metrics as well as inclusion of all QC flags and convenience metrics such as lambda GC. By default, when loading using load_final_sumstats_mt, the best practice QC parameters are used, which removes traits with a lambda GC < 0.5 or > 5 as well as applying all QC filters. This results in importing summary statistics for 527 traits; if it is preferable to load all traits with exported summary statistics (e.g., only applying the lambda GC < 0.5 or > 5 filter), use load_final_sumstats_mt(filter_pheno_h2_qc=False), resulting in 7,228 traits. If any filtering is undesirable, use load_final_sumstats_mt(filter_pheno_h2_qc=False, filter_phenos=False), which will import all 7,271 traits.
Additional arguments can be used to adjust the format of the p-values. Enabling exponentiate_p will produce absolute-scale p-values, while legacy_exp_p_values will produce legacy p-values that are in the format ln p.
Results schema#
note
Please note that p-values are now stored as -log10 p-values to avoid underflow.
The basic summary statistics have the following schema:
Columns (phenotypes)#
The columns are indexed by phenotype using a composite key of trait type, phenocode, pheno_sex, coding, and modifier. Trait types have one of the values below. phenocode typically corresponds to the Field from UK Biobank, or the specific ICD code or phecode, or a custom moniker. pheno_sex designates which sexes were run, and is marked as both_sexes for most traits, though some phecodes were restricted to females or males. The coding field is primarily used for categorical variables, to indicate which one-hot encoding was used (e.g. coding 2 for field 1747). Finally, modifier refers to any downstream modifications of the phenotype (e.g. irnt for inverse-rank normal transformation).
By default, the MatrixTable loaded by load_final_sumstats_mt returns one column per phenotype-population pair. We can see the number of unique phenotypes for each trait_type by:
You can explore the population-level data in more detail using (several fields removed for brevity):
More information about the GWAS run is found in the pheno_data struct. This struct also includes all heritability information and QC flags. More details on heritability estimation methods are forthcoming, but can be previewed here. Descriptions of the heritability fields can be found below and more information on QC flags can be found here.
Rows (variants)#
The rows are indexed by locus and alleles. Direct annotations can be found in the vep schema, but we also provide a nearest_genes annotation for ease of analysis. Additionally, variant QC annotations are provided in the high_quality field (which is filtered to by default using load_final_sumstats_mt and can be switched off in the filter_variants parameter in that function).
Entries (association tests)#
note
Please note that p-values are now stored as log p-values to avoid underflow.
The entry fields house the summary statistics themselves. Note that there is a low_confidence annotation that indicates a possible low-quality association test (allele count in cases or controls <= 3, or overall minor allele count < 20).
The resulting dataset can be filtered and annotated as a standard Hail MatrixTable:
Meta-analysis files#
note
Please note that p-values are now stored as -log10 p-values to avoid underflow.
The meta-analysis results are in a similarly structured file which can be obtained as such:
By default, this function imports both the meta-analyses for all phenotype-ancestry pairs with completed GWAS as well as the "high-quality" meta-analyses, which are meta-analyses of only ancestry groups passing all QC filters. Naturally the high-quality meta-analyses are only available for those phenotypes for which at least two ancestries passed all QC (since a requirement for QC PASS is that a trait passes QC in EUR and at least one other ancestry). The meta-analyses for all pairs are found under the meta_analysis struct, while the high-quality meta-analyses are found in the meta_analysis_hq struct.
Meta-analyses using only the high-quality pairs or using all pairs can be imported seperately by using load_meta_analysis_results(h2_filter='pass') or load_meta_analysis_results(h2_filter='none') respectively. These arguments always produce tables with the following entry schema:
All meta-analysis tables have results provided in an array, which includes the all-available-population meta-analysis in the 0th element meta_mt.meta_analysis[0] and leave-one-out meta-analyses in the remainder of the array.
As for the per-population summary statistics, additional arguments can be used to adjust the format of the p-values. Enabling exponentiate_p will produce absolute-scale p-values, while legacy_exp_p_values will produce legacy p-values that are in the format ln p.
Combining the datasets#
We also provide a function to annotate the overall sumstats MatrixTable with the largest meta-analysis for that phenotype. Specification of h2_filter will determine if the annotated meta analysis is for all included phenotypes or for just those passing QC.
If your analysis requires the simultaneous analysis of summary statistics from multiple populations (and not the meta-analysis), you can load the data with a similar structure to the meta-analysis MatrixTable (one column per phenotype, with population information packed into an array of entries and columns) using load_final_sumstats_mt(separate_columns_by_pop=False).
LD matrices#
The LD matrices are in BlockMatrix format. Please refer to Hail's documentation for available operations on BlockMatrix.
We note that the LD matrices were sparsified to a upper triangle (all elements of the lower triangle were zeroed out using BlockMatrix.sparsify_triangle).
Variant indices#
To determine which row/column corresponds to which variant, we provide variant indices for BlockMatrix in Hail Table format.
The variant indices table has the following schema and idx corresponds to a row/column index in BlockMatrix.
Extracting a subset of LD matrix#
To extract a subset of LD matrix, you first need to identify indices of your variants of interest. Here, we provide two examples:
Then, you can filter the LD matrix into a subset using BlockMatrix.filter:
Exporting a LD matrix to a flat file#
Finally, to export a LD matrix to a flat file (txt file), you can use BlockMatrix.export:
If your matrix is small enough to fit on memory, you can also directly export it to numpy via BlockMatrix.to_numpy.
LD scores#
The LD scores are in Hail Table format. For LDSC-compatible flat files, you can find them here.
The LD score table has the following schema.
Heritability estimates#
The heritability estimates can be found as a flat file manifest or as a Hail Table.
This table has the following schema:
Note that this is very similar to the heritability struct in the results schema -- the load_final_sumstats_mt() automatically uses get_h2_ht() to import the heritability table and annotate it into the column schema of the summary statistics results table.
Here the heritability field is an array of structs, where the elements of the array represent the ancestries for which heritability estimates are available. The corresponding ancestry is found in the heritability.pop field. To obtain one row for each ancestry-trait pair, use the following commmand:
The heritability.estimates struct contains point estimates and significance test results for:
- Univariate LD score regression (
ldsc), run using LD score flat files using high-quality HapMap3 SNPs with MAF 0.01 with summary statistics exported from the results table. - Stratified LD score regression (
sldsc_25bin), run using the same summary statistcs asldscwith LD scores generated from SNPs in 5 MAF and 5 LD score bins. - Randomized Haseman-Elston (
rhemc_8bin) using genotype data with 2 MAF and 4 LD score bins using default settings. This was predominantly used to analyze non-EUR ancestry groups but includes a small set of estimates for traits in EUR. - Randomized Haseman-Elston (
rhemc_25bin) using genotype data with 5 MAF and 5 LD score bins using default settings. Only run for non-EUR ancestry groups. - Randomized Haseman-Elston (
rhemc_25bin_50rv) using genotype data with 5 MAF and 5 LD score bins using 50 random variables to reduce run-to-run variability at a slightly higher computational cost. Only run for non-EUR ancestry groups.
Within each of the above methods, we produce observed- and liability-scale estimates as well as standard errors and the z-score for the test of . We also produce LDSC intercept and ratio estimates when relevant.
The heritability.qcflags struct contains the results from our sequential QC filtering scheme -- see quality control for more details. N_ancestry_QC_pass refers to the number of ancestries passing all QC for a given trait.
More information can be found here on the heritability estimation approach, and important caveats can be found here.