Add --per-block-alpha: shrink omega per connected component - #12
Draft
jeffreyswigert wants to merge 1 commit into
Draft
Add --per-block-alpha: shrink omega per connected component#12jeffreyswigert wants to merge 1 commit into
jeffreyswigert wants to merge 1 commit into
Conversation
Omega off-diagonal is block-diagonal over the connected components of the relatedness graph, since a pair only contributes to the block containing both of its members. The eigenvalues of a block-diagonal matrix are the union of its blocks, so positive-definiteness is a PER-BLOCK property and there is no mathematical requirement for alpha to be a single global scalar. With one global alpha the single worst component sets the shrinkage for every individual. On All of Us educational attainment at 3rd degree we measure 13 of 43,903 components breaking PD, holding 1.45% of the sample, forcing alpha to 0.4284 and so a 57% shrinkage on the other 98.5%. One 1,351-person component accounts for almost all of it. Shrinking each component only as far as it requires preserves PD identically: every block independently satisfies pheno_variance + alpha_c*lambda_min_c > 0, so the union of eigenvalues is positive. Opt-in. Default behaviour is unchanged and tested to be so. Implementation notes: - get_block_alphas returns per-component alphas plus the component labels. calculate_omega then scales each entry of the off-diagonal by its own component alpha, which needs no block reassembly since both members of a pair share a component. - A Gershgorin screen skips the eigendecomposition for any block whose largest absolute row sum is already below pheno_variance, since such a block cannot break PD. That covers the overwhelming majority of ordinary families and keeps the cost near the single global call it replaces. A test verifies the screen never skips a block that needed shrinking, by checking every alpha against a direct eigendecomposition. Five tests added, and the existing suite passes unchanged (202 total). One pre-existing issue noticed while testing, NOT addressed here: get_lambda_min calls ARPACK without a fixed starting vector, so repeated calls on the same matrix differ in the last ULP or two (-1.0 vs -0.99999999999999989 observed). Results are therefore not bitwise reproducible on the global path either. Happy to fix separately if wanted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Omega off-diagonal is block-diagonal over the connected components of the
relatedness graph, since a pair only contributes to the block containing both
of its members. The eigenvalues of a block-diagonal matrix are the union of
its blocks, so positive-definiteness is a PER-BLOCK property and there is no
mathematical requirement for alpha to be a single global scalar.
With one global alpha the single worst component sets the shrinkage for every
individual. On All of Us educational attainment at 3rd degree we measure 13 of
43,903 components breaking PD, holding 1.45% of the sample, forcing alpha to
0.4284 and so a 57% shrinkage on the other 98.5%. One 1,351-person component
accounts for almost all of it.
Shrinking each component only as far as it requires preserves PD identically:
every block independently satisfies pheno_variance + alpha_c*lambda_min_c > 0,
so the union of eigenvalues is positive.
Opt-in. Default behaviour is unchanged and tested to be so.
Implementation notes:
calculate_omega then scales each entry of the off-diagonal by its own
component alpha, which needs no block reassembly since both members of a
pair share a component.
absolute row sum is already below pheno_variance, since such a block cannot
break PD. That covers the overwhelming majority of ordinary families and
keeps the cost near the single global call it replaces. A test verifies the
screen never skips a block that needed shrinking, by checking every alpha
against a direct eigendecomposition.
Five tests added, and the existing suite passes unchanged (202 total).
One pre-existing issue noticed while testing, NOT addressed here: get_lambda_min
calls ARPACK without a fixed starting vector, so repeated calls on the same
matrix differ in the last ULP or two (-1.0 vs -0.99999999999999989 observed).
Results are therefore not bitwise reproducible on the global path either. Happy
to fix separately if wanted.