Skip to content

Add --per-block-alpha: shrink omega per connected component - #12

Draft
jeffreyswigert wants to merge 1 commit into
mainlinefrom
per-block-alpha
Draft

Add --per-block-alpha: shrink omega per connected component#12
jeffreyswigert wants to merge 1 commit into
mainlinefrom
per-block-alpha

Conversation

@jeffreyswigert

Copy link
Copy Markdown
Collaborator

Omega off-diagonal is block-diagonal over the connected components of the
relatedness graph, since a pair only contributes to the block containing both
of its members. The eigenvalues of a block-diagonal matrix are the union of
its blocks, so positive-definiteness is a PER-BLOCK property and there is no
mathematical requirement for alpha to be a single global scalar.

With one global alpha the single worst component sets the shrinkage for every
individual. On All of Us educational attainment at 3rd degree we measure 13 of
43,903 components breaking PD, holding 1.45% of the sample, forcing alpha to
0.4284 and so a 57% shrinkage on the other 98.5%. One 1,351-person component
accounts for almost all of it.

Shrinking each component only as far as it requires preserves PD identically:
every block independently satisfies pheno_variance + alpha_c*lambda_min_c > 0,
so the union of eigenvalues is positive.

Opt-in. Default behaviour is unchanged and tested to be so.

Implementation notes:

  • get_block_alphas returns per-component alphas plus the component labels.
    calculate_omega then scales each entry of the off-diagonal by its own
    component alpha, which needs no block reassembly since both members of a
    pair share a component.
  • A Gershgorin screen skips the eigendecomposition for any block whose largest
    absolute row sum is already below pheno_variance, since such a block cannot
    break PD. That covers the overwhelming majority of ordinary families and
    keeps the cost near the single global call it replaces. A test verifies the
    screen never skips a block that needed shrinking, by checking every alpha
    against a direct eigendecomposition.

Five tests added, and the existing suite passes unchanged (202 total).

One pre-existing issue noticed while testing, NOT addressed here: get_lambda_min
calls ARPACK without a fixed starting vector, so repeated calls on the same
matrix differ in the last ULP or two (-1.0 vs -0.99999999999999989 observed).
Results are therefore not bitwise reproducible on the global path either. Happy
to fix separately if wanted.

Omega off-diagonal is block-diagonal over the connected components of the
relatedness graph, since a pair only contributes to the block containing both
of its members. The eigenvalues of a block-diagonal matrix are the union of
its blocks, so positive-definiteness is a PER-BLOCK property and there is no
mathematical requirement for alpha to be a single global scalar.

With one global alpha the single worst component sets the shrinkage for every
individual. On All of Us educational attainment at 3rd degree we measure 13 of
43,903 components breaking PD, holding 1.45% of the sample, forcing alpha to
0.4284 and so a 57% shrinkage on the other 98.5%. One 1,351-person component
accounts for almost all of it.

Shrinking each component only as far as it requires preserves PD identically:
every block independently satisfies pheno_variance + alpha_c*lambda_min_c > 0,
so the union of eigenvalues is positive.

Opt-in. Default behaviour is unchanged and tested to be so.

Implementation notes:
- get_block_alphas returns per-component alphas plus the component labels.
  calculate_omega then scales each entry of the off-diagonal by its own
  component alpha, which needs no block reassembly since both members of a
  pair share a component.
- A Gershgorin screen skips the eigendecomposition for any block whose largest
  absolute row sum is already below pheno_variance, since such a block cannot
  break PD. That covers the overwhelming majority of ordinary families and
  keeps the cost near the single global call it replaces. A test verifies the
  screen never skips a block that needed shrinking, by checking every alpha
  against a direct eigendecomposition.

Five tests added, and the existing suite passes unchanged (202 total).

One pre-existing issue noticed while testing, NOT addressed here: get_lambda_min
calls ARPACK without a fixed starting vector, so repeated calls on the same
matrix differ in the last ULP or two (-1.0 vs -0.99999999999999989 observed).
Results are therefore not bitwise reproducible on the global path either. Happy
to fix separately if wanted.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant