Skip to content
View drkalexander1's full-sized avatar

Block or report drkalexander1

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. Markdown is supported. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
drkalexander1/README.md

Hi, I'm Dan πŸ±πŸ‘‹

Researcher and data scientist working at the intersection of economics, data science, and AI evaluation. I bring an economist's causal-inference toolkit to questions about how AI systems behave and what we can reliably learn from large-scale data β€” dual graduate training in Economics (Iowa State) and Information/Data Science (Michigan).

πŸ”­ Currently

Running an ongoing weekly LLM calibration & uncertainty evaluation program β€” original evaluations of whether frontier models represent their own uncertainty, spanning probabilistic forecast calibration, interval calibration as question specificity decreases, and cross-level belief consistency. Scored with proper scoring rules (CRPS, log loss, pinball), ECE, and ground-truth-free self-consistency checks; power-analyzed rather than anecdotal (effect sizes, scenarios-needed-for-80%-power, multiple-comparison caveats). Across the program so far: frontier models stay overconfident on rare/underspecified cases, and self-reported confidence is largely uninformative. The anchoring arm closed at Round 10, and the finding it ended on wasn't the anchoring effect itself but a same-turn contradiction β€” models answer "greater" on turn one, then emit an interval lying entirely on the other side of the anchor. Whether clearer labels fix it turns out to be model-specific, token priming was ruled out with a matched control, and the Round 10 confirmation was predicted in a git-committed file before the runs and held on a new item set.

πŸ“Š The eval program

An ongoing weekly program on whether LLMs can represent their own uncertainty. Each round reuses the previous round's design or data β€” newest first.

Round Eval Question Builds on
R6–R10 llm-anchoring-evals Does an irrelevant anchor move a model's estimate and its own interval β€” and do the two turns agree with each other? R3 taxon substrate
R5 eval-meta-consistency In a fresh context, does a model know where its own uncertainty is highest? R3 p10/p50/p90 CIs
R3 bird-taxonomy-evals Are independent beliefs about genus/family/order mutually coherent? R1 species domain
R2 florida-weather-evals Do intervals widen as questions get less specific? (known unknowns) R1 design
R1 michigan-bird-evals Probabilistic calibration against eBird ground truth (unknown unknowns) β€”

Round 10 was pre-registered β€” the prediction sits in commit f09a6c2, committed before the confirmatory runs, and it held.

Side branch (R4) β€” same harness, different question: model output quality and judge bias rather than uncertainty. haiku-evals (5-7-5 form + subject grounding via embedding similarity) β†’ haiku-judge-evals (blind pairwise judging β€” does a model pick its own haiku more often than an independent judge would?)

πŸ—‚οΈ Other work

🧭 Background

  • Research integrity & reproducibility β€” independent statistical replications and forensic data analysis at Cornell supporting institutional research-integrity investigations; co-launched a cross-disciplinary reproducibility initiative, independently reproducing 15+ peer-reviewed studies across economics, sociology, communication, and veterinary medicine; presented methodology at IASSIST 2019.
  • Large-scale applied ML β€” billion-observation HPC pipelines (SLURM, Great Lakes cluster) for ecological modeling: collaborative-filtering and multi-task neural models over 13.6B observations, directed-network migration bottleneck analysis via max-flow min-cut over 3.2M observations.
  • Information retrieval β€” multi-signal (BM25 + SBERT + Jaccard) plagiarism/reuse detection pipeline evaluated on the PAN 2011 benchmark, addressing a gap standard tools like Turnitin miss.
  • Agentic systems β€” technical lead on an LLM-powered natural-language-to-simulation-code interface for a biological cell-modeling platform (Compucell3D / UMich).

πŸ› οΈ Stack

Python PyTorch scikit--learn R SQL HPC/SLURM

πŸ“« Reach me

Email LinkedIn

Pinned Loading

  1. llm-anchoring-evals llm-anchoring-evals Public

    Independent LLM evaluation series: do irrelevant anchors move a model's estimate and its own p10/p50/p90 interval? Jacowitz & Kahneman (1995) adapted to six models in Inspect AI.

    Python

  2. eval-meta-consistency eval-meta-consistency Public

    R5 β€” Inspect eval: meta-consistency of LLM uncertainty representations (stated CI widths vs fresh-context meta relative widths).

    Python

  3. ebird-species-prediction ebird-species-prediction Public

    Large-scale species occurrence prediction: collaborative-filtering and multi-task neural models over 13.6B eBird observations, on SLURM/HPC.

    Python

  4. haiku-evals haiku-evals Public

    R4 β€” LLM benchmark for English haiku: 5-7-5 syllable form and subject grounding measured by embedding similarity (full-text and word-sum cosine).

    Python

  5. sandhill-crane-migration-networks sandhill-crane-migration-networks Public

    Directed-network migration bottleneck analysis for sandhill cranes: max-flow min-cut over 3.2M observations.

    Python 2

  6. bird-taxonomy-evals bird-taxonomy-evals Public

    R3 β€” LLM calibration benchmark for taxonomic hierarchy consistency: do a model's independently elicited genus, family, and order beliefs nest coherently?

    Python