Customer Financial Risk: MDP Simulation and Reinforcement Learning

A 5-state Markov chain (Good, At-Risk, Delinquent, Default, Closed) models how a neobank's customers actually move toward or away from default, plus a reinforcement learning environment where an agent has to decide, at every step, whether to do nothing, warn a customer, or freeze their account. Built with AVO, a neobank focused on emerging markets, in mind.

What running this actually showed

78% of the population exits into Default or Closed within 30 days. That's the real headline once you account for the full 3,000-customer population correctly, and it wasn't visible in the original chart. The simulator stops writing rows for a customer once they hit an absorbing state, so a plain groupby-by-day undercounts every later day, the population looks like it's shrinking rather than redistributing. Forward-filling each customer's last known state through day 29 gives the true picture.

Expected loss across the 30-day window: $799,135, on a starting population of 3,000 and initial balances drawn from a log-normal distribution.

Training an agent on the intervention environment surfaced a real, non-obvious RL bug, not in the code, in the hyperparameter choice, and it's the most interesting thing in this repo.

Why gamma matters here

rl_env.py's CustomerRiskEnv has a property that's easy to miss reading the code once: the next state a customer moves to depends only on their current state, never on the action the agent takes. Actions change the reward (did you make the right call), never the trajectory. That means the standard Q-learning update, Q(s,a) += alpha * (reward + gamma * max(Q(s')) - Q(s,a)), has a gamma * max(Q(s')) term that is mathematically identical across every action available in state s, since s' doesn't depend on which action was picked. It carries zero information about which action is better. It's pure noise stacked on top of the real signal.

Trained at gamma=0.95, the standard default in most Q-learning examples, the agent gets worse with more training: mean reward over the last 100 of 3,000 episodes (-100.70) is worse than the first 100 (-65.60), and the resulting policy overreacts in the wrong places, freezing Closed accounts and warning Good ones, while getting the one state the reward function is explicit about, Delinquent, wrong (warns instead of freezing, where freezing pays +10 and ignoring it costs -20). Trained at gamma=0.3, same environment, same seed, same everything else, the agent converges cleanly: reward climbs from -64.20 to -32.70 over training, and it correctly learns to warn At-Risk customers, the reward function's single clearest signal (+5, nothing else in that state pays anything).

Delinquent is more interesting. Even at gamma=0.3, the agent lands on warning rather than freezing, the "wrong" call by the hand-derived reading of the reward function, but only by 0.72 in Q-value (-7.25 for warn versus -7.97 for freeze, against a real spread of -25.78 for doing nothing). That gap is a near-tie, not a real preference, and it traces back to a different mechanism than gamma: decaying epsilon-greedy visited "freeze" in this state only a fraction as often as "warn" by the time exploration wound down, so its Q-value never fully converged. Running the same setup for 20,000 episodes with slower epsilon decay resolves it to the theoretically correct freeze (confirmed by hand, not part of the committed scripts). Kept the 3,000-episode result here because it is the realistic first-attempt budget for a problem this size, and because the near-tie is a genuine, separate finding worth showing rather than tuning away: gamma controls whether the agent learns the right thing at all, exploration schedule controls whether it fully resolves close calls once it does.

q_learning_gamma95_demo.py is kept in this repo, not deleted, specifically so this comparison is something you can actually reproduce rather than take on faith.

Other real issues found and fixed in this pass

Every script wrote to or read from a data/ folder that doesn't exist anywhere in this repository. simulate_data.py, model_logic.py, ai_predictor.py, and the old visualize.py all had data/ hardcoded into a file path. Running any of them from a fresh clone failed immediately. Fixed across all of them.

Two scripts ran a blocking Tkinter popup as an unconditional module-level side effect, not inside an if __name__ == "__main__": guard. ai_predictor.py had no guard at all: importing it, not even running it, would train a full Random Forest and then try to open a GUI window. rl_env.py was worse: the popup code referenced variables (obs, env) that were only ever defined inside the properly-guarded block above it, so importing this file anywhere would raise a NameError immediately. Both popups were also strictly redundant, the exact text they displayed was already being printed to the console two lines earlier, and a Tkinter popup can't be included in a README or reviewed by anyone who doesn't have this exact repo open with a display attached. Removed both, kept the console output, which was already correct.

Both training scripts set random.seed(42) and np.random.seed(42) but were still not actually reproducible. env.action_space.sample(), the call both scripts use for epsilon-greedy exploration, draws from gym's own internal RNG, a separate stream that neither of those two seed calls touches. Two runs with "the same seed" could and did produce different reward trajectories and different learned policies. Caught this by rerunning the full pipeline fresh as a final check and diffing the output against the previous run. Fixed by calling env.action_space.seed(42) right after the environment is constructed in both scripts; verified by running each twice and diffing the resulting CSVs byte for byte.

ai_predictor.py's classification report could fail entirely depending on how the train/test split landed. It passed all 5 state names as target_names unconditionally, but classification_report infers its label set from what actually appears in the test split, and Default and Closed are rare enough that a given random split isn't guaranteed to contain both. Fixed to pass the labels actually present in y_test explicitly, so the report always matches the classes it's reporting on regardless of how a given split happens to land.

The state prediction model, honestly

ai_predictor.py trains a Random Forest to predict a customer's next recorded state from their current state, balance, and day. Overall accuracy: 73%, driven almost entirely by how well it predicts "stays Good" (87% precision), which is also the most common outcome. On the states that actually matter for risk management, Default (13% precision) and Closed (2% precision), it's barely better than guessing. Three features, current state, balance, and day, apparently aren't enough signal to predict the rare transitions that matter most. That's a real finding about this feature set, not a tuning problem to paper over.

Running this

pip install pandas numpy matplotlib scikit-learn gym cairosvg

python3 simulate_data.py             # 3,000 synthetic customers, 30 days -> simulated_data.csv
python3 model_logic.py               # daily and cumulative expected loss
python3 visualize_risk.py            # state_distribution.png and loss_chart.png
python3 ai_predictor.py              # Random Forest next-state predictor, prints its own report

python3 rl_env.py                    # sanity-check the environment with 10 random steps
python3 q_learning_gamma95_demo.py   # the naive attempt, kept for comparison
python3 q_learning.py                # the corrected training run -> q_table.csv
python3 visualize_rl.py              # gamma_comparison.png and learned_policy.png

Repository structure

simulate_data.py               generates the 5-state customer population
model_logic.py                 expected and cumulative loss from absorbing states
visualize_risk.py              population and loss charts, forward-fill accounted for
ai_predictor.py                Random Forest next-state predictor

rl_env.py                      CustomerRiskEnv, the custom gym environment
q_learning.py                  the corrected training run, gamma=0.3
q_learning_gamma95_demo.py     the naive attempt, gamma=0.95, kept for comparison
visualize_rl.py                the gamma comparison chart and the learned policy chart

dope_viz.py                    shared hand-built SVG chart primitives

Future improvements

Add customer metadata (region, account type) to the simulation for richer segmentation. Give the Random Forest more features, current state, balance, and day clearly aren't enough to predict the rare Default and Closed transitions well. Extend the RL environment so actions actually do affect the customer's trajectory (a real freeze should change what happens next, not just how the freeze itself is scored), which would make gamma actually matter for a genuine long-horizon reason instead of being a pitfall to avoid.