Conversation
read_header now returns the frame's timestep for every format (XYZ step:, EXYZ Time=, LAMMPS ITEM: TIMESTEP), defaulting to 0 when the token is absent (a plain input config is implicitly t=0). load_configuration surfaces it as config_dict[:t]. Also fix EXYZ Properties parsing: greedy (.*) swallowed the trailing 'Time=' token, so switch to (\S+).
StorePhiTrajectories opens its files in append mode when simulation.t_start > 0, and no longer re-fires store_first on restart (which would duplicate the frame at t_start), matching the convention used by Arianna's stores.
Route the TOML 'restart' key to run\!(; wall_time) (seconds budget; Inf = uncapped), and detect t_start from the StoreLastFrames checkpoint: restart_format reads the configured fmt, and t_start is read from chains/1/lastframe<ext>'s :t when it exists. Detection is gated on a finite wall_time so non-chunked runs always start fresh and ignore stale output.
init_path may now be a Vector of file paths, loaded in the given order (in addition to a single file or a directory). This lets the caller control chain order explicitly — needed on restart so chain c resumes into index c rather than relying on walkdir's ordering.
On restart, build the per-chain lastframe list in numeric order (sort(readdir(chains); by=Int)) and load it instead of config, so every chain resumes from its own checkpoint. Also translate run\!'s status to a process exit code (:need_restart -> 1) so a bash loop can resubmit.
Add read_phi_frame (+ read_block / row builders) to parse lastphiframe.dat, and branch ComputeRotation.initialise on simulation.t_start > 0: reload R_ref, Φ_acc and system.Φ per chain from chains/c/lastphiframe.dat instead of fresh-starting, so the accumulated rotation is continuous across a restart. Also rename the theta_T kwarg to θ_T (constructor + call site) consistently. Verified end-to-end: a molecular job stopped at t=133 and resumed at t_start=133 with Φ continuous across the seam (mol 1 ‖Φ‖ 0.162 -> 0.165, not reset to 0).
V-Francois
reviewed
Sep 23, 2026
V-Francois
left a comment
V-Francois
left a comment
Contributor
There was a problem hiding this comment.
Good stuff!
The description mentions a lack of bit by bit reproducibility, but the rng states are written/read from json. I guess you added this after. Can you update the description and graphs to represent this?
Also, some tests fail.
cyfraysse
force-pushed
the
native-restart
branch
from
September 23, 2026 12:55
bc9a2c0 to
df15e57
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a simulation hits a wall-time budget (< CPU time), it stops cleanly, saves checkpoints, and can be resubmitted by a bash script to continue from these exact checkpoints.
Description :
1- Add a
restartfield in the params.toml (in seconds) that sets the wall-time budget. If the simulation has not finished by this time,run!returns:need_restartand Julia exits1. Else Julia exits0.2- When starting, Julia checks if checkpoints exist. If yes then we are in a restart case (load checkpoints). Else, fresh start.
3- On restart positions are loaded from
lastframe.fmt(fmt = xyz,exyz,lmp) andload_chainstakes care that checkpoints are loaded in numeric order so chain N resumes into index N.4- The restart time
tis parsed depending on the checkpoints format and is used to sett_starton a restart so schedulers are conserved.5- On restart
ComputeRotationreloadsR_ref,ɸ_accandɸfromlastphiframe.datto keep the MSAD continuity.6-
StoreTrajectoriesandStorePhiTrajectoriesopen files in append mode instead of write mode on a restart.7-
load_chainsgets afoldkeywordtrueby default andfalseifrestart=trueso positions are not folded back into the box on a restart. This way MSD is conserved through a restart.8- After
run!the chain's RNG state is serialised tochains/N/rng_state.jls. On restart it is deserialised back intosimulation.algorithm[1].rngs[c]. Requires addingSerializationtoProject.toml.I join the results of a validation test: the blue curve is a simulation restarted every 10 minutes without saving RNG sequence, the orange curve is a single uninterrupted run and the green curve is a simulation restarted every 10 minutes and saving the RNG sequence.
Keeping the RNG sequence prevent bias introduced by restarting from the same seed at each checkpoints (blue curve large time divergence vs green curve statistically equivalent to the orange curve : zscore < 1). However, due to round approximation when saving a checkpoint we are still losing the bit for bit reproducibility.