From 8a04aa707bf631e62808e8c0f9dfc4b1b7134f2f Mon Sep 17 00:00:00 2001 From: James Date: Tue, 25 Aug 2026 03:46:27 +0000 Subject: [PATCH] Move the maintenance notes out of the evaluation repo docs/maintenance-notes.md was maintainer history: a 31 KB account of the 2026-07 cleanup, opening with its own disclaimer that "none of this is needed to use the demos". It was linked six times from the README, including second in the top nav bar, so the earliest thing a visitor could click -- before running anything -- was a table reading 28 open Dependabot advisories, 25 committed jars, 3 tests failing and documented as expected, and 263 files whose path contradicted their own package line. All of that is fixed, and the candour is what made the document good. But this repository is what people clone to decide whether to buy the parser, and a public catalogue of past defects in the nav bar works against that. The audience for the content is whoever maintains this repo next, which is not the same audience the repo is for. It also demonstrated the cost of being on check-stale-docs.sh's ALLOW list. It had to be there -- its job is naming things that were deleted -- which meant the one guard we have could not see it drift, and it had: a test count two releases stale (144, now 156) and five references to connector/, renamed the day before. That list is now README.md alone, with a comment explaining why to keep it that way. The file now lives at docs/gsp_demo_java-maintenance-notes.md in the private sqlparser/gsp_java repository, with a header recording where it came from, the two facts corrected on the way over, and the fact that its change log stops at 2026-08-12. Its full history stays in this repository's git log. Nothing is lost from the README: the two deep links that carried substance are inlined where they were referenced -- the four package roots and 263 contradicting files, and all four pom_dlineage.xml breakages -- so both warnings now stand on their own. Also fixes two relative links that had been broken for some time and that nothing was checking: columnImpact pointed at ../dlineage from a directory where that resolves to antiSQLInjection/dlineage, and search pointed at ./visitors from inside search/. Every relative link in all 64 tracked markdown files resolves now. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01UK3dBXFXqiDRYJxz1YxWCr --- .github/scripts/check-stale-docs.sh | 8 +- README.md | 25 +- docs/maintenance-notes.md | 561 ------------------ .../antiSQLInjection/columnImpact/readme.md | 2 +- .../gsqlparser/demos/search/readme.md | 2 +- 5 files changed, 23 insertions(+), 575 deletions(-) delete mode 100644 docs/maintenance-notes.md diff --git a/.github/scripts/check-stale-docs.sh b/.github/scripts/check-stale-docs.sh index c6b9a0a1..e31cf7ad 100755 --- a/.github/scripts/check-stale-docs.sh +++ b/.github/scripts/check-stale-docs.sh @@ -23,9 +23,15 @@ set -euo pipefail cd "$(dirname "$0")/../.." # Documents whose subject IS the removal. They have to name what was removed. +# +# docs/maintenance-notes.md was the other entry until 2026-08-25, when it was +# moved out of this repository -- it was maintainer history, not evaluation +# material, and it sat in the README nav bar of a repo people clone to decide +# whether to buy the parser. It also showed the cost of being on this list: it +# was exempt from this check and had quietly drifted (a stale test count, and a +# directory renamed out from under it). Keep this list short. ALLOW=( "README.md" - "docs/maintenance-notes.md" ) DEAD=( diff --git a/README.md b/README.md index dd64ee59..9a0cdc16 100644 --- a/README.md +++ b/README.md @@ -23,8 +23,7 @@ mvn package -DskipTests [Project layout](#project-layout) · [Standalone lineage tool](#the-standalone-lineage-tool) · [Windows .bat scripts](#the-windows-bat-scripts) · [CI](#what-ci-checks) · -[Contributing](#contributing) · -[Maintenance notes](docs/maintenance-notes.md) +[Contributing](#contributing) ## What the library does @@ -112,7 +111,9 @@ built-in Oracle query when no file is supplied. > > Instructions written before 2026-07-27 say things like > `demos.checksyntax.checksyntax`. Those need the `gudusoft.gsqlparser.` prefix -> now; see [One package root](docs/maintenance-notes.md#one-package-root). +> now. Before that rename this tree carried four package roots and 263 files +> whose path contradicted their own `package` line; there is one root now, and +> the path under `src/main/java/` *is* the package. If you have older notes telling you to add `-Dexec.classpathScope=compile`, you no longer need it. It worked around `system`-scope dependencies that are gone. @@ -337,7 +338,6 @@ licensed-only/Connector/ JDBC modules, licensed parser lib-repo/ in-project Maven repository setenv/ + per-demo *.bat the Windows route .github/scripts/ CI checks, all runnable locally -docs/maintenance-notes.md why things are the way they are ``` Four rules worth knowing before you add anything: @@ -388,8 +388,13 @@ Check it yourself with `.github/scripts/smoke-dlineage-jar.sh`, which is what CI runs. > This replaced `pom_dlineage.xml`, a second POM that broke four separate times -> without a build ever going red. If you are tempted to add a second POM, read -> [that story first](docs/maintenance-notes.md#pom_dlineagexml-four-breakages-and-a-merge). +> without a build ever going red: it pinned a parser jar predating the APIs its +> own source used; it inherited from a private parent published nowhere, so +> nobody outside Gudu could read it at all; its documented run command needed a +> directory only the Windows route creates; and it lacked a JAXB dependency the +> root POM already had, so it compiled and then died on every JDK past 8. Each +> was invisible because CI only ever *built* it. Think hard before adding a +> second POM. ## The Windows .bat scripts @@ -489,9 +494,9 @@ drive with arguments. ## Contributing -- **Wire new things into CI in the same commit.** Every bug in - [the maintenance notes](docs/maintenance-notes.md) shares one cause: something - nothing ran. Building is not running — assert on output, not exit status. +- **Wire new things into CI in the same commit.** Nearly every bug this + repository has had shares one cause: something nothing ran. Building is not + running — assert on output, not exit status. - **One package root**, path equals package. Don't add a second. - **Don't commit jars**; add dependencies by coordinate. - **Don't add a live-JDBC path to a demo** under `src/main/java`. It can't run @@ -509,5 +514,3 @@ compile against the parser `pom.xml` pins. - Java documentation: - Quick start: - .NET demos: -- [Maintenance notes](docs/maintenance-notes.md) — why this repository is shaped - the way it is, and the mistakes that are easy to make again diff --git a/docs/maintenance-notes.md b/docs/maintenance-notes.md deleted file mode 100644 index b66f87b1..00000000 --- a/docs/maintenance-notes.md +++ /dev/null @@ -1,561 +0,0 @@ -# Maintenance notes - -Why this repository is shaped the way it is. - -Most of what follows documents a concentrated cleanup between **2026-07-26 and -2026-07-28**, when this tree was merged with the library-side demo tree and then -made to build, test and run for someone outside Gudu. Almost every item here is -a bug that was invisible until something ran it, which is the recurring lesson: - -> **Anything CI does not run is load-bearing only by luck.** Building is not -> running. If you add a script, a POM, or a workflow path, wire it into CI in -> the same commit — and make the check assert on output, not on exit status. - -None of this is needed to *use* the demos; start with the -[README](../README.md). It is here for whoever maintains the repository next, -and because several of these mistakes are easy to make again. - -## Contents - -- [The 2026-07 cleanup at a glance](#the-2026-07-cleanup-at-a-glance) -- [One package root](#one-package-root) -- [The library-side test tree](#the-library-side-test-tree) -- [The "3 known failures" that were not](#the-3-known-failures-that-were-not) -- [samples/ and src/main/resources/](#samples-and-srcmainresources) -- [The vendored expression library](#the-vendored-expression-library) -- [Demos that needed a live database](#demos-that-needed-a-live-database) -- [dbConnect was removed](#dbconnect-was-removed) -- [Dependencies and security advisories](#dependencies-and-security-advisories) -- [pom_dlineage.xml: four breakages and a merge](#pom_dlineagexml-four-breakages-and-a-merge) -- [What putting the .bat scripts under CI found](#what-putting-the-bat-scripts-under-ci-found) -- [Two demos that crashed instead of explaining themselves](#two-demos-that-crashed-instead-of-explaining-themselves) -- [Change log](#change-log) -- [Appendix: organising multiple demos as Maven modules](#appendix-organising-multiple-demos-as-maven-modules) - -## The 2026-07 cleanup at a glance - -| | before | after | -|---|---|---| -| `.java` under `src/main/java` | 273 | 190 | -| files whose path contradicts their `package` | 263 | **0** | -| package roots | 4 | **1** | -| open Dependabot advisories | 28 | **0** | -| committed `.jar` files | 25 | **1** (in `lib-repo/`, as a resolvable artifact) | -| POMs building this tree | 2 | **1** | -| tests | 123 run, **3 failing** and documented as expected | **144 run, 0 failing, 0 skipped** | -| workflows | 1, build-only | 2, and they *run* what they build | - -## One package root - -`src/main/java` had grown four competing package roots — `demos.*`, -`gudusoft.*`, `gsp.demos.dlineage` and a stray `com.gudusoft.gsqlparser.demo` -(`App.java`, sitting loose at the very top of the source tree). Java's one hard -rule is that a file's directory path mirrors its package, and **263 of 273 -files broke it**. Two packages were even split across both roots, so their -halves shared package-private access from unrelated directories. - -There is now exactly one root, `gudusoft/`, and **path equals package for all -190 files**: - -| | session start | now | -|---|---|---| -| `.java` under `src/main/java` | 273 | 190 | -| files whose path contradicts their `package` | 263 | **0** | -| package roots | 4 | **1** | -| packages split across roots | 2 | **0** | -| JDBC driver `.jar` files inside `src/main/java` | 10 | **0** | -| Maven projects nested in the compile root | 1 | **0** | - -Everything moved to `gudusoft.gsqlparser.demos.`, which is the naming the -repository had been half-migrated toward for years. **This changes every -`-Dexec.mainClass` value**, so older instructions that say -`demos.checksyntax.checksyntax` need `gudusoft.gsqlparser.demos.checksyntax.checksyntax`. -The per-demo `readme.md` files, the `.bat` scripts and the CI workflow were all -updated with it. - -A side effect worth knowing: the `.bat` scripts now work again. They had been -stale twice over — compiling `src\main\java\demos\\` and `cd`-ing up five -levels, both correct only before the demos moved under -`gudusoft/gsqlparser/demos/`. Since the rename had to touch them anyway, their -paths and directory depths were corrected to match where each script actually -sits (7 levels for most, 8 for the nested ones). - - -## The library-side test tree - -`src/test/java/gudusoft/gsqlparser/` used to hold 222 files, but 204 of them -were a stale copy of the library's own test tree, carried in by the 2026/7/26 -merge. They have been returned to `gsp_java_core`, which is where they belong -and where they actually work: - -- They test the parser, not any demo here — none referenced demo code. -- They resolve fixtures through `gspCommon.BASE_SQL_DIR`, which points at - `../gsp_java_core/gsp_sqlfiles/TestCases/`. That path does not exist relative - to this repository, so their data was never found here. -- Most are named `testXxx.java` (lowercase), which Surefire's default include - patterns (`Test*.java`, `*Test.java`, `*TestCase.java`) do not match. Only 3 - of the 204 were ever executed by `mvn test` here. `gsp_java_core`'s POM adds - `**/test*.java`, so they run there. - -169 were byte-identical to, or an older revision of, a file already in -`gsp_java_core` and were simply deleted; 35 existed only here and were moved -over. What remains are the 18 files that genuinely belong to this repository: -13 tests that import demo classes, `commonTest/SqlFileList.java` and -`commonTest/gspCommon.java` which those tests use, and three files kept back -because moving them would have required adding `org.jdom` and -`com.alibaba.fastjson` to the library's POM (`commonTest/testXmlXSD.java`, -`sqlenvTest/TJSONSQLEnv.java`, `sqlenvTest/testJSONEnv.java`). - -`src/test/java/demos/visitors/` — the last test package still sitting outside -the `gudusoft/gsqlparser/` tree — has since been folded into it as -`gudusoft/gsqlparser/visitorsTest/`, following the `Test` convention -the other directories already use (`joinConvertTest` covers the `joinConvert` -demo, `antiSQLInjectionTest` covers `antiSQLInjection`, and so on). All 20 test -files now live under one root. - - -## The "3 known failures" that were not - -Until 2026-07-28 the README said three tests were expected to fail. They were -not. - -> **Correction.** Until 2026-07-28 the README said three tests in -> `gudusoft.gsqlparser.demosTest.analyzespTest` failed because their expected -> output was written for an older parser and had since drifted, and that they -> were kept red on purpose as a drift signal. That was wrong, and it sent -> people looking at parser output for a bug that was never there. -> -> Those tests read stored procedures from the library's SQL corpus over a -> relative path, `gspCommon.BASE_SQL_DIR`. The path read `../gsp_java_core/`, -> one directory level short: it named a sibling of this checkout rather than -> the module inside `gsp_java`, so it resolved to nothing at all. `Analyze_SP` -> never found an input file, returned an empty string, and comparing that with -> the expected output failed. The expected strings match the current parser's -> output exactly, character for character, on all three. `testSample7` "passed" -> throughout only because it expects empty output, which is also what a missing -> file produces, so it was asserting nothing. -> -> The four scripts now live in `src/test/resources/sqlscripts/analyze_sp/`, -> copied byte for byte from the corpus, which removes the sibling-checkout -> requirement rather than just correcting it. See that directory's `readme.md`. - -## samples/ and src/main/resources/ - -`src/main/java` is a *source* root: Maven compiles what is in it and does not -put anything else on the classpath. 229 files in there were not `.java`, and -that turned out to be hiding a real bug. - -**`snowflake.js` was broken.** `SnowflakeSQLExtractor` loads it with -`getResourceAsStream("snowflake.js")`, but the file sat in `src/main/java`, so -Maven never copied it to `target/classes` — the call returned `null` and the -demo died on an NPE. It is the only classpath resource in the repository, and -it now lives at -`src/main/resources/gudusoft/gsqlparser/demos/snowflake/sqlextract/snowflake.js`, -mirroring its package so the same lookup resolves. (The demo also uses Nashorn, -removed from the JDK in 15, so it still needs Java 8–14 to run.) - -**Sample SQL moved to `samples/`.** The 85 `.sql` files were input data passed -on the command line, never read from the classpath, so `src/main/resources` is -the wrong home for them too — they belong outside `src/` entirely: - -``` -samples/dlineageBasic/{oracle,mysql,mssql}/… 81 files -samples/tracedatalineage/ 2 -samples/dlineage/ 1 -samples/callgraph/ 1 -``` - -**Build output deleted.** `demos/dlineage/class/` was created and `rm -rf`'d by -`buildJar.sh` on every run, and `data-lineage-result.xml` was a generated -lineage report nothing referenced. Both are now in `.gitignore`. - -`buildJar.sh`, `buildJar.bat` and `demos/dlineage/MANIFEST.MF` went with them on -2026-07-28. All three existed to hand-assemble the same `data_flow_analyzer.jar` -that `maven-shade-plugin` now produces from the normal build, and all three -named files that do not exist: `buildJar.sh` copied -`../../../../../lib/gudusoft.gsqlparser.jar`, which was never in `lib/` under -that name, and the manifest's `Class-Path` listed -`lib/gudusoft.gsqlparser.jar` and `lib/sqlflow-exporter.jar`, neither of which -survived. Nothing in CI ran any of them, which is why nobody noticed. - -What deliberately stays next to its demo: each demo's `readme.md`, its -`compile_*.bat` / `run_*.bat` (which `cd` relative to their own location, so -they cannot move), and a handful of per-demo assets (`tree-view.xsl` and -`tree-view.css`, referenced by relative href from generated XML; -`sqlflow-settings.png`; the dlineage PDF). Co-locating documentation and -per-demo tooling with the demo is the point of this repository's layout. - - -## The vendored expression library - -`src/main/java/gudusoft/gsqlparser/demos/antiSQLInjection/` used to carry 365 -`.java` files of a third-party expression evaluator under `org/boris/expr/`. -That was **57% of every source file in this repository**, and it duplicated -`lib/expr4j.jar`, which was already a declared dependency: the jar holds the -same 365 top-level classes and nothing else, so the library shipped twice and -javac quietly compiled the sources while the jar sat unused. - -The sources are gone; the jar supplies `org.boris.expr`. Only one file ever -imported it (`GEval.java`), the `antiSQLInjection` tests cover the path, and -they pass against the jar. One consequence worth knowing: **`mvn package` no -longer puts `org/boris/expr/**` inside the project jar.** Nothing here consumes -that jar as a library, so this only matters if you start doing so. (The shaded -`-dlineage` jar does contain it, along with every other runtime dependency — -that is what an uber jar is.) - -The dependency's coordinates were also wrong: it was declared as -`tk.pratanumandal:expr4j`, a different library entirely, which would have -pointed SBOM and vulnerability tooling at the wrong project. It now names what -is actually on disk, with a checksum recorded alongside it, since the jar -carries no version metadata of its own. - -It is also the one dependency here with no coordinate in any public repository — -searched and confirmed — so it is served from `lib-repo/`, a file-based Maven -repository inside this checkout, laid out exactly like `~/.m2/repository`. See -[`lib-repo/readme.md`](../lib-repo/readme.md). It was a `system` jar -under `lib/` until 2026-07-28, which is worse than it sounds: `system` scope is -deprecated, invisible to Dependabot, **and skipped by `maven-shade-plugin`, -`maven-assembly-plugin` and `dependency:copy-dependencies` alike**, so it could -never appear in a packaged artifact or a generated classpath. That last part is -most of why the dlineage demo needed a hand-assembled `java -cp` -([#46](https://github.com/sqlparser/gsp_demo_java/issues/46)). - -If Gudu ever uploads it to `https://www.sqlparser.com/maven/`, delete `lib-repo/` -and declare it normally — publishing to a repository you control is the better -answer, and this project already resolves from that one. - - -## Demos that needed a live database - -Two demos used to be excluded because they read metadata straight out of a -running database over JDBC, and so needed a driver and a live server to do -anything. **Both are now in the build**, since 2026-07-28: they take that -metadata offline instead, which also let the two vendored jars behind it leave -`lib/`. - -- `gettablecolumns/runGetTableColumn.java` — dropped its `/h /P /u /p` flags; - supply a `TSQLEnv` in code or from JSON if you want column disambiguation -- `columninspect/ColumnInspect.java` — `/jdbc /u /p` became `/metadata `, - reading the same JSON the connection used to return. Every line of the - inspection already worked off that JSON string, so nothing else changed. - `samples/columninspect/` has a runnable metadata file and script - -## dbConnect was removed - -`dbConnect` was a **complete Maven project nested inside `src/main/java`** — its -own `pom.xml`, its own `src/main/java`, and 10 JDBC driver jars, all sitting in -the compile root of the project that contains it. It was moved out to -`connector/dbConnect/` and then deleted outright, because it had stopped being -revivable: - -- It is written against `gudusoft.gsqlparser.sqlenv.util`, which the public - parser no longer ships, so it needs an API migration rather than a rebuild. -- `pom.xml` excluded it from the build, and nothing else built it either. -- Its own parser dependency was pinned to `latest`, which - Maven cannot resolve. -- Its 10 vendored JDBC drivers accounted for **16 of this repository's 28 - Dependabot advisories**, and none of them could be patched by upgrading a - declared version, because the jars were files in git rather than resolvable - dependencies. - -Deleting it removed all 16 advisories, 10 binaries and 83 Java files in one go. -It is in git history if it ever needs reviving, but reviving it means the API -migration, not a checkout. - -Moving it out first also resolved one of the two split packages: 16 of -`demos.sqlenv`'s files were dbConnect's and 1 (`runSQLEnv.java`) was not. - - -## Dependencies and security advisories - -GitHub reported 28 open Dependabot advisories against this repository. They came -from one root cause: dependencies declared with `system` and a -`` into `lib/`. **Dependabot cannot patch those** — a system-scope -dependency is a file on disk, not something Maven resolves, so there is no -version for a bot to bump. - -All 28 are now closed: - -| where | count | what was done | -|---|---|---| -| `pom.xml` — `junrar` | 4 | `0.7` system-scope → `7.5.10` from Maven Central, `test` scope | -| `pom.xml` — `jdom` | 1 | every version of `org.jdom:jdom` is affected; migrated to the successor artifact `org.jdom:jdom2:2.0.6.1`, whose classes are `org.jdom2.*` | -| `pom.xml` — `junit` | 1 | `4.12` → `4.13.2` | -| `connector/snowflakeConnector` | 6 | `snowflake-jdbc` `3.12.9` → `4.3.2`, resolved from Central instead of a jar you drop into `lib/` | -| `connector/dbConnect` | 16 | module deleted ([see below](#dbconnect-was-removed)) | - -Two things worth keeping in mind for next time: - -- **`junrar` and `jdom` were only ever used by tests**, so they are `test` scope - now. Both jars are gone from `lib/`. -- `fastjson` is pinned at `1.2.83` and is **not** flagged — that is the final - 1.x release, and it is where the 1.x deserialization advisories were fixed. - Leave it, or move to `fastjson2`; do not "upgrade" it within 1.x. - -**`lib/` is now gone entirely, and with it the last `system`-scope dependency.** -Three jars were left in it, believed to have no public coordinate. Two of them -did: - -| jar | outcome | -|---|---| -| `simple-xml-2.7.1.jar` | on Maven Central as `org.simpleframework:simple-xml:2.7.1`, and **byte-for-byte identical** to the committed copy (`sha256 7a43d2d5…f4e429f9`). Declared normally; jar deleted. | -| `fastjson-1.2.83.jar` | on Maven Central as `com.alibaba:fastjson:1.2.83`, likewise byte-identical (`sha256 641a4d65…5fe692631d`). Declared normally; jar deleted. | -| `expr4j.jar` | genuinely not on Central under any groupId. Moved to `lib-repo/` as a real artifact, `org.boris:expr:0.0.0-vendored`. | - -Since the two Central jars are the same bytes, that swap changed nothing at -runtime and gained three things: Dependabot can see and patch them, packaging -plugins stop skipping them, and their **transitive** dependencies now resolve — -`simple-xml` pulls in `stax` and `xpp3`, which under `system` scope were simply -absent, waiting to surface as a `NoClassDefFoundError` on some code path nobody -had exercised yet. - -The rule that follows: **add a dependency by coordinate, not by file.** Check -Maven Central first — two of the three "no public coordinate" jars here were on -it all along. If it truly has no coordinate, publishing it to Gudu's own Maven -repository beats committing it; `lib-repo/` is the fallback for when nobody can -upload one. - -`sqlflow-exporter.jar` and `sqlflow-library.jar` were dropped on 2026-07-28. -They supplied `gudusoft.dbadapter.TSQLDataSource`, which two demos used -to read table metadata out of a running database — so both demos needed a live -server and a JDBC driver, and `pom.xml` excluded them from the build for exactly -that reason. Both now take their metadata offline instead -(`runGetTableColumn` via `TSQLEnv`, `ColumnInspect` via a `/metadata` JSON file), -so both build and run like every other demo, and the jars had nothing left -holding them in. Two dead helpers went with them, `dlineage/SqlflowIngester.java` -and `dlineage/DataSourceProvider.java`: neither had a `main`, and the only -reference to either in the tree was a commented-out call in `DataFlowAnalyzer`. - -Eight more jars used to sit alongside them, 5.5 MB of the directory's 7.1 MB. -None was declared by any POM, imported by any source, or named by any `.bat` -script, so nothing resolved them — `lib/` was just where they had been dropped: - -| removed | why it was dead | -|---|---| -| `lib/jdbc/ojdbc-1.1.1.jar`, `lib/jdbc/sqljdbc4-4.0.jar` | the connector POMs resolve `${project.basedir}/lib/`, which is `connector//lib/` — never this directory. Download the driver into the module's own `lib/`, as its readme says. | -| `lib/proguard/proguard.jar`, `proguard.pro` | byte-identical to `gsp_java_core/proguard/`, and the `.pro` obfuscates `gudusoft.gsqlparser.jar` — the *library* artifact. Release tooling for the other repository. | -| `lib/jdk1.5/junit-4.5.jar`, `junit.jar` | JUnit comes from Maven at `4.13.2`. | -| `lib/commons-logging-1.1.3.jar` | referenced only by `fastjson`'s `support/spring/*` adapters; there is no Spring here. | -| `lib/jarLoader.jar` | Eclipse's jar-in-jar export loader, used by the IDE wizard, not by any build. | - -Vendored driver jars in particular should not come back. They are invisible to -Dependabot for the same reason the system-scope entries above were, so they age -in place with no bot to flag them. - -The one exception, `lib-repo/`, is deliberately not a place to drop jars: it is a -Maven repository with a strict layout, every artifact needs a hand-written -`.pom`, and its readme says to check Central and Gudu's own repository first. -The friction is the point. - - -## pom_dlineage.xml: four breakages and a merge - -Until 2026-07-28 this was a second POM, `pom_dlineage.xml`, built with -`mvn -f pom_dlineage.xml package`. It broke four times, and **every one of them -was invisible to a green build**, because until the very end nothing in CI ran -the thing — and for most of that time nothing even built it. - -1. **It pinned a parser jar from 2019.** It declared - `lib/gsqlparser-3.1.1.0.jar` on `system` scope while `DataFlowAnalyzer` had - moved on to `getOption().setTraceTablePosition(...)` and - `ProcessUtility.generateColumnLevelLineageCsvSimple(...)`, neither of which - that jar has. -2. **It kept a `` published nowhere**, `gudusoft:gsp_java:1.0-SNAPSHOT`, - the private library reactor. Anyone without that POM in their local - repository got `Non-resolvable parent POM` before Maven read a line of the - file. It looked fine to everyone at Gudu, who all have the library checked - out. The root `pom.xml` had been cut loose from that parent years earlier; - this one was missed. -3. **[#46](https://github.com/sqlparser/gsp_demo_java/issues/46) — the - documented run command could not work.** It put `external_lib/*` on the - classpath, a directory only the Windows `.bat` route ever creates, so anyone - following the Maven instructions in order hit `NoClassDefFoundError` on the - first command in the section. The jar Maven had actually downloaded sat in - `~/.m2`, under a filename containing a version number that changes with every - release, so it could not be guessed either. -4. **[#47](https://github.com/sqlparser/gsp_demo_java/issues/47) — it was - missing the JAXB dependency `pom.xml` already had.** JAXB left the JDK after - Java 8; `pom.xml` had carried `jakarta.xml.bind-api` and `jaxb-runtime` for - ages, and `pom_dlineage.xml` had neither. Since `pom.xml` *excluded* - `DataFlowAnalyzer` from its own build, this second POM was the only thing - that ever compiled the class, so no other build path could catch it. - -Merging it into `pom.xml` removes the category rather than the four instances. -There is no second POM to drift, no second parser version to forget, no second -dependency list to fall behind — and no -[#39](https://github.com/sqlparser/gsp_demo_java/issues/39) either, the bug -where the second POM's incremental-compile cleanup deleted the root build's -`target/classes` because both POMs shared `${project.basedir}`. One POM cannot -collide with itself, so `target-dlineage/` is gone too. - -The exclusion that made all this necessary was circular: `DataFlowAnalyzer` was -excluded from the root build *because* a separate POM compiled it, and that POM -existed to produce a standalone jar. `maven-shade-plugin` produces the -standalone jar from the one build, so nothing is excluded from `pom.xml` any -more. - -**What actually keeps it working now is that CI runs it.** -`.github/scripts/smoke-dlineage-jar.sh` executes the jar on -`samples/dlineage/demo.sql` in both `build.yml` and `nightly.yml`, on JDK 8 and -21, and asserts on the *output*: non-empty, parses, and contains lineage -relationships. That last part is not pedantry — in #47 the process created -`lineage.json`, died before writing a byte into it, and still exited `0`. A file -that exists and an exit code of zero were both, at that moment, lies. It checks -XML as well as JSON, because those are separate code paths and the JAXB one is -the one that broke. - - -## What putting the .bat scripts under CI found - -They had been stale for years — compiling `src\main\java\demos\\` and -`cd`-ing up five levels, both correct only before the demos moved under -`gudusoft/gsqlparser/demos/`. Nothing noticed, because nothing ran them. Putting -them under CI turned up five faults, none of which anything else would have -caught: - -| fault | scripts | -|-------|---------| -| Still compiled `src\main\java\demos\*.java`, a directory the package rename deleted | 2 | -| Doubled path `analyzesp\sybase\sybase\` | 1 | -| Named a package that no longer existed after `ColumnImpact` moved | 2 | -| Passed only their own folder to `javac`, so cross-demo imports failed to resolve — fixed with `-sourcepath src\main\java` | 6 | -| No `-encoding`, so Windows `javac` used the platform codepage against UTF-8 sources | 39 | - -The last one is the reason a real Windows runner was worth the trouble. `javac` -there defaults to `Cp1252`, and two demos failed with `unmappable character for -encoding Cp1252`. `pom.xml` has always declared `project.build.sourceEncoding` -as UTF-8, so Maven was never affected, and on Linux the default is UTF-8 anyway -— simulating all 39 scripts there passed cleanly. It only reproduces with the -encoding forced. - -Also deleted along the way: `TGetTableColumn_notUsed.java`, which redefined -three classes from the `TGetTableColumn.java` beside it and so broke any -wildcard compile of that folder, which is exactly how -`compile_gettablecolumns.bat` compiles it. - - -## Two demos that crashed instead of explaining themselves - -Found on 2026-07-28 while documenting them, both invisible to CI at the time. - -**`scriptwriter`** went straight to `sqlstatements.get(0)` without checking what -`parse()` returned. Its built-in query is a deliberately hard ~49 KB Oracle -statement, and **the trial parser refuses input over 10,000 bytes** — so with -the published jar the parse failed, the statement list was empty, and the demo -died with `IndexOutOfBoundsException: Index 0 out of bounds for length 0`. That -message names neither the real cause (a licensing limit) nor the fix. It now -checks the return code, prints the parser's own error, says so explicitly when -the input is over the limit, and takes an optional `` so it can -actually demonstrate something with the trial jar. - -This is also why the README no longer claims the trial build is "enough for -every demo here". It is enough for every demo but this one. - -**`events/ProcessSQLStatement`** had this as its input: - -```java -String sqlfile = "C:\\Users\\DELL\\Downloads\\20240311110800487_mssql_sql\\data.sql"; -``` - -A hardcoded absolute path into one developer's Downloads folder, so the demo -could not run anywhere else. The file was missing, `parse()` failed, and the -error branch then called `getErrormessage().substring(0, 1000)` unconditionally -on a message shorter than 1000 characters — throwing -`StringIndexOutOfBoundsException` from the code whose job was to report the -error. It now takes the file as an argument, prints a usage line without one, -and bounds the substring. - -**The general check.** `run-all-demos.sh` matched only linkage failures -(`NoClassDefFoundError` and friends), so both of these passed. It now also fails -any demo that throws `Exception in thread "main"` when run with **no -arguments**: with nothing to work on, a demo should print its usage line, not a -stack trace. All 77 pass it. - -A caution for whoever checks this next: the first survey run for this said "0 of -77 throw", and it was wrong — it used a relative `-cp target/classes` from -another working directory, so every demo failed to load and matched no pattern. -`run-all-demos.sh` builds an absolute classpath for exactly this reason. - -## Change log - -Newest first. - -- **[2026/8/12]** A pre-commit hook (`.githooks/pre-commit`, opt in with - `git config core.hooksPath .githooks`) and a red-master tracker - (`.github/workflows/red-master.yml`). Both come out of one incident and - neither is a new check: `afb6f3a` bumped `${gsp.core.version}` by hand and - missed the three connector POMs, the push build went red **within the same - minute**, and master stayed broken for 21 hours anyway because nobody opens - the Actions tab of a repository that is usually green. The hook moves the - existing check earlier — it reads the *index*, since staging `pom.xml` while - leaving the connector edits unstaged is the exact drift in question — and the - tracker turns a failure into an assigned issue that closes itself when both - watched workflows are green again. The hook fails open by design (no python, - no POM staged, no readable index and it stands aside), so - `test-pre-commit-hook.sh` drives it through four cases in a throwaway clone - and `build.yml` runs that; a hook that quietly stopped working would - otherwise be invisible until the next bad bump. - -- **[2026/7/28]** `pom_dlineage.xml` merged into `pom.xml`; the standalone - lineage tool is now a `maven-shade-plugin` uber jar produced by the normal - build, and CI *runs* it on JDK 8 and 21 rather than only building it. Closed - [#46](https://github.com/sqlparser/gsp_demo_java/issues/46) and - [#47](https://github.com/sqlparser/gsp_demo_java/issues/47). See - [pom_dlineage.xml: four breakages and a merge](#pom_dlineagexml-four-breakages-and-a-merge). -- **[2026/7/28]** `lib/` deleted. `simple-xml` and `fastjson` became ordinary - Maven Central dependencies (byte-identical to the committed copies); - `org.boris:expr`, the only one with no public coordinate, moved to - `lib-repo/`. Last `system` dependency gone, so - `-Dexec.classpathScope=compile` is no longer needed by any demo. -- **[2026/7/28]** `jaxb-runtime` `2.3.3` → `2.3.9`: 2.3.3 calls - `sun.misc.Unsafe.defineClass`, removed in JDK 11, so XML lineage output - silently produced a 0-byte file on every supported JDK except 8. -- **[2026/7/28]** `scriptwriter` and `events/ProcessSQLStatement` fixed: both - crashed with an unhandled exception rather than reporting why. - `run-all-demos.sh` now fails any demo that throws when run with no arguments. -- **[2026/7/28]** The "3 known failures" were a wrong fixture path, not parser - drift. Fixtures checked in; the suite is 144 passing, 0 skipped, and - self-contained. `connector/dbConnect` deleted, closing 16 advisories. -- **[2026/7/27]** One package root: all 190 sources are now - `gudusoft.gsqlparser.demos.`, and path equals package everywhere. **This - changed every `-Dexec.mainClass` value.** `snowflake.js` moved to - `src/main/resources/`, where `getResourceAsStream` can actually find it. -- **[2026/7/26]** Merged with the library-side demo tree, which had diverged. - That tree is authoritative, so its version wins wherever the two differed (61 - files). Nothing was dropped: the 282 files that existed only here — 143 under - `demos/dbConnect`, 130 tests, and a few data files — were kept. Test count - went from 123 to 153. Most demos moved to `src/main/java/gudusoft/gsqlparser/demos/`. -- **[2026/7/26]** The build no longer inherits from the private `gudusoft:gsp_java` - parent POM, which was why nobody outside Gudu could build this repository. - It now resolves `com.gudusoft:gsqlparser` from - and builds standalone. -- **[2024/9/17]** Test packages moved from `src/test/java/` to - `src/test/java/gudusoft/gsqlparser/`, so unit tests now live under package - names like `gudusoft.gsqlparser.xxxTest`. The demos followed later, in the - 2026/7/26 merge above, and that move is now complete — see - [One package root](#one-package-root). - -## Appendix: organising multiple demos as Maven modules - -Design note carried over from the library-side tree. This describes a structure -the repository does **not** currently use; it is kept as guidance. - -When a project contains multiple demos that need to be built separately, the -usual approach is a Maven multi-module project: one parent `pom.xml` with -`pom` and a `` section, plus a child `pom.xml` -per demo declaring its own dependencies and inheriting shared versions from the -parent's ``. - -``` -gsp_demo_java/ -├── pom.xml <-- parent POM, manages the modules -├── dlineage-demo/ -│ ├── pom.xml -│ └── src/main/java/demos/dlineage/DataFlowAnalyzer.java -└── another-demo/ - ├── pom.xml - └── src/main/java/demos/another/AnotherDemo.java -``` - -Build everything with `mvn clean package` from the root, or a single module with -`mvn -pl dlineage-demo clean package`. diff --git a/src/main/java/gudusoft/gsqlparser/demos/antiSQLInjection/columnImpact/readme.md b/src/main/java/gudusoft/gsqlparser/demos/antiSQLInjection/columnImpact/readme.md index 894d6849..deec7dde 100644 --- a/src/main/java/gudusoft/gsqlparser/demos/antiSQLInjection/columnImpact/readme.md +++ b/src/main/java/gudusoft/gsqlparser/demos/antiSQLInjection/columnImpact/readme.md @@ -100,7 +100,7 @@ Please try your own SQL with [this online live demo](http://www.sqlparser.com/li ## Related projects If you need to handle more complicated SQL and get all dataflow information of table and columns, -please check this demo [Dataflow Analyzer](../dlineage#dataflowanalyzer). +please check this demo [Dataflow Analyzer](../../dlineage/readme.md). ## Changes - [2012-01-11, first version](https://github.com/sqlparser/wings/issues/1) \ No newline at end of file diff --git a/src/main/java/gudusoft/gsqlparser/demos/search/readme.md b/src/main/java/gudusoft/gsqlparser/demos/search/readme.md index b6d3b353..086be81f 100644 --- a/src/main/java/gudusoft/gsqlparser/demos/search/readme.md +++ b/src/main/java/gudusoft/gsqlparser/demos/search/readme.md @@ -2,7 +2,7 @@ Search the specified parse tree node name in sql files under the directory recursively . The file name will be printed out if it include the specified parse tree node name. -Please check [toXML demo](./visitors) to find out more information on how to use the visitor pattern introduced in this library. +Please check [toXML demo](../visitors/readme.md) to find out more information on how to use the visitor pattern introduced in this library. ## Usage `java searchClause parse_tree_node_name directory`