Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .githooks/pre-commit
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
#
# It checks the *index*, not the working tree, because the index is what you
# are about to commit. Staging pom.xml while leaving the matching
# connector/*/pom.xml edits unstaged is precisely the drift being guarded
# licensed-only/*/pom.xml edits unstaged is precisely the drift being guarded
# against, and a working-tree check would call that clean.

set -uo pipefail
Expand Down Expand Up @@ -77,7 +77,7 @@ pre-commit: the parser version disagrees across the POMs you are committing.
git add -u && git commit then commit again

Never edit a version by hand: it is written in pom.xml and in each of the three
connector/*/pom.xml, which are separate builds with no parent to inherit it.
licensed-only/*/pom.xml, which are separate builds with no parent to inherit it.
In practice you should not be editing one at all -- the nightly tests the newest
release and opens a pre-verified bump PR.

Expand Down
65 changes: 65 additions & 0 deletions .github/scripts/check-licensed-only-guard.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
#!/usr/bin/env bash
#
# Prove the licensed-only modules refuse to build with an explanation.
#
# licensed-only/{oracle,snowflake,sqlServer}Connector need a LICENSED parser:
# the public trial artifact this repository resolves does not ship
# gudusoft.gsqlparser.sqlenv.T*SQLDataSource. Before 2026-08-24 they simply
# failed -- javac's "cannot find symbol" for snowflake, and for the other two a
# dependency-resolution error about a JDBC jar nobody had, thrown before any
# plugin could speak. A first-time evaluator read that as "this library does not
# compile" and said so in an evaluation report.
#
# Now each POM stops at validate with a message naming the licensing boundary
# and pointing at the trial-friendly alternative. That message is the whole
# feature, so this asserts on the message, not on the exit status: a build that
# fails for the old confusing reason also exits non-zero.
#
# Usage:
# check-licensed-only-guard.sh

set -euo pipefail

cd "$(dirname "${BASH_SOURCE[0]}")/../.."

MODULES=(oracleConnector snowflakeConnector sqlServerConnector)
NEEDLE="needs a LICENSED General SQL Parser"

failed=0

for m in "${MODULES[@]}"; do
pom="licensed-only/$m/pom.xml"
if [ ! -f "$pom" ]; then
echo " FAIL $pom is missing"
failed=1
continue
fi

out=$(mvn -B -f "$pom" validate 2>&1 || true)

if ! grep -qF "$NEEDLE" <<<"$out"; then
echo " FAIL $m built or failed without explaining the licence boundary"
echo "$out" | tail -20 | sed 's/^/ /'
failed=1
continue
fi

# It must be the guard that stopped it, not something incidental that
# happened to print the same words.
if ! grep -q "BUILD FAILURE" <<<"$out"; then
echo " FAIL $m printed the message but the build succeeded"
failed=1
continue
fi

echo " ok $m stops at validate and says why"
done

echo

if [ "$failed" -ne 0 ]; then
echo "::error::a licensed-only module no longer refuses the trial build with an explanation"
exit 1
fi

echo "ok: all ${#MODULES[@]} licensed-only modules refuse the trial build with an explanation"
11 changes: 6 additions & 5 deletions .github/scripts/set-parser-version.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,10 @@
# it down.
#
# The version lives in four files. The root build declares it as a
# ${gsp.core.version} property, and the three connector/ modules hardcode the
# version in their gsqlparser dependency, because they are separate builds with
# no parent to inherit a property from. Nothing made them agree, so they could
# ${gsp.core.version} property, and the three licensed-only/ connector modules
# hardcode the version in their gsqlparser dependency, because they are separate
# builds with no parent to inherit a property from. (They lived under
# connector/ until 2026-08-24; see licensed-only/README.md for why they moved.) Nothing made them agree, so they could
# drift apart silently -- and a bump meant four hand edits, which is most of why
# bumping felt expensive.
#
Expand Down Expand Up @@ -61,8 +62,8 @@ TARGETS = [
]
for mod in ("oracleConnector", "snowflakeConnector", "sqlServerConnector"):
TARGETS.append((
"connector/%s/pom.xml" % mod,
"connector/%s dependency" % mod,
"licensed-only/%s/pom.xml" % mod,
"licensed-only/%s dependency" % mod,
# Anchor on the gsqlparser dependency so we never touch the JDBC
# driver's <version> sitting a few lines below it.
r"(?s)(<artifactId>gsqlparser</artifactId>\s*<version>)([^<]+)(</version>)",
Expand Down
39 changes: 39 additions & 0 deletions .github/scripts/smoke-dlineage-jar.sh
Original file line number Diff line number Diff line change
Expand Up @@ -96,4 +96,43 @@ if not rel:
print("ok: XML output, %d relationships" % len(rel))
PY

# --- a whole schema, under the trial cap -----------------------------------
# samples/dlineage/demo.sql is 366 bytes. Every real schema dump under
# samples/dlineageBasic/ is over the trial parser's 10,000-byte limit, so a
# visitor who tried "lineage on a real schema" got a licence error and no
# lineage. hr_mini exists to be the one that fits, which makes its size the
# thing worth guarding: grow it past the cap and the demo silently stops
# working for every trial user, while the tool goes on exiting 0.
SCHEMA="samples/dlineageBasic/oracle/hr_mini/hr_mini.sql"
CAP=10000

[ -f "$SCHEMA" ] || fail "$SCHEMA is missing"

bytes=$(wc -c <"$SCHEMA")
if [ "$bytes" -ge "$CAP" ]; then
fail "$SCHEMA is $bytes bytes, at or over the trial parser's $CAP-byte limit; it exists precisely so schema-scale lineage is runnable on the trial jar. Shrink it, or put the addition in one of the licensed-only dumps."
fi
echo "ok: $SCHEMA is $bytes bytes, under the $CAP-byte trial limit"

java -jar "$JAR" /f "$SCHEMA" /t oracle /o "$OUT/schema.json" /json

[ -s "$OUT/schema.json" ] || fail "schema.json is empty or missing"

python3 - "$OUT/schema.json" <<'PY'
import json, sys
d = json.load(open(sys.argv[1]))
rel = d.get("relationships") or []
if not rel:
sys.exit("hr_mini parsed but produced no relationships")
# A licence rejection is not an exception: the tool reports it as an error
# inside an otherwise well-formed document and exits 0, which reads as
# "no lineage found" unless something goes looking for it.
if "trial version can only process" in json.dumps(d):
sys.exit("hr_mini hit the trial size limit; it is no longer trial-evaluable")
if len(rel) < 50:
sys.exit("hr_mini produced only %d relationships; it produced 120 when it "
"was added, so the schema or the analyzer has regressed" % len(rel))
print("ok: schema-scale lineage, %d relationships from hr_mini" % len(rel))
PY

echo "ok: the standalone dlineage jar runs and produces lineage in both formats"
11 changes: 11 additions & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,17 @@ jobs:
echo
.github/scripts/check-stale-docs.sh

# The three licensed-only/ modules cannot compile against the public
# trial parser, and used to say so only in javac's words -- "cannot find
# symbol", or worse, a dependency-resolution error about a JDBC jar
# nobody had. An evaluator read that as a broken library. Each POM now
# stops at validate with a message naming the licensing boundary, and the
# message is the feature, so this asserts on the message: failing for the
# old confusing reason also exits non-zero.
- name: Licensed-only modules refuse the trial build, with a reason
if: matrix.java == '21'
run: .github/scripts/check-licensed-only-guard.sh

- name: Build
run: mvn -B package -DskipTests

Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -240,7 +240,7 @@ jobs:
--title "Bump the parser from $pinned to $NEW" \
--body "\`$NEW\` is the newest release on sqlparser.com. Tonight's nightly ran it through the same checks the pinned \`$pinned\` gets, and it passed all of them: the full test suite with no failures, every demo starting, and every case in \`.github/scripts/demo-cases.tsv\` producing its expected output.

This touches the four places the version is written -- \`pom.xml\` and the three \`connector/*/pom.xml\` -- via \`.github/scripts/set-parser-version.sh\`, so they cannot drift apart.
This touches the four places the version is written -- \`pom.xml\` and the three \`licensed-only/*/pom.xml\` -- via \`.github/scripts/set-parser-version.sh\`, so they cannot drift apart.

Merge to move the demos to \`$NEW\`. Close to stay on \`$pinned\`; the nightly will go on testing \`$NEW\` and will not reopen this until a newer release appears.

Expand Down
42 changes: 31 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,14 +146,23 @@ any individual demo.

`samples/` holds sample `.sql` files to feed them.

### Demos that connect to a database
### Demos that connect to a database — licensed parser only

`connector/{oracleConnector,snowflakeConnector,sqlServerConnector}/` are
`licensed-only/{oracleConnector,snowflakeConnector,sqlServerConnector}/` are
separate, independently built Maven modules showing JDBC-connected metadata
extraction. They are **not** part of `mvn package` or `mvn test` at the root;
build each on its own. Each module's `lib/` holds only a readme — you download
the JDBC driver yourself, and the version in that module's `pom.xml` tells you
which.
extraction. **They cannot be built with the trial parser**, which does not ship
`gudusoft.gsqlparser.sqlenv.T*SQLDataSource`; each stops at `validate` with a
message saying so, and `-Plicensed` turns that guard off once you have a
licensed parser. They are not part of `mvn package` or `mvn test` at the root.
See [`licensed-only/README.md`](licensed-only/README.md).

They sat in `connector/` until 2026-08-24, where they read as part of the
ordinary demo set: a first-time evaluation started there, met `cannot find
symbol`, and concluded the library did not compile.

**On the trial parser, `columninspect` is the thing to run instead.** It does
the same metadata-aware column resolution from a JSON catalog export rather
than a live connection, and `samples/columninspect/` has a runnable pair.

## Rewriting SQL through the parse tree

Expand Down Expand Up @@ -217,9 +226,19 @@ version (e.g. `4.1.9`) does not necessarily match the one in the release notes.

> **The trial build refuses input larger than 10,000 bytes**, reporting
> `trial version can only process query with size of at most 10000 bytes`.
> The limit is on a single parse, not on total throughput.
>
> Every demo here works within that except `scriptwriter`, whose built-in query
> is ~49 KB on purpose — give it your own smaller file, or use a licensed
> parser. The limit is on a single parse, not on total throughput.
> parser. **16 of the 89 `.sql` files under `samples/` are also over the
> limit**, all of them vendor schema dumps under `samples/dlineageBasic/`
> (10,378 to 99,139 bytes). Those are for licensed evaluation; the rejection
> arrives as an `<error>` inside otherwise-normal output, which reads as "no
> lineage found" rather than as a licence limit. For schema-scale lineage on
> the trial jar use
> [`samples/dlineageBasic/oracle/hr_mini/`](samples/dlineageBasic/oracle/hr_mini/readme.md)
> — 5,212 bytes, 120 relationships, added for that purpose and size-checked in
> CI.

### Published versions are kept; one batch was recalled in July 2026

Expand Down Expand Up @@ -265,7 +284,7 @@ One command, never by hand:

The version lives in **four** files — the `${gsp.core.version}` property in
`pom.xml`, plus a hardcoded `<version>` in each of the three
`connector/*/pom.xml`, which are separate builds with no parent to inherit a
`licensed-only/*/pom.xml`, which are separate builds with no parent to inherit a
property from. Both workflows run `--check`, so a missed file is a red build
rather than a connector quietly compiling against an older parser.

Expand Down Expand Up @@ -314,7 +333,7 @@ src/main/java/gudusoft/gsqlparser/demos/<demo>/ the demos, one dir per topic
src/main/resources/ classpath resources (one file)
src/test/java/gudusoft/gsqlparser/ tests, all exercising demos
samples/ sample .sql for the demos
connector/<vendor>Connector/ separate JDBC-connected modules
licensed-only/<vendor>Connector/ JDBC modules, licensed parser only
lib-repo/ in-project Maven repository
setenv/ + per-demo *.bat the Windows route
.github/scripts/ CI checks, all runnable locally
Expand Down Expand Up @@ -408,9 +427,10 @@ than only building them.
| Parser version consistency | `set-parser-version.sh --check` across all four POMs |
| The pre-commit hook | `test-pre-commit-hook.sh`: a drifting bump is refused in a throwaway clone |
| Documentation | `check-stale-docs.sh`: no readme names `pom_dlineage.xml`, `gudusoft.dlineage.jar` or the old `demos` package root; `--self-test` first, so a check that matches nothing cannot pass as a clean repo |
| Licensed-only guard | `check-licensed-only-guard.sh`: each `licensed-only/*` module stops at `validate` **with the licence message**, not with `cannot find symbol` |
| Build and test | JDK 8 and 21; 156 tests, and a run that skipped everything fails |
| Demo smoke test | `checksyntax` against known SQL |
| Standalone lineage jar | `smoke-dlineage-jar.sh` on JDK 8 and 21 — asserts on **output**, in JSON *and* XML |
| Standalone lineage jar | `smoke-dlineage-jar.sh` on JDK 8 and 21 — asserts on **output**, in JSON *and* XML, and that `hr_mini.sql` stays under the trial parser's 10,000-byte cap |
| Windows `.bat` | `windows-latest`: bootstrap, 39 compile scripts, 50 run scripts, 4 driven with real arguments |

`.github/workflows/nightly.yml` — at 03:17 UTC, because the parser is the moving
Expand Down Expand Up @@ -476,7 +496,7 @@ drive with arguments.
- **Don't commit jars**; add dependencies by coordinate.
- **Don't add a live-JDBC path to a demo** under `src/main/java`. It can't run
in CI, and it is what got two demos excluded from the build for years. The
`connector/*` modules are where database connections belong.
`licensed-only/*` modules are where database connections belong.
- **Don't add parser tests here**; they belong in `gsp_java_core`.

`master` tracks released GSP versions from
Expand Down
1 change: 0 additions & 1 deletion connector/oracleConnector/lib/readme.md

This file was deleted.

74 changes: 0 additions & 74 deletions connector/oracleConnector/pom.xml

This file was deleted.

1 change: 0 additions & 1 deletion connector/snowflakeConnector/lib/readme.md

This file was deleted.

1 change: 0 additions & 1 deletion connector/sqlServerConnector/lib/readme.md

This file was deleted.

Loading
Loading