feat: migrate to Fess 15.8 - #5
Open
marevol wants to merge 1 commit into
Open
Conversation
Pin Fess 15.8.0 / OpenSearch 3.8.0, and bring along the pieces that do
not follow those pins on their own.
- .env: FESS_VERSION=15.8.0, OPENSEARCH_VERSION=3.8.0. The
fess_config.properties base is re-fetched from the fess-15.8.0 tag by
render-fess-config.sh with no further change; every key the codesearch
overlay redefines still exists in the 15.8 base.
- bin/setup.sh: fess-ds-git 15.8.0, downloaded from
maven.codelibs.org/release. Fess plugins are no longer published to
Maven Central from the 15.8 line on, so the previous URL 404s for
15.8.0 and setup.sh would abort. FESS_PLUGIN_REPO_URL overrides the
repository for pinning an older plugin release.
- fess_indices/_codesearch: refresh the hand-maintained fork with what
upstream added in 15.8 - index.knn (plus knn.derived_source.enabled
and the merge policy) in fess.json, and the content_chunk_vector /
content_chunk_status document fields in fess/doc.json. The fork is now
identical to codelibs/fess@fess-15.8.0 apart from the codesearch
tuning it exists for. Codesearch does not use semantic search - the
Content Chunk Vector Indexer job and content_chunker.search.enabled
are both off by default - but a mapping is only ever applied to a
brand-new index, so carrying it now is what keeps enabling it later
from requiring a full re-crawl.
- bin/verify.sh: read record_count out of the {"response": {...}}
envelope that /api/v2/search actually returns. The probe read it at
the top level, so it never found a count and silently SKIPped instead
of running - the one check that catches the "HTTP 200 with 0 hits"
failure mode this script exists for.
- README, register_github.sh, compose.yaml: version references, the
plugin repository change, and a note that an in-place upgrade keeps
the old index mapping.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves this deployment to Fess 15.8.0 / OpenSearch 3.8.0, and brings along the pieces that do not follow those pins on their own.
Changes
Version pins —
.env:FESS_VERSION=15.8.0,OPENSEARCH_VERSION=3.8.0(Fess 15.8.0 targets OpenSearch 3.8.0, perfess-parent-15.8.0).render-fess-config.shre-fetches thefess_config.propertiesbase from thefess-15.8.0tag with no change needed: every key the codesearch overlay redefines still exists in the 15.8 base. Theapi.v2.*→api.*renames in 15.8 do not touch any key the overlay owns.Plugin download —
bin/setup.shpinsfess-ds-git15.8.0 and downloads fromhttps://maven.codelibs.org/release. Fess plugins are no longer published to Maven Central from the 15.8 line on, so the old URL 404s for 15.8.0 andsetup.shwould abort.FESS_PLUGIN_REPO_URLoverrides the repository for pinning an older plugin release (≤ 15.7.0, still on Central).Index schema —
fess_indices/_codesearchis a hand-maintained fork, so it does not track the version. Refreshed with what upstream added in 15.8:fess.json:index.knn,index.knn.derived_source.enabled,index.merge.policyfess/doc.json:content_chunk_vector(nestedknn_vector),content_chunk_statusThe fork is now identical to
codelibs/fess@fess-15.8.0apart from the codesearch tuning it exists for (line_number_filter,operator_filter/dotnum_filter/code_stop_filter, the seven codesearch document fields,important_contentunindexed, nocontent_minhash_bitscopy).Codesearch does not use semantic search — the Content Chunk Vector Indexer job and
content_chunker.search.enabledare both off by default in 15.8 — but a mapping is only ever applied to a brand-new index, so carrying it now is what keeps enabling it later from requiring a full re-crawl. It does make the k-NN plugin a hard requirement;ghcr.io/codelibs/fess-opensearchbundles it.bin/verify.shfix — the search API probe readrecord_countat the top level of the response, but/api/v2/searchhas always wrapped its payload in a{"response": {...}}envelope. The probe therefore never found a count and silentlySKIPped — the one check that catches the "HTTP 200 with 0 hits" failure mode this script exists for (#3 / Redmine 6708). Now reads through the envelope, with a fallback to the flat shape.Docs — README version references, the plugin-repository change, the 15.8 index-schema note, and an explicit warning that an in-place upgrade keeps the old index mapping (Fess applies a mapping only to an index that has none yet).
Verification
Run against an isolated stack (
ghcr.io/codelibs/fess:15.8.0+ghcr.io/codelibs/fess-opensearch:3.8.0, separate compose project / container names / ports) built from exactly this branch's tracked files.bin/setup.shcompletes: fess-ds-git 15.8.0 downloaded from maven.codelibs.org, theme packaged from fess-themesmain(codesearch 1.0.9), base config fetched fromcodelibs/fess@fess-15.8.0.index.knn=true,index.knn.derived_source.enabled=false,index.merge.policy.floor_segment=16mb,max_merge_at_once=30. Live mapping: all 7 codesearch fields pluscontent_chunk_statusandcontent_chunk_vector(dimension=768,engine=lucene,space_type=cosinesimil,method=hnsw). No reindex needed.bin/verify.sh: all checks pass, includingthe codesearch schema has every core field of codelibs/fess@fess-15.8.0and the now-live API probe (the theme's facet request returns hits (18 for 'fess')). The one remainingWARN(duplicatedynamic_templatesnameslang_ca/lang_en/lang_hu) is pre-existing and unrelated.bin/register_github.shand crawled: 20 documents,domain/organization/repository/filetypeall populated./api/v2/searchreturns real facet payloads fororganization,repository,filename,filetype; the theme's facet qualifiers work (repository:→ 10,filetype:java→ 5,organization:→ 18). An unsupported facet field correctly answers HTTP 400invalid_request.record_count, org/repo/path crumbs, filetype badges, the facet rail with counts, and line-number gutters — confirming the codesearchline_number_filterandappendLineNumberhandler still line up. Home and/helprender clean;/search/advanceserves the SPA (/advanceis not an allowlisted theme UI prefix, and the theme links to the canonical path).Content Chunk Vector Indexerpresent andavailable=false, as expected.ERRORlines infess.logacross boot, crawl and search.