Task/347 dev manual for codeserving - #348
Merged
Merged
Conversation
Genos 를 처음 접하는 타사 개발자가 공개 배포본 repo 만으로 전처리기 코드를
수정·재배포할 수 있도록 개발 매뉴얼을 작성했다.
매뉴얼 (genon/preprocessor/facade/gitbook_doc/code_serving_dev_manual.md, 1,725줄)
- 0~3장: 시작 상태(이미 배포된 코드서빙 전제) · Genos/코드서빙/전처리기 개념
- 4장: 개발환경 준비 — 로컬(실측 검증) / 코드스페이스 / 로컬에서 되는 것·안 되는 것
- 5장: 코드 이해 — main.py 처리 순서, DocumentProcessor 계약,
parser/chunker 내부 구조와 "여기를 고치면 무엇이 바뀌는가", 복제 범위
- 6장: config yaml 옵션 / 7장: 코드 수정 레시피·제약 / 8장: 재배포 / 9장: 퀵 가이드
- 부록: 용어집 · 환경값 확인 목록 · 컨테이너 경로 · 참고 문서 · 신규 생성
- docx 는 genon/tools/md2docx.py 로 md 에서 생성 (단일 소스 유지)
코드 변경
- sync-serving-repo.sh: requirements-dev.txt 에 로컬 개발 deps 추가
(pymupdf/langchain-*/markdown2/pydub/chardet) — 기존 4개로는 parser·chunker
facade import 자체가 실패
- parse_chunk_test.sh: venv 탐색 후보에 저장소 루트 .venv 추가
- 공개 repo 로 나가는 예제 스크립트·문서에서 게이트웨이 인증키·주소·serving_id 제거
(환경변수 또는 필수 인자로 전환)
검증: 배포본 트리를 복제해 실제 설치 → 5개 facade import → PDF 파싱 → 청킹 E2E 확인.
링크·앵커는 원본/배포본 두 기준 모두 검증. 타사 개발자 페르소나 리뷰 2라운드 반영.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closed
|
Warning Review limit reached
Next review available in: 20 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (10)
📝 WalkthroughWalkthroughThe pull request updates generated code-serving deployment artifacts, changes CSV/XLSX processing to row-based parse elements and chunks, adds a Markdown-to-DOCX CLI, and adjusts runtime error and dependency handling. ChangesDeployment and code-serving updates
Row-based tabular processing
Document tooling and runtime handling
Estimated code review effort: 5 (Critical) | ~90 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
🧹 Nitpick comments (2)
genon/tools/md2docx.py (2)
390-411: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRemove the unused
header_countvariable, or use it to repeat header rows.Lines 392 and 398-399 compute
header_count, but no later code reads it. Either delete it, or apply it so header rows repeat on page breaks in Word.♻️ Proposed fix (mark header rows as repeating)
if src.tag == "th": _set_shading(cell._tc.get_or_add_tcPr(), "EDEDED") for run in p.runs: run.bold = True + for r_idx in range(header_count): + tr_pr = table.rows[r_idx]._tr.get_or_add_trPr() + tr_pr.append(OxmlElement("w:tblHeader")) self.doc.add_paragraph()🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@genon/tools/md2docx.py` around lines 390 - 411, Remove the unused header_count variable and its related updates from _table, since no later logic consumes it. Leave the existing table creation and autofit behavior unchanged.
586-607: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winDeclare the converter dependencies and handle file I/O errors.
genon/tools/pyproject.tomldoes not declaremarkdownorpython-docx. Add both dependencies so thegenos-toolsenvironment can runmd2docx.py.
_insert_imagealready handles image insertion failures. HandleOSErrorfrom input reading, output directory creation, and document saving, then return1with a concise error message.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@genon/tools/md2docx.py` around lines 586 - 607, Update genon/tools/pyproject.toml to declare both markdown and python-docx dependencies. In main, handle OSError from input reading, out_path.parent.mkdir, and convert/document saving by reporting a concise stderr error and returning 1, while preserving successful conversion behavior and the existing _insert_image handling.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@build-script/code-serving-README.md`:
- Around line 67-72: Update the generated README content around the GenOS Docker
image instructions to remove the internal registry URL and internal repository
path; replace them with deployment-time placeholders or relocate the
build-and-push guidance to an internal document while preserving the public
deployment steps.
- Around line 212-213: Update the --chunk-size note in code-serving-README.md to
state that omitting the option leaves the field unset, allowing the server
configuration to apply instead of sending 0 and forcing a single chunk.
Reconcile the chunk_size table description so it reflects the same default and
omission behavior.
In `@build-script/sync-serving-repo.sh`:
- Around line 246-249: Extend the generated ignore entries near the existing
preprocessor example outputs to include the result_serving_gateway_test/
directory produced by
genon/preprocessor/examples/code_serving/serving_gateway_test.sh. Preserve the
existing ignore patterns and add only this generated output path.
In `@genon/preprocessor/converters/xlsx_processor.py`:
- Around line 559-563: Update the row-dictionary construction in the XLSX
processing flow to prevent duplicate blank headers from overwriting values:
either reject duplicate blank headers before calling dict(zip(headers, values)),
or assign positional col_N keys for blank headers as the neutral row format
does. Preserve all column values distinctly.
In `@genon/preprocessor/examples/code_serving/serving_gateway_test.py`:
- Around line 104-106: Prevent the argparse help output from revealing
GENOS_AUTH_KEY in both parsers. Update the auth-key defaults in
serving_gateway_test.py (lines 104-106) and serving_gateway_preprocess_test.py
(lines 70-72) to use a non-secret argparse default or suppress the default in
help, then resolve GENOS_AUTH_KEY after argument parsing while preserving
authentication behavior.
In `@genon/preprocessor/examples/code_serving/serving_gateway_test.sh`:
- Around line 8-10: Remove --auth-key "$GENOS_AUTH_KEY" from both parser and
chunker invocations in
genon/preprocessor/examples/code_serving/serving_gateway_test.sh at lines 8-10,
relying on the Python environment default; in
genon/preprocessor/examples/code_serving/serving_gateway_preprocess_test_curl.sh
at lines 21-23, replace interpolated AUTH in the curl header with a protected
curl config or header file so bearer tokens never appear in child-process
arguments.
In `@genon/preprocessor/facade/gitbook_doc/parser_processor.md`:
- Around line 954-955: Update the tabular_row example in the parser
documentation to use valid JSON string escaping: represent pipe characters as
raw | characters rather than \|, matching the formatter’s actual output. Keep
the existing sheet, header, and value content unchanged.
In `@genon/preprocessor/facade/README.md`:
- Around line 205-207: Sanitize the public configuration examples: in
genon/preprocessor/facade/README.md lines 205-207, replace adjacent internal API
URLs and the /nfs-root/models/223/760 path with placeholders or configurable
values; in genon/preprocessor/facade/gitbook_doc/code_serving.md line 326,
replace serving ID 139 and the implicit authentication default with required
environment or argument values.
In `@genon/README.md`:
- Around line 193-198: Keep the README blockquote continuous by removing the
blank line between the quoted manual guidance and the following quoted A~C
statement, or prefixing that line with “>” so the entire block remains one
blockquote.
---
Nitpick comments:
In `@genon/tools/md2docx.py`:
- Around line 390-411: Remove the unused header_count variable and its related
updates from _table, since no later logic consumes it. Leave the existing table
creation and autofit behavior unchanged.
- Around line 586-607: Update genon/tools/pyproject.toml to declare both
markdown and python-docx dependencies. In main, handle OSError from input
reading, out_path.parent.mkdir, and convert/document saving by reporting a
concise stderr error and returning 1, while preserving successful conversion
behavior and the existing _insert_image handling.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: ec090520-33a8-42e2-8bb6-05bac54dbab8
⛔ Files ignored due to path filters (6)
genon/preprocessor/facade/gitbook_doc/code_serving_dev_manual.docxis excluded by!**/*.docxgenon/preprocessor/facade/gitbook_doc/images/codeserving_setup_create_code_serving_revision.pngis excluded by!**/*.pnggenon/preprocessor/facade/gitbook_doc/images/codeserving_setup_create_codespace_button.pngis excluded by!**/*.pnggenon/preprocessor/facade/gitbook_doc/images/codeserving_setup_gitea_id.pngis excluded by!**/*.pnggenon/preprocessor/facade/gitbook_doc/images/codeserving_setup_serving_id.pngis excluded by!**/*.pnggenon/preprocessor/facade/gitbook_doc/images/codeserving_setup_vscode.pngis excluded by!**/*.png
📒 Files selected for processing (30)
build-script/code-serving-README.mdbuild-script/sync-serving-repo.shgenon/README.mdgenon/preprocessor/converters/xlsx_processor.pygenon/preprocessor/examples/code_serving/serving_gateway_preprocess_test.pygenon/preprocessor/examples/code_serving/serving_gateway_preprocess_test_curl.shgenon/preprocessor/examples/code_serving/serving_gateway_test.pygenon/preprocessor/examples/code_serving/serving_gateway_test.shgenon/preprocessor/examples/code_serving/serving_gateway_test_curl.shgenon/preprocessor/examples/parse_chunk/parse_chunk_test.pygenon/preprocessor/examples/parse_chunk/parse_chunk_test.shgenon/preprocessor/facade/README.mdgenon/preprocessor/facade/attachment_processor.pygenon/preprocessor/facade/chunking_processor.pygenon/preprocessor/facade/gitbook_doc/code_serving.mdgenon/preprocessor/facade/gitbook_doc/code_serving_dev_manual.mdgenon/preprocessor/facade/gitbook_doc/guardrail_workflow_setup.mdgenon/preprocessor/facade/gitbook_doc/installation.mdgenon/preprocessor/facade/gitbook_doc/intro.mdgenon/preprocessor/facade/gitbook_doc/parser_processor.mdgenon/preprocessor/facade/parser_processor.pygenon/preprocessor/resource/parser_processor_config.yamlgenon/preprocessor/resource_dev/parser_processor_config.yamlgenon/preprocessor/tests/smoke/test_parser_processor_smoke.pygenon/preprocessor/tests/unit/test_chunking_processor_unit.pygenon/preprocessor/tests/unit/test_custom_fields_routing.pygenon/preprocessor/tests/unit/test_parser_processor_unit.pygenon/preprocessor/tests/unit/test_xlsx_processor.pygenon/tools/md2docx.pymain.py
Comment on lines
+67
to
+72
| curl http://192.168.74.164:30500/v2/mnc/template-code-serving-doc-parser/tags/list | ||
| ``` | ||
| - GenOS **도커 이미지**에 등록하고, 이미지 타입은 반드시 **`Code_Serving`** 으로 지정합니다. | ||
| (이 타입이 아니면 4번 리비전 생성 화면의 이미지 목록에 나타나지 않습니다.) | ||
| - 레지스트리에 없으면 **사내 원본 저장소의 `build-script/code-serving-doc-parser/README.md`** 절차로 | ||
| 빌드·푸시합니다. 공개 배포본에는 빌드 도구가 포함되지 않습니다. |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Remove internal registry details from the generated public README.
build-script/sync-serving-repo.sh copies this file into the public deployment repository. These lines expose http://192.168.74.164:30500/... and the internal path build-script/code-serving-doc-parser/README.md. Replace them with deployment-time placeholders, or move the internal build instructions to an internal document.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@build-script/code-serving-README.md` around lines 67 - 72, Update the
generated README content around the GenOS Docker image instructions to remove
the internal registry URL and internal repository path; replace them with
deployment-time placeholders or relocate the build-and-push guidance to an
internal document while preserving the public deployment steps.
Comment on lines
+8
to
+10
| # python serving_gateway_test.py --mode parser_upload --upload-file "../sample_files/hwp_sample_table.hwp" --out-doc result_serving_gateway_test/doc.json --serving-id "$GENOS_SERVING_ID" --auth-key "$GENOS_AUTH_KEY" | ||
| python serving_gateway_test.py --mode parser --file-path "/app/src/service/genon/preprocessor/sample_files/pdf_sample.pdf" --out-doc result_serving_gateway_test/doc.json --serving-id "$GENOS_SERVING_ID" --auth-key "$GENOS_AUTH_KEY" | ||
| python serving_gateway_test.py --mode chunker --doc-json result_serving_gateway_test/doc.json --out result_serving_gateway_test/chunks.json --serving-id "$GENOS_SERVING_ID" --auth-key "$GENOS_AUTH_KEY" |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift
Keep bearer tokens out of child-process arguments.
Both shell examples read secrets from environment variables and then expand them into Python or curl arguments. This exposes tokens to process inspection and command logging.
genon/preprocessor/examples/code_serving/serving_gateway_test.sh#L8-L10: remove--auth-key "$GENOS_AUTH_KEY"and rely on the Python environment default.genon/preprocessor/examples/code_serving/serving_gateway_preprocess_test_curl.sh#L21-L23: use a protected curl config or header file instead of interpolatingAUTHinto--header.
📍 Affects 2 files
genon/preprocessor/examples/code_serving/serving_gateway_test.sh#L8-L10(this comment)genon/preprocessor/examples/code_serving/serving_gateway_preprocess_test_curl.sh#L21-L23
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@genon/preprocessor/examples/code_serving/serving_gateway_test.sh` around
lines 8 - 10, Remove --auth-key "$GENOS_AUTH_KEY" from both parser and chunker
invocations in genon/preprocessor/examples/code_serving/serving_gateway_test.sh
at lines 8-10, relying on the Python environment default; in
genon/preprocessor/examples/code_serving/serving_gateway_preprocess_test_curl.sh
at lines 21-23, replace interpolated AUTH in the curl header with a protected
curl config or header file so bearer tokens never appear in child-process
arguments.
Comment on lines
+954
to
+955
| | `tabular_row` | CSV/XLSX 데이터 행 1개. `시트명 → (제목) → 헤더 라인 → 값 라인` 파이프 텍스트 | `"시트명: Sheet1\n\| name \| age \|\n\| Alice \| 30 \|"` | | ||
| | `custom_fields_row` | CSV/XLSX 데이터 행 1개(`doc_type` 매핑 적용). `text_fields` 로 지정한 목표필드 값을 줄바꿈으로 이어붙임 | `"카드 발급은 어떻게 하나요?\n앱에서 신청 가능합니다."` | |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Fix the invalid JSON example.
Line 954 uses \| inside a JSON string. \| is not a valid JSON escape. The row formatter also outputs raw pipe characters.
Proposed fix
- "시트명: Sheet1\n\| name \| age \|\n\| Alice \| 30 \|"
+ "시트명: Sheet1\n| name | age |\n| Alice | 30 |"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| | `tabular_row` | CSV/XLSX 데이터 행 1개. `시트명 → (제목) → 헤더 라인 → 값 라인` 파이프 텍스트 | `"시트명: Sheet1\n\| name \| age \|\n\| Alice \| 30 \|"` | | |
| | `custom_fields_row` | CSV/XLSX 데이터 행 1개(`doc_type` 매핑 적용). `text_fields` 로 지정한 목표필드 값을 줄바꿈으로 이어붙임 | `"카드 발급은 어떻게 하나요?\n앱에서 신청 가능합니다."` | | |
| | `tabular_row` | CSV/XLSX 데이터 행 1개. `시트명 → (제목) → 헤더 라인 → 값 라인` 파이프 텍스트 | `"시트명: Sheet1\n| name | age |\n| Alice | 30 |"` | | |
| | `custom_fields_row` | CSV/XLSX 데이터 행 1개(`doc_type` 매핑 적용). `text_fields` 로 지정한 목표필드 값을 줄바꿈으로 이어붙임 | `"카드 발급은 어떻게 하나요?\n앱에서 신청 가능합니다."` | |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@genon/preprocessor/facade/gitbook_doc/parser_processor.md` around lines 954 -
955, Update the tabular_row example in the parser documentation to use valid
JSON string escaping: represent pipe characters as raw | characters rather than
\|, matching the formatter’s actual output. Keep the existing sheet, header, and
value content unchanged.
Comment on lines
+205
to
+207
| "toc_api_key": "<API_KEY>", | ||
| "metadata_api_key": "<API_KEY>", | ||
| "toc_model": "<MODEL_NAME_OR_PATH>" |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Complete the public-repository configuration sanitization.
Both changes partially remove deployment-specific values but leave public examples with internal endpoints, paths, or serving defaults.
genon/preprocessor/facade/README.md#L205-L207: replace the adjacent internal API URLs and/nfs-root/models/223/760path with placeholders or configurable values.genon/preprocessor/facade/gitbook_doc/code_serving.md#L326-L326: replace the remaining139serving ID and implicit auth default with required environment or argument values.
📍 Affects 2 files
genon/preprocessor/facade/README.md#L205-L207(this comment)genon/preprocessor/facade/gitbook_doc/code_serving.md#L326-L326
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@genon/preprocessor/facade/README.md` around lines 205 - 207, Sanitize the
public configuration examples: in genon/preprocessor/facade/README.md lines
205-207, replace adjacent internal API URLs and the /nfs-root/models/223/760
path with placeholders or configurable values; in
genon/preprocessor/facade/gitbook_doc/code_serving.md line 326, replace serving
ID 139 and the implicit authentication default with required environment or
argument values.
Comment on lines
+193
to
+198
| > **타사(외부) 개발자에게 전처리기 코드 수정을 넘길 때는** | ||
| > [`preprocessor/facade/gitbook_doc/code_serving_dev_manual.md`](preprocessor/facade/gitbook_doc/code_serving_dev_manual.md) | ||
| > 를 안내한다 — 공개 배포본 repo 만으로 개발환경 세팅 → parser/chunker 코드 이해·수정 → gitea push → | ||
| > 리비전 재배포까지 할 수 있게 쓴 매뉴얼이다(이미지 빌드·원본 repo 접근은 전제하지 않음). | ||
| > **아래 A~C(이미지 빌드·최초 등록)는 사내 담당자 몫**이며 그 매뉴얼에는 포함되지 않는다. | ||
|
|
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Keep the blockquote continuous.
The blank line at Line [198] splits the blockquote and triggers MD028. Remove the blank line or add > on that line.
Suggested fix
> **아래 A~C(이미지 빌드·최초 등록)는 사내 담당자 몫**이며 그 매뉴얼에는 포함되지 않는다.
+>
> **미리 코드서빙용 도커이미지를 보유하고 있다면 아래의 A, B 단계는 수행하지 않아도 된다.**🧰 Tools
🪛 markdownlint-cli2 (0.23.2)
[warning] 198-198: Blank line inside blockquote
(MD028, no-blanks-blockquote)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@genon/README.md` around lines 193 - 198, Keep the README blockquote
continuous by removing the blank line between the quoted manual guidance and the
following quoted A~C statement, or prefixing that line with “>” so the entire
block remains one blockquote.
Source: Linters/SAST tools
HeechanKim-Genon
approved these changes
Aug 11, 2026
HeechanKim-Genon
approved these changes
Aug 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
task(#347): 코드서빙 전처리기 개발 매뉴얼(타사 개발자용) + xlsx tabular 행별 청크
개요
Genos 코드서빙으로 배포된 전처리기의 코드를 타사(외부) 개발자가 직접 수정할 수 있도록,
개발환경 세팅 → 코드 이해 → 수정 → gitea push → 리비전 재배포까지 한 문서로 안내하는
개발 매뉴얼을 새로 작성했다. 이 매뉴얼은 공개 배포본 repo(
doc_parser_code_serving)만가지고 따라갈 수 있는 것을 전제로 하며(이미지 빌드·사내 원본 repo 접근 불필요), 그 전제를
실제로 성립시키기 위해 배포본 생성 스크립트·예제 스크립트·기존 문서의 사내 의존(하드코딩된
접속정보, 배포본에 없는 상대링크, 사내 전용 문서)을 함께 정리했다.
매뉴얼을 쓰면서 발견한 동작 문제 하나도 같이 고쳤다 —
formats.xlsx.processing_mode: tabular인데도
doc_type을 주지 않으면 xlsx/csv 가 시트 통짜 1청크로 나오던 문제. 이제processing_mode만으로 행 1개 = 청크 1개가 된다.주요 변경
1) 코드서빙 전처리기 개발 매뉴얼 신규 (본체)
facade/gitbook_doc/code_serving_dev_manual.md(신규, 2055줄)배포되어 동작 중"으로 고정하고, 출발점을 "돌고 있는 전처리기를 내 요구에 맞게 고치기"로 둠.
0문서 안내(대상·시작 상태·준비물·읽는 순서) →1Genos 기본(모델서빙·게이트웨이·리비전·코드스페이스) →
2코드서빙 동작 모델·경로 제약 →3전처리기 5종·처리 흐름·저장소 2종 →4개발환경(인터넷 연결 / 인터넷 단절 오프라인 키트 / 코드스페이스 / 모델서빙 연결) →5코드 이해(코드 지도·main.py처리 순서·parser/chunker 읽기·출력 스키마) →6config yaml 옵션 →7코드 수정 가이드(개발 루프·수정 레시피·알아둘 제약) →8재배포(gitea push → 리비전 → 호출 검증) →9퀵 가이드(시나리오 A/B) →부록
A용어집 ·B환경값 확인 목록 ·C컨테이너 경로/환경변수 ·D참고 문서 ·E코드서빙·코드스페이스 신규 생성.외부 개발자가 어디서 실행해야 하는지 헷갈리지 않게 함.
<GENOS_HOST>처럼 꺾쇠 placeholder 로 통일 + 미치환 placeholder 확인명령을 각 장에 배치.
code_serving_dev_manual.docx, 캡처 이미지 5장(
gitbook_doc/images/codeserving_setup_*.png— 코드스페이스 생성/VSCode/gitea id/serving id/리비전 생성).genon/tools/md2docx.py(신규, 611줄) — 위 docx 를 뽑는 md→docx 변환기.새 패키지 설치 없이 저장소
.venv의markdown+python-docx만 사용.지원: h1~h4 · 문단 · 중첩 목록 · 표 · 펜스 코드블록 · blockquote · hr · 인라인 서식 · 상대경로 이미지.
genon/README.md(타사 개발자에게 넘길 때 안내 + A~C 이미지 빌드는 사내 몫임을 명시),facade/gitbook_doc/intro.md,gitbook_doc/code_serving.md(호출 문서 ↔ 개발 문서 분리 명시),build-script/code-serving-README.md(독자별로 읽을 곳 표 신설).2) 공개 배포본 정리 (매뉴얼 전제 성립)
build-script/sync-serving-repo.shgenon/README.md,genon/MAINTAINERS.md,genon/dotsocr_vllm_max_num_seqs.md,genon/preprocessor/facade/README.md.배포본에 없는 폴더(
build-script/·docling/·docs/)를 안내해 혼란만 주던 문서들..gitignore생성 + 스모크 검증 추가 — 로컬 개발 산출물(.venv·result.json·캐시)과 오프라인 설치 키트(
offline-dev-kit/·wheelhouse/)가 gitea 배포 저장소에실수로 커밋되지 않게.
requirements-dev.txt보강 — 기존 4개(fastapi·httpx·grpcio·protobuf)에pymupdf·langchain-community·langchain-core·langchain-text-splitters·markdown2·pydub·chardet추가. parser·chunking·attachment·convert facade 는 모듈 최상위에서 이들을import 해, 없으면 로컬에서 facade import 자체가 실패했다(외부 개발자의 주 작업 대상이 parser/chunker).
build-script/code-serving-README.md1번 단계를 "설치 담당 엔지니어용(사이트당 1회)"로 명시.
모델 서빙 외부 게이트웨이 URL + API 키 설정 방식으로 갱신(매뉴얼 4장 참조).
--chunk-size생략 시 청크가 1개만 나오는 함정 경고 추가.3) 접속 정보 하드코딩 제거 · 사내 링크 정리
examples/code_serving/serving_gateway_test.py,serving_gateway_preprocess_test.pyGENOS_BASE_URL/GENOS_SERVING_ID/GENOS_AUTH_KEY환경변수 또는--base-url/--serving-id/--auth-key.없으면 무엇을 지정해야 하는지 안내하고 exit 2.
--chunk-size기본값0→None— 생략하면 전송하지 않아 서버chunking.chunk_size가쓰인다(종전엔
0이 전송돼 의도치 않게 크기 기반 분할이 꺼졌다).examples/code_serving/*_curl.sh2종 —${GENOS_*:?}로 필수 환경변수화.examples/parse_chunk/parse_chunk_test.py—--doc_type/--chunk-size도움말을 변경된의미(행 분할은
processing_mode,0=크기 기반 병합·분할 끄기)로 갱신.examples/parse_chunk/parse_chunk_test.sh— venv 탐색에 저장소 루트.venv추가(배포본 레이아웃 = 매뉴얼 4장이 만드는 위치). 순서:
genon/preprocessor/.venv→ 루트.venv→ 시스템 python.gitbook_doc/installation.md·guardrail_workflow_setup.md,facade/README.md—배포본에 존재하지 않는 상대링크(
../../../README.md,../../scripts/…)를 "사내 원본 저장소 …"텍스트 참조로 바꾸고, 예시 API 키·모델 경로를 placeholder 로 치환.
4) 기타 안정화
main.py—_error_response()가getattr(exc, 'error_code', None)를 먼저 사용.facade 는
GenosServiceException로컬 사본을 던져 전용 핸들러가 아닌 generic 경로로오는데, 그때 facade 가 부여한 코드가
INTERNAL_ERROR로 뭉개지고 있었다.facade/attachment_processor.py— weasyprint import 가드를except (ImportError, OSError)로확장(패키지는 있지만
libgobject-2.0등 네이티브 라이브러리가 없는 환경).parser_processor.py의 동일 블록과 형태 일치.resource/·resource_dev/parser_processor_config.yaml—processing_mode: tabular주석을"데이터 행마다 parse element/청크 생성"으로 갱신.
5) 문서 / 테스트
gitbook_doc/parser_processor.md— CSV/XLSX 절 전면 갱신:tabular_row스키마와 JSON 예시,행
metadata·column_map, 예약어 alias 정책,doc_type과의 관계,page의미(시트 순번),category 목록·
content형식 표에tabular_row/custom_fields_row추가,table은"예전 산출물 재청킹 시에만 나타남"으로 표기.
elements[].metadata필드 설명 추가.gitbook_doc/code_serving.md— 비-docling 청킹 설명을csv/xlsx → [DA] 단일→행마다 청크 1개로 수정, 접속 정보 예시 placeholder 화, 개발 매뉴얼 링크 추가.tests/unit/test_xlsx_processor.py(5종)doc_type없이 행별 element 생성 /multi_table에서도 행 parity 유지title등) →field_<hash>alias 회피_RESERVED_FIELDS↔ chunkerGenOSVectorMeta드리프트 감지_2suffix) 고정tests/unit/test_chunking_processor_unit.py(2종)tabular_row2행 → 청크 2개(i_chunk_on_doc연속, metadata 가 벡터 property 로 승격)tests/smoke/test_parser_processor_smoke.py(csv/xlsx category →tabular_row),tests/unit/test_parser_processor_unit.py(_sheet_to_html테스트 삭제 → 행별 element·metadata키 검증),tests/unit/test_custom_fields_routing.py(tabular_row파라미터 추가).동작 변경 / 하위 호환
processing_mode: tabular(기본)에서 csv/xlsx 의 parse 결과가시트당
tableelement → 데이터 행당tabular_rowelement 로 바뀐다.이어지는 청킹도
[DA]단일 벡터 → 행마다 벡터 1개. 벡터 수·검색 단위가 달라지므로기존 적재분과 섞이는 인덱스가 있으면 재적재 필요 여부를 확인해야 한다.
category="table"만으로 이루어진 예전 parse 산출물을 다시 청킹하면 종전대로[DA]단일 벡터가 된다(legacy 가드 유지).faq_row·custom_fields_row경로도 그대로.doc_type매칭tabular_mapping이 있는 호출은 종전과 동일하게custom_fields_row로 처리된다(우선순위 불변).인자 없이 실행하면 안내 메시지와 함께 exit 2.
serving_gateway_test.py --chunk-size를 생략하던 호출은 이제 서버 config 값이 쓰인다(종전엔
0전송). 종전 동작을 원하면--chunk-size 0을 명시.테스트
tests/unit/test_xlsx_processor.py5종 ·tests/unit/test_chunking_processor_unit.py2종 신규 통과.test_parser_processor_smoke.py)에서 모든 element 가tabular_row임을 확인.parse_chunk_test.py--doc_type없이 → 행마다 1청크(변경 전에는 시트 통짜 1청크),metadata에 컬럼 키 +column_map.--doc_type faq→ 종전대로custom_fields_row목표필드 부착(회귀 없음).serving_gateway_test.py --mode parser_upload(xlsx) →--mode chunker로행별 청크 결과 확인. 접속 정보 미지정 시 exit 2 동작 확인.
sync-serving-repo.sh를 throwawaySERVING_DIR로 실행해.gitignore·requirements-dev.txt·VERSION생성과 사내 문서 4개 제외를 스모크로 확인.md2docx.py로 매뉴얼 docx 생성(표·코드블록·이미지·중첩목록 렌더 확인).영향 파일
신규
genon/preprocessor/facade/gitbook_doc/code_serving_dev_manual.md(2055줄)genon/preprocessor/facade/gitbook_doc/code_serving_dev_manual.docxgenon/preprocessor/facade/gitbook_doc/images/codeserving_setup_*.png(5장)genon/tools/md2docx.py(611줄)genon/preprocessor/tests/unit/test_xlsx_processor.py코드
genon/preprocessor/converters/xlsx_processor.py—tabular_data_to_parse_format,_build_column_keys,_RESERVED_FIELDS확장,sheet_indexgenon/preprocessor/facade/parser_processor.py—_sheet_to_html제거, 행별 위임genon/preprocessor/facade/chunking_processor.py—tabular_row경로 + 드롭 WARNINGgenon/preprocessor/facade/attachment_processor.py— weasyprintOSError가드main.py—error_code보존빌드/배포
build-script/sync-serving-repo.sh,build-script/code-serving-README.mdconfig
genon/preprocessor/resource/parser_processor_config.yaml,genon/preprocessor/resource_dev/parser_processor_config.yaml(주석)예제 스크립트
examples/code_serving/serving_gateway_test.py|.sh|_curl.sh,examples/code_serving/serving_gateway_preprocess_test.py|_curl.sh,examples/parse_chunk/parse_chunk_test.py|.sh문서
genon/README.md,genon/preprocessor/facade/README.md,gitbook_doc/intro.md·code_serving.md·parser_processor.md·installation.md·guardrail_workflow_setup.md테스트
tests/smoke/test_parser_processor_smoke.py,tests/unit/test_parser_processor_unit.py,tests/unit/test_chunking_processor_unit.py,tests/unit/test_custom_fields_routing.pySummary by CodeRabbit