Gemma 4 Text-Only Local Quantization Pipeline

This repository builds a local, text-only Gemma 4 pipeline for Hugging Face checkpoints, GGUF conversion, and Ollama import without patching Ollama internals in place.

The supported production target is google/gemma-4-E4B-it. The google/gemma-4-26B-A4B-it path remains available, but it is experimental and not a clean 16 GB VRAM target even after stripping the vision branch and quantizing to Q4_K_M.

Some Gemma 4 GGUF artifacts may retain upstream metadata tags such as image-text-to-text. In this repository, all produced models are explicitly stripped to text-only at the weight level. These tags are informational carryovers and do not imply multimodal capability.

Purpose

• Start from a multimodal Gemma 4 checkpoint.
• Strip the vision branch and keep only text-generation weights for Gemma4ForCausalLM.
• Convert the stripped checkpoint to GGUF with f16 base precision.
• Quantize to Q4_K_M.
• Write an Ollama Modelfile.
• Import the model into Ollama.
• Run a smoke test and record machine-readable manifests.

Supported Targets

• Production: Gemma 4 E4B text-only
• Experimental: Gemma 4 26B text-only

Architecture Overview

The pipeline is split into two Python stages and two wrapper scripts.

1. gemma4_text_only_pipeline.py Strips a multimodal Hugging Face checkpoint down to a text-only Gemma4ForCausalLM checkpoint.
2. gemma4_text_only_to_gguf.py Validates the stripped checkpoint, converts it with llama.cpp, quantizes it, writes a Modelfile, optionally runs ollama create, and optionally runs an Ollama HTTP API smoke test.
3. build_gemma4_textonly_e4b.sh Supported end-to-end production wrapper.
4. build_gemma4_textonly_26b.sh Experimental end-to-end wrapper.

Both Python stages write resumable manifests and machine-readable provenance.

Pipeline Stages

Stage 1: Strip to Text-Only HF

Input:

• multimodal Gemma 4 Hugging Face checkpoint

Output:

• text-only config.json
• sharded safetensors
• model.safetensors.index.json
• tokenizer and chat-template files
• strip_manifest.json

Stage 2: Convert to GGUF and Import into Ollama

Input:

• stripped text-only Hugging Face checkpoint

Output:

• model-f16.gguf
• model-q4.gguf
• Modelfile
• gguf_manifest.json
• command_log.txt
• optional smoke_test.json

Prerequisites

• Linux
• Python 3.11+ recommended
• CUDA-capable GPU if you want GPU-backed inference or faster smoke tests
• Ollama installed and running locally
• build tools for llama.cpp:
  • git
  • cmake
  • C/C++ compiler toolchain
• Enough disk space for cached sources and generated artifacts

The scripts perform practical disk-space preflight checks before large strip and conversion steps and fail with a clear message if free space looks insufficient.

Installation / Setup

cd /home/denis/ki/quant

python3 -m venv gemma4-local-pipeline/.venv
. gemma4-local-pipeline/.venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

If llama.cpp is missing, the wrapper scripts will clone and build it under gemma4-local-pipeline/llama.cpp.

Quick Start

Supported production build:

./build_gemma4_textonly_e4b.sh

Experimental 26B build:

./build_gemma4_textonly_26b.sh

Example Commands

Strip E4B to a text-only Hugging Face checkpoint:

python gemma4_text_only_pipeline.py \
  --source google/gemma-4-E4B-it \
  --cache-root artifacts/cache \
  --output-dir artifacts/gemma4-e4b-text-only \
  --manifest-path artifacts/gemma4-e4b-text-only/strip_manifest.json \
  --max-shard-size 2GiB \
  --resume

Convert the stripped E4B checkpoint to GGUF, quantize it, and import it into Ollama:

python gemma4_text_only_to_gguf.py \
  --source artifacts/gemma4-e4b-text-only \
  --output-dir artifacts/gemma4-e4b-gguf \
  --manifest-path artifacts/gemma4-e4b-gguf/gguf_manifest.json \
  --llama-cpp-dir gemma4-local-pipeline/llama.cpp \
  --quantization Q4_K_M \
  --temperature 0.7 \
  --num-ctx 8192 \
  --write-modelfile \
  --ollama-model-name gemma4-textonly-e4b \
  --ollama-create \
  --smoke-test \
  --resume

Unified CLI Behavior

Both Python stages share the same top-level control flags:

• --source
• --output-dir
• --manifest-path
• --resume
• --overwrite-output
• --smoke-test
• --smoke-test-prompt

The legacy --input-dir alias is still accepted as a compatibility path to --source.

Resume Behavior

Resumability is manifest-driven.

• Both stages write explicit manifest schema versions and reject incompatible manifest versions on resume.
• If a matching manifest is already complete, --resume reuses the completed outputs where possible.
• If outputs exist but the manifest is missing, the scripts attempt a controlled bootstrap from existing artifacts.
• If outputs are partial or inconsistent, the scripts clean only known stage outputs and rebuild in place.
• Resume state is inspectable through strip_manifest.json, gguf_manifest.json, command_log.txt, and the recorded step statuses.

The manifests also record:

• source model identifier and revision when available
• tokenizer provenance when available
• shard filenames and sizes
• key artifact sizes and SHA-256 hashes
• machine facts such as OS, Python version, CUDA visibility, GPU name, reported VRAM, Ollama version, and llama.cpp revision when obtainable
• normalized smoke-test data for the GGUF/Ollama stage

Output Artifacts

Primary artifact roots:

• artifacts/gemma4-e4b-text-only/
• artifacts/gemma4-e4b-gguf/
• artifacts/gemma4-26b-text-only/
• artifacts/gemma4-26b-gguf/

Important files:

• strip_manifest.json
• gguf_manifest.json
• machine_info.json
• artifact_hashes.json
• smoke_test.json
• command_log.txt
• root Modelfile

Hardware Expectations

E4B is the supported target for a 16 GB VRAM machine.

The 26B path is first-class as an experimental artifact flow, but it should not be treated as a clean 16 GB VRAM deployment target. In practice it is expected to require hybrid CPU/GPU execution, slower throughput, and significantly more disk and host RAM pressure.

Limitations

• The 26B path is experimental and may be blocked by source download size, conversion time, or runtime memory pressure.
• The pipeline assumes current transformers, huggingface_hub, and llama.cpp support for Gemma 4 layouts.
• The GGUF stage patches the local llama.cpp converter to accept Gemma4ForCausalLM if needed. This is a local checkout change, not an upstream release guarantee.
• Disk usage is substantial. Cached sources and generated outputs can consume tens of gigabytes.

Repository Structure

.
├── README.md
├── BUILD_REPORT.md
├── Modelfile
├── requirements.txt
├── gemma4_pipeline_utils.py
├── gemma4_text_only_pipeline.py
├── gemma4_text_only_pipeline_README.md
├── gemma4_text_only_to_gguf.py
├── gemma4_text_only_to_gguf_README.md
├── build_gemma4_textonly_e4b.sh
├── build_gemma4_textonly_26b.sh
├── artifacts/
└── gemma4-local-pipeline/

Current Status

• E4B is the supported production target.
• 26B remains experimental.
• The repository favors explicit artifacts and reproducible manifests over mutating Ollama’s internal model blobs.