[SC-17911] Send logged Markdown without script tags

notebooks/code_sharing/markdown_html_logging_regression.ipynb

@@ -0,0 +1,364 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "copyright-sc17911",
+   "metadata": {},
+   "source": [
+    "<!-- VALIDMIND COPYRIGHT -->\n",
+    "\n",
+    "<small>\n",
+    "\n",
+    "***\n",
+    "\n",
+    "Copyright © 2023-2026 ValidMind Inc. All rights reserved.<br>\n",
+    "Refer to [LICENSE](https://github.com/validmind/validmind-library/blob/main/LICENSE) for details.<br>\n",
+    "SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial</small>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "title",
+   "metadata": {},
+   "source": [
+    "# Markdown and HTML logging regression (SC-17911)\n",
+    "\n",
+    "This notebook verifies the compatibility contract for `log_text()` and `/tracking/log_metadata` after the WAF-safe TeX change. It covers plain Markdown, inline and display TeX, Markdown extensions, mixed Markdown/HTML, explicit HTML, and explicit MathJax HTML.\n",
+    "\n",
+    "The first two test matrices are deterministic and do not require credentials or a running backend:\n",
+    "\n",
+    "- When the backend advertises `log_metadata_markdown`, Markdown remains raw on the wire and carries `text_format=\"markdown\"`. The SDK does not put generated MathJax `<script>` tags in the request.\n",
+    "- When the capability is absent, the SDK preserves compatibility with older backends by converting Markdown locally and omitting `text_format`.\n",
+    "- Explicit caller-supplied HTML remains unchanged in both modes. This intentionally includes explicit `<script>` input; the WAF-safe change only prevents the SDK from generating script tags from Markdown before transport.\n",
+    "\n",
+    "The final section is an opt-in end-to-end check against a real backend. Set `VM_RUN_E2E=1` and the usual `VM_API_HOST`, `VM_API_KEY`, `VM_API_SECRET`, and `VM_API_MODEL` variables before executing the notebook. The E2E check creates or updates `test_description:sc17911_notebook_*` metadata records."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "imports-and-cases",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "import os\n",
+    "from unittest.mock import patch\n",
+    "\n",
+    "import pandas as pd\n",
+    "from bs4 import BeautifulSoup\n",
+    "\n",
+    "import validmind as vm\n",
+    "import validmind.api_client as api_client\n",
+    "from validmind.client_config import client_config\n",
+    "from validmind.utils import is_html, md_to_html\n",
+    "\n",
+    "CASES = [\n",
+    "    {\n",
+    "        \"name\": \"plain_markdown\",\n",
+    "        \"text\": \"## Compatibility heading\\n\\nA **bold** paragraph and a list:\\n\\n- one\\n- two\",\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"inline_tex\",\n",
+    "        \"text\": r\"Weight of Evidence is $WOE = \\ln\\dfrac{\\%\\ of\\ Events}{\\%\\ of\\ Non-Events}$.\",\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"display_tex\",\n",
+    "        \"text\": \"## Equation\\n\\n\" + r\"$$G(x) = \\sum_{i=1}^{n} w_i x_i$$\",\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"markdown_extensions\",\n",
+    "        \"text\": \"~~Deprecated~~ current value.[^1]\\n\\n| Metric | Value |\\n| --- | ---: |\\n| AUC | 0.91 |\\n\\n[^1]: Regression footnote.\",\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"mixed_markdown_html\",\n",
+    "        \"text\": \"## Mixed input\\n\\nMarkdown before an <em>inline HTML fragment</em>.\",\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"explicit_html\",\n",
+    "        \"text\": \"<h2>Explicit HTML</h2><p><strong>Keep this markup unchanged.</strong></p>\",\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"explicit_mathjax_html\",\n",
+    "        \"text\": r'<p><script type=\"math/tex\">WOE = \\ln\\dfrac{\\%\\ of\\ Events}{\\%\\ of\\ Non-Events}</script></p>',\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "assert {is_html(case[\"text\"]) for case in CASES} == {False, True}\n",
+    "pd.DataFrame(\n",
+    "    {\n",
+    "        \"case\": [case[\"name\"] for case in CASES],\n",
+    "        \"detected_as\": [\"html\" if is_html(case[\"text\"]) else \"markdown\" for case in CASES],\n",
+    "    }\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "capture-helper",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "async def capture_log_text_request(case, supports_markdown):\n",
+    "    \"\"\"Run alog_text while capturing the serialized log_metadata body.\"\"\"\n",
+    "    captured = {}\n",
+    "\n",
+    "    async def capture_post(path, params=None, data=None, **kwargs):\n",
+    "        captured[\"path\"] = path\n",
+    "        captured[\"params\"] = params\n",
+    "        captured[\"body\"] = json.loads(data)\n",
+    "        return {\"content_id\": captured[\"body\"][\"content_id\"], \"text\": captured[\"body\"].get(\"text\")}\n",
+    "\n",
+    "    original_flags = client_config.feature_flags\n",
+    "    try:\n",
+    "        client_config.feature_flags = {\"log_metadata_markdown\": supports_markdown}\n",
+    "        with patch.object(api_client, \"_post\", new=capture_post):\n",
+    "            await api_client.alog_text(\n",
+    "                content_id=f'test_description:sc17911_{case[\"name\"]}',\n",
+    "                text=case[\"text\"],\n",
+    "            )\n",
+    "    finally:\n",
+    "        client_config.feature_flags = original_flags\n",
+    "\n",
+    "    assert captured[\"path\"] == \"log_metadata\"\n",
+    "    return captured[\"body\"]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "new-backend-heading",
+   "metadata": {},
+   "source": [
+    "## New backend: WAF-safe Markdown transport\n",
+    "\n",
+    "Markdown and TeX must stay raw in the request. Fully formed HTML must retain the legacy pass-through behavior."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "new-backend-matrix",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "new_backend_rows = []\n",
+    "for case in CASES:\n",
+    "    body = await capture_log_text_request(case, supports_markdown=True)\n",
+    "    html_input = is_html(case[\"text\"])\n",
+    "\n",
+    "    assert body[\"text\"] == case[\"text\"]\n",
+    "    if html_input:\n",
+    "        assert \"text_format\" not in body\n",
+    "    else:\n",
+    "        assert body[\"text_format\"] == \"markdown\"\n",
+    "        assert \"<script\" not in body[\"text\"].lower()\n",
+    "\n",
+    "    new_backend_rows.append(\n",
+    "        {\n",
+    "            \"case\": case[\"name\"],\n",
+    "            \"text_format\": body.get(\"text_format\", \"omitted\"),\n",
+    "            \"unchanged_on_wire\": body[\"text\"] == case[\"text\"],\n",
+    "            \"script_on_wire\": \"<script\" in body[\"text\"].lower(),\n",
+    "        }\n",
+    "    )\n",
+    "\n",
+    "pd.DataFrame(new_backend_rows)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "legacy-backend-heading",
+   "metadata": {},
+   "source": [
+    "## Older backend: compatibility fallback\n",
+    "\n",
+    "When the backend does not advertise support, Markdown must be converted locally and `text_format` must be omitted. Explicit HTML still passes through unchanged. This avoids silently storing raw Markdown on older customer-managed backends."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "legacy-backend-matrix",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "legacy_backend_rows = []\n",
+    "for case in CASES:\n",
+    "    body = await capture_log_text_request(case, supports_markdown=False)\n",
+    "    html_input = is_html(case[\"text\"])\n",
+    "    expected_text = case[\"text\"] if html_input else md_to_html(case[\"text\"], mathml=True)\n",
+    "\n",
+    "    assert \"text_format\" not in body\n",
+    "    assert body[\"text\"] == expected_text\n",
+    "\n",
+    "    legacy_backend_rows.append(\n",
+    "        {\n",
+    "            \"case\": case[\"name\"],\n",
+    "            \"converted_locally\": not html_input,\n",
+    "            \"matches_legacy_output\": body[\"text\"] == expected_text,\n",
+    "            \"script_on_wire\": \"<script\" in body[\"text\"].lower(),\n",
+    "        }\n",
+    "    )\n",
+    "\n",
+    "pd.DataFrame(legacy_backend_rows)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e2e-heading",
+   "metadata": {},
+   "source": [
+    "## Optional end-to-end backend check\n",
+    "\n",
+    "Run this section with `VM_RUN_E2E=1` to initialize the SDK from `VM_API_*`, log every case to the selected model's documentation, and verify the backend persists semantically equivalent rendered HTML while preserving explicit HTML byte-for-byte. It runs in either capability state and reports whether it exercised backend Markdown conversion or the legacy SDK fallback."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "initialize-client-heading",
+   "metadata": {},
+   "source": [
+    "### Initialize the ValidMind client\n",
+    "\n",
+    "The client is initialized only for the optional end-to-end run. Credentials are read from environment variables and are never stored in the notebook."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "initialize-client",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "run_e2e = os.getenv(\"VM_RUN_E2E\", \"0\").lower() in {\"1\", \"true\", \"yes\"}\n",
+    "\n",
+    "if run_e2e:\n",
+    "    required = [\"VM_API_HOST\", \"VM_API_KEY\", \"VM_API_SECRET\", \"VM_API_MODEL\"]\n",
+    "    missing = [name for name in required if not os.getenv(name)]\n",
+    "    assert not missing, f\"Missing required environment variables: {missing}\"\n",
+    "\n",
+    "    vm.init(\n",
+    "        api_host=os.environ[\"VM_API_HOST\"],\n",
+    "        api_key=os.environ[\"VM_API_KEY\"],\n",
+    "        api_secret=os.environ[\"VM_API_SECRET\"],\n",
+    "        model=os.environ[\"VM_API_MODEL\"],\n",
+    "        document=\"documentation\",\n",
+    "    )\n",
+    "else:\n",
+    "    print(\"Client initialization skipped. Set VM_RUN_E2E=1 to enable it.\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e2e-check",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "e2e_rows = []\n",
+    "\n",
+    "def assert_backend_render(case, stored_text):\n",
+    "    \"\"\"Assert stable rendering semantics without requiring byte-identical HTML.\"\"\"\n",
+    "    if is_html(case[\"text\"]):\n",
+    "        assert stored_text == case[\"text\"]\n",
+    "        return\n",
+    "\n",
+    "    assert stored_text != case[\"text\"]\n",
+    "    soup = BeautifulSoup(stored_text, \"html.parser\")\n",
+    "    name = case[\"name\"]\n",
+    "\n",
+    "    if name == \"plain_markdown\":\n",
+    "        assert soup.find(\"h2\").get_text(strip=True) == \"Compatibility heading\"\n",
+    "        assert soup.find(\"strong\").get_text(strip=True) == \"bold\"\n",
+    "        assert [item.get_text(strip=True) for item in soup.find_all(\"li\")] == [\"one\", \"two\"]\n",
+    "    elif name == \"inline_tex\":\n",
+    "        script = soup.find(\"script\", attrs={\"type\": \"math/tex\"})\n",
+    "        assert script is not None and \"WOE\" in script.get_text()\n",
+    "    elif name == \"display_tex\":\n",
+    "        script = soup.find(\"script\", attrs={\"type\": \"math/tex\"})\n",
+    "        assert script is not None and \"G(x)\" in script.get_text()\n",
+    "    elif name == \"markdown_extensions\":\n",
+    "        assert soup.find(\"del\").get_text(strip=True) == \"Deprecated\"\n",
+    "        assert soup.select_one(\"figure.table table\") is not None\n",
+    "        assert soup.select_one(\"section.footnotes\") is not None\n",
+    "        assert [cell.get_text(strip=True) for cell in soup.find_all(\"th\")] == [\"Metric\", \"Value\"]\n",
+    "    elif name == \"mixed_markdown_html\":\n",
+    "        assert soup.find(\"h2\").get_text(strip=True) == \"Mixed input\"\n",
+    "        assert soup.find(\"em\") is None\n",
+    "        assert \"&lt;em&gt;inline HTML fragment&lt;/em&gt;\" in stored_text\n",
+    "    else:\n",
+    "        raise AssertionError(f\"Missing Markdown assertion for {name}\")\n",
+    "\n",
+    "if run_e2e:\n",
+    "    markdown_capability = client_config.supports_log_metadata_markdown()\n",
+    "    transport_mode = (\n",
+    "        \"backend Markdown conversion\"\n",
+    "        if markdown_capability\n",
+    "        else \"legacy SDK conversion\"\n",
+    "    )\n",
+    "\n",
+    "    for case in CASES:\n",
+    "        response = await api_client.alog_text(\n",
+    "            content_id=f'test_description:sc17911_notebook_{case[\"name\"]}',\n",
+    "            text=case[\"text\"],\n",
+    "        )\n",
+    "        stored_text = response[\"text\"]\n",
+    "        assert_backend_render(case, stored_text)\n",
+    "\n",
+    "        e2e_rows.append(\n",
+    "            {\n",
+    "                \"case\": case[\"name\"],\n",
+    "                \"content_id\": response[\"content_id\"],\n",
+    "                \"transport_mode\": transport_mode,\n",
+    "                \"persisted_as_expected\": True,\n",
+    "            }\n",
+    "        )\n",
+    "\n",
+    "    await api_client._get_session().close()\n",
+    "else:\n",
+    "    e2e_rows.append(\n",
+    "        {\n",
+    "            \"case\": \"skipped\",\n",
+    "            \"content_id\": \"Set VM_RUN_E2E=1 to run against a backend\",\n",
+    "            \"transport_mode\": None,\n",
+    "            \"persisted_as_expected\": None,\n",
+    "        }\n",
+    "    )\n",
+    "\n",
+    "pd.DataFrame(e2e_rows)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "execution-notes",
+   "metadata": {},
+   "source": [
+    "## Headless execution\n",
+    "\n",
+    "Run the deterministic transport regression from the repository root:\n",
+    "\n",
+    "```bash\n",
+    "uv run jupyter nbconvert --execute --to notebook \\\n",
+    "  --output /tmp/markdown_html_logging_regression.out.ipynb \\\n",
+    "  notebooks/code_sharing/markdown_html_logging_regression.ipynb\n",
+    "```\n",
+    "\n",
+    "Add `VM_RUN_E2E=1` and valid `VM_API_*` variables to the command environment to include the backend check. A failed assertion makes notebook execution exit non-zero."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}