feat(anthropic): Gate prompt collection on data_collection option - #7054
feat(anthropic): Gate prompt collection on data_collection option#7054ericapisani wants to merge 5 commits into
Conversation
Replace include_prompts and send_default_pii checks with the new data_collection configuration for controlling whether messages and system instructions are captured. Maintain backwards compatibility: when data_collection is not configured, fall back to the legacy pii/include_prompts behavior. Tools are always collected regardless of the message collection setting. Refs PY-2588
Codecov Results 📊✅ 99957 passed | ⏭️ 6515 skipped | Total: 106472 | Pass Rate: 93.88% | Execution Time: 358m 35s 📊 Comparison with Base Branch
All tests are passing successfully. ✅ Patch coverage is 88.46%. Project has 2509 uncovered lines. Files with missing lines (1)
Coverage diff@@ Coverage Diff @@
## main #PR +/-##
==========================================
- Coverage 90.00% 89.97% -0.03%
==========================================
Files 193 193 —
Lines 24991 25003 +12
Branches 9012 9026 +14
==========================================
+ Hits 22490 22494 +4
- Misses 2501 2509 +8
- Partials 1437 1438 +1Generated by Codecov Action |
|
This is currently failing tests so I'll review when they're green |
|
@alexander-alderman-webb Got it sorted - the original tests I had written didn't work on older versions of the Anthropic SDK because the Should be ready for 👀 now |
| @@ -456,23 +487,12 @@ def _set_common_input_data( | |||
| ) | |||
There was a problem hiding this comment.
Anthropic response text/tool calls ignore data_collection gen_ai.outputs opt-out
This change gates prompt (input) collection on data_collection["gen_ai"]["inputs"], but _set_output_data (line 609) still records GEN_AI_RESPONSE_TEXT and GEN_AI_RESPONSE_TOOL_CALLS based solely on the legacy should_send_default_pii() and integration.include_prompts check, so a user who sets _experiments={"data_collection": {"gen_ai": {"outputs": False}}} while still having send_default_pii=True will keep sending LLM response content — which frequently echoes user PII — to Sentry despite the explicit opt-out and despite the SDK warning that send_default_pii "is ignored when data_collection is set".
Evidence
_set_common_input_data(this hunk) now resolvesrecord_inputsfromclient.options["data_collection"]["gen_ai"]["inputs"], falling back toshould_send_default_pii() and integration.include_promptsonly whenhas_data_collection_enabled()is false._set_output_data(line 609) gatesGEN_AI_RESPONSE_TEXT/GEN_AI_RESPONSE_TOOL_CALLSonly onshould_send_default_pii() and integration.include_prompts; it never readsoptions["data_collection"]["gen_ai"]["outputs"](confirmed via grep — nodata_collectionreference in the output path).data_collection._gen_ai_from_valuesupports an explicitoutputs: False, and_resolve_data_collectiondocuments thatdata_collectionwins oversend_default_pii, emitting a DeprecationWarning thatsend_default_pii"is ignored whendata_collectionis set."- With
send_default_pii=Trueplusdata_collection={"gen_ai": {"outputs": False}}, the resolved config says outputs are off, yet the legacy check at line 609 still evaluates true and response text is attached to the span and sent to Sentry.
Identified by Warden · security-review · JXZ-LMZ
Replace include_prompts and send_default_pii checks with the new data_collection configuration for controlling whether messages and system instructions are captured. Maintain backwards compatibility: when data_collection is not configured, fall back to the legacy pii/include_prompts behavior. Tools are always collected regardless of the message collection setting.
Refs PY-2588