DependencyManager purges the `google` namespace package but keeps `google.protobuf`, leaving the parent unbound (function load fails)

[pr1te]

Summary

DependencyManager.prioritize_customer_dependencies() purges the google namespace package from sys.modules while leaving the regular subpackage google.protobuf cached. Any later import google.protobuf then takes the sys.modules fast path, which does not re-bind the submodule onto its parent, while __import__ creates a fresh, empty google module. The result is a process where import google.protobuf succeeds but google.protobuf.__version__ raises.

To be explicit about where each half lives, since the failing line is not in this repository: the defect is the purge in azure_functions_worker/utils/dependency.py, and the symptom surfaces in the third-party google-api-core package, which reads that attribute at module scope and has no reason to guard it:

# google/api_core/grpc_helpers.py
29  import google.protobuf
30
31  PROTOBUF_VERSION = google.protobuf.__version__   # <-- AttributeError

So every function whose import chain reaches google.api_core fails to load, for the whole life of that worker process. In our case that is any function importing a Google Cloud client (Discovery Engine). Because it fails during function_load, no retry can clear it: an affected worker fails every invocation it is given.

Why the purge is asymmetric

All references below are to workers/azure_functions_worker/utils/dependency.py at tag 4.45.1.

prioritize_customer_dependencies() re-adds the worker tree via _add_to_sys_path(..., add_to_first=False) and then calls _remove_module_cache(), which filters cached modules by set(getattr(module, '__path__', None) or []) and pops anything whose path starts with the tree being removed. The ordering is the problem:

• google is a namespace package. Its __path__ is a _NamespacePath, whose __iter__ re-computes against the just-mutated sys.path, so it re-acquires <worker_tree>/google, matches the prefix test, and is popped.
• google.protobuf is a regular package with a static __file__. If it first resolved from the customer tree it does not match, and survives.

CPython then does the rest: import a.b with sys.modules['a.b'] present skips both the parent import and the setattr(parent, 'b', mod).

Note that from google.cloud import storage on an adjacent line keeps working, because from X import Y has a sys.modules fallback that plain attribute access does not. That makes this look intermittent when it is not.

Reproduction

Deterministic (200/200 across randomised PYTHONHASHSEED), driving the real DependencyManager at tag 4.45.1. No threads and no concurrency involved.

FROM python:3.10-slim
RUN pip install --no-cache-dir --target=/trees/app \
      google-cloud-discoveryengine==0.11.8 google-api-core==2.20.0 \
      grpcio==1.83.0 protobuf==4.25.5
RUN pip install --no-cache-dir --target=/trees/worker \
      protobuf==4.25.5 grpcio==1.83.0 azure-functions==1.21.3
RUN pip install --no-cache-dir azure-functions-worker==4.45.1 || true
WORKDIR /w

"""Minimal repro: DependencyManager purges the `google` namespace package but keeps
`google.protobuf`, leaving the parent unbound. Run inside the image built from the Dockerfile."""
import os
import sys

# The purge below is gated on this; it is the documented setting for apps whose dependencies
# collide with the worker's, and is the default from Python 3.13.
os.environ['PYTHON_ISOLATE_WORKER_DEPENDENCIES'] = '1'

WORKER_TREE, APP_TREE = '/trees/worker', '/trees/app'

# Layout the Functions host produces: the app's .python_packages and the worker's own tree are
# both importable, and both contain part of the `google` namespace.
sys.path.insert(0, WORKER_TREE)
sys.path.insert(0, APP_TREE)

# The worker imports protobuf for its own gRPC channel before any customer code runs.
import google, google.protobuf                                    # noqa: E401
print('before  : google in sys.modules =', 'google' in sys.modules,
      '| google.protobuf bound on parent =', hasattr(google, 'protobuf'))

sys.path.insert(0, '/azure-functions-worker-src/workers')          # worker source checkout
from azure_functions_worker.utils.dependency import DependencyManager as DM  # noqa: E402

DM.initialize()
DM.worker_deps_path = WORKER_TREE
DM.cx_deps_path = APP_TREE
DM.cx_working_dir = '/wwwroot'
DM.prioritize_customer_dependencies('/wwwroot')

print('after   : google in sys.modules =', 'google' in sys.modules,
      '| google.protobuf in sys.modules =', 'google.protobuf' in sys.modules)

import google.protobuf                                            # succeeds: sys.modules fast path
print('line 29 : import google.protobuf  -> OK')
try:
    v = google.protobuf.__version__                               # the failing read
    print('line 31 : google.protobuf.__version__ ->', v)
except AttributeError as e:
    print('line 31 : google.protobuf.__version__ -> AttributeError:', e)

Clone the worker source at the tag and mount it at /azure-functions-worker-src, then:

docker build -t afpw-repro:3.10 .
docker run --rm -v "$PWD:/w" -v "<worker-src>:/azure-functions-worker-src" -w /w afpw-repro:3.10 python /w/repro.py

Output:

before  : google in sys.modules = True | google.protobuf bound on parent = True
after   : google in sys.modules = False | google.protobuf in sys.modules = True
line 29 : import google.protobuf  -> OK
line 31 : google.protobuf.__version__ -> AttributeError: module 'google' has no attribute 'protobuf'

The after line is the defect: parent evicted, child retained.

The Dockerfile pins google-api-core to 2.20.0 so the line numbers match the trace above. The unguarded read is not specific to that release: it is present unchanged in 2.11.0 through 2.29.0 (currently at lines 25 and 30), so upgrading the library does not avoid this.

Impact seen in production

A Python 3.10 Linux Consumption function app, host build 4.1052.300.26370:

• 224,951 occurrences over seven days, 81% of all exceptions in the window
• 66.8% of invocations failed on the affected function
• 319 to 547 distinct instances affected per day. Incidence tracked instance count almost linearly, because every cold start re-runs the specialization that triggers this.

On a staging app with the same shape, a diagnostic added at the top of every function module reported the namespace already broken in 4 of 4 cold-started worker processes. On this configuration the condition is not occasional, it is universal. What varies is only whether the module a given worker loads happens to traverse google.api_core.

Suggested fix

In _remove_module_cache(), either purge parent and children together, or skip namespace packages whose __path__ is dynamic. Alternatively, do the sys.path mutation after computing the set of modules to evict, so a _NamespacePath cannot re-acquire the path being removed. Leaving sys.modules with a child but no parent binding is the state that makes the failure possible.

Workaround, for anyone who lands here

Re-bind the submodule before importing any Google client, as the first statement of every function module. A bare import google.protobuf is not enough, because that is precisely the statement that succeeds without binding:

import importlib
import google

if not hasattr(google, 'protobuf'):
    google.protobuf = importlib.import_module('google.protobuf')

Verified in a real environment: with this in place, cold-start bursts produce zero occurrences, and the diagnostic still shows the guard re-binding on every fresh worker. It is repairing the condition, not masking its absence.

A question we cannot answer from outside

The purge is gated behind PYTHON_ISOLATE_WORKER_DEPENDENCIES in the classic worker, and both ..._DEFAULT and ..._DEFAULT_310 are False in 4.45.1 and 4.46.0. Yet our production incident occurred with that app setting absent, where the published gating says prioritize_customer_dependencies should never run. A staging app shows the process-visible value matching its app setting, so we see no injection there.

Does the Linux Consumption placeholder image set PYTHON_ISOLATE_WORKER_DEPENDENCIES at container level, or did host hotfix 4.1052.300.26370 bundle a worker whose gating differs from the tagged sources? Worker system logs are not forwarded to Application Insights, so we cannot tell from our side.

Environment

Worker	4.45.1 (repro), also checked 4.46.0
Host	4.1052.300.26370
Python	3.10
Plan	Linux Consumption (Y1)
Programming model	v1
Relevant packages	grpcio 1.83.0, protobuf 4.25.5, google-api-core 2.20.0, google-cloud-discoveryengine 0.11.8

[pr1te]

Some follow-up measurements that narrow the defect, and one detail suggesting it should not be fixed by special-casing protobuf.

The purge evicts exactly two entries

Instrumenting prioritize_customer_dependencies() against 4.45.1 with the same two trees:

purged google.* modules : ['google', 'google._upb']
api_core still cached?  : True
protobuf still cached?  : True

Everything else survives: google.api_core, google.api_core.grpc_helpers, google.protobuf, google.cloud.*. Only the entries whose __path__ is dynamic are hit, which is consistent with the _NamespacePath re-computation described above.

google._upb is left unbound too, so this is not protobuf-specific

After the purge, hasattr(google, '_upb') is also False. Nothing reads google._upb as an attribute today, because protobuf's C extension is referenced as the already-cached google._upb._message, so it is latent rather than a second live failure. It matters for the fix though: the defect is not "protobuf gets unbound", it is "any child of a purged namespace package is left orphaned". A fix that special-cases google.protobuf would leave the same trap for the next package that reads a namespace sibling as an attribute.

The failure window is first-import only

PROTOBUF_VERSION = google.protobuf.__version__ is module-scope code, and grpc_helpers survives the purge, so its body never re-executes. A later purge cannot re-trigger the read. The failure needs both a broken parent binding and a first-time import of a module that reads the attribute. That is why it presents as a load failure and why it is sticky for the life of the process rather than intermittent within it.

For completeness, google-api-core 2.20.0 has three such reads, all of the same attribute: grpc_helpers.py:31, operations_v1/transports/base.py:36, and operations_v1/transports/rest.py:39.