Is your feature request related to a problem?
An uppercase file extension (e.g. report.PDF) leads to issues when uploading to OpenAI's vector store, as it can result in mismatches during file searches. This could cause a single document failure to roll back the entire collection job.
Describe the solution you'd like
- Ensure the file extension is lowercased before being sent to OpenAI's upload function.
- Maintain the original case of the filename for downloading purposes.
- Prevent the failure of one document from affecting the entire collection job during indexing.
Original issue
Describe the bug
A knowledge base document whose filename carries an uppercase extension (e.g. report.PDF) is accepted at upload but sends an uppercase suffix to OpenAI when its collection is built.
get_file_format lowercases the suffix for validation (backend/app/services/doctransform/registry.py:63), so the upload succeeds.fname is stored with its original case, since it drives the download attachment name.OpenAIProvider.upload_files then passed doc.fname verbatim to client.files.create, so the uppercase suffix reached the provider.
OpenAI file search may not match an uppercase suffix in every ingestion path. Because any failed file in a vector store batch is a hard failure that rolls back the whole vector store (see docs/wiki/modules/knowledge-base.md), a single such document can take an entire collection job down rather than failing on its own.
To Reproduce
- Upload a knowledge base document named with an uppercase extension, e.g.
report.PDF.
- Create a collection over that document.
- The name sent to
files.create carries the uppercase suffix, unlike the equivalent lowercase document.
Expected behavior
Extension casing in the stored filename should not influence vector store indexing. The name sent to OpenAI should carry a lowercased extension, while fname keeps its original case so downloads are unaffected.
Additional context
- Affected code:
backend/app/services/collections/providers/openai.py, OpenAIProvider.upload_files
- The blast radius is the whole collection job, not the single document, because of the batch rollback semantics noted above.
- The mechanism is stated as a possibility rather than a confirmed provider rejection: normalizing the case is a cheap invariant either way.
Is your feature request related to a problem?
An uppercase file extension (e.g.
report.PDF) leads to issues when uploading to OpenAI's vector store, as it can result in mismatches during file searches. This could cause a single document failure to roll back the entire collection job.Describe the solution you'd like
Original issue
Describe the bug
A knowledge base document whose filename carries an uppercase extension (e.g.
report.PDF) is accepted at upload but sends an uppercase suffix to OpenAI when its collection is built.get_file_formatlowercases the suffix for validation (backend/app/services/doctransform/registry.py:63), so the upload succeeds.fnameis stored with its original case, since it drives the download attachment name.OpenAIProvider.upload_filesthen passeddoc.fnameverbatim toclient.files.create, so the uppercase suffix reached the provider.OpenAI file search may not match an uppercase suffix in every ingestion path. Because any failed file in a vector store batch is a hard failure that rolls back the whole vector store (see
docs/wiki/modules/knowledge-base.md), a single such document can take an entire collection job down rather than failing on its own.To Reproduce
report.PDF.files.createcarries the uppercase suffix, unlike the equivalent lowercase document.Expected behavior
Extension casing in the stored filename should not influence vector store indexing. The name sent to OpenAI should carry a lowercased extension, while
fnamekeeps its original case so downloads are unaffected.Additional context
backend/app/services/collections/providers/openai.py,OpenAIProvider.upload_files