Here is the short version.
Yes, but only if the FAQ already has the usual problems of a living internal knowledge base: duplicated answers, slightly different wording for the same issue, outdated entries that still rank high on keywords, and questions from teammates that never match the exact title of the article. On a real 340-entry support FAQ used by a six-person team, a hybrid EmbeddingGemma 2 + BM25 retriever reduced the number of “I can’t find it” messages in the team chat and cut the average time-to-correct-article for new hires. Pure keyword search was not enough; pure dense search missed too many exact error codes and product names. The combination was the first version the team actually kept using.
This post describes the before-and-after measurements, the data cleaning that mattered more than the model, the integration points that made the system usable inside an existing Slack-and-browser workflow, and the limits that still remain.
The Starting Point: A Typical Small-Team FAQ
The FAQ lived in a shared Markdown repository and was rendered as a simple internal website. It had grown organically for two years. Typical symptoms:
Multiple articles that answered the same question with slightly different wording
Titles written by the person who solved the issue once, not by the person who would search for it later
Error codes and product SKUs that appeared in the body but not in the title
No reliable way to know which articles were still accurate
New team members defaulting to asking in chat because search felt slower than asking a human
Keyword search (the built-in Git-based search plus browser Ctrl-F) found exact matches quickly and failed on everything else. The team had already tried a hosted semantic-search plugin; they turned it off after two weeks because it sent internal content to a third party and still returned plausible but wrong articles often enough to erode trust.
What We Measured Before Changing Anything
I instrumented the existing workflow for two weeks with the lightest possible logging (local only):
Number of FAQ-related questions asked in the team chat
Whether the asker eventually found an article or received a human answer
Time from question to the correct article (self-reported by volunteers)
Queries that produced zero useful results
Baseline (two-week median):
18 FAQ-related chat questions per week
11 of them resolved by a human rather than by search
Median time-to-correct-article when search succeeded: ~70 seconds
Several recurring questions that had good articles but were never found by the people who needed them
These numbers are small. That is the point. A six-person team does not need a sophisticated ranking research platform; it needs the existing knowledge to be findable by the people who already work there.
The Intervention
We kept the Markdown files as the source of truth and added a local hybrid retrieval service exactly like the one built in the earlier posts:
EmbeddingGemma 2 for dense vectors
BM25 on the same chunks
Simple weighted fusion (alpha 0.6)
Qdrant for persistence
A single internal endpoint that returned the top five articles with title, path, and a short highlight
Two integration points made the difference between “interesting prototype” and “tool the team actually used”:
A /faq slash command in the team chat that posted the top three results back into the thread.
A search bar on the existing FAQ website that called the same local endpoint and rendered results in the familiar layout.
No content left the company network. The service ran on a small internal Linux machine already used for other development tooling.
Data Work That Moved the Numbers
The model alone produced only modest gains. The larger improvements came from three data changes:
Deduplication and canonicalization. Near-duplicate articles were merged or explicitly cross-linked. This reduced the “two almost-right answers” problem that dense retrieval sometimes amplifies.
Error-code and SKU promotion. Distinctive tokens that previously lived only in the body were copied into a lightweight metadata field that both BM25 and the embedding text could see.
Query logging and description editing. We examined the first two weeks of real queries that still failed and edited a few dozen article openings so that the language of the question and the language of the answer sat closer together.
None of these steps required retraining a model. All of them improved the hybrid results more than changing alpha or switching to a larger embedding model would have.
Results After Four Weeks
Metric | Before | After (hybrid) |
|---|---|---|
FAQ-related chat questions / week | 18 | 11 |
Resolved by human rather than search | 11 | 4 |
Median time-to-correct-article | ~70 s | ~35 s |
Queries with zero useful results | frequent | rare |
The reduction in chat questions was the number the team cared about most. People still asked humans when the issue was truly new or politically sensitive; they stopped asking when the answer already existed and was now findable.
Qualitative feedback was consistent: new hires found the FAQ usable within the first few days instead of the first few weeks. Experienced members noticed that older, oddly titled articles started appearing again when they were actually relevant.
Where the Hybrid System Still Failed
Outdated articles that were semantically close.
Dense retrieval happily surfaces an old answer that uses similar language. We mitigated this with a lightweight “last reviewed” date and a visual flag in the result list, but the system cannot know that a policy has changed unless a human updates the document.
Highly specific internal jargon that never appears in the article.
If the team invents a nickname for a process and never writes it down, no retriever will find the official article. The fix is cultural: put the nickname in the document.
Multi-hop questions.
“How do I do X after the new Y process?” still requires the reader to combine two articles. The retriever can surface both; it cannot yet synthesize them.
Very short queries.
Single-word searches remain BM25’s territory. The hybrid respects that by keeping a non-zero BM25 weight.
Operational Reality for a Six-Person Team
The service has been running for several months with roughly one hour of maintenance per month (mostly adding new articles and re-embedding).
Full re-index of the 340 articles takes under two minutes on ordinary hardware.
No one has requested a cloud migration. Privacy and the absence of per-query cost remain valued.
The same evaluation discipline used in the Retrieval Bench posts (small held-out query set, manual inspection of failures) is what keeps the system from quietly degrading.
Practical Advice If You Try the Same Thing
Start with the existing FAQ files. Do not rebuild the knowledge base from scratch.
Instrument the current pain for at least a week before you change anything. Small teams often overestimate how broken search is and underestimate how broken the documents are.
Make the new search appear inside the tools people already open (chat, existing FAQ site). A separate search page is usually ignored.
Budget time for data cleaning. It will produce larger gains than model swapping.
Keep a short “known hard queries” list and re-test them after every change.
Treat the hybrid service as infrastructure that must survive the person who built it. Document the exact model revision, alpha, and chunk settings.
Who Benefits Most
This pattern works well for teams that:
Already maintain an internal FAQ or wiki in text-friendly formats
Are small enough that informal chat is the current fallback
Care about keeping internal content internal
Can tolerate a simple local or self-hosted service
It is less necessary when the team already has strong keyword search discipline and consistently titled articles, or when the knowledge base is so large and formal that a heavier enterprise platform is already justified.
Closing
EmbeddingGemma 2 can improve a small-team FAQ workflow, but it does so most effectively as part of a hybrid system and only after the obvious data problems are addressed. The measurable gains on this six-person team were fewer chat interruptions, faster time to the correct article, and better onboarding for new hires. The system stayed private, inexpensive, and simple enough that it is still running without heroic effort.
The same stack that powers personal document search and a children’s book library also works for internal knowledge. The differences are mainly in integration surface and the social habits around keeping articles up to date.
The next post looks at what happens when a weekend homelab project is asked to become a maintainable tool that other people rely on. After that we close the launch series with the lessons learned when real, messy documents replace the clean evaluation sets we started with.
Until then, if your team’s FAQ is already the place answers are supposed to live, make those answers findable by the people who need them—without sending the content to a third party. The hybrid pipeline you already know how to run is usually sufficient.
The benchmark is the conversation starter, not the verdict.
No feedback yet — submit the first.