Back to Enloq

Enloq, my own project, measured on 27 and 29 October 2026

How Ask this group finds and checks an answer.

Ask this group is Enloq's retrieval-augmented generation (RAG) feature. This page shows how it retrieves, how it checks its citations, what was measured and what wasn't.

The result, and its limits.

What it doesAnswers a member's question from that group's own cached messages, as up to four statements with source links. When the messages don't settle a question, it says so.
Measured on100 authored questions over six synthetic groups. Not real groups or live traffic.
Result78 of the 80 answerable questions correct, graded by a model against authored answers.
Tool calling78 against 73 without tools, and none lost. That gain isn't statistically significant.
Not shownProduction use, adoption or quality on real groups. Enloq is a personal deployment, shared with a few approved members.

What Ask does.

A daily briefing summarizes one day. Finding one answer across a group's recent days still means scrolling. Ask this group answers a member's question from that group's own messages, in up to four short statements. Each statement links to the messages it rests on. A link opens the group at that message, and Back returns to the answer.

It searches only what Enloq already caches for that group: text, captions, voice transcripts and image descriptions, from the group's first opt-in day, within a 30-day cache. Hidden and ignored bots, flagged ads and service events stay out. It reads no Telegram history, no other chat and no web. Enloq's database keeps neither the question nor the answer: they exist only on the open screen, and receipts keep usage without content.

How an answer is made.

A question goes through these steps. The first three are the fixed pass.

StepWhat happens
RankThe question's embedding ranks the group's messages by similarity.
RerankA reranker reorders the first 40. If reranking fails, the embeddings' order stands.
ReadThe writer model, gpt-6.1-sol, reads the best 25.
Search furtherIf those don't settle the question, the writer calls its tools.
WriteIt writes its statements, citing only messages it has read. Any other ID fails validation.
CheckA decision model checks each statement against its citations. Nothing streams: an answer shows once checked.

Embeddings, picked by measurement.

The measurement ran on 27 October 2026 over six synthetic groups, 665 messages in Russian, English and both. It used 100 authored questions, labelled before any run. 80 are answerable, 11 of those asked in another language than the group's, and 20 aren't.

Recall at k is the share of the relevant messages in the first k. MRR is the mean reciprocal rank of the first relevant one. nDCG at 10 rewards relevant messages ranked early.

RetrievalRecall at 5at 10at 25MRRnDCG at 10
Keywords alone (FTS5 trigram, BM25)0.530.630.670.380.41
Qwen3 Embedding 8B0.800.850.920.630.66
Perplexity Embed V1 4B (adopted)0.800.890.920.660.69
bge-m30.760.870.940.580.62
Qwen3 Embedding 8B with keywords0.750.810.880.610.63
Perplexity Embed V1 4B with keywords0.750.830.870.620.63
bge-m3 with keywords0.730.810.900.580.60

Perplexity Embed V1 4B ranked best on MRR and nDCG and was the fastest on these routes. One request of up to 64 messages took 1.3 seconds at the median, against 2.4 for bge-m3 and 9.2 for Qwen3 at Nebius. bge-m3 found slightly more by recall at 25.

The reranker stayed. Keyword search didn't.

Fusing keywords with the embeddings by reciprocal rank lowered every embedding model's ranking. Across languages, keywords found almost nothing: recall at 25 of 0.05 on those 11 questions, where the fusion reached 0.95. The final run compared the reranker's two possible lists.

CandidatesRecall at 40 before rerankingnDCG at 10 beforeRecall at 10 afternDCG at 10 after
The embeddings' first 40 (adopted)0.950.690.930.78
The keyword fusion's first 400.940.630.910.77

Reranking lifted nDCG at 10 from 0.69 to 0.78. After it the two lists rank about alike: one rerank in that run fell back after a provider refused it, and one question moves these means by up to 0.0125. Before it the embeddings' list is better. So the application reranks the embeddings' list and leaves keyword search out.

The reranker is Qwen3 Reranker 8B at Fireworks, the one zero-retention reranker listed when it was chosen. Voyage's rerank-3 ranked alike on the fused list, but on 27 October its terms allowed training on submitted content unless the customer opted out. So it ran on authored messages only and never receives real group messages.

When the fixed pass doesn't settle it.

After the 25, the writer can call three tools. It can search again in its own words, for the best 10. It can read up to five messages on each side of one. And it can read a message with the one it replies to and up to ten replies. The tools read only this group's searchable messages, and their results are framed as untrusted data, like the messages. Per question the writer gets at most five requests, four searches and twelve tool calls, and the fifth request must answer.

On 29 October 2026, each of the 100 questions was answered three ways through the application's own steps: the fixed pass, the tools alone with no messages to start with, and the fixed pass then the tools. One judge request graded the three answers together against the authored answer, in a shuffled order. The rule was fixed before the run: a way qualifies with at least two more answerable questions correct than the fixed pass, no fewer unanswerable ones and no more statements judged unsupported. The qualifying way with the most correct answers is adopted if its median time is at most twice the fixed pass's.

ResultFixed passTools aloneFixed pass, then tools
Answerable: correct73 of 80 (82.8% to 96.4%)74 of 80 (84.4% to 97.2%)78 of 80 (91.3% to 99.7%)
Unanswer­able: correct20 of 2020 of 2020 of 20
Unanswer­able: said no answer3 of 203 of 203 of 20
Statements judged unsupported4 of 1352 of 1242 of 127
Seconds: median (95th)6.5 (12.8)13.8 (25.7)7.7 (22.4)
Relevant messages read0.950.960.99
OutcomeThe baselineDidn't qualifyAdopted

Ranges are two-sided 95% exact (Clopper-Pearson). Relevant messages read is the mean share of a question's labelled messages that its writer read, over the 80 answerable questions.

The fixed pass then the tools met the rule and was adopted. It answered five questions the fixed pass missed and lost none. An exact two-sided McNemar test gives p = 0.0625. So it met the practical bar set before the run, not significance at 5% on 80 questions. In those five, the message that settles the question wasn't among the 25: for example, a correction posted right after an instruction, or the reply claiming a lost multimeter. Tools alone didn't qualify, and the application doesn't use that way.

The price is waiting. A question took 17.9 seconds at the median when the writer called tools, against 6.7 when it didn't. Averaged over all 100 questions, the adopted way cost about $0.0014 on the metered routes, against $0.0009 for the fixed pass. That leaves out the writer, which runs on my ChatGPT subscription. Vectors were computed before the run, so these figures also leave out a group's first question, which pays for its messages' vectors.

The citation check.

The check is a decision model, Perplexity's Decider. It answers fixed yes-or-no questions with probabilities and writes no text. For each statement it's asked whether each cited message supports part of it, and whether together they support all of it.

A citation below 0.5 is removed, and a statement left with none is dropped. A statement its citations don't fully support shows a muted line: "Not all of this is in its sources." If the check can't run, no answer shows, only "Couldn't check the answer."

On 70 authored claims, labelled before the runCount (95% range)
Supporting citations removed8 of 93 (3.8% to 16.3%)
Unsupported citations kept1 of 69 (0.04% to 7.8%)
Correct claims judged unsupported1 of 32 (0.08% to 16.2%)
Wrong claims judged supported3 of 38 (1.7% to 21.4%)

So a wrong claim can get through. In the tool-calling run the check dropped no statement and marked 6 of the adopted way's 127 as not fully supported. The judge found 2 of those 127 unsupported. One added "that date has passed" to a power cut, which the messages don't say.

When the messages hold no answer.

If the group has nothing to search, no model is called. If the writer returns no statements, or no statement survives the check, the answer is one line that names the searched range: "No answer in this group's messages from 3 Oct to today."

Edited and deleted messages.

A message's vector is computed when a question first needs it, and kept per message revision with the 30-day cache. An edit or a deletion drops it. An edited message is embedded again the next time a question needs it, and a deleted one leaves the search.

Where the messages go.

The embedding and rerank requests go through OpenRouter. Each is pinned to one provider with no fallback and requires zero data retention: Perplexity for embeddings and Fireworks for reranking. That rests on how OpenRouter lists those routes.

Only those two requests require zero retention. The writer runs on my ChatGPT subscription, so messages reach OpenAI under that plan's data settings. The check goes to Perplexity's decision model. On 27 October, Perplexity published no retention terms for it, and OpenRouter listed it as zero retention. My own proxy logs the check's requests, which hold each statement and its cited messages, for up to 30 days.

What this doesn't show.

These are authored questions over synthetic groups, not real groups or live traffic. Each group holds about 110 messages, so the writer's 25 are nearly a quarter of it. A busy group's 30 days can hold far more, and nothing here measures retrieval or waiting time there.

The same questions chose each component, so 78 of 80 isn't a held-out estimate. A model graded the answers, the same one that wrote them, so the grades aren't an independent human rating. The samples are small, as the ranges show, and the times and costs are indicative, not a benchmark. Tool calling's gain isn't statistically significant.

This isn't production use. Enloq is a personal deployment, shared with a few approved members, and nothing here measures how they use Ask.

Evidence basis.

This account is based on Enloq's public decision records, measurements and code. The authored groups, questions and labels are public in the repository, and the measurements list the tool-calling run's misses. The run reports, which hold each model's answers, are private.

Privacy