S

01 / 04 JJRAG · Jul 2026

A retrieval chatbot for Jiu Jitsu rules.

Competition rules for Jiu Jitsu live in three rulebooks (IBJJF, ADCC, AJP) that disagree with each other and change every season. JJRAG answers a rules question with the section it came from, so the answer can be checked against the book rather than trusted.

Role
Sole author
When
Jul 2026
Stack
Python, MiniLM, LoRA, FastAPI
Result
[headline number, e.g. recall@5 or hallucination rate]
Repositorygithub.com/AreedAdmin/[jjrag]
  • Demo: [url]
  • Write-up: [url or none]

01 — Problem

Three rulebooks, one referee, no time.

A competitor, coach or referee needs a ruling in seconds: is this grip legal at this belt, does this position score, how long is the penalty. The answer depends on which federation is running the event, and the three rulebooks are long, structured differently, and revised often.

A general chatbot answers fluently and is often wrong, because it has no way to say which book and which section it is quoting. The requirement was an answer that names its source, and a measured hallucination rate rather than an impression of one.

02 — Method

Retrieve the section, then answer from it.

Chunking follows the structure. Each rulebook is split along its own headings and numbered sections rather than by fixed token windows, so a chunk is a rule, not the second half of one. Each chunk carries its book, section number and title as metadata.

Embedding and retrieval. Chunks are embedded with MiniLM. A query is embedded the same way and the top-k chunks by cosine similarity are retrieved. The retriever is then LoRA fine-tuned with a multiple-negatives ranking loss on question–section pairs, which pulls a question's true section ahead of the near-misses from the other two books.

A semantic cache, swept over τ. Queries repeat with different wording. A new query whose cosine similarity to a cached one exceeds τ is served from the cache. τ was swept to find the threshold where hit rate rises without wrong-answer rate rising with it.

Citation-grounded generation. The model answers only from the retrieved chunks and must cite section numbers. Answers whose citations do not resolve to a retrieved chunk are flagged.

Evaluation. An LLM judge scores answers for correctness and grounding. The judge was validated against human labels with Cohen's κ before its scores were used, and the hallucination rate is reported from it.

0.92

Fig. III aTop-k retrieval by cosine on a ring of chunk embeddings. The arcs are the semantic cache at the τ set above.

03 — Results

Measured, not felt.

[0.00]recall@5, fine-tuned retriever
[0.00]recall@5, base MiniLM
[0%]cache hit rate at chosen τ
[0.00]Cohen's κ, judge vs human
[0%]hallucination rate

[One paragraph: what moved when the retriever was fine-tuned, what the τ sweep looked like, and where the judge and the humans disagreed.]

04 — Stack

What it is made of.

01 Interface 02 Generation 03 Retrieval 04 Embedding 05 Data 06 Serving QUERY VECTOR MISS HIT CHUNKS CITED SCORE SECTIONS WEIGHTS QUESTION CITED ANSWER LLM HALLUCINATION GATE LLM JUDGE · κ SEMANTIC CACHE · τ TOP-K COSINE LORA RETRIEVER MINILM SECTION CHUNKER IBJJF ADCC AJP FASTAPI HOST 01 Interface 02 Generation 03 Retrieval 04 Embedding 05 Data 06 Serving QUESTION CITED ANSWER LLM HALLUCINATION GATE LLM JUDGE · κ SEMANTIC CACHE · τ TOP-K COSINE LORA RETRIEVER MINILM SECTION CHUNKER IBJJF ADCC AJP FASTAPI HOST

System design6 layers, 13 flows. Arrows are the data path; dashed ones are side channels. Hover a layer.