SeterusnyaPanduan seterusnya
Metrik Penilaian ROUGE dan BLEU
Teknikal
PANDUAN Teknikal
RAG evaluation scores a retrieval-augmented generation system separately on retrieval and on generation, most commonly with four metrics: faithfulness, answer relevance, context precision and context recall.
Dengar panduan iniAI-diceritakan · ~3 min
RAGAS is an open-source framework that computes these metrics, largely using an LLM as a judge. Separate scores matter because they show whether a bad answer came from fetching the wrong passages or from misusing the right ones.
A RAG system can fail in two places: retrieval can fetch the wrong passages, or generation can ignore or distort the right ones. A single overall quality score hides which happened, so RAG evaluation splits the pipeline into measurable parts. RAGAS, an open-source framework introduced in a 2023 paper by Es and colleagues, popularized a set of core metrics that cover both halves. Faithfulness asks whether every claim in the answer is supported by the retrieved context; it catches hallucination even when the answer sounds plausible. Answer relevance asks whether the answer actually addresses the question, penalizing evasive or padded replies. Context precision asks whether the retrieved chunks that matter are ranked near the top rather than buried among noise. Context recall asks whether the retrieved context contains everything needed to produce a reference answer, so it requires a ground-truth answer for each test question. Most of these scores are computed by an LLM acting as a judge, broken into small steps. For faithfulness, the judge extracts individual statements from the answer and checks each against the context; the score is the fraction supported. For answer relevance, RAGAS generates questions that the answer would respond to and compares them with the original question using embeddings. Similar ideas appear in other tools, such as the TruLens RAG triad (context relevance, groundedness and answer relevance) and DeepEval. Two misconceptions are common. First, faithful does not mean correct: an answer can faithfully repeat an outdated or wrong document. Second, LLM-judge scores are not ground truth. They vary with the judge model and prompt, so teams spot-check them against human labels and compare systems under identical settings rather than treating a score like 0.85 as an absolute grade.
A bank's policy assistant scores 0.95 on context recall but 0.70 on faithfulness, which tells the team retrieval is fine and the model is adding unsupported details, so they tighten the prompt rather than rebuild the index.
A support team compares two embedding models on the same 200-question test set and keeps the one with higher context recall and context precision.
A hospital knowledge tool runs faithfulness scoring on a daily sample of real questions and flags answers with unsupported claims for human review before the pattern spreads.
A startup adds a reranker after seeing high context recall but low context precision, meaning the right chunks were retrieved but ranked below irrelevant ones.
RAG evaluation is shifting from occasional offline reports toward continuous checks, where a sample of production traffic is scored and regressions trigger alerts. Frameworks have been adding metrics for multi-turn conversations, agents and tool use, and metric definitions change between library versions, so pinning versions matters for comparable results. The largest open issue is judge reliability: LLM judges can show biases such as favoring longer answers, and agreement with human reviewers varies by domain. Calibrating judges against small human-labeled sets and pairing them with deterministic retrieval metrics is likely to remain good practice.
Newsletter percuma
Tiga cerita AI yang disahkan setiap pagi hari bekerja, ditulis dalam Bahasa Inggeris biasa. Percuma selamanya, tiada iklan.
Satu e-mel setiap hari bekerja. Berhenti melanggan dalam satu klik. Kami tidak pernah menjual atau berkongsi alamat anda.
Uji diri anda
Maklum balas segera pada setiap jawapan, dan sijil yang boleh dikongsi dengan ID yang boleh disahkan sebaik sahaja anda lulus kursus.
Sokong pendidikan AI percuma. AI Understanding ialah pertubuhan bukan keuntungan 501(c)(3) — tiada iklan, tiada bayaran dinding, selamanya. Buat derma
RAG evaluation scores a retrieval-augmented generation system separately on retrieval and on generation, most commonly with four metrics: faithfulness, answer relevance, context precision and context recall. RAGAS is an open-source framework that computes these metrics, largely using an LLM as a judge. Separate scores matter because they show whether a bad answer came from fetching the wrong passages or from misusing the right ones.
Faithfulness measures the fraction of answer claims backed by the context, which targets hallucination.
Context recall checks whether the retrieved context contains what is needed for the reference answer, so a reference must exist.
Context precision rewards putting useful chunks high in the ranking instead of burying them among irrelevant ones.
Jika soalan yang dijana daripada jawapan menyerupai soalan sebenar, jawapan itu relevan; embeddings mengukur kesamaan tersebut.
Chunk yang betul sedang diambil tetapi terletak rendah, yang merupakan masalah ranking yang biasanya diatasi oleh reranker.
Teruskan belajar
Lebih banyak panduan dipilih untuk topik ini
SeterusnyaPanduan seterusnya
Metrik Penilaian ROUGE dan BLEU
Teknikal