iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
After updating an n8n chatbot’s knowledge base, rerun a saved set of representative questions and compare the answers with expected facts. Include questions whose answers should change, stable controls that should not, and edge cases where the bot should acknowledge uncertainty. For a retrieval-augmented generation (RAG) workflow, check ingestion and retrieved documents as well as the final response.
Build a repeatable set of questions
Record which source documents changed, their versions or dates, and what answers you expect to change. Pair questions about those changes with unaffected control questions. This makes it easier to see both whether the update worked and whether it disturbed answers that should remain stable.
For each case, save the question and an expected answer or list of key facts. Add an expected source or citation when that matters. Use varied phrasings, production-like edge cases, and questions the knowledge base cannot answer; for those, define what an appropriate fallback or expression of uncertainty looks like.
Free tools Windows power users keep installed
One-click scans. No signup required.
n8n’s evaluation documentation describes datasets of sample inputs and often expected outputs. It recommends representative inputs that reflect production edge cases, rather than relying only on easy examples. There is no universal test-set size or passing threshold: set both according to the chatbot’s risk and use.
Check that the updated material is available to the chatbot
If the chatbot uses RAG, an incorrect answer may begin before generation: the updated content may not have been loaded, or retrieval may return an irrelevant or outdated chunk. Verify that the updated source entered the vector store and that the retrieved chunks contain the current information.
n8n’s vector database documentation describes loading documents, splitting them with a data loader, embedding and storing them, and querying through an agent or vector-store node. Check the document metadata and splitting as well as the retrieval result. Use the same embedding model for insertion and querying.
Chunk size depends on the source material and the task. n8n gives 200–500 tokens as an example range for fine-grained retrieval; treat it as a starting point to test on your corpus, not a universal setting. Larger chunks can preserve more context but may also add irrelevant material.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Choose a practical evaluation method
A few development cases: light evaluation
For a small set of questions while developing, run the cases and inspect the answers visually against your references. This is a useful quick check, but it will not provide the aggregate trends that become valuable across a larger set of cases.
Rank #3
- Conversational AI with Rasa: Build, test, and deploy AIpowered, enterprisegrade virtual assistants and chatbots
- ABIS BOOK
- Packt Publishing
Larger datasets: metric-based evaluation
For broader datasets, use metric-based evaluations to score cases and compare runs. n8n documents data tables or Google Sheets as dataset sources, along with score summaries and case-level execution review. The documentation lists availability for n8n Cloud Pro and Enterprise and self-hosted Enterprise; it also says registered Community and Starter users can use the feature for a single workflow. Check current access for your plan because product availability can change.
Metrics can add latency and cost. Keep the test set and scoring method consistent between runs so score changes are interpretable. n8n notes that metrics may be deterministic or AI-calculated and can be compared across runs.
Rank #4
Compare the answers on the dimensions that matter
- Correctness: Are the supported facts consistent with the reference? n8n documents an AI-based correctness metric on a 1-to-5 scale.
- Coverage and helpfulness: Did the answer address the full question? n8n documents an AI-based helpfulness metric on a 1-to-5 scale.
- Retrieval relevance: Did the workflow retrieve the updated, relevant material? RAG document relevance is one example of a custom metric.
- Unsupported claims and abstention: Did the chatbot avoid inventing details missing from the knowledge base, and respond appropriately when it lacked support? Define a human-review rubric or custom metric for this; the documented built-in metrics do not by themselves establish that unsupported claims are absent.
- Operational behavior: Did the workflow complete and use the expected tools? n8n documents a tools-used metric, and execution history can be filtered by status and workflow.
String similarity or an exact match can help with particular tasks, but wording similarity alone does not prove factual correctness: a correct answer may use different words, while a similar-sounding answer can still be wrong. Treat automated scores as signals, not substitutes for reviewing high-impact factual failures.
Investigate failures before changing the baseline
Open failed or materially changed cases and inspect both the answer and its execution. n8n lets you open test-case execution details from evaluation results. For workflow troubleshooting, use the Executions view to filter by workflow, status, and start time, and load prior execution data when debugging.
Trace the failure to its cause before deciding what to change:
- Missing or stale content: Confirm that the updated document was loaded into the store.
- Irrelevant retrieval: Inspect the retrieved chunks, metadata, and splitting behavior.
- Answer-generation change: Check whether prompt or model behavior changed alongside the knowledge base.
- Outdated reference: Revise the expected answer only when the authoritative knowledge itself has changed.
After fixing the cause, rerun the full set rather than only the failed question. Add real production failures as new cases so they can help catch regressions in later updates.
Preserve the dataset and results outside execution history
Execution history is useful for debugging, but n8n states that deleting a workflow also deletes its execution history. Keep the evaluation dataset and any reports you need as a durable audit record separate from that history. When a run passes your team’s risk-based threshold, preserve its results as the comparison baseline for the next knowledge-base change.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
