iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Pen-testing a GenAI application means testing the entire system that turns input into an answer or action—not just sending jailbreak prompts to a chat endpoint. A useful engagement covers the model and prompts, retrieval and vector permissions, output handling, tools, identity boundaries, infrastructure, and runtime controls. The practical goal is to prove whether an attacker can cross a data or action boundary, then produce reproducible evidence and a regression test.
This guide uses OWASP’s four-part structure—model evaluation, implementation testing, infrastructure assessment, and runtime behavior analysis—and applies it to direct and indirect prompt injection, RAG data leakage, unsafe output handling, excessive agency, system-prompt leakage, vector weaknesses, and unbounded consumption.
What exactly are you pentesting in an LLM application?
You are testing an application stack, not a model in isolation. A model can produce an unsafe suggestion without creating a security impact; the impact appears when application code, permissions, retrieval, or tools give that output authority.
Map the complete attack surface
- Model layer: provider, model and version, temperature or equivalent controls, context window, safety settings, fallback models, and multimodal inputs.
- Instruction layer: system and developer prompts, user templates, memory, conversation summaries, routing prompts, and hidden policy text.
- Knowledge layer: parsers, loaders, chunkers, embeddings, vector stores, metadata filters, hybrid search, rerankers, citation builders, and cache layers.
- Action layer: function calls, plugins, HTTP clients, databases, shells, file operations, email, ticketing, payment, and other write-capable connectors.
- Identity and data layer: users, roles, tenants, document ACLs, secrets, service accounts, network paths, and outbound destinations.
- Operations layer: rate limits, token and cost quotas, logging, tracing, alerting, retries, timeouts, rollback, and incident response.
Documents, web pages, emails, retrieved chunks, tool results, and uploaded files are all untrusted input channels. Include them in the threat model even when users cannot edit the system prompt.
#1 Best Overall
Define success before sending payloads
Write down the boundaries that must not be crossed: a tenant must not read another tenant’s records; a user must not invoke an administrative tool; a model output must not become executable SQL or shell syntax; and a retrieval payload must not trigger a consequential write without authorization and confirmation. Also define permitted actions, rate and cost limits, logging requirements, and stop conditions for destructive tests.
How do you build a threat model and scope for the engagement?
Start with an inventory and a set of abuse cases tied to real identities and data. A prompt-only test is incomplete until you know what the application can reach and which account performs the downstream operation.
- Inventory components and trust boundaries. Record every model provider, prompt template, retriever, index, parser, tool, connector, secret, service account, and write path. Draw where data changes from untrusted to trusted and where authorization is enforced.
- Enumerate principals. Create test identities for anonymous users, ordinary users, managers, administrators, service accounts, and separate tenants. Include users with access to overlapping and non-overlapping datasets.
- List assets and prohibited outcomes. Include personal data, confidential documents, system prompts, credentials, billing records, source code, outbound network access, and destructive operations. State the maximum data and action exposure that is acceptable.
- Model attack channels. Cover direct chat input, uploaded files, indexed documents, web pages, email, tool output, memory, error messages, logs, and protocol or API requests. Treat each as a place an attacker may plant instructions.
- Agree on controls and stop rules. Set quotas, test windows, sandbox accounts, canary records, confirmation requirements, rollback procedures, and contacts for an observed real-world side effect.
What are OWASP’s four assessment areas?
OWASP separates LLM security testing into four complementary areas. Use all four; a model evaluation cannot reveal a stale tenant filter, and an infrastructure review cannot show that a retrieved instruction causes a tool call.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11| Assessment area | What to examine | Typical evidence |
|---|---|---|
| Model evaluation | Instruction following, refusal behavior, jailbreak resistance, multilingual and encoded input, context-boundary handling, and leakage of hidden instructions | Exact prompt, model/version, settings, context, output, and repeatability |
| Implementation testing | Prompt construction, output parsing and escaping, authorization, memory, retries, fallbacks, schemas, and error paths | Request/response traces, parser behavior, authorization decisions, and safe or unsafe sink reachability |
| Infrastructure assessment | Identity, network paths, vector and embedding services, storage, secrets, isolation, API exposure, and operational limits | Role and tenant test results, configuration evidence, access logs, and quota behavior |
| Runtime behavior analysis | Live retrieval, tool calls, loops, latency, token use, cost, anomalous destinations, and destructive actions | Correlated traces, alerts, side-effect records, and rollback results |
OWASP notes that applications built on large language models fail in ways that SAST, DAST, and SCA were not designed to find. Traditional scanning remains useful for the surrounding code and infrastructure, but it does not replace behavioral testing.
How do you test the model and prompt layer?
Use a controlled matrix of benign canaries and adversarial variations, then verify whether any apparent bypass reaches protected data or an action. Record the exact input and conditions so another tester can reproduce it.
Direct prompt-injection cases
- Ask the assistant to ignore prior instructions, change its role, or reveal hidden policy text.
- Try role confusion, nested quotations, long context, multilingual wording, homoglyphs, Base64 or other encodings, and instructions split across messages.
- Request a sensitive canary value that is present only in a protected system prompt or test record. Use a unique, non-production value so disclosure is unambiguous.
- Attempt policy evasion through fictional framing, translation, continuation requests, or tool-result impersonation.
- Repeat tests with different temperatures, fallback models, conversation memory states, and truncation boundaries.
What to record for every case
- Input messages and any attached files or retrieved context.
- Model name and version, sampling settings, routing decision, and timestamp.
- System, developer, and user message hashes or approved redacted copies.
- Output, refusal, tool proposal, actual tool call, and side effects.
- Whether the result reproduces across clean sessions and whether a lower-privilege user can obtain it.
Do not label a prompt-only response an application compromise until you verify permissions and downstream effects. A model may repeat a secret-looking string supplied by the tester without having accessed a protected asset.
Rank #2
- Used Book in Good Condition
How do you test implementation and unsafe output handling?
Follow model output to every consumer. The key question is not whether text looks dangerous, but whether the application inserts it into a sensitive interpreter without contextual validation and authorization.
Recommended Free Tools
| Output destination | Tests | Expected control |
|---|---|---|
| Browser or template | Markup, script-like text, URL schemes, and attribute-bound output | Contextual escaping, sanitization, content security policy, and safe URL handling |
| SQL or search filter | Quotes, operators, wildcards, and malformed structured values | Parameterized queries, typed builders, and server-side authorization |
| Shell or process runner | Separators, substitutions, paths, and unexpected arguments | No shell interpolation, strict argument allowlists, and a restricted service account |
| HTTP client | Unexpected schemes, internal addresses, redirects, and header values | Destination allowlists, network egress controls, redirect validation, and timeouts |
| File or object store | Traversal markers, absolute paths, oversized content, and type confusion | Canonicalized paths, sandboxing, size limits, and content-type validation |
| Structured tool parser | Extra fields, wrong types, duplicate keys, nulls, and schema drift | Strict schema validation, rejection of unknown fields, and independent authorization |
Exercise retry logic, fallback models, memory updates, logging, and error paths. An error message can disclose a prompt or credential, while a retry can duplicate a write. Validate tool parameters again in application code; a persuasive model instruction must never grant access.
How do you pentest a RAG chatbot?
Test retrieval authorization independently from generation. A chatbot can produce a perfectly formatted answer while leaking another tenant’s chunk because the index query, metadata filter, cache, or citation layer ignored the user identity.
Test retrieval scope for every identity
- Create records with unmistakable canary values in each tenant and access tier.
- Query the index or retriever directly as each test identity, bypassing the chat interface where possible.
- Repeat through the normal application path with exact-match, semantic, hybrid, reranked, and citation requests.
- Change filters, sort fields, chunk sizes, pagination, case, Unicode normalization, and missing or duplicated tenant metadata.
- Check caches, asynchronous ingestion, deleted documents, stale ACLs, and replicas for residual access.
Retrieval authorization is a separate control from tool-call validation. A tool may be correctly restricted while the assistant still discloses unauthorized text in its answer.
Test poisoning and parser abuse
- Upload documents containing plausible instructions aimed at the assistant, including instructions placed in footers, tables, metadata, OCR text, and quoted correspondence.
- Use near-duplicate and conflicting documents to test which version is indexed and cited.
- Exercise malformed files, deeply nested markup, decompression expansion, oversized documents, unusual encodings, and parser error handling within agreed limits.
- Test whether deleted or reclassified material remains searchable through embeddings, caches, summaries, or conversation memory.
Make indirect injection realistic
Do not rely only on a malicious chunk engineered to rank first. Place payloads in documents that are plausible but not guaranteed to be top-ranked, then vary chunk boundaries, hybrid search weights, reranking, query wording, and citation requirements. Scan retrieved chunks for injection patterns before inserting them into the model context, while recognizing that scanning is an additional signal rather than an authorization mechanism.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Can an injected document make an agent call a tool?
It can, if retrieved text is treated as an instruction and the agent has a reachable tool with insufficiently independent controls. Prove the complete chain in a sandbox: corpus content to retrieval, context insertion, model decision, tool request, authorization, and side effect.
- Plant a benign canary instruction in a test document, such as asking the agent to request a non-destructive “list status” operation.
- Ask a low-privilege user a question that causes the document to be retrieved, and capture the full trace.
- Check whether the model merely quotes the text, proposes a tool call, or actually invokes the tool.
- Verify the tool service evaluates the real user and tenant identity, not only the model-supplied role or document text.
- Repeat with the document ranked lower, split across chunks, returned by a reranker, or delivered as a tool result.
- Confirm that consequential operations require explicit user confirmation and that the canary action can be rolled back.
Findings are materially different: an instruction echoed in an answer is an injection weakness; an unauthorized read is a data-boundary failure; and an unconfirmed write is excessive agency with a demonstrated business impact.
How should you test agent tools and runtime behavior?
For each tool, test both the model’s ability to request it and the service’s ability to reject an unauthorized or unsafe request. Runtime traces should show who initiated an action, what data influenced it, and which policy decision allowed it.
Tool and protocol cases
- Parameter tampering: change IDs, tenant fields, recipients, amounts, paths, filters, and pagination values; add unknown fields and duplicate keys.
- Confused deputy: ask a low-privilege user to make a highly privileged connector perform an action on their behalf.
- Privilege escalation: replace user or role claims in model arguments, metadata, headers, and conversation state.
- Secret exposure: request credentials through arguments, errors, logs, traces, prompt context, or tool results.
- SSRF and injection paths: test destination allowlists, redirects, internal hostnames, and content returned from remote systems.
- Replay and duplication: resend tool requests, interrupt after an acknowledgement, and trigger retries around a write.
- Loops and exhaustion: create plans that repeatedly search, call tools, or expand context until token, latency, or cost limits are reached.
- Destructive actions: verify dry-run modes, confirmation gates, idempotency keys, rollback, and separation of read and write credentials.
Runtime signals worth monitoring
Alert on unusual retrieval volume, cross-tenant lookup attempts, new outbound destinations, repeated tool failures, sudden token or latency growth, high-cost loops, abnormal error rates, and action sequences that differ from a user’s normal workflow. Keep enough correlated trace data to reconstruct the prompt, retrieved sources, decision, authorization result, and side effect without storing unnecessary sensitive content.
Which OWASP risk categories should findings map to?
Use OWASP’s LLM risk taxonomy to give each issue a common language, then add an optional MITRE ATLAS technique when an adversarial behavior needs more granular mapping. The seven areas most directly relevant to this testing program are:
| Risk area | Example evidence |
|---|---|
| Prompt injection | A direct or retrieved instruction changes behavior or bypasses policy |
| Sensitive information disclosure | System prompts, secrets, personal data, or another tenant’s chunks are returned |
| Improper output handling | Model text reaches a browser, query, shell, HTTP client, or file operation unsafely |
| Excessive agency | An agent performs a consequential action without independent authorization or confirmation |
| System prompt leakage | Hidden instructions or policy text are disclosed beyond the intended audience |
| Vector and embedding weaknesses | Poisoning, cross-tenant retrieval, stale ACLs, or metadata-filter bypasses succeed |
| Unbounded consumption | Prompts, documents, loops, or retries consume excessive tokens, time, or cost |
Taxonomy labels do not establish severity by themselves. Severity should reflect preconditions, affected identities or tenants, data sensitivity, actionability, reproducibility, and business impact.
What safeguards should be verified during runtime testing?
Controls should remain effective when the model is wrong, manipulated, unavailable, or replaced by a fallback.
- Least privilege: separate read and write credentials, restrict tools by user and tenant, and remove unused network access.
- Independent policy enforcement: authorize document retrieval and tool execution in services that do not trust model text, prompt claims, or document instructions.
- Confirmation and reversibility: require explicit confirmation for consequential operations, show the exact target and parameters, use dry runs, idempotency, and rollback.
- Bounded execution: enforce context-size, document-size, recursion, tool-count, token, latency, rate, and cost quotas.
- Input and context hygiene: validate uploads, isolate tenants, scan retrieved chunks for injection indicators, label untrusted content, and minimize context.
- Observability: retain trace identifiers linking identity, retrieval, model decision, tool request, authorization, and side effect.
What belongs in a professional red-team report?
Each finding should let an engineer reproduce the behavior and an owner decide what to fix first.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →| Field | What to include |
|---|---|
| Attack precondition | Required identity, tenant, document state, feature flag, and network position |
| Payload and path | Exact input, file or chunk placement, model/version, settings, and request sequence |
| Affected component | Prompt, retriever, vector store, parser, tool, connector, policy, or infrastructure control |
| Observed result | Output, retrieved records, tool arguments, authorization decision, and side effect |
| Scope and impact | Users or tenants affected, data exposed, action performed, and operational or financial consequence |
| Reproducibility | Clean-session rate, variations tested, timestamps, trace IDs, and preserved canaries |
| Remediation owner | Specific service or team, proposed control, priority, and verification test |
Redact production secrets and personal data, but preserve enough context to prove the boundary crossing. Include a negative test showing the intended control working when possible.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How do you turn findings into CI/CD regression tests?
Automate deterministic cases and reserve human review for novel behavior, ambiguous intent, and business-impact validation.
- Normalize each confirmed case into a fixture containing identity, tenant, corpus state, prompt or file, expected authorization decision, and a safe expected outcome.
- Run fast tests on pull requests for prompt templates, parsers, authorization code, tool schemas, and retrieval filters.
- Run broader integration tests on a schedule against realistic indexes, rerankers, fallback models, and connector sandboxes.
- Fail the build when a protected canary is disclosed, an unauthorized chunk is retrieved, a tool is called without the required identity or confirmation, or quotas are exceeded.
- Store traces and model versions with the test result so a behavior change is attributable rather than mistaken for random model variation.
- Keep manual red-team review for multilingual, encoded, multi-turn, cross-channel, and previously unseen attacks; feed confirmed cases back into the fixture set.
How do assessment approaches differ?
A useful comparison is based on coverage and evidence, not on the number of prompts generated.
| Approach | Strength | Blind spot if used alone |
|---|---|---|
| Model-focused evaluation | Fast coverage of refusal, jailbreak, leakage, and instruction-following behavior | Cannot prove tenant authorization, sink safety, or real side effects |
| Application penetration test | Connects identity, code paths, data stores, tools, and business impact | May miss broad multilingual or model-version behavior without a dedicated test set |
| RAG security assessment | Deep coverage of retrieval scope, poisoning, parsers, chunks, filters, and citations | Does not automatically cover non-RAG tools, infrastructure, or prompt-only routes |
| Automated CI regression | Repeatable detection of known failures after code or model changes | Weak at novel attacks, ambiguous context, and judgment about real-world impact |
| Runtime red team | Validates live traces, quotas, loops, outbound behavior, and operational safeguards | Requires careful sandboxing and can be expensive or disruptive without stop rules |
The strongest program combines these approaches: broad model cases, application and infrastructure verification, RAG-specific authorization tests, live runtime observation, and repeatable CI checks.
Free tools Windows power users keep installed
One-click scans. No signup required.
What common mistakes make an LLM pentest misleading?
- Testing only the public chat endpoint while never querying the retriever or tool service with separate identities.
- Declaring victory because the model refused a jailbreak, even though a stale ACL still returns the protected chunk.
- Using a malicious document that always ranks first instead of testing plausible indirect-injection placement and ranking variation.
- Trusting model-supplied tenant, role, approval, or confirmation fields.
- Ignoring fallback models, retries, memory, summaries, logs, and error responses.
- Running destructive actions against production without canaries, confirmation, quotas, and rollback.
- Reporting a prompt response without the model version, settings, context, trace, or reproducibility information.
- Assuming a scanner or one-time benchmark proves safety; the relevant behavior changes with prompts, indexes, permissions, and model updates.
No authoritative general prevalence or attack-success rate establishes how often these failures occur. Treat the taxonomy as a way to organize testing, not as a statistic about your own exposure.
Best Value
Frequently Asked Questions
How often should an LLM application be retested?
Retest after model, prompt, retriever, index, parser, tool, permission, or connector changes, and after any security incident. Run deterministic regression cases continuously in CI, then schedule broader adversarial and runtime exercises at an interval appropriate to the application’s data sensitivity and change rate.
Should a red-team test use production data?
Use isolated tenants, synthetic records, and unique canaries whenever possible. If production behavior is essential, minimize data, obtain written authorization, enforce read-only or dry-run controls, and define an immediate stop and rollback procedure.
How can I tell whether a leaked string came from the model or from retrieval?
Place different canary values in the system prompt, each tenant’s documents, tool results, and application logs. Correlate the output with retrieval and tool traces; a canary appearing without a corresponding authorized fetch points to a different leakage path than one returned in a retrieved chunk.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Does scanning retrieved text for prompt injection solve RAG security?
No. Scanning can flag suspicious content before context insertion, but it cannot replace tenant authorization, output validation, or tool-side policy checks. Attackers can evade pattern rules, and benign text can still become dangerous when combined with excessive agency.
What is the safest first tool to connect to an agent?
Begin with a read-only, narrowly scoped operation in a sandbox, using a dedicated low-privilege identity and strict parameter allowlists. Add write capability only after independent authorization, explicit confirmation, quotas, idempotency, monitoring, and rollback have been tested.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
