How to Troubleshoot SIEM Integrations Returning Incomplete or Delayed Data

To find why a SIEM is missing or receiving late events, trace a few known records from the source through transport, connector or agent, collection rules, ingestion, and parsing. The first point where those records disappear—or where their volume or timestamps change unexpectedly—is the boundary to investigate. Measure event time separately from arrival time, and distinguish an ingestion problem from a query or detection that simply fails to include the data.

First determine what is actually missing

Choose a representative source and time range, then describe the symptom precisely. A feed that has stopped entirely calls for a different investigation than one producing fewer records than expected, records arriving late, or data that is stored but absent from a dashboard or alert.

#	Preview	Product	Price
1		Juniper SSG 520M Security Appliance (SSG-520M-SH)	$229.00	Buy on Amazon

• Record the source, event type, time range, and expected volume, if known.
• Collect a small set of identifiable events, including their source timestamps and any timestamps recorded by a forwarder or connector.
• Check whether the issue affects all events from the source or only a category, facility, account, or event type.
• Note whether the records are missing from storage or only from a query, normalized view, dashboard, or detection.

Use those known events or counts to compare each stage of the path. This turns “the SIEM is missing logs” into a testable question: at which hand-off do the records stop appearing?

Trace the event through the complete data path

Follow the same event from the producing system to the place where analysts query it. Check the boundaries in order: source, network, forwarder, connector or agent, collection rule, SIEM table or index, and parser or downstream query. Use the diagnostics provided by your platform and connector; commands and log locations are not interchangeable across products.

#1 Best Overall

Juniper SSG 520M Security Appliance (SSG-520M-SH)

• Juniper ssg 520m security appliance - 4 x 10/100/1000base-t
• Juniper ssg 520m security appliance
• 4 x 10/100/1000base-t

$229.00

Buy on Amazon

1. Source: Verify that the system is generating the expected event class and is configured to send it to the intended destination. Check source-side errors and configuration for category or severity filters.
2. Network: Confirm that messages reach the next component. Inspect relevant firewall, load-balancer, security-group, and routing rules as well as the receiving component’s logs.
3. Forwarder or connector: Check that it is enabled, healthy, and connected to the intended endpoint or tenant. Review its local diagnostics and service logs for errors, retries, or rejected messages.
4. Collection rules and destination: Confirm that rules include the required event types and route them to the expected workspace, table, or index. Check that credentials and permissions allow reading from the source and writing to the destination.
5. Stored record and query: Search the raw destination for a known event identifier or distinctive field. If it is present there but absent from a normalized view, dashboard, or alert, investigate transformations, parsing, and downstream filters rather than source transport.

Example: Microsoft Sentinel CEF/Syslog through AMA

For the documented Microsoft Sentinel CEF/Syslog path, the route is source → RSyslog or Syslog-ng forwarder → Azure Monitor Agent (AMA) → Data Collection Rule (DCR) → Log Analytics/Sentinel workspace. Microsoft’s CEF/Syslog troubleshooting guide recommends packet capture on port 514 as an initial check that traffic reaches the forwarder, along with checks of firewalls and network components. Then verify agent status and local diagnostics, DCR selections and routing, and the resulting records in the workspace. Microsoft notes that logs on this path can take up to 20 minutes to appear after configuration; that is connector-path guidance, not a general SIEM delivery guarantee or an SLA for every source.

Measure event time and ingestion time separately

A record’s event timestamp tells you when the source says the event occurred. Its ingestion or arrival timestamp tells you when the SIEM received or stored it. Comparing the two reveals whether a record is genuinely late or whether a query is filtering it out based on the wrong time field. Measure the delay by data type over representative periods; different feeds can have different behavior, so one global latency assumption can mislead—especially in queries that join multiple feeds.

In Microsoft Sentinel

Microsoft documents comparing TimeGenerated with ingestion_time() to investigate delay. Its data connector health monitoring guidance also describes the Workspace Usage Report for viewing latency and delay by data type. Establish a baseline for the affected feed and compare it with the source and connector’s expected behavior.

In Elastic

For Elastic ingest pipelines, Elastic recommends temporarily using a data view based on event.ingested to investigate ingestion lag. For certain anomaly-detection datafeeds, Elastic documents a delayed-data check and increasing query_delay when the error reports that documents were missed due to ingest latency. These are Elastic-specific mechanisms; do not apply the setting to other platforms or assume it addresses missing events caused earlier in the pipeline.

Elastic references: data quality and ingestion lag and delayed data detection.

Check connector settings, health, and permissions

Once the failing boundary is narrowed down, validate the integration itself. A connector may be running while collecting the wrong categories, using an outdated endpoint, or lacking access to the source. Check the SIEM-side integration logs and source-system logs rather than assuming that a healthy-looking status means the expected records are flowing.

• Confirm the endpoint, tenant, workspace, credentials, and destination.
• Review selected event categories, facilities, polling or streaming settings, and any filters.
• Check connector or agent health, extension status, and version requirements for the deployment.
• Verify network reachability and the permissions needed to read source data and write to the SIEM.
• Look for source-side rate limits, errors, or changes that coincide with the drop or delay.

Microsoft’s Sentinel data connector reference describes connector-specific checks; the exact steps vary by integration. For sources without a suitable built-in connector, Microsoft’s planning guidance covers custom ingestion using an agent, Logstash, or API, while its solution guidance describes the Codeless Connector Framework for partner connectors: plan data collection and create a codeless connector. Choose an integration method based on supportability, health monitoring, infrastructure, filtering, and permissions—not only whether it can move data.

Separate collection failures from parsing and query failures

If the message reaches the destination but fields are missing, malformed, or unavailable to expected detections, inspect the raw payload before changing transport settings. Compare it with the format and schema the connector or parser expects. Check timestamp interpretation, delimiters, escaping, field mappings, transformations, and parser version.

For Sentinel’s CEF/Syslog path, the troubleshooting guide includes CEF validation and DCR checks. If raw records are present but absent from a normalized table, dashboard, or detection, inspect the relevant transformation and downstream filters. A transport fix will not correct a parser or query that drops otherwise-ingested events.

Adjust scheduled detection windows only for measured delay

A scheduled detection can miss a late record even when ingestion eventually succeeds. For example, a rule may search a short event-time interval; by the time the event arrives, its timestamp can fall outside that interval. Microsoft’s Sentinel example illustrates this with a two-minute ingestion delay and a five-minute rule look-back. It expands the event-time search to seven minutes, then uses ingestion time to restrict processing to the normal five-minute interval and avoid reprocessing the overlap:

let ingestion_delay = 2min;
let rule_look_back = 5min;
CommonSecurityLog
| where TimeGenerated >= ago(ingestion_delay + rule_look_back)
| where ingestion_time() > ago(rule_look_back)

Those durations are illustrative parameters in Microsoft’s example, not recommended defaults or a platform-wide latency statistic. Measure delay for the affected data type, then test the rule against known late events. Consider the effects on query cost and duplicate handling. Microsoft’s ingestion delay guidance also discusses near-real-time analytics rules for applicable Sentinel cases.

Choose a fix at the boundary that is failing

Match the remedy to the evidence. Before changing an integration, check what it changes, whether it addresses absence or timing, and how it affects event-time meaning and duplicate or backfill behavior.

For a built-in, partner, or custom integration, also compare supportability, health monitoring, infrastructure requirements, filtering controls, and permissions. Microsoft recommends prioritizing sources during Sentinel deployment planning and notes that custom connectors may suit unsupported sources; the choice depends on the source and operational needs, not on a universal best method.

Quick Recap

Bestseller No. 1

Juniper SSG 520M Security Appliance (SSG-520M-SH)

Juniper ssg 520m security appliance - 4 x 10/100/1000base-t; Juniper ssg 520m security appliance

$229.00

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.