iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Antihost offers Linux VPS plans with full root access, unlimited traffic on a 1 Gbps port and a 14-day free trial.
Put Logstash filters between your inputs and outputs to turn inconsistent host logs into searchable, time-correct events. A typical centralized pipeline receives Beats or Kafka events, parses each message with grok or json, assigns the event time with date, normalizes fields with mutate, and forwards the result to Elasticsearch.
This guide shows how to design that chain, write a mixed-format pipeline, validate it safely, and diagnose the failures that commonly appear when many senders share one Logstash pipeline.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Where filters fit in a centralized Logstash pipeline
A Logstash configuration has three sections in this order: input, optional filter, and output. Events enter through input plugins, pass through every filter plugin in the order written, and then leave through the output plugins. The filter section is not a separate product or service; it is the transformation stage inside the shared pipeline.
In a common centralized arrangement, agents on many hosts send events through a Beats input (or another input such as Kafka). One Logstash pipeline receives those events, applies the same parsing and normalization rules, and sends the resulting events to an Elasticsearch output. Centralization means that the parsing policy is maintained at the shared Logstash point rather than duplicated on every host.
#1 Best Overall
- Powerful Turbo Fan:CAGIWIRU Electric Air Duster have reached speed up to 130000RPM,Can efficient remove dust and debris.With LED light and Three Adjustable Speed to meet with different cleaning tasks.
- With Attached USB C Fasting Charging cable.Charing 3 hours,Enjoy up to 240 minutes of use on the lowest setting, For third speed can use for 25mins.with four charging options to meet with your needs.
- Wide Application:Upgraded attached 5 different nozzles,making it suitable for a variety of scenes, such as car,pet, pc, keyboards, and other electronic devices. It also serves for office and home clean duster.
- Handy Design & Portable:Cordless air duster electric nozzle is Light weight and small size, design and comfortable to hold and space saving, convenient to carry around. You can put it in your room or inside the car without taking up space.
- Warranty & Support: Our Electric air duster comes with 5 years warranty and 24/7 customer service. During this period, if you have any questions,pls kindly feel free to contact us.
What a filter should accomplish
- Extract useful fields from an unstructured
messagestring. - Convert the source timestamp into Logstash’s canonical
@timestampfield. - Give fields consistent names and data types so searches and aggregations behave predictably.
- Expand a JSON payload into fields instead of leaving it as an opaque string.
- Preserve enough information to troubleshoot a malformed event without flooding the destination with unusable data.
Define the event shape before writing filters
Filtering is easier when you decide what a successful event should look like. The following contract covers the fields used in the examples in this guide.
| Field | Purpose | Typical source |
|---|---|---|
@timestamp |
Event time used by time-based searches and indexes | Parsed from a source timestamp by the date filter |
level |
Normalized severity such as info or error |
Grok capture or a JSON property |
msg |
Human-readable log text | Grok capture from the remainder of a line |
status |
Numeric response or outcome code | Converted by mutate |
event |
Nested object for fields decoded from a JSON message | json with a target |
Keep the original message while you are developing the pipeline. Removing it too early makes it harder to understand why a pattern or conversion failed. Once the parser is stable, you can remove it deliberately if storage or privacy requirements call for that change.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Start with a minimal, testable pipeline
Use a small configuration to verify connectivity before adding parsing logic. The Elastic 8.19 “Parsing Logs with Logstash” guide demonstrates the same basic shape: a Beats input listening on port 5044, an optional filter block, and a stdout output for inspection.
input {
beats {
port => 5044
}
}
filter {
# Add filters here.
}
output {
stdout {
codec => rubydebug
}
}Save the file, for example as first-pipeline.conf, and check its syntax before starting a long-running instance:
bin/logstash -f first-pipeline.conf --config.test_and_exitA successful check means Logstash accepted the configuration syntax. It does not prove that every incoming line matches your patterns or that an output endpoint is reachable, so continue with sample events and runtime observation.
Rank #2
- Screen Cleaner is Your Screen’s New Bestfriend! - EVEO is proud to present a screen cleaner that’s perfectly suited for your TV. Meet your new screen cleaner, computer screen cleaner, laptop cleaner, iPad cleaner, Macbook,tv cleaner or any other electronic device. Our screen cleaner spray and wipe along with the included plush microfiber cloth, easily removes dust, fingerprints & other smudges on the screen’s surface. Clean effectively for a streak-free screen
- Gentle on your Screens - Our screen cleaner spray and wipes is compatible to be used even for the most sensitive LCD, LED, CRT, and OLED screens in proper use. Be at ease knowing that this product is compatible with all the major brands: LG, SONY, Samsung, Sharp, iPad, iPhone, Android screens, Kindle, PC monitors - we’ve got you covered! Can be used as LCD screen cleaner, tv cleaner, smart tv screen cleaner, and more
- Premium Super Soft Plush Microfiber Cloth - Included in this efficient screen cleaning kit. This screen cleaner spray and wipe has our signature plush microfiber cloth specially designed to comply with screens and monitors and leave them streak-free. Has been shown to effectively absorbs stubborn stains, zero streaks, and no fiber shedding. Always use this together with our screen cleaner for the best results
- Easy-to-Use Screen Cleaner Kit + microfiber cleaning cloth - The EVEO Screen Cleaning Kit will easily be your new favorite tool when it comes to taking care of the things you love most, quick and easy solution for a streak-free screen. Simply screw on the sprayer gun on the screen cleaner bottle. Spray the cleaning liquid onto the microfiber cloth, then wipe the screen until desired cleanliness & shine is achieved.
- Quality Screen Care Kit Worth Purchasing - Enjoy a convenient Screen Cleaner anytime, anywhere you need it. Your satisfaction is our top priority. We stand behind the quality of our products and that’s why, you shouldn’t be worrying at all. We manufacture the Best Screen Cleaner. and can be used for laptop cleaning kit, laptop screen cleaner, tv screen cleaner for smart tv. MacBook screen cleaner, monitor cleaner or screen cleaner spray for electronic devices with microfiber cleaning cloth
Add the filters in the order the event needs them
1. Use grok to extract fields from text
The grok filter matches text with named patterns. Predefined patterns include TIMESTAMP_ISO8601, LOGLEVEL, and GREEDYDATA. Each pattern can write a capture directly into a field.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutegrok {
match => {
"message" => "%{TIMESTAMP_ISO8601:ts} %{LOGLEVEL:level} %{GREEDYDATA:msg}"
}
}For a source line such as 2026-10-14T07:02:58Z ERROR Database connection refused, this pattern creates ts, level, and msg. It does not yet change @timestamp; that is the job of the date filter.
When senders use more than one line format, provide an array of patterns. Grok tries the alternatives until one matches:
grok {
match => {
"message" => [
"%{TIMESTAMP_ISO8601:ts} %{LOGLEVEL:level} %{GREEDYDATA:msg}",
"%{TIMESTAMP_ISO8601:ts} [%{LOGLEVEL:level}] %{GREEDYDATA:msg}"
]
}
}Keep alternatives specific. A very broad pattern can accept malformed lines and produce misleading fields. Events that match none of the alternatives receive a grok failure tag, which you can inspect during testing.
2. Use date to set @timestamp
Once a source time has been captured, convert it explicitly:
Rank #3
- 【Large size】：Measuring 800 x 300 mm/ 31. 5×12 inches, which is wider and taller than others. Large mouse pad can perfectly fit your keyboard and mouse of any size, and there is also ample space for peripherals such as phones, tablets, etc.
- 【LED backlight mouse pad】：Blade Hawks RGB mouse pad provides 7 static modes and 7 dynamic modes will give you the options you are looking for. Power-off memory function remembers the last lighting mode you selected. The super glow fiber Chroma will give you a colorful gaming setup that you want.
- 【Anti-slip rubber base】：Made from Natural rubber, this gaming mousepad will firmly grip your desktop, allowing you to enjoy the ultimate precision in the games you are most passionate about. It also comes with a texture that enhances this ability even further.
- 【Smooth and waterproof surface】: Micro-textured cloth surface, for extremely precise mouse control and a smooth movement. When liquid splashes on the gaming mouse pad, it forms water droplets and slides down. Will not delay your work or gameplay.
- 【Plug and play】: This mouse and keyboard pad is powered by USB, plug and play, no driver required. One button control light modes. Press one time to change the lighting mode, press twice to change the brightness, press and hold about 3 seconds turn ON/OFF.
date {
match => ["ts", "ISO8601"]
target => "@timestamp"
remove_field => ["ts"]
}ISO8601 handles standard ISO-formatted timestamps. For another format, supply the appropriate custom format string. If the source time is local rather than UTC, set the filter’s timezone option to the zone that applies to those logs; otherwise events can be shifted by the local offset.
Do not assume the timestamp in the arrival metadata is the time the application wrote the log. Parsing the source time lets time-based searches, retention policies, and dashboards reflect the actual event instead of network delay.
3. Use mutate for names, types, and cleanup
The mutate filter handles routine normalization. For example:
mutate {
convert => { "status" => "integer" }
lowercase => ["level"]
remove_field => ["message"]
}Conversion matters because a field stored as text sorts differently from a number and cannot be aggregated reliably as a numeric value. Lowercasing severity values prevents separate buckets for INFO, Info, and info. Remove the raw message only after you have confirmed that the parsed fields are sufficient for your retention and investigation needs.
4. Use json for a JSON payload
When the entire message is JSON, point the filter’s source at message:
Rank #4
- 【Softer and More Comfortable】Vaydeer wrist rest has unique diamond pattern, which is the combination of softness and aesthetics. The materials of wrist rest are improved into higher quality memory foam and covered with silky smooth lycra. The computer wrist rest makes you as comfortable and cushiony as like rest your wrists on clouds.
- 【Ergonomic Wrist Saver】The wrist rests for keyboard and mouse comes with a 17.32×3.15×0.83 inch keyboard wrist pad and a 5.94×3.15×0.83 inch mouse wrist support. Based on ergonomic design, the unique concave shape is the perfect fit for your wrist joints. The wrist rest pad fits most computer keyboards and laptops, improve hand and wrist posture, release your wrist and arm stress.
- 【Non-Slip Rubber Bottom】Featuring an anti-skid silicone base on the bottom, this wrist keyboard support stays firmly in place on your desk, preventing the padding from sliding around, ensuring stable and consistent wrist support during extended computer sessions.
- 【Better Experience & Pain Relief】Our keyboard arm rest is beneficial to alleviate the soreness caused by direct contact and friction between your arm and a hard desk surface, reducing the risk of wrist fatigue or carpal tunnel. The soft texture of memory foam can evenly distribute the pressure around your wrists and provide good support with just enough give.
- 【Helpful in Multiple Scenarios】Whether you're working, studying, writing, typing, gaming, this keyboard and mouse rest combo is an essential accessory to add comfort and support to your hands and wrists. It’s also a great gift for men, women, family, friend, coworker, gamer, teacher, etc.
json {
source => "message"
target => "event"
}The target option keeps the decoded properties under event, avoiding accidental overwrites of top-level fields. Without a target, properties are expanded into the event’s top level. A malformed or non-JSON message receives a JSON parse-failure tag, so do not run this filter unconditionally when the same pipeline also receives plain-text lines.
A mixed text-and-JSON pipeline
The following configuration illustrates a safer pattern for a centralized pipeline that receives both plain-text application logs and JSON application logs. It chooses a branch by checking whether the message begins with a JSON object, parses each branch, assigns the appropriate timestamp, normalizes severity, and keeps the raw message for inspection.
input {
beats {
port => 5044
}
}
filter {
if [message] =~ /^s*{/ {
json {
source => "message"
target => "event"
}
if [event][timestamp] {
date {
match => ["[event][timestamp]", "ISO8601"]
target => "@timestamp"
}
}
mutate {
convert => { "[event][status]" => "integer" }
lowercase => ["[event][level]"]
}
} else {
grok {
match => {
"message" => "%{TIMESTAMP_ISO8601:ts} %{LOGLEVEL:level} %{GREEDYDATA:msg}"
}
}
date {
match => ["ts", "ISO8601"]
target => "@timestamp"
remove_field => ["ts"]
}
mutate {
lowercase => ["level"]
convert => { "status" => "integer" }
}
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => ["https://elasticsearch.example:9200"]
index => "logs-%{+YYYY.MM.dd}"
}
}This is a template, not a claim that every sender uses these exact fields. Adapt the JSON timestamp path, text pattern, index policy, credentials, and TLS settings to your environment. Validate the configuration and feed representative events before routing production traffic.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsUnderstand ordering at two levels
Logstash runs separate filter plugins in the order they appear. A common text-log chain is grok, then date, then mutate: grok creates the source fields, date consumes the parsed time, and mutate applies final naming and type cleanup. JSON messages use the json filter first so later filters can read the decoded object.
Ordering inside one mutate block is also defined. Elastic documents this internal sequence:
Best Value
- 【No Drill Mounted】Cable management tray can be assembled in as little as 3 minutes or less-simply. Side clips easily clampe the edge of desk, pass through the holes on the sides, and secure excess cables with baskets. Desk cable management supports inward or outward installation to meet your needs in different scenarios, and provides you with great convenience when collecting and organizing wires.
- 【Rubber Pad Protects Desk】Wire organizer under desk built-in soft rubber pads on the clamp protect your desk from scratches, so you will not damage your furniture or office.
- 【Sturdy and Beautiful】Cord management under desk is made of high-quality carbon steel and holds about 15 lbs. The wire basket has 2 holes on each side of the tray to enable your cables to pass freely. Cable baskets are suitable for desk thickness use from 0.4" to 2.4".
- 【Space Saving & Keep Organized】Desk cable management tray holds all your power cords, outlet strips, USB hubs and computer transformers hidden from your feet, and the data cables will not enter the vacuum cleaner to keep your desk clean and tidy. Desk wire organizer is also suitable for use in the kitchen and outdoors.
- 【Reduce Safety Risks】Cable tray under desk keeps all plugs away from your kids, no more worry about tripping. Using 2 holes wire tray saves extra space, helps you easily cabling and wire protection. The mesh design is not easy to accumulate dust, but avoids safety accidents caused by messy wires.
- coerce
- rename
- update
- replace
- convert
- gsub
- uppercase
- capitalize
- lowercase
- strip
- split
- join
- merge
- copy
For example, a rename placed in the same mutate block runs before a convert. If you need a value renamed and then transformed by an operation whose interaction is unclear, use two mutate blocks and make the dependency explicit. This also makes the configuration easier to review.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Validate filters before rollout
Syntax check
Run the configuration test command every time you change the pipeline:
bin/logstash -f first-pipeline.conf --config.test_and_exitFix every reported syntax error before testing event content. A syntax check does not exercise a grok match, a date parse, or an Elasticsearch connection.
Use representative fixtures
Prepare samples for each format and edge case your centralized pipeline will receive:
- A normal text line with an ISO8601 timestamp and a conventional level.
- A text line using the second format in a grok pattern array.
- A JSON line with a timestamp, level, and numeric status.
- A malformed JSON line.
- A text line with a missing or invalid timestamp.
- A line containing a level in uppercase or mixed case.
Send each sample through a non-production instance or inspect the stdout rubydebug output. Confirm that @timestamp is the source event time, numeric fields are actually numeric, and the expected nested paths exist.
Inspect failure tags instead of hiding them
Grok and JSON parsing failures are valuable diagnostics. During rollout, preserve the tags and route a small sample to stdout or a separate diagnostic destination. Removing failure tags or deleting the original message before you understand the cause turns a recoverable parsing problem into an investigation gap.
Recommended Free Tools
Troubleshoot the most common filter failures
| Symptom | Likely cause | Action |
|---|---|---|
_grokparsefailure appears |
The line does not match any configured pattern, or the field is not the one you intended to parse. | Inspect the original message, verify whitespace and delimiters, then add or refine a pattern alternative. |
_jsonparsefailure appears |
The message is plain text, truncated JSON, or JSON is nested in another field. | Apply the JSON filter conditionally, point source at the actual field, and test the raw payload. |
@timestamp is wrong or unchanged |
The date filter is reading the wrong field, the format does not match, or the source timezone was omitted. | Inspect the captured time field, use ISO8601 or the correct custom format, and set timezone for local timestamps. |
| A number sorts alphabetically | The field remained a string. | Convert it with mutate, then verify the resulting field type in the destination mapping. |
| Fields disappear unexpectedly | A remove operation ran before another filter needed the field. | Move cleanup to the end of the relevant branch and keep the original message during diagnosis. |
| JSON fields overwrite top-level data | The JSON filter expanded into the root event. | Set target => "event" or another deliberate namespace and update later field paths accordingly. |
| One sender works while another fails | The centralized pipeline assumes one schema for multiple producers. | Use conditional branches, grok pattern arrays, or producer-specific tags to select the correct parser. |
Roll out a centralized filter safely
- Inventory producers. Record which hosts send plain text, JSON, multiline messages, or different timestamp conventions. The filter cannot infer a schema that is not documented.
- Choose stable field names. Decide whether JSON stays nested under
eventand which severity, status, and timestamp paths are canonical. - Build branches for genuinely different formats. Do not force a single grok expression to accept unrelated line structures.
- Keep a diagnostic output during the first rollout. Stdout with
rubydebugmakes field paths and failure tags visible. - Run the syntax test. Use
--config.test_and_exitbefore restarting or deploying. - Canary a subset of senders. Compare parsed fields, timestamps, and failure tags before moving every host to the new configuration.
- Watch volume and failures. A sudden rise in parse-failure tags usually means a producer changed its format, not that the destination is malfunctioning.
- Remove or retain raw data deliberately. Keep
messagewhen investigations require it; remove it only after considering storage, privacy, and replay needs.
Maintain plugin and documentation compatibility
Filter plugins are versioned gems and do not share the same release number as Logstash core. RubyGems lists logstash-filter-grok 4.4.4, released on February 11, 2026, and logstash-filter-mutate 3.6.0, released on March 26, 2026. Those dates identify plugin releases, not a Logstash core release. Confirm which versions your installed Logstash distribution bundles before relying on an option or planning an upgrade.
The official documentation used for the pipeline shape is for Elastic 8.19. Your installation may use another Logstash or Elasticsearch version, so check the documentation and bundled plugin versions that match your deployment. Treat examples as configuration starting points and validate them against the exact runtime you will operate.
Quick Recap
Practical checklist
- Inputs, filters, and outputs appear in the correct configuration sections.
- Every text format has a matching grok pattern or a deliberate failure path.
- JSON parsing is conditional when non-JSON messages share the pipeline.
- The date filter reads the parsed source field and applies the correct timezone.
- Mutate conversion and cleanup happen after the fields they need have been created.
- Nested JSON fields have an intentional target and consistent paths.
- The original message and failure tags remain available during validation.
bin/logstash -f first-pipeline.conf --config.test_and_exitpasses before deployment.- Plugin versions are checked against the Logstash release in use.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
