Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more

Collect NGINX and Apache logs with Logstash by first identifying the active access and error-log files, then routing each stream to its own parser. Use Logstash’s file input for files it can read locally, Filebeat when an agent already ships the files, or the syslog input when another system forwards them. Parse standard Apache combined access lines with %{COMBINEDAPACHELOG}, match NGINX against the deployed log_format, normalize the source timestamp, and test rotation and registry recovery before indexing the events.

What the pipeline should do

A reliable web-log pipeline has four distinct stages:

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
  1. Read: obtain events from local files, Filebeat, or syslog.
  2. Identify: attach metadata for the web server and stream, such as apache/access or nginx/error.
  3. Parse: apply a parser that matches that exact format. Access and error records must not share a generic parser.
  4. Normalize and deliver: convert the original timestamp, retain the raw message, and send validated events to Elasticsearch or another supported destination.

The most common failures happen when a pipeline assumes a default path, treats every web log as Apache combined format, or starts indexing before its fields and timestamp conventions are settled.

Find the real log files and formats

Inspect NGINX rather than trusting a copied path

NGINX access and error logging are configured independently. The access_log directive determines the access-file location and its log_format; error_log determines the error-file location and the minimum severity recorded. Package defaults differ between operating systems, containers, source builds, and custom virtual-host configurations.

#1 Best Overall
Sale
Compressed Air Duster,130000RPM Super Power Cordless Air Duster,Rechargeable Brushless Blower,3 Gear Adjustable Blower with LED Light,Dust Remover for Computer/House/Outdoor/Car
  • Powerful Turbo Fan:CAGIWIRU Electric Air Duster have reached speed up to 130000RPM,Can efficient remove dust and debris.With LED light and Three Adjustable Speed to meet with different cleaning tasks.
  • With Attached USB C Fasting Charging cable.Charing 3 hours,Enjoy up to 240 minutes of use on the lowest setting, For third speed can use for 25mins.with four charging options to meet with your needs.
  • Wide Application: attached 5 different nozzles,making it suitable for a variety of scenes, such as car,pet, pc, keyboards, and other electronic devices. It also serves for office and home clean duster.
  • Handy Design & Portable:Cordless air duster electric nozzle is Light weight and small size, design and comfortable to hold and space saving, convenient to carry around. You can put it in your room or inside the car without taking up space.
  • Economical &Reusable: Built durable materials with a long-lasting battery,our CAGIWIRU air duster it's a sustainable alternative to disposable air cans.

Inspect the running configuration and each enabled server block. Record every active access file, error file, and format name. A host may have several files, for example one access log per virtual host and a shared error log. Do not assume that a file named access.log uses the standard fields: operators often add upstream response time, forwarded addresses, request IDs, or other variables.

Locate Apache files from the running installation

Apache commonly writes files below an installation-specific log directory. Debian-derived layouts often resemble /var/log/apache2/, while other packages use a different directory and names. Treat these as examples, not universal paths. Confirm the active CustomLog and ErrorLog directives and note the format assigned to each CustomLog.

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Stream Typical content Parsing implication
Access One HTTP request per line, usually a configurable access format Use the exact configured format. Standard combined access lines can use COMBINEDAPACHELOG.
Error Severity, time, process or module context, and a diagnostic message Use a separate, format-specific Grok pattern; access patterns are not appropriate.

Use narrow globs

For a local pipeline, a starting path list might be /var/log/nginx/*.log and /var/log/apache2/*_log. Narrow the patterns if those directories contain rotated archives, audit records, or unrelated application files. A broad recursive glob can ingest files that have no matching parser and can create duplicate or misleading events.

Choose how Logstash receives the events

Method Use it when Main considerations
Logstash file input Logstash runs on the web host or has dependable filesystem access Logstash tracks file positions in its registry. Permissions, rotation, truncation, and registry recovery are your responsibility.
Filebeat to Logstash A lightweight agent already runs on each web host or Logstash should not read application disks Filebeat handles local harvesting and forwards events over the Beats protocol. Parser ownership can remain in Logstash, or move to the agent if you standardize it there.
Logstash syslog input Another service already forwards the logs through syslog Choose TCP or UDP deliberately, expose only the required listener, and supply a Grok pattern that captures a parseable timestamp.

File input is the shortest path for a single host. Filebeat is usually easier to scale across many hosts because collection stays close to the files and Logstash receives a consistent transport. Syslog avoids adding a file harvester when forwarding already exists, but its framing and transport behavior are controlled by the sender and can vary substantially.

Build a local-file pipeline

Prepare access and state

  1. Identify the user that runs Logstash and grant it read access to the selected files and traverse access to every parent directory.
  2. Choose a persistent path.data location so the file-input registry survives a restart. Losing that registry can cause old content to be read again.
  3. Start with one representative access line and one representative error line from every deployed format. Keep the original lines in a secure test file if production permissions prevent repeated reads.
  4. Validate the configuration before starting the service. The exact command depends on the installation, but Logstash supports a configuration-test mode such as bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/web.conf.

Use conditional routing

This example demonstrates the design. Adjust the path tests and parsers to your actual filenames; the path test is not a substitute for inspecting the web-server configuration.

Rank #2
EVEO® Screen Cleaner Spray Kit (Streak-Free) TV Screen Cleaner for Smart TV, Lcd, LEd, Oled | Computer Screen Cleaner Spray and Wipe Microfiber Cloth - TV, Monitor, Laptop, iPad, Car - 6.7oz 1 Pack
  • Screen Cleaner is Your Screen’s New Bestfriend! - EVEO is proud to present a screen cleaner that’s perfectly suited for your TV. Meet your new screen cleaner, computer screen cleaner, laptop cleaner, iPad cleaner, Macbook,tv cleaner or any other electronic device. Our screen cleaner spray and wipe along with the included plush microfiber cloth, easily removes dust, fingerprints & other smudges on the screen’s surface. Clean effectively for a streak-free screen
  • Gentle on your Screens - Our screen cleaner spray and wipes is compatible to be used even for the most sensitive LCD, LED, CRT, and OLED screens in proper use. Be at ease knowing that this product is compatible with all the major brands: LG, SONY, Samsung, Sharp, iPad, iPhone, Android screens, Kindle, PC monitors - we’ve got you covered! Can be used as LCD screen cleaner, tv cleaner, smart tv screen cleaner, and more
  • Premium Super Soft Plush Microfiber Cloth - Included in this efficient screen cleaning kit. This screen cleaner spray and wipe has our signature plush microfiber cloth specially designed to comply with screens and monitors and leave them streak-free. Has been shown to effectively absorbs stubborn stains, zero streaks, and no fiber shedding. Always use this together with our screen cleaner for the best results
  • Easy-to-Use Screen Cleaner Kit + microfiber cleaning cloth - The EVEO Screen Cleaning Kit will easily be your new favorite tool when it comes to taking care of the things you love most, quick and easy solution for a streak-free screen. Simply screw on the sprayer gun on the screen cleaner bottle. Spray the cleaning liquid onto the microfiber cloth, then wipe the screen until desired cleanliness & shine is achieved.
  • Quality Screen Care Kit Worth Purchasing - Enjoy a convenient Screen Cleaner anytime, anywhere you need it. Your satisfaction is our top priority. We stand behind the quality of our products and that’s why, you shouldn’t be worrying at all. We manufacture the Best Screen Cleaner. and can be used for laptop cleaning kit, laptop screen cleaner, tv screen cleaner for smart tv. MacBook screen cleaner, monitor cleaner or screen cleaner spray for electronic devices with microfiber cleaning cloth
input {
  file {
    path => ["/var/log/nginx/*.log", "/var/log/apache2/*_log"]
    start_position => "beginning"
  }
}

filter {
  if [path] =~ /apache/ and [path] =~ /access/ {
    mutate {
      add_field => {
        "[service][name]" => "apache"
        "[log][type]" => "access"
      }
    }
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
      match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
    }
  } else if [path] =~ /nginx/ and [path] =~ /access/ {
    mutate {
      add_field => {
        "[service][name]" => "nginx"
        "[log][type]" => "access"
      }
    }
    # Match the deployed NGINX log_format here.
  } else if [path] =~ /apache/ and [path] =~ /error/ {
    mutate {
      add_field => {
        "[service][name]" => "apache"
        "[log][type]" => "error"
      }
    }
    # Apply an Apache error-log pattern here.
  } else if [path] =~ /nginx/ and [path] =~ /error/ {
    mutate {
      add_field => {
        "[service][name]" => "nginx"
        "[log][type]" => "error"
      }
    }
    # Apply an NGINX error-log pattern here.
  } else {
    mutate { add_tag => ["unclassified_web_log"] }
  }
}

output {
  elasticsearch {
    hosts => ["${ELASTICSEARCH_HOSTS}"]
  }
  stdout { codec => rubydebug }
}

The metadata field is deliberately [log][type]. A malformed field name in a copied example can prevent the configuration from behaving as intended. Keep the stdout output during development so you can inspect fields before relying on index mappings and dashboards; remove or restrict it in production if it would expose sensitive request data.

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Parse Apache access logs

Use the built-in combined pattern when it truly matches

Apache’s standard combined access format contains the client address, identity and authenticated-user fields, timestamp, request line, response status, response size, referrer, and user agent. For that exact grammar, the Grok pattern %{COMBINEDAPACHELOG} separates the values into named fields. Follow it with a date filter:

grok {
  match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
  match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
}

The date filter should run after Grok has created timestamp. The Apache timestamp includes a numeric UTC offset, so the filter can convert it to Logstash’s event time without guessing the host’s local timezone.

Know when combined parsing is wrong

Apache formats are configurable. Added request IDs, forwarded-client fields, virtual-host names, or different quoting will cause a combined-pattern mismatch or misaligned fields. Compare a real line with the active LogFormat, then create a custom pattern that captures each field in its actual order. Keep the unmodified message even after successful parsing; it is valuable when a format changes or a field needs forensic review.

Parse NGINX access logs against the deployed format

Why a default pattern is unsafe

NGINX writes whatever variables the configured log_format contains. Two servers can both be called “default NGINX” while emitting different fields. A parser that expects only the common request, status, byte count, referrer, and user-agent fields will fail when upstream timing, forwarded addresses, or request IDs are inserted.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Rank #3
Sale
BladeHawks RGB Gaming Mouse Pad Extra Large Extended 31.5x12 Inch-Black | LED mouse pad with 14 lighting modes, covers keyboard & mouse, anti-slip rubber base, waterproof surface, USB powered
  • 【Large size】:Measuring 800 x 300 mm/ 31. 5×12 inches, which is wider and taller than others. Large mouse pad can perfectly fit your keyboard and mouse of any size, and there is also ample space for peripherals such as phones, tablets, etc.
  • 【LED backlight mouse pad】:Blade Hawks RGB mouse pad provides 7 static modes and 7 dynamic modes will give you the options you are looking for. Power-off memory function remembers the last lighting mode you selected. The super glow fiber Chroma will give you a colorful gaming setup that you want.
  • 【Anti-slip rubber base】:Made from Natural rubber, this gaming mousepad will firmly grip your desktop, allowing you to enjoy the ultimate precision in the games you are most passionate about. It also comes with a texture that enhances this ability even further.
  • 【Smooth and waterproof surface】: Micro-textured cloth surface, for extremely precise mouse control and a smooth movement. When liquid splashes on the gaming mouse pad, it forms water droplets and slides down. Will not delay your work or gameplay.
  • 【Plug and play】: This mouse and keyboard pad is powered by USB, plug and play, no driver required. One button control light modes. Press one time to change the lighting mode, press twice to change the brightness, press and hold about 3 seconds turn ON/OFF.

Start with a format-specific Grok expression

For a conventional combined-style NGINX line, a custom expression can look like this:

grok {
  match => {
    "message" => "%{IPORHOST:clientip} - %{DATA:auth} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{DATA:request}(?: HTTP/%{NUMBER:httpversion})?" %{NUMBER:response:int} %{NUMBER:bytes:int} "%{DATA:referrer}" "%{DATA:agent}""
  }
}
date {
  match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
}

This is only a starting point. If the NGINX configuration emits a different timestamp, an upstream field, or a request containing characters this expression does not cover, revise the pattern and test it against representative lines. Version custom patterns with the deployment configuration so a later change to log_format does not silently corrupt fields.

Tag failures instead of discarding data

Configure Grok to add a failure tag, or rely on the standard _grokparsefailure tag, and monitor its rate. A failed parse should retain the raw message and source metadata. Dropping the event makes it impossible to distinguish a quiet server from a parser that stopped matching.

Handle NGINX and Apache error logs separately

NGINX error records

NGINX error lines generally contain a date and time, bracketed severity, process information, and a diagnostic message. Some lines add a connection identifier, client address, server name, request, or host. A deliberately broad first pass can capture the stable outer structure:

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
grok {
  match => {
    "message" => "(?<timestamp>d{4}/d{2}/d{2} d{2}:d{2}:d{2}) [%{LOGLEVEL:severity}] %{GREEDYDATA:error_detail}"
  }
}
date {
  match => ["timestamp", "yyyy/MM/dd HH:mm:ss"]
}

Once the outer parse is reliable, add fields for process IDs, client address, request, and host only if your deployed lines contain them consistently. Preserve error_detail or the original message for lines that include a different shape.

Rank #4
Sale
Vaydeer Wrist Rest for Keyboard and Mouse, Computer Ergonomic Wrist Support Pad, Soft Memory Foam Arm Cushion for Desk, Palm Hand Office Laptop Typing
  • 【Softer and More Comfortable】Vaydeer wrist rest has unique diamond pattern, which is the combination of softness and aesthetics. The materials of wrist rest are improved into higher quality memory foam and covered with silky smooth lycra. The computer wrist rest makes you as comfortable and cushiony as like rest your wrists on clouds.
  • 【Ergonomic Wrist Saver】The wrist rests for keyboard and mouse comes with a 17.32×3.15×0.83 inch keyboard wrist pad and a 5.94×3.15×0.83 inch mouse wrist support. Based on ergonomic design, the unique concave shape is the perfect fit for your wrist joints. The wrist rest pad fits most computer keyboards and laptops, improve hand and wrist posture, release your wrist and arm stress.
  • 【Non-Slip Rubber Bottom】Featuring an anti-skid silicone base on the bottom, this wrist keyboard support stays firmly in place on your desk, preventing the padding from sliding around, ensuring stable and consistent wrist support during extended computer sessions.
  • 【Better Experience & Pain Relief】Our keyboard arm rest is beneficial to alleviate the soreness caused by direct contact and friction between your arm and a hard desk surface, reducing the risk of wrist fatigue or carpal tunnel. The soft texture of memory foam can evenly distribute the pressure around your wrists and provide good support with just enough give.
  • 【Helpful in Multiple Scenarios】Whether you're working, studying, writing, typing, gaming, this keyboard and mouse rest combo is an essential accessory to add comfort and support to your hands and wrists. It’s also a great gift for men, women, family, friend, coworker, gamer, teacher, etc.

Apache error records

Apache error formats vary with the server version, module, and configured error-log format. They may include a human-readable date, module and severity labels, process or thread identifiers, and a message. Route these records to a separate Grok expression rather than trying to force them through COMBINEDAPACHELOG. Build the expression from actual samples and use a date pattern that matches the emitted date, including fractional seconds when present.

Keep severity useful

Store the source severity as a dedicated field and do not infer severity from the text of an access request. NGINX’s error_log directive controls the minimum severity written, so a missing debug message may reflect configuration rather than a collection failure.

Choose ECS field names before indexing

Grok composite patterns can emit different field names depending on the plugin’s ecs_compatibility setting. Decide whether your index templates, dashboards, alert rules, and queries expect ECS-compatible fields or the legacy names before changing that setting. Apply the same decision to every pipeline that writes the destination index.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

A practical naming policy is to keep source identity in fields such as [service][name] and [log][type], retain the raw line in message, and use consistent timestamp and HTTP-field conventions across NGINX and Apache. If you migrate to ECS, update mappings and dashboards together; changing only the parser can make existing searches appear empty even though ingestion is working.

Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Send events through Filebeat instead of local file input

Use Filebeat when each web host should harvest its own files and forward events to a central Logstash tier. A minimal Filebeat input can resemble:

Best Value
Sale
Under Desk Cable Management Tray No Drill, Metal Mesh Cable Management Under Desk with Clamp Mount, 2 Hole Cable Tray with Wire Management and Cord Organizer for Home Office Standing Desk
  • 【No Drill Mounted】Cable management tray can be assembled in as little as 3 minutes or less-simply. Side clips easily clampe the edge of desk, pass through the holes on the sides, and secure excess cables with baskets. Desk cable management supports inward or outward installation to meet your needs in different scenarios, and provides you with great convenience when collecting and organizing wires.
  • 【Rubber Pad Protects Desk】Wire organizer under desk built-in soft rubber pads on the clamp protect your desk from scratches, so you will not damage your furniture or office.
  • 【Sturdy and Beautiful】Cord management under desk is made of high-quality carbon steel and holds about 15 lbs. The wire basket has 2 holes on each side of the tray to enable your cables to pass freely. Cable baskets are suitable for desk thickness use from 0.4" to 2.4".
  • 【Space Saving & Keep Organized】Desk cable management tray holds all your power cords, outlet strips, USB hubs and computer transformers hidden from your feet, and the data cables will not enter the vacuum cleaner to keep your desk clean and tidy. Desk wire organizer is also suitable for use in the kitchen and outdoors.
  • 【Reduce Safety Risks】Cable tray under desk keeps all plugs away from your kids, no more worry about tripping. Using 2 holes wire tray saves extra space, helps you easily cabling and wire protection. The mesh design is not easy to accumulate dust, but avoids safety accidents caused by messy wires.
filebeat.inputs:
  - type: filestream
    paths:
      - /var/log/nginx/*.log
      - /var/log/apache2/*_log

output.logstash:
  hosts: ["${LOGSTASH_HOST}:5044"]

Configure Logstash with a Beats listener on the chosen port, then perform the same source classification and parsing there. You can add fields at the agent, but keep the ownership of parser definitions clear. If Logstash is responsible for parsing, send the original line and enough host/path metadata to select the correct NGINX or Apache pattern.

Filebeat does not remove the need to test rotation, permissions, multiline behavior, and back-pressure. Confirm that the agent’s registry is persistent and that its harvester settings match the way your logrotate or container runtime renames and recreates files.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Accept forwarded logs with the syslog input

If a relay already sends web logs through syslog, Logstash can listen on TCP, UDP, or both. Use a dedicated port and network policy rather than exposing a listener broadly. A custom pattern must capture a timestamp that Logstash can parse:

input {
  syslog {
    port => 5514
    grok_pattern => "<%{POSINT:priority}>%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:sender} %{GREEDYDATA:message}"
  }
}

Adapt the expression to the sender’s actual framing and payload. If the sender wraps an entire NGINX or Apache line inside the syslog message, parse the syslog envelope first and then apply the web-server parser to the remaining message. UDP can lose events under congestion; TCP provides a connection and back-pressure but still requires queue and failure monitoring.

Understand file positions, rotation, and duplication

start_position is not a replay switch

start_position => "beginning" tells a new file-input registration where to begin reading. It does not force Logstash to reread content whose position is already stored in the file-input registry. To deliberately replay a file, you need a controlled test environment and a clear registry procedure; never delete production state casually.

Test the rotation patterns you actually use

  • Rename the active file and create a new file with the original name.
  • Truncate a file in place if your rotation tool uses copy-truncate.
  • Compress or remove an older file and verify that the glob does not ingest the archive.
  • Restart Logstash during rotation and confirm that the registry resumes from the expected byte offset.
  • Check that the service account can read newly created files, not just the files present during installation.

Watch for duplicate events after a registry reset and missed events after an inode or rename-handling mismatch. Keep event identifiers or source offsets available in your operational logs so an apparent application spike can be compared with collector behavior.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Validate a pipeline before production indexing

  1. Send one known Apache combined access line and verify every expected field and the converted event timestamp.
  2. Send one Apache error line and confirm that it bypasses the access parser.
  3. Repeat with one line from every NGINX log_format and one NGINX error line.
  4. Intentionally send a malformed line and verify that the raw message remains and a parse-failure tag appears.
  5. Inspect the event with stdout { codec => rubydebug } before enabling dashboards or index templates.
  6. Stop and restart Logstash, then confirm that already-read content is not unexpectedly replayed.
  7. Exercise log rotation, permission changes, truncation, and temporary destination failure.
  8. Measure queue growth and parse-failure rates after deployment, not only successful event counts.

Troubleshoot by symptom

Symptom Likely cause Action
No events arrive Wrong path, insufficient permissions, inactive directive, or a registry position at end-of-file Confirm the running configuration, read the file as the Logstash user, inspect the registry, and test with a newly appended line.
Apache fields are empty The line is not standard combined format Compare it with LogFormat and replace COMBINEDAPACHELOG with a custom pattern.
NGINX fields shift or parse fails The deployed log_format has added or reordered fields Capture a real line, revise the pattern, and version it with the NGINX configuration.
Events have the wrong time Date filter ran before Grok, used the wrong layout, or ignored an offset Parse first, then apply the exact date layout and timezone information emitted by the source.
Error lines look like access requests A single parser is being applied to both streams Route by source metadata or path and give each error format its own parser.
Events duplicate after restart File-input registry was lost or moved Restore persistent state, identify the replay window, and avoid resetting the registry without a replay plan.
New rotated files are ignored Glob, permissions, or harvester behavior does not match the rotation method Test the exact rename/truncate sequence and inspect collector logs while rotating a test file.

A production-ready decision checklist

  • Have you confirmed every active NGINX access_log, error_log, and log_format?
  • Have you confirmed Apache’s active CustomLog and ErrorLog paths and formats?
  • Are access and error streams tagged and parsed independently?
  • Does each parser preserve the original message?
  • Is the source timestamp converted only after successful parsing?
  • Is the chosen ECS compatibility mode consistent with index mappings and dashboards?
  • Is file-input state persistent, or is the Filebeat registry persistent on every host?
  • Have rotation, truncation, permissions, multiline behavior, restart recovery, and destination back-pressure been tested?
  • Are parse failures visible through tags and monitoring rather than silently discarded?

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.