Serving Markdown alongside HTML attracted two different kinds of traffic on one site: corpus-oriented crawlers mostly requested explicit .md URLs, while agent-oriented answer tools and a coding agent made greater use of HTTP content negotiation. That is what GoodBarber co-founder and engineering lead Dominique Siacci reports from a two-week measurement—not a general forecast for how agents behave across the web.
His September 4–18, 2026 logs also show why raw request counts need careful interpretation: redirects, uncertain bot identities, and the limits of access logs all complicate the picture.
How the site served Markdown
Siacci’s site is a Django application behind nginx with a full-page cache. Rather than maintain separate Markdown templates or a second copy of each page, its middleware converts the HTML already in the cache. The converted response is cached using a fingerprint of the HTML, so conversion is performed once for each distinct HTML response. Middleware placement lets the feature run before later minification and CSRF placeholder substitution.
There are two request paths to the same Markdown content:
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstall#1 Best Overall
- Includes Raspberry Pi 5 with 2.4Ghz 64-bit quad-core CPU (8GB RAM)
- Includes 128GB Micro SD Card pre-loaded with 64-bit Raspberry Pi OS, USB MicroSD Card Reader
- CanaKit Turbine Black Case for the Raspberry Pi 5
- CanaKit Low Noise Bearing System Fan
- Mega Heat Sink - Black Anodized
- Content negotiation: A client requests the canonical URL with
Accept: text/markdown. When the request’s quality values prefer Markdown, the server returns Markdown and includesVary: Accept. - A .md URL: A client adds
.mdto a public path, such as/pricing.mdor/blog/<slug>.md. The root page uses/index.md. The route resolves to the canonical page and uses the same conversion middleware.
Eligible HTML pages advertise their Markdown counterparts in both an HTML rel="alternate" link and an HTTP Link header. Markdown responses include YAML front matter for the title, description, canonical URL, and dates sourced from JSON-LD; they link back to the canonical page and return X-Robots-Tag: noindex. Unknown .md paths receive a Markdown 404 document, while 24 back-office and partial routes are excluded.
To check whether the feature changed ordinary page output, Siacci fetched three pages with it enabled and disabled using seven client types. He reports that all 21 responses were byte-identical, and that browsers incurred roughly eight microseconds of additional per-request processing. These are implementation-specific checks and measurements, not an independent benchmark. The same conversion also feeds nightly generation of llms.txt and llms-full.txt in eleven languages, with local currency forced by country.
What the two request paths revealed
In Siacci’s September 4–18, 2026 measurement, the .md path and negotiated path drew different observed client populations. He reports approximately 80,000 .md requests, compared with about 22,500 negotiated requests. The counts are approximate and describe requests in these logs, not distinct readers or confirmed content consumption.
Rank #2
- Includes Raspberry Pi 5 16GB with 2.4Ghz 64-bit quad-core CPU (16GB RAM)
- Includes 128GB Micro SD Card pre-loaded with 64-bit Raspberry Pi OS, USB MicroSD Card Reader
- CanaKit Turbine Black Case for the Raspberry Pi 5
- CanaKit Low Noise Bearing System Fan
- Mega Heat Sink - Black Anodized
| Measure | .md URL requests | Negotiated requests |
|---|---|---|
| Approximate requests | 80,000 | 22,500 |
| Addresses reported | 5,800 | 5,700 |
| Visitors as defined by the author | 20,000 | 6,700 |
| Host-and-path pairs | About 26,000 | Not stated |
| 200 response rate | 93.8% | 52.7% |
Siacci defines a visitor as a combination of IP address, user agent, and UTC day. A log line counts as a request. Those definitions matter: neither figure represents a verified unique human, and a request alone does not establish that a client read or used the content.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →The 200 rates are not a like-for-like measure of Markdown delivery success. The negotiated requests included about 10,500 redirects, approximately 8,500 of them missing-trailing-slash redirects. A redirected request contributes to the status count even if the client can follow it; therefore, the lower negotiated 200 rate partly reflects route handling, not necessarily failure to obtain Markdown.
.md URLs: more corpus-oriented crawling
In the measured logs, explicit .md URLs were used mainly by corpus crawlers. Siacci reports that GPTBot followed the site’s alternate link, while the real Googlebot made no .md requests during the window. These are observations about this domain’s links and this short period; they do not establish how those crawlers behave on other sites or at other times.
Rank #3
- CanaKit Raspberry Pi 5 Essentials Starter Kit
Before September 4, the site had received about 900 unsuccessful .md probes over three and a half months. Many targeted repository-style names such as README.md, agents.md, and CLAUDE.md. Those probes are not evidence of demand for the site’s eventual Markdown twins.
Negotiation: agent-oriented tools and a coding agent
About 80% of negotiated requests came from ShapBot and ExaSearchBot, according to Siacci. Their identities were treated as declared rather than verified because the method described had no published address ranges for them.
He also reports about 1,400 requests identified as Claude Code. It sent the Markdown Accept header on 99.1% of its page requests, and Markdown was received in 97.7% of determinable cases; three quarters of these requests were for the help center. That result describes one client’s behavior against this site, not a general uptake rate for coding agents.
Rank #4
- All-in-One Complete Kit: This SANOOV RPi 5 bundle comes with Raspberry Pi 5 4GB RAM single board, active cooler, durable ABS case and screwdriver. No extra parts needed, ready to use right out of the box for beginners and hobbyists
- Powerful Single Board Computer: Equipped with 4GB RAM and high-performance processor, delivers fast running speed for 4K playback, AI projects, programming and daily computing tasks. SANOOV for raspberry pi 5 4GB is equipped with broadcom 64 quad-core Arm Cortex A76 processor with gigabit ethernet and upgraded with IEEE 802.11ac Wi-Fi, Bluetooth 5.0 dual-band 2.4Ghz and 5Ghz and Power Over Ethernet (POE). Upgrading delivers 2-3 x speed vs Pi 4, redefining the experience
- Efficient Active Cooler: Effectively lowers operating temperature and prevents performance throttling. Runs quietly even under long-time heavy load, ensures stable operation all day long. SANOOV RPi 5 4GB kit offer an active cooler, which combines an aluminium heatsink with a high-performance PWM fan. Active cooler is fully compatible with the Pi OS, which can effectively reduce the temperature of RPi5 and ensure its good performance during long-term high load operation
- Sturdy ABS Protective Case: Well-fitted for Raspberry Pi 5 board, can be secured with 4 screws to effectively protect the Pi 5 motherboard from damage, reserves full access to all ports and buttons. SANOOV uses ABS material to produce the case, which has a softer texture and feel. Meanwhile, SANOOV case adopts a layered design for easy disassembly and installation. (Tip: The Case cannot install M.2 HAT Add on Board and Solid State Drive!)
- Wide Application & Full Compatibility: Seamlessly compatible with official OS and mainstream peripheral accessories for Raspberry Pi 5. Whether you are a beginner, student, electronics hobbyist or professional developer, this all-in-one kit meets your diverse needs. It excels in IoT projects, robotics design, retro gaming devices, home media servers and other DIY creations. Backed by a large global community, you can easily find guides, technical support and shared projects online
Assistants fetching pages for people still requested HTML
Siacci counted roughly 10,000 ChatGPT-User, 2,700 hosted Claude-User, and 900 Perplexity-User HTML page fetches. He says one .md or negotiated fetch among these was his own ChatGPT test. This does not show what an assistant did with HTML after retrieving it: access logs reveal the requested response, not the system’s downstream processing. He suggests assistants may convert HTML themselves, but the observed traffic cannot establish that explanation or show that their behavior will remain the same.
How much smaller were the Markdown responses?
Across about 22,000 pages, Siacci reports that Markdown had a 78.7% smaller median response size than HTML; on the help center, the median was 83.9% smaller. These figures refer to bytes on the wire and include compression when accepted. They are not token counts, and they do not establish a corresponding reduction in model costs or processing time.
He also reports about 1,200 requests for llms.txt since early August, roughly one hundred attributed to declared AI-operator crawlers or fetchers. That is a reported count drawn from his earlier tally, not an independently verified figure.
Best Value
- 【What you Get】You will get 1*Pi 5 8GB Single Board,1*RasTech Case,1*Active Cooler,1*Screwdriver,1*Installation instructions,12-month free warranty, lifetime service, 24-hour prompt and friendly response.
- 【More Connectors】There are two USB 3.0 ports(5Gbps simultaneously) and two USB 2.0 ports, which triple total bandwidth ,support any combination of up to two cameras or displays. Peak SD card performance is doubled through support for the SDR104 high-speed mode. It provides a smooth desktop experience for you. Offer Gigabit Ethernet and a PCIe interface, along with dual-band Wi-Fi and Bluetooth 5.0/BLE wireless capability. The RasTech Pi 5 Kit use the new 27W 5.1V 5A USB-C power connector.
- 【 Support Dual 4Kp60 Display 】Each of the two microHDMI sockets can control a 4K display at 60 Hertz, now support HDR, offering super HD video for media streaming projects. RPi 5 is the first RPi model that comes with a PCI Express port (PCIe 2.0 x1 with 500 MB/s) to attach SSDs (requires separate M.2 HAT).
- 【 Excellent Chips And Applications】Pi 5 is a full-size Pi computer using silicon built in-house at Pi. The RP1 “southbridge” provides the bulk of the I/O capabilities for Pi 5. Pi 5 is more friendly and convenient in the development of Internet of Things, Web development, machine identification, automatic control and other electronic equipment applications and network.
- 【 Faster CPU, Better GPU 】 Pi 5 features a Broadcom BCM2712 64-bit quad-core Arm Cortex-A76 processor running at 2.4GHz, it delivers a 2–3× increase in CPU performance relative to RaspberryPi 4. The 800MHz VideoCore VII GPU is compatible to OpenGL ES 3.1 and Vulkan 1.2, substantial uplift in graphics performance. Pi 5 Offers lightning-fast CPU speed, a PCI Express interface, a Real Time Clock (RTC) and a power button and runs significantly cooler than Pi 4.
What the logs can—and cannot—prove
Siacci used three log sources, with Paris-local timestamps converted to UTC. The two request-specific app-server logs covered September 4–18, 2026; a front-end syslog covering May through September 18 supplied broader traffic context.
| Log source | Window and approximate volume | Fields noted |
|---|---|---|
| Dedicated app-server log for .md URL requests | September 4–18, 2026; 82,000 lines | Host, Accept, X-Forwarded-For, X-Real-IP, scheme, response time |
| Dedicated app-server log for requests whose Accept header contained text/markdown | September 4–18, 2026; 23,000 lines | Same fields |
| Front-end syslog for all traffic | May–September 18, 2026; 260 million lines | Host and X-Forwarded-For |
For negotiated requests, the analysis inferred whether Markdown had been returned by comparing response size with the .md twin. Siacci notes that logging $sent_http_content_type alongside the status would make this a direct lookup rather than a size-based inference. His team excluded internal test tools and eight internal addresses regardless of user-agent value.
Bot attribution has limits too. The author checked some identities against operator-published IP ranges downloaded on a dated snapshot, or through forward-confirmed reverse DNS for some crawlers. User-agent strings alone do not verify a bot. In the method he describes, Anthropic published no ranges, so ClaudeBot was inferred from traffic blocks in the site’s own logs and treated as declared, not proven. ShapBot and ExaSearchBot were likewise treated as declared. Operator ranges can change; Siacci says one OpenAI list changed on the day it was downloaded, making the dated snapshot important for anyone trying to replay the analysis.
What to take from this case study
- One content source can serve both formats. This implementation derives Markdown from cached HTML rather than maintaining a second set of page content.
- The two discovery paths are not interchangeable in practice. .md URLs and content negotiation exposed the same converted pages, but the observed client mix differed on this site.
- Design the response signals and edge cases deliberately. Alternate links help expose the .md path;
Vary: Acceptdistinguishes negotiated variants; canonical links, noindex signaling, exclusions, redirects, and Markdown error responses all affect how the feature behaves. - Instrument responses, not just requests. Separate logs for the two paths helped the author analyze traffic. Logging the returned content type would avoid inferring negotiated response format from response size.
- Report identity confidence honestly. Distinguish verified crawler traffic from user-agent claims or other declared identities, and retain the dated ranges used for verification.
- Keep the scope narrow. Two weeks of traffic on one domain cannot predict adoption elsewhere or reveal what an assistant does after a fetch.
For a team considering the same approach, Siacci’s implementation offers a concrete design pattern, but the traffic figures are a baseline for that site alone. The practical question for another site is not simply how many .md requests to expect; it is whether its own routes, caches, headers, logging, and bot-verification method can show which format was actually served and to whom.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
