To extract links from a website, fetch its HTML, select every <a href="..."> element, read the raw href, and then decide how to resolve, filter, normalize, and deduplicate the results. Beautiful Soup is ideal for one page; Scrapy’s LxmlLinkExtractor is better for a controlled crawl.
This guide shows both approaches, including relative URLs, fragments, query strings, non-HTTP schemes, duplicate handling, metadata, dynamic pages, and failure recovery.
What counts as a link?
HTML defines an anchor’s href as a destination for web pages, files, email addresses, in-page locations, and other URL-addressable resources. It is therefore unsafe to assume every value is an HTTP or HTTPS page.
https://example.com/docsandhttp://example.comare web URLs./pricing,../images/logo.svg, and//cdn.example.com/app.jsare references that need a base URL.#featurespoints to a fragment in the current document.mailto:team@example.com,tel:+15551234567, andsms:+15551234567are contact links.javascript:void(0)and a bare#are commonly UI controls, not useful destinations.- Files, data URLs, custom schemes, and other values may be valid for a particular application.
MDN notes that fake href values can break copying, dragging, bookmarking, new-tab behavior, and no-JavaScript operation; use a button for actions that are not navigation.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
Choose the extraction method
| Need | Best fit | Why |
|---|---|---|
| Links from one fetched document | Beautiful Soup | Small, explicit parser with full control over raw and resolved values. |
| Domain, pattern, file-type, or crawl filtering | Scrapy LxmlLinkExtractor |
Built-in scope, regex, extension, canonicalization, and unique-link controls. |
| Links created only after JavaScript runs | Browser rendering, then parse the rendered HTML | Normal HTML parsers process the response they receive and do not promise browser execution. |
Extract every href from one page with Beautiful Soup
Minimal pattern
Beautiful Soup documents this basic loop:
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "html.parser")
for link in soup.find_all("a"):
print(link.get("href"))get() returns None when an anchor has no href. For production code, select only anchors with an attribute and validate blank values.
Production extractor with URL resolution
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urldefrag
def extract_links(html: str, page_url: str):
soup = BeautifulSoup(html, "html.parser")
results = []
for tag in soup.find_all("a", href=True):
raw_href = tag["href"].strip()
if not raw_href:
continue
absolute = urljoin(page_url, raw_href)
without_fragment, fragment = urldefrag(absolute)
results.append({
"raw_href": raw_href,
"url": without_fragment,
"fragment": fragment,
"text": tag.get_text(" ", strip=True),
"rel": tag.get("rel", []),
})
return results
html = """Install
Email"""
for item in extract_links(html, "https://example.com/products/page"):
print(item)The output keeps three useful representations: the exact author-supplied value, an absolute URL suitable for requests, and a fragment stored separately. Keeping the raw value matters when auditing markup or reproducing a page exactly.
Respect the document base URL
A page can contain a <base href="..."> element. Browsers resolve relative references against that base rather than necessarily against the address shown in the request. If browser-compatible behavior matters, read the document’s base element and pass its resolved value to urljoin before processing links.
Decide what to keep before normalizing
Fragments
Fragments normally identify a location within a representation and are not sent in an HTTP request. Remove them when calculating crawl identity, but retain them in a separate field when section-level navigation matters.
Free tools Windows power users keep installed
One-click scans. No signup required.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
Query strings
Do not remove a query string automatically. Parameters can carry search terms, pagination, filters, authentication state, or file versions. Tracking parameters can be removed only under an explicit allowlist or denylist that fits your project.
Relative and protocol-relative references
Resolve /path, ../path, and page.html with the fetched page URL. A reference beginning with // inherits the page’s scheme through urljoin.
Non-navigation and unsafe values
For a real destination list, exclude blank values, #, and values beginning with javascript:. Decide separately whether mailto:, tel:, sms:, data:, or custom schemes belong in your output. Never execute a JavaScript URL merely to extract it.
Remove duplicates without losing evidence
There are three legitimate policies:
- Preserve occurrences: keep every anchor, including repeated navigation links.
- Deduplicate exact values: collapse identical raw strings while retaining a count.
- Deduplicate crawl targets: resolve URLs, apply your fragment and query policy, and use the normalized result as the key.
If auditing or accessibility work is involved, store occurrence count, anchor text, source element position, and rel values even after producing a unique destination list. Canonicalization can change the URL visible to a server, so treat it as a deliberate crawl policy rather than a harmless formatting step.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallRank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Simple ordered deduplication
seen = set()
unique = []
for item in extract_links(html, page_url):
key = item["url"]
if key not in seen:
seen.add(key)
unique.append(item)Extract links at crawl scale with Scrapy
Scrapy describes a link extractor as an object that extracts links from responses. Its LxmlLinkExtractor defaults to tags=('a', 'area') and attrs=('href',), and supports domain restrictions, allow/deny regular expressions, CSS or XPath restrictions, extension filters, custom value processing, whitespace stripping, canonicalization, and unique filtering.
Domain-limited crawler
import scrapy
from scrapy.linkextractors import LinkExtractor
class DocsSpider(scrapy.Spider):
name = "docs"
start_urls = ["https://example.com/"]
def parse(self, response):
extractor = LinkExtractor(
allow_domains={"example.com"},
deny_extensions={"pdf", "zip"},
unique=True,
)
for link in extractor.extract_links(response):
yield {
"url": link.url,
"text": link.text,
"fragment": link.fragment,
"nofollow": link.nofollow,
}
yield response.follow(link.url, callback=self.parse)Scrapy’s Link object exposes the destination, visible text, fragment, and whether the source carries nofollow. Use allow and deny for URL patterns, allow_domains and deny_domains for scope, and extension filters for assets you do not want to crawl.
When canonicalization helps—and when it hurts
Canonicalization is useful for duplicate checking when equivalent spellings should map to one crawl target. Keep the original or non-canonical URL when exact markup, server behavior, signed URLs, or query ordering matters.
Dynamic pages and browser-rendered links
Beautiful Soup and Scrapy parse HTML delivered in a response. They do not, by themselves, guarantee execution of JavaScript that inserts anchors later. If the links appear only after a script runs, obtain rendered HTML with a browser-capable workflow and then apply the same extraction and normalization rules. Also check whether the page requires a consent interaction, login, bot challenge, or a wait for a selector or network idle.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
Or skip the browser setup
ScreenshotNeo can render a URL through a website screenshot API when you need a reliable visual capture or rendered-page step before inspecting links. Its API accepts options for waiting on a selector, a delay, or network idle; custom headers, cookies, user agents, authorization, timezone, geolocation, JavaScript, and CSS can handle pages that do not expose their final state immediately. Cookie and consent banners, newsletter popups, and chat widgets are removed before capture. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and the response identifies the result with X-Page-Verdict and X-Billed headers. An MCP server provides take_screenshot, get_page_info, and capture_pdf tools for Claude, Cursor, and other MCP clients.
For a direct capture, see the ScreenshotNeo API documentation:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpYou can also use Python or Node.js:
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);The Free plan includes 1,000 shots per month with no card; paid plans start at $5 for 3,000 shots. Create a free ScreenshotNeo account to try it.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Troubleshooting common extraction failures
No links are returned
Confirm that the response is HTML, not a redirect, error page, or API payload. Print the response status and a short prefix of the body. If a browser shows links absent from the response source, they are likely injected by JavaScript.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Relative URLs point to the wrong host
Pass the final response URL, not the original redirecting URL, to urljoin. Then account for any HTML base element.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
Too many duplicates
Choose whether fragments, tracking parameters, trailing slashes, and query ordering define identity. Normalize once, then deduplicate; retain occurrence metadata separately.
Important links are missing
Inspect area elements, nonstandard attributes, embedded JSON, and links revealed after interaction. Scrapy’s default extractor includes a and area, while a Beautiful Soup loop over only a does not.
Requests fail while parsing works locally
Respect robots and access rules, use appropriate timeouts, identify your client honestly, and handle redirects, compressed responses, rate limits, and authentication. A parser cannot recover links from content it never received.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Operational checklist
- Fetch the final HTML and record status, content type, and final URL.
- Extract anchors with an explicit missing and blank-
hrefpolicy. - Preserve raw values when provenance matters.
- Resolve relative references against the correct base.
- Store fragments separately if crawl identity excludes them.
- Define a query and tracking-parameter policy before deduplication.
- Filter fake controls and decide which non-HTTP schemes to retain.
- Keep anchor text,
rel, fragment, and occurrence data when needed. - Use Scrapy for crawl scope and browser rendering for JavaScript-created links.
Frequently Asked Questions
Should I extract href or link text?
Extract both when possible: href identifies the destination, while visible text supports audits, search indexing, and accessibility checks.
Are links in JavaScript source the same as HTML links?
No. A URL string in a script or JSON object is not necessarily an anchor destination; parse it with a format-specific method unless the browser turns it into an element.
Can I crawl every extracted URL?
Only after applying scope, robots, authentication, rate, file-type, and safety policies appropriate to your project.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Recommended Free Tools
