Filter and sort parameters can multiply one product page into thousands of near-identical URLs — the most common cause of crawl-budget waste. Here’s how parameter order and hex casing create the duplicates, and the fixes Google actually recommends today.
Faceted navigation lets users filter a listing by attributes — color, size, price, brand. Each filter combination usually produces a distinct URL:
/shoes?color=red&size=10
/shoes?color=red&size=10&sort=price
/shoes?brand=nike&color=red&size=10
With even a handful of facets, the number of possible combinations runs into the thousands. Google describes faceted navigation as by far the most common source of overcrawling reported by site owners: crawlers burn budget on endless permutations instead of discovering your genuinely new content.
Beyond the combinatorial explosion, two subtle formatting issues double the duplicate count — and both are about how the URL string is written, not what it means.
A URL is matched as a literal string, so these are two different URLs to a crawler even though they render the identical page:
/shoes?color=red&size=10
/shoes?size=10&color=red
If your navigation code appends filters in whatever order the user clicks them, you generate a fresh URL for every ordering. The fix is to enforce a single canonical parameter order server-side — sort the query keys the same way every time before the URL is emitted.
Percent-encoding hex digits are case-insensitive per RFC 3986 — %3A and %3a decode to the same byte. But crawlers key on the raw string, so /caf%C3%A9 and /caf%c3%a9 can be indexed as two separate pages, splitting link equity between them. If different parts of your stack emit different casing, you’ve doubled the URL space for no reason. Normalize to one case (uppercase is the RFC recommendation) at your CDN, proxy, or application layer. You can confirm two encoded URLs are equivalent by decoding both in our URL decoder — if they produce the same text, they’re the same resource.
Google’s URL Parameters tool in Search Console was deprecated in 2022, so ignore any advice to configure parameters there. The current controls, from Google’s own faceted-navigation documentation, are:
robots.txt. If filtered pages have no search value, a disallow rule is the most direct way to stop crawl waste. Note that blocked URLs can still appear in results if other pages link to them.rel="canonical". For filter pages you want crawlable but not indexed separately, point the canonical at the unfiltered or primary combination. Over time this reduces crawl volume of the non-canonical variants.rel="nofollow". Adding nofollow to filter links can reduce crawling — but every link to a given URL must carry it for the signal to hold, and it’s a hint, not a directive.Don’t send conflicting signals: canonicalizing to a URL you’ve also blocked in robots.txt is self-defeating, because the crawler can’t fetch the canonical target to honor the relationship.
Parameter bloat distorts reporting as well as crawling. Because each parameterized URL is tracked separately, filter permutations can dominate your “top pages” report while the real page’s traffic is scattered across variants. In GA4, use parameter-based URL grouping or regex filters to roll the data up to the canonical URL so your numbers reflect actual pages.
rel="canonical", rel="nofollow", consistent parameter order, and 404s for empty results. developers.google.com/searchrel="canonical" and other canonicalization signals work. developers.google.com/searchHonest scope: search-engine behaviour evolves, and Google increasingly clusters URLs by intent rather than crawling every permutation — the mechanics here (canonical, robots.txt, nofollow) are current best practice, but treat ranking outcomes as guidance, not guarantees. This site’s role is the encoding half (order and casing); the indexing decisions are yours.
No. Google deprecated the URL Parameters tool in Search Console in 2022. Its algorithms now interpret parameters automatically, so the current controls are robots.txt disallow rules, rel="canonical", and rel="nofollow" — not a Search Console setting. Any guide telling you to configure parameters in Search Console is out of date.
Because a URL is matched as a literal string. Reordering the parameters produces a different string, so crawlers treat the two as separate URLs even though they render the same page. Multiply that across every filter combination and one product listing can explode into thousands of near-duplicate URLs. Enforce a consistent parameter order server-side so one canonical form exists.
Percent-encoding hex digits are case-insensitive per RFC 3986 (%3A equals %3a), but crawlers track the raw strings as distinct keys. If your site emits uppercase hex on some links and lowercase on others, each variant can be indexed separately, splitting link equity. Normalize to one case — uppercase is the RFC recommendation — at your CDN or application layer.
It depends on whether you want them indexable. If the filtered pages have no search value, block them with robots.txt — the most direct way to stop crawl waste. If some combinations should rank (like “red running shoes”), leave them crawlable and point rel="canonical" at the appropriate anchor. Don’t canonicalize to a URL you’ve also blocked in robots.txt — the crawler can’t read the canonical target to honor it.
Yes. Because each parameterized URL is tracked separately, your “top pages” report can fill with filter permutations that are really the same page, hiding true performers and muddying conversion paths. In GA4, consolidate with parameter-based URL grouping or regex filters so the data rolls up to the canonical version.
Written and maintained by the urlencodedecode.com team. Every technical claim on this page is verified against primary sources — the RFCs (3986, 3629, 4648, 7578), the WHATWG URL Standard, and official vendor or language documentation — rather than second-hand summaries. When a source contradicts a common assumption, we follow the source and note the discrepancy. Corrections: contactus@urlencodedecode.com.