When people post on social media, they often shrink links using URL shorteners. So instead of https://help.crowdtangle.com/faqs-and-troubleshooting/the-secrets-of-url-redirection, we'd post something like https://fb.me/URLsAreLong.
We do this by having our crawler bot follow the short link, as if it were clicking on it. That often leads to intermediary links, which we keep following until they stop. Once we get to a URL that no longer redirects, we store that as the final URL.
The only two URLs that remain in our system are the initial short one and the one at the end. As a result, searches for either of those will lead us to the post.
But there are some ways you could unexpectedly not find what you wanted:
- You ask for an intermediate link, not the short link or the final URL.
- The host's system stops redirecting us too early, and we think we're at a final URL when the host believes there are more.
- The host initially publishes the post at one URL, we store the first and last links in our system, and then the host changes the link to a new final URL. At that point, we will not recognize the new URL.