How I Use the Internet Marketing Ninjas Broken Links, Redirects and Google Sitemap Generator Tool

In SEO, tools can help make evaluation infinitely more efficient. In a lot of cases, you have to use a combination of tools to really see all of the angles. Well, we hate having to go a lot of places to gather intelligence, so we recently released a tool called the Find Broken Links, Redirects & Google Sitemap Generator Free Tool or the Site Crawler Tool for short.

It’s pretty extensive, with a lot of data (ok, it’s like a fire hose of data), but it packs a ton of recon into one report. Of course, a tool is only helpful when you can make use of the information it gives you. Like anything, everyone will have their own reasons for using the tools and way of reading the information. But to help you generate some ideas, here are the ways I use each section of this expansive tool.

URL of Pages Being Spidered

Just like a site: command, this column lets you look at all of the URLs on your site in a nice ordered list. Aside from using this list as the guide for the figures detailed in the rest of the report, this column is a great way to check your URLs for a few key elements:

• Use of relevant keywords
• Use of best practices like hyphens instead of underscores and no spaces
• Mixed case letters
• Obvious instances of duplicate URLs

This close-up of your top URLs is an easy place to spot anything unusual, or to locate a site-wide issue.

On Page Report

The next column here links to the IMN On-page optimization Tool. That’s a completely different tool which probably deserves its own post, but for now I’ll summarize by saying this tools looks at all of the following on-page elements:

• Title tag
• Meta Description
• Meta Keywords
• Total words on page
• Words that are links on the page
• Words that are not links on the page
• Number of external links
• Total distinct Words
• Number of links on the page
• Keyword density for specific phrases

There’s a lot you can do with all of that information, but I want to stay focused on the bigger picture, so that will have to wait for another day.

Level from Home

This category will give you a quick insight into how deep your pages go from your homepage. Pages that are linked from the home page will be Level 1; pages that are linked from those pages will be Level 2 and so on. And why does this matter to me? Well, based on the concept that your home page is your most powerful and authoritative page, then from a logical, and search engine standpoint, pages that are closest to your home page are the most important to your site.

Joost De Valk has a much more detailed post on site structure here. But the basic reward from reading this section of the tool report is that you’ll be able to tell if you have valuable pages buried too deep on your site. Your important pages could be those that you want to show up in searches or pages that typically convert well or bring in a significant amount of traffic. If it takes 3 or more clicks to reach these pages, you may want to consider moving them “up” in your site structure. It tells search engines that these pages are of greater importance to you and if those pages truly do matter to your business, bringing them closer to the home page could be helpful for users as well.

Status

Pretty simply, this column tells you where your broken links and redirects are. After a long list of crawled pages, the bottom of the report (and the floating table) gives you a list of internal and external errors. The 200 statuses are fine. Mostly I’m looking for are:

404 Errors – Internal and external broken links need to be fixed; it’s just part of routine site maintenance. But I’ve gotten a false positive or two on 404s, so click the link before you change anything.

301 & 302 Redirects – A 301 is ok, but I’m actually more interested in 302 redirects and making sure that a 302 (temporary redirect) wasn’t accidentally put in place when a 301 (permanent redirect) should have been. For more on the 301 vs 302 issue, check our Rick DeJarnette’s Ultimate Guide to 301 Redirects.

408 and 500 Errors – These errors may represent a bigger problem that needs investigation and work on the server side, so they are certainly helpful to identify. But again, click the link before you call your IT guy.

Internal links

I love this column. It shows the number of internal links pointing to a page, and when you click on it, you get a list of all of those pages. So the main things I’m looking for are:

• Do my most important pages have enough internal links?
• Do pages have links from outside the main navigation?
• Are the most powerful pages being used well?

These first two questions are usually a yes or no situation based on the numbers. If you have a page which you want to rank well that only has a few internal links, it needs more.

I’m also looking for variations in the numbers. Linking to pages from the main nav is good, but having in-content links from other pages is also powerful. So if every page in the main nav has the same number of links it means there probably should be more in-content, internal links pointing to those pages too.

To answer the third question (and to determine where to place internal links to the pages which need more), you may need to go to another tool like the ahrefs site explorer, which lets you look at a breakdown of external links coming into all of your pages. With that info and the internal links column from the Site Crawler Tool, you can make sure that your pages with a lot of back links are sending a lot of internal link love to the pages which are of the most commercial value to you.

Link Text

Aside from just looking at the raw number of internal links, the tool also shows you the first instance of anchor text it locates to a page. This allows you can evaluate some of your choices in anchor text and decide if there may be a better choice of words you could be using. In some cases, you’ll find that you can better optimize your link text to reflect relevant keywords to the linked page. In other cases, you may want to cut back on a string of keywords or fix a situation where multiple pages are using overly similar internal link text. No matter what you find, this column gives you a valuable overview of how you may be able to sharpen your internal link anchor text strategy.

Internal and External Links on Page

The next 2 columns give you another look at on-page links. While the first internal links column tells you how many internal links are coming IN to a page, this one tells you how many internal links are going out of a page. So again you can use this information to make sure your powerful pages are sending out plenty of internal links.

The external links column of course looks at instances where you link out to pages. Clicking on that number, you can see exactly what those outgoing links are. From there, you can evaluate whether or not you actually want to link out to those places and also see if you have opportunities to add more authoritative external links. Linking out to trusted websites can be another signal of trust that you send to search engines and provide practical resources for your users. So don’t be afraid to link out. Just choose wisely who you decide to link to. They should be as trusted as you would like to be.

Size of page

When it comes to web pages, bigger isn’t always better. The size of the page is the total of the file sizes of all the elements that comprise the page. Because slow page load times can be a deterrent for users and can be a negative ranking factor for search engines, page size is a factor. In the tool report, you not only get the file size but you can click on it to test the load time of any single page. If a page is loading slowly, it may be time to clean up your HTML and CSS, compress some images or get rid of some unnecessary elements.

Check Images

This link takes you to our Check Image Sizes, Alt Text, Header Checks, and More Tool. This is another useful, little tool that lets you scan a page for broken links and redirects, but also alt text. This is probably one of the fastest ways to check to make sure all of your images have useful alt text.

Since search engines can’t read pictures, alt text can tell them what an image is. And when used well, alt text can incorporate both an accurate description of an image and a keyword or two. Again Rick does a great job talking image optimization in a recent post.

Title Tag of Page

There’s no understating the value of a title tag in terms of on-page optimization. So this column lets you take a clear look at all of your title tags. I’ve offered a number of title tag optimization tips in the past, but the basic benefit of this column is that, in a quick glance, you can look for the best use of keywords as well as variation in your titles. There may be other ways to get this info, like a site: command for example, but this is a comprehensive, one-page view of all of your page titles. The nice thing about a close comparison like this is that it tends to make any gaping holes jump out at you.

Meta Description

This final column is the meta descriptions of all of your pages. Even though meta descriptions aren’t a ranking factor like title tags, seeing them laid out this way will help you see right away if you’re overusing the same meta description for too many pages. Even though the words in your meta description may not help you rank better, they can help entice users to click on you over other listings in the SERPs.

So they deserve some attention for, at the very least, their marketing value. This column can help you evaluate the descriptions on all of your pages, but it’s most useful to help you zero in on which ones really need to be changed.

Sitemap

Yeah, that’s the other thing. When the report is finished running, this tool can also give you an XML Sitemap to help search engines find all of your pages.

And The Latest Development…

Another column actually just got added to  this tool today! The new feature is a spellchecker, which can locate possible misspellings on the pages of your site. When you click on those results you’ll get a full list of possibly misspelled words on your page. You’ll be able to discount some words like “Googlebot” or “AdWords”. But when you’re optimizing, you might as well clean up those pesky typos as well.

So those are the highlights of what I look for with one of the reports from our site crawler tool. Of course, with a tool like this, there are always going to be different discoveries on a case-by-case basis. But that’s what makes it exciting. When you run the tool, you never know what’s going to grab your attention and make you go “Well that’s odd…” But whatever it is, you’re bound to find something that inspires you and helps you make changes for the better.

So if you’ve read this post, go use the Broken Link, Redirects and Sitemap Generator Tool and tell me what you see! And to all my ninjas out there, if you have any other ways you use our tools, let’s hear them.