Website Crawling and Your Content Writing for Humans and Search Engine Spiders

Content. Content. Content. The way Google previously structured their SEO scoring system for keywords meant everyone produced loads of content. Good or bad…they would just write something, publish it, and get started on the next piece. For them, it was all about getting website crawling by Google’s SEO spiders.

While they had the right idea of wanting to be found by Google and show up at the top of searches, it’s important to note now that you can’t ignore the basics of copywriting in exchange for a better position on the search engine results page (SERP). It’s necessary to focus on the basics of good writing and swing the pendulum back from writing for arachnids to writing for humans. After all, are they not the ones who will ultimately make a purchase to buy your products or services?

Read on for 6 questions to help you write for humans and search engine spiders.

Search engines have digital web crawlers that browse the internet in order to index pages. These website crawling spiders move rapidly from one page to another, reading every page and making copies that are then stored in an index.

When someone uses a search engine, the search phrase is compared to the most recent index of each indexed page. The most relevant pages are selected by the search engine, with the best pages appearing at the top of search. Website crawling is the main way search engines know what each page is about, allowing them to connect to millions of search results at once.

Search-engine algorithms continue to grow more sophisticated in how they experience the world with their goals starting to more closely mirror those of the human reader.

In the past, digital marketers approached SEO strategy as something completely separate from their content marketing plan. They would prioritize factors like keyword frequency over audience need so that they might rank as high as possible in the search results. However, today’s web crawler algorithms are much more sophisticated and seek to prioritize searchers’ intent.

For us marketers, this is great news. We no longer have to choose between creating strong content for humans or for web crawlers-we can now reach both audiences at once.

Even if these advancements hadn’t been made in recent years, we should still seek to write for our target audiences because their opinion is the one that matters. THEY are the ones reading, understanding, and deciding to do business with you or not.

Before you ever pick up your pen or make your first keystroke, ask these six questions to see if what you’re about to create will add any value in the minds of your buyers.

Why does this information matter? Why would your prospects or customers be interested in reading or listening to it? Is there a problem that you’re trying to address, an issue you’re trying to raise, or a role you’re trying to fill?

Leadership expert Simon Sinek has a great TED Talk on starting with the why. While it focuses on how leaders can inspire, consider the following quote from his presentation and how it’s equally applicable to the content creation process:

“What’s your purpose? What’s your cause? What’s your belief? Why does your organization exist? Why do you get out of bed in the morning? And why should anyone care? As a result, the way we think, we act, the way we communicate is from the outside in, it’s obvious. We go from the clearest thing to the fuzziest thing. But the inspired leaders and the inspired organizations…all think, act and communicate from the inside out.”

When you think of web crawlers evaluating your content as another member of your brand’s target audience, you are more likely to prioritize user experience over high-traffic jargon. When trying to convey importance for both reader and crawler, remember that first impressions matter.

According to Steve Krug — author of Don’t Make Me Think: A Common Sense Approach to Web Usability — you have about two seconds to capture a person’s interest. In this short window of time, you need to convince the reader your content is worth interacting with.

Assuming you passed through the first gate and found good answers to the “why” question, then it’s time to ask yourself who would care about it. Is it best suited for your current customers? Which ones — long-time users or first-time buyers? Or maybe it’s better for prospects. Is there a certain buyer persona that needs to hear this message? Or maybe several groups need to hear it, and so a one-size-fits-all message just won’t do.

Your content will need to be refined per each audience. Be sure to clearly identify who would be interested in your content marketing piece before you start writing.

Once you’ve clarified the “why” and the “who,” then you can consider the best type of content to create for these groups. Will a series of podcasts they can listen to suffice? Maybe adding a collection of CTAs on your website that point to a well-designed landing page will work better. Or maybe it’s time to invest your time and energy into creating an ebook or whitepaper. Only you know what deliverable will best send your message and be best consumed by your audience.

Parkinson’s Law says “work expands so as to fill the time available for its completion.” Very true. Figure out when this content needs to be created and set some deadlines. Create some milestones for you and your team to reach so that your great content ideas actually come to fruition.

When it comes to content strategy, we also recommend setting up a content plan and calendar so you know what’s coming down the road.

Will it sit on your website? In how many places? How will it be found? Will it be sent in a newsletter? Or maybe you’ll also end up creating a hard-copy piece that needs to be delivered to your brick-and-mortar locations. Does a display need to be created for the content?

Think through all the locations this content will be posted and consider all the corresponding elements to make this content shine.

The idea of human-machines taking over society is a staple of science fiction. Put aside the demise of humanity by machines, and consider the demise of your content abandoned by both humans AND machines due to irrelevance and poor writing.

A person confronted with a wall of text may have a more negative visceral reaction than a search engine algorithm, but the consequence is the same: a low ranking and irrelevant content.

However, if you choose to think more carefully about your content strategy using the questions above, you can arrange your content so it won’t cause people and machines to leave, but will instead engage, inform, and even refresh your audience and increase their appetite to know more.