BoroLook Search - How Spider index your website

Search Menu

| Meta Using | HTML Coding | Live Crawling Data Sample |

Spider looks of a website:

BoroLook spider stores data from Title tag, Page Body, Meta Description, Meta Keywords and Anchor hyperlink texts.

Title Crawling:

BoroLook spider makes change Title Tag of a page if the contents of <title>…</title> are less than 20 characters. For example: if you have a page where the Title Tag is like “my site”, then BoroLook spider will convert it to “Welcome to: yoursitename.com” because “my site” have only 7 characters.

We noticed that sometime webmasters make a mistake when they make websites or pages using third party software like Dreamwaver, FrontPage etc. They forget to change the Title Tag according to their necessary. For this cause, the Title name remains like “New Page 1”, “New Page 2”, “New Page 3”, “Untitled Document”, “Untitled Page” etc. These types of Title Tag have no meaning. So, when BoroLook spider gets these types of page Title at the indexed time, it converts to “Welcome to: yoursitename.com” automatically.

BoroLook spider collects maximum 70 characters from a Title Tag.

Pre-Description Crawling:

BoroLook has two types of crawler for indexing Home Page and internal pages for collection Pre-Description of a page. BoroLook takes maximum 180 characters for page Pre-Description.

(a) Home Page: Home Page crawler gives more importance on Meta Description for collecting Pre-Description of a page. See the following diagram to know how BoroLook spider gives importance for Home Page crawling.

Meta Description -----------------> Page Contents -----------------> Hyperlink Texts

BoroLook spider stores data from Meta Description, if the total contents are greater than 20 characters. If the Meta Description contents are less than 20 characters then BoroLook spider moves to Page Contents. The Page Contents (total characters of under <body>and </body>) also should be greater than 20 characters; if not then it will move to Hyperlink Texts.

Hyperlink Texts should be anchor links. If BoroLook spider does not get any sufficient contents from Meta Description and Page Contents then it collect all the anchor link texts of the pages and make separate by commas. When all the anchor link texts will more than 20 characters then BoroLook spider will add those contents as Pre-Description, otherwise not.

(b) Internal Page: Internal Page crawler gives more importance on Page Contents. See the following diagram –

Page Contents -----------------> Meta Description -----------------> Hyperlink Texts

Reasons: For Internal Page, first BoroLook spider collects information from Page Contents. We have found, more than 14% websites have same Meta Description on the every Internal Pages. We always try to collect more and more information from every pages rather than collection same contents.

Sometimes BoroLook uses both methods for Home Page and Internal Page indexing.

Finalization:

If BoroLook spider not gets any sufficient contents from one of above (Title and Pre-Description) then it will be not store the page information. BoroLook will make red line on those urls where can’t be crawl by BoroLook spider. You can give the page Title and Pre-Description manually or delete form the Webmaster Account (For enabling Webmaster Account, you’ll need to submit your website and need to login using your First Name, Last Name and Email)

bottom