read before at : https://ext-5743301.livejournal.com/4655.html
Web crawlers, similar to Google and A9, keep an immense informational collection of Web pages and available records. To do this, they devise a program called a web crawler, or bug. This item normally and reliably surfs and pursues content in the Web. Pages that the bug finds are recuperated and recorded by text content, giving more weight to titles and section headers. Creepy crawlies explore continually the web starting with one page then onto the next, to list the significant substance of the Internet. Other than looking at the substance of titles and headers, a couple of ventures can recognize default names and keep a library of these page watchwords or key articulations in the rundown.
------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------
