Robots: How do they know ??

[Ric]

Whenever I post a new 'post' on this website and I look on the 'who's online' link, there are always "robots" there...

I understand in a way how the Google one "knows" that there is new data on the page(there is Google code on the page for the Adsense Adverts), but how does MSN, Yahoo slurp etc know that pages have been changed and therefore come and have a look and reindex if needed ???


Freaky !!!

 

[Ric]

More spiders than people, hmmmm....

...must stop leaving food out for them.

 

[Cory Cooper]

Go

I really think it's a case of so many hackers searching for/harvesting IP addresses of bulletin boards/discussion forums from the legit web crawlers of search engines. It's inflating the hits on sites that have boards, thus, they are tricked into ranking them higher, thus visiting more often.

C

(Nice image...Ric, Cory, and the spiders...the others are going to talk!)

 

[Ric]

hmmm...not exactly sure what you mean ?

I get my fair share of 'kiddy scripters' running port scans. (Note to kiddy scripters: before trying to 'hack' a website try and learn how to become invisible on the net first...I can see who you are, and I can find you if I want ! ) But not sure how that would inflate hits/ie page rank?

What I was trying in my head to work out, (without Googling for an answer!) was that there 'is' a piece of code (Spider) say the MSNBot... that visits this page as soon as I press 'submit reply'

If I press it now ...look at 'whos online' and there she is (too inquisitive to be male !), how does 'she' know to come have a look?

 

[Ric]

Wow, spiders are going mad -just me and 11 Spiders in here !


Who's Gigabot, she was very hungry 8 of them in here !

 

[Ric]

New Record, Ric and 19 spiders....18 were Yahoo Slurp Spiders ?

 

[Ric]

Well MSN spiders seem to be doing a good job...We're number 1 and number 2 depending on search keywords...

...not sure though how many Mac users will be using MSN as thier search engine of choice.