Finally, somebody in a chicken costume who will do whatever you want. Check it out: http://www.subservientchicken.com
Thursday, July 31, 2008
Subservient Chicken
We knew the web was big...
Thoughts about - How many web pages are there on the web?
After reading the article, i think its not relevant to say that web has infinite web pages. fact is we cant count it. Nothing is infinite - its just our inability/incompetency to count.
Btw the example too was not satisfactory for me. a calender page should be seen as a single page rather then a page with multiple states. But the fact that identifying such pages would be difficult for a web spider.
How do we find all those pages? We start at a set of well-connected initial pages and follow each of their links to new pages. Then we follow the links on those new pages to even more pages and so on, until we have a huge list of links. In fact, we found even more than 1 trillion individual links, but not all of them lead to unique web pages. Many pages have multiple URLs with exactly the same content or URLs that are auto-generated copies of each other. Even after removing those exact duplicates, we saw a trillion unique URLs, and the number of individual web pages out there is growing by several billion pages per day.
So how many unique pages does the web really contain? We don't know; we don't have time to look at them all! :-) Strictly speaking, the number of pages out there is infinite -- for example, web calendars may have a "next day" link, and we could follow that link forever, each time finding a "new" page. We're not doing that, obviously, since there would be little benefit to you. But this example shows that the size of the web really depends on your definition of what's a useful page, and there is no exact answer.
We don't index every one of those trillion pages -- many of them are similar to each other, or represent auto-generated content similar to the calendar example that isn't very useful to searchers. But we're proud to have the most comprehensive index of any search engine, and our goal always has been to index all the world's data.
To keep up with this volume of information, our systems have come a long way since the first set of web data Google processed to answer queries. Back then, we did everything in batches: one workstation could compute the PageRank graph on 26 million pages in a couple of hours, and that set of pages would be used as Google's index for a fixed period of time. Today, Google downloads the web continuously, collecting updated page information and re-processing the entire web-link graph several times per day. This graph of one trillion URLs is similar to a map made up of one trillion intersections. So multiple times every day, we do the computational equivalent of fully exploring every intersection of every road in the United States. Except it'd be a map about 50,000 times as big as the U.S., with 50,000 times as many roads and intersections.
As you can see, our distributed infrastructure allows applications to efficiently traverse a link graph with many trillions of connections, or quickly sort petabytes of data, just to prepare to answer the most important question: your next Google search.
Posted by Jesse Alpert & Nissan Hajaj, Software Engineers, Web Search Infrastructure Team 
After reading the article, i think its not relevant to say that web has infinite web pages. fact is we cant count it. Nothing is infinite - its just our inability/incompetency to count.
Btw the example too was not satisfactory for me. a calender page should be seen as a single page rather then a page with multiple states. But the fact that identifying such pages would be difficult for a web spider.
Sent to you by happy via Google Reader:
via The Official Google Blog by Karen on 7/25/08
We've known it for a long time: the web is big. The first Google index in 1998 already had 26 million pages, and by 2000 the Google index reached the one billion mark. Over the last eight years, we've seen a lot of big numbers about how much content is really out there. Recently, even our search engineers stopped in awe about just how big the web is these days -- when our systems that process links on the web to find new content hit a milestone: 1 trillion (as in 1,000,000,000,000) unique URLs on the web at once!How do we find all those pages? We start at a set of well-connected initial pages and follow each of their links to new pages. Then we follow the links on those new pages to even more pages and so on, until we have a huge list of links. In fact, we found even more than 1 trillion individual links, but not all of them lead to unique web pages. Many pages have multiple URLs with exactly the same content or URLs that are auto-generated copies of each other. Even after removing those exact duplicates, we saw a trillion unique URLs, and the number of individual web pages out there is growing by several billion pages per day.
So how many unique pages does the web really contain? We don't know; we don't have time to look at them all! :-) Strictly speaking, the number of pages out there is infinite -- for example, web calendars may have a "next day" link, and we could follow that link forever, each time finding a "new" page. We're not doing that, obviously, since there would be little benefit to you. But this example shows that the size of the web really depends on your definition of what's a useful page, and there is no exact answer.
We don't index every one of those trillion pages -- many of them are similar to each other, or represent auto-generated content similar to the calendar example that isn't very useful to searchers. But we're proud to have the most comprehensive index of any search engine, and our goal always has been to index all the world's data.
To keep up with this volume of information, our systems have come a long way since the first set of web data Google processed to answer queries. Back then, we did everything in batches: one workstation could compute the PageRank graph on 26 million pages in a couple of hours, and that set of pages would be used as Google's index for a fixed period of time. Today, Google downloads the web continuously, collecting updated page information and re-processing the entire web-link graph several times per day. This graph of one trillion URLs is similar to a map made up of one trillion intersections. So multiple times every day, we do the computational equivalent of fully exploring every intersection of every road in the United States. Except it'd be a map about 50,000 times as big as the U.S., with 50,000 times as many roads and intersections.
As you can see, our distributed infrastructure allows applications to efficiently traverse a link graph with many trillions of connections, or quickly sort petabytes of data, just to prepare to answer the most important question: your next Google search.
Posted by Jesse Alpert & Nissan Hajaj, Software Engineers, Web Search Infrastructure Team
Things you can do from here:
- Subscribe to The Official Google Blog using Google Reader
- Get started using Google Reader to easily keep up with all your favorite sites
Monday, July 28, 2008
Thursday, July 10, 2008
Tuesday, July 01, 2008
July 1 2008 - Canada Day
Movies
The Garden State
- nice paced movie. its a story but a boy who doesn't have memory of his life thanks to his father prescription to drugs(medicine). During the movie, you realize that everyone has some sad story to tell which now a part of there life. No body is lucky and no body is unlucky, its just a matter of time. And sometimes you have to take sides/steps in life which might not be right but are for good.
No Country For Old Men(2007)
- true to its name. its violent yet sensible. its not like a everyday movie where in the end everything is turns out to be good. This story tells me that, if you are ruthless then be it, but if you have heart you it all the way and don't leave any stone's unturned, because one unturned stone may haunt you and can decide the course of your life (a character in the movie, had to give a guy water, he forgot it while thinking about money, when he realized he had to do it, he went back and got himself into a mess. which took his and his wife's life).
The Bank Job(2008)
- different style of robbery....by the way there is no style to it, its just plain lucky. But what ever happens after the robbery is more interesting. Robbery is just a ground built up for the mayhem. Has a good end to the story not that everyone got out of it but someone who went all the way were saved by the script writer .....lol
The Garden State
- nice paced movie. its a story but a boy who doesn't have memory of his life thanks to his father prescription to drugs(medicine). During the movie, you realize that everyone has some sad story to tell which now a part of there life. No body is lucky and no body is unlucky, its just a matter of time. And sometimes you have to take sides/steps in life which might not be right but are for good.
No Country For Old Men(2007)
- true to its name. its violent yet sensible. its not like a everyday movie where in the end everything is turns out to be good. This story tells me that, if you are ruthless then be it, but if you have heart you it all the way and don't leave any stone's unturned, because one unturned stone may haunt you and can decide the course of your life (a character in the movie, had to give a guy water, he forgot it while thinking about money, when he realized he had to do it, he went back and got himself into a mess. which took his and his wife's life).
The Bank Job(2008)
- different style of robbery....by the way there is no style to it, its just plain lucky. But what ever happens after the robbery is more interesting. Robbery is just a ground built up for the mayhem. Has a good end to the story not that everyone got out of it but someone who went all the way were saved by the script writer .....lol
Subscribe to:
Posts (Atom)
