COLLECTED BY

Organization: Internet Archive

The Internet Archive discovers and captures web pages through many different web crawls. At any given time several distinct crawls are running, some for months, and some every day or longer. View the web archive through the Wayback Machine.

Collection: Wide Crawl started March 2011

Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.

What’s in the data set:

Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069

The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.

However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.

We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done some further analysis of the content.

If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered.

TIMESTAMPS

Caribbean kick off for RWC 2011 qualifying

(IRB.COM) Thursday 3 April 2008

Mexico will compete in the Caribbean tournament which will double as the first RWC qualifier

For the first time ever Rugby World Cup Qualification will begin in the Caribbean.

Qualification for Rugby World Cup 2011 in New Zealand starts on April 20 in the Cayman Islands as the 2008 Caribbean Championship doubles as a RWC 2011 qualifying event.

The host nation Cayman Islands will play Trinidad & Tobago on April 20 in the first match of the Championship. This is six months to the day after a global audience of billions watched the Rugby World Cup 2007 final in France when South Africa became World Champion.

The six-day knock-out qualifying Tournament will feature Barbados, Bahamas, Bermuda, Guyana, Jamaica, Mexico, Trinidad & Tobago and the Cayman Islands, with the final played on April 26. The winner will progress to meet the winner of the South America Division B Championship involving Colombia or Venezuela, Paraguay, Brazil and Peru.

“Rugby World Cup 2007 was a memorable Tournament,” said Rugby World Cup Chairman Dr Syd Millar. “The matches were competitive throughout, the venues full and a record global audience of over 4 billion watched the drama unfold on television. The 2011 Tournament may still be over three years away but planning and preparation for New Zealand is already well advanced. The beginning of the qualifying process kicks off the next phase as the Rugby community looks ahead to RWC 2011.”

Exciting journey

“The launch of the qualifying process also marks the start of a long and exciting journey for the IRB’s Member Unions across six global regions. RWCL will mark the historic occasion by taking the Webb Ellis Cup to the Cayman Islands, the first of many planned global visits to key RWC 2011 qualifiers,” added Dr Millar.

IRB Chairman, Bernard Lapasset, said, “Tony Spreadbury, who refereed the opening match of RWC 2007 in France, has temporarily come out of international retirement to take charge of the first RWC 2011 qualification match between Cayman Islands and Trinidad & Tobago. The IRB is very much looking forward to launching the RWC qualifying process later this month.”

“Eight teams will be determined through the global regional qualifying system, which will be completed in 2010 one year before the Tournament. These teams will join South Africa, England, Argentina, France, Australia, New Zealand, Scotland, Fiji, Wales, Ireland, Tonga and Italy who have automatically qualified for RWC 2011,” said Lapasset.

Watching with interest

Rugby New Zealand 2011 Chief Executive Officer Martin Snedden said: “While planning is well underway in New Zealand to host RWC 2011, the 2008 Caribbean Championship in the Cayman Islands will be the first step for a number of Unions on their road towards the 2011 Tournament. We will be following the RWC Qualifying Process with keen interest.”

The Tournament in the Cayman Islands marks the start of a new-look RWC qualifier process that has been streamlined to incorporate established Tournaments across the IRB’s six regions (North America & West Indies, South America, Europe, Africa, Asia and Oceania). This will remove duplication of matches in an already busy international calendar and includes Tournaments such as the European Nations Cup, the Africa Cup and the restructured Asian Rugby Football Union Tournaments including the Asian 5 Nations.

In terms of the regions, Europe and the Americas have been allocated two qualifying places, Africa, Asia and Oceania one place each, with the last place determined by a play-off.

RWC 2011 Caribbean Championship

April 20

Cayman Islands v Trinidad & Tobago (1030)
Guyana v Jamaica (1230)
Bahamas v Bermuda (1430)
Barbados v Mexico (1630)

April 23

Semi-finals (1430 & 1630)