Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.
What?s in the data set:
Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069
The seed list for this crawl was a list of Alexa?s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.
However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.
We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available ?warts and all? for people to experiment with. We have also done some further analysis of the content.
If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you?re hoping to do with it. We may not be able to say ?yes? to all requests, since we?re just figuring out whether this is a good idea, but everyone will be considered.
RED alert at Conservative Central Office: Eric Pickles, the bullish Tory party chairman, has outed himself as a former communist.
Pickles has revealed for the first time his passion for Leon Trotsky and Karl Marx as a teenager growing up in Yorkshire in the 1960s. He admits that he switched to the Tories only in protest at the Soviet Union’s invasion of Czechoslovakia.
“I was massively inclined that way,” says Pickles in a radio interview to be broadcast tomorrow. “It was part of my upbringing.
“I was a pretty serious young chap. For my 14th birthday I got Trotsky’s History of the Russian Revolution as a present — and I read the damn thing.”
During his youth in Keighley, Pickles also read the Communist Manifesto and Das Kapital, and even compiled an admiring school project on Marx.
In the Radio 4 documentary, The ’89 Generation, leading MPs discuss the impact of the fall of the Berlin Wall on their political beliefs.
Michael Gove, the shadow schools secretary, also reveals a radical streak. He was on a workplace picket line when the Wall came down.
Pickles’s flirtation with communism is not mentioned on the official Conservative website; nor is it raised on his own Wikipedia entry, although the site does mention that his great-grandfather was a founder of the Independent Labour party.
The MP for Brentwood and Ongar tells the presenter Anne McElvoy that it was Moscow’s violent response to the Prague Spring, the liberal reforms introduced by Alexander Dubcek, the Czech leader, that drove him into the Tory fold.
“I was 16 years old in 1968 when Dubcek’s Spring was crushed. I was very interested in Dubcek and thought it was the natural evolution of communism. So I felt a tremendous shock when the tanks rolled into Prague.
“I thought the [inaction from the] British government [of the Labour prime minister Harold Wilson] was useless, in the way that only a 16-year-old can think the British government is useless. And I thought: ‘What’s the most outrageous thing I can do to protest? I know, I’ll join the Conservative party’.”
Pickles, 57, who discouraged colleagues from drinking champagne at this year’s Tory conference, says he did not initially intend to remain a Conservative but found himself naturally moving towards the right.
The MP, who is often deployed as an attack dog against the government, also confesses to shedding a tear when the Berlin Wall finally fell in 1989 — as do Nick Clegg, the Liberal Democrat leader, and Gisela Stuart, the German-born Labour MP.
Asked later if there was any danger of a left-wing relapse, Pickles guffawed: “That would shock them, wouldn’t it? After doing 40 years in the Conservative party, I think I’ll stay.”
Militancy also played a brief role in the career of another Cameron loyalist. Gove, a former journalist, reveals in the programme that he was on the picket line in a regional newspaper dispute when the Wall fell. The documentary unearths a clip of “Red Gove” berating the management of a local newspaper for reneging on its workplace promises.
“I was warming my toes ... by the brazier as I joined my colleagues in demanding the right to be represented by a trade union,” he says. “I was also a member of the Aberdeen South Young Conservatives — which caused no end of ribaldry.”
In the clip, the young Gove intones: “Management are denying people the chance to be governed by a collective bargaining deal and denying freedom of choice in the workplace.”
The fall of the Berlin Wall divided the Conservatives in power over how to deal with the aftermath. David Willetts, the shadow universities secretary, says Margaret Thatcher, then prime minister, was wrong to oppose German unification at the time.
One long-lasting impact of the Wall’s demise was a wave of immigration from eastern Europe, an issue seized by William Hague after he became Tory leader.
But Hague, now shadow foreign secretary, tells McElvoy, a columnist for the Evening Standard newspaper, that he regrets his controversial claim in 2001 that Britain was turning into “a foreign land”. He admits: “Partly we didn’t get it right and partly it was easily misrepresented.
“Talking about a country that’s going to change into something you don’t want was easily misrepresented as saying we were being overrun by foreigners. That was our fault.”
His policy on asylum was prescient, but the tone and wording were mistaken, he says. “I regret it among hundreds of things I did when I was leader of the Conservative party.”
Anne McElvoy presents The ’89 Generation at 11am tomorrow on BBC Radio 4
Contact us | Terms and Conditions | Privacy Policy | Site Map | FAQ | Syndication | Advertising
© Times Newspapers Ltd 2010 Registered in England No. 894646 Registered office: 3 Thomas More Square, London, E98 1XY