Using Stanford Phoenix++ MapReduce to count the most frequent words in Wikipedia Database

Log of a quick and fun afternoon project.

1. Download Wikipedia English-language Database:

http://en.wikipedia.org/wiki/Wikipedia:Database_download

Download pages-articles.xml.bz2

Uncompress it ro pages-articles.xml (around 45GB)

2. Download Phoenix++:

http://mapreduce.stanford.edu/

http://mapreduce.stanford.edu/plus/phoenix++-1.0.tar.gz

make binary file and then find ‘word_count’ in ‘tests’ folder

3. copy wordcount and pages-articles.xml to the same folder

./word_count pages-articles.xml 1000 >wikiwords.txt

Using wordcount program to count top 1000 popular words from wiki database

4. Result: (Using 8 Xeon E5 server, 128 processing cores)

—————-

Job Name: myjob.pbs
Session:
Limits: neednodes=1,nodes=1,procs=128
Resources: cput=00:09:30,mem=49870344kb,vmem=51225052kb,walltime=00:02:57

—————-

Wikiwords.txt:

Wordcount: Running…
Wordcount: Calling MapReduce Scheduler Wordcount
Wordcount: MapReduce Completed

Wordcount: Results (TOP 1000 of 34425700):
THE – 193399750
QUOT – 124981071
GT – 122310147
LT – 119499174
OF – 119433785
ID – 91232877
IN – 79301110
AND – 78304198
TO – 68535720
A – 67833125
TITLE – 56486124
REF – 49681363
TEXT – 47596541
PAGE – 41273141
AMP – 39064116
USER – 35322581
IS – 33185078
HTTP – 31938499
FOR – 30390474
FORMAT – 30158636
MODEL – 29339983
TIMESTAMP – 29309382
NS – 29050522
REVISION – 28964076
CONTRIBUTOR – 28846548
SHA – 28800926
USERNAME – 28605557
COMMENT – 27578561
ON – 27358434
CATEGORY – 27120495
NAME – 25772995
WAS – 24528542
AS – 23756873
X – 23528969
WWW – 23189689
BY – 22963410
TALK – 22949011
COM – 22768372
WITH – 19990960
PARENTID – 19708590
T – 19512107
THAT – 18856762
WIKIPEDIA – 18397499
S – 18188906
IT – 17835250

….

Total: 6318735075

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s