matpalm - Latest Comments

Re: brain of mat kelcey

Srikant Jakilinki — Thu, 03 May 2012 05:27:54 -0000

Of course you did not find any words in common. The WordBags API explicitly gives unique word usage per-user. Simple GIGO really

Re: brain of mat kelcey

John — Fri, 30 Mar 2012 05:15:02 -0000

Well, I think I just noticed the change as 15 minutes ago something that used to take me to philosophy is now taking me to truth... o.0 I think someone changed something.

Re: brain of mat kelcey

mat kelcey — Wed, 21 Mar 2012 16:08:49 -0000

Sorry Kiran I no longer have access to a large enough cluster to do this.
The closest I have is http://bit.ly/GJgylB which is the highest frequency top level domains.

Re: brain of mat kelcey

Kiran J — Wed, 21 Mar 2012 15:38:03 -0000

Hi Matt,

I would like to have the all the domain names present in the common crawl. I have asked the admins at commoncrawl repeatedly for this without success.

I would be thrilled if you could create a dump of this data and post it somewhere.

Thanks in advance.

Re: sketching

lalakis — Fri, 16 Mar 2012 07:26:47 -0000

I could be nice if you update your table with a context triggered pairwise hashing algorithm like ssdeep

Re: brain of mat kelcey

mat kelcey — Sun, 04 Mar 2012 19:08:32 -0000

Nice pick up Tom, I did in fact have an error in some parsing which meant I was dropping numbers; stupid silly bug.

I agree Wikipedia is a bit of a special corpus and lots of people don't appreciate that it's not fully representative of common language, though what is I guess...

Thanks for the comment!

Mat

Re: resemblance with the jaccard coefficient

Tom Morris — Sun, 04 Mar 2012 12:26:48 -0000

Using the processor's POPCNT instruction would be much faster than your little Kernighan inspired loop.

Re: brain of mat kelcey

Tom Morris — Sun, 04 Mar 2012 12:08:07 -0000

Both of your longest phrase examples seems suspicious to me. Did you drop all numbers in your initial parse? Intentionally?

A little Googling confirms my suspicions that these were actually of the form:

United Nations Security Council Resolution 1699, adopted unanimously on August 8, 2006, after recalling
As of the census [ 1 ] of 2000, there were 9536 people, 3922households, and 2517 families residing in the city.

To answer your question about templates vs cut & paste, templates (infoboxes) are excluded from the body text, but this type of stylized pro forma structure is pretty common in Wikipedia. Some of it's from cut & paste or a single author working on a series of related articles, but often it's a semi-formal convention adopted by a group of authors.

There are a number of Wikipedia-isms that would be fascinating to study statistically if there was an equivalent corpus to compare against. For example, I suspect the frequency of the word "notable" is much higher in Wikipedia than elsewhere because they've got a notability requirement for inclusion, so authors writing about marginal cases take pains to stress why their subject is "notable."

Re: sketching

lalakis — Tue, 28 Feb 2012 05:19:18 -0000

And one last question. Broder's algorithm propose to hash multiple times each shingle so i.e: shingle1 = {(a,b,c),(e,f)} then according to sketching approach i hash this: H(shingle1) = dig1 then i hash this dig1 again H(dig1) or i rehash shingle1 and i choose the min value from all the digests of this shingle?

Re: sketching

mat kelcey — Mon, 27 Feb 2012 10:58:56 -0000

i used a simple modulo hash based on some code from "introduction to algorithms". i've actually since found out it has a bug in it and it produces very poorly distributed values, i guess it worked well enough for this experiment :/ this was a few years ago now. i'm not sure you can use SHA hashing as the hash function needs to be seedable. when i run something like this now i use murmur hashing; seedable, fast and very well distributed.

Re: sketching

lalakis — Mon, 27 Feb 2012 10:33:16 -0000

Which hash algorithm do you use to hash many times the shingles and compute its sketch in the end?I see in bibliography a lot of rabin's fingerprints but why not just a simple SHA-1 won't work?

Re: simhash

lalakis — Thu, 23 Feb 2012 08:02:17 -0000

Besides the length of each n-gram there is another property of construction gthe features which is the sliding window. Did you choose this ramndomly as ewll or there is a rational in this?I.e:
{ "Bobs cafe", "cafe is", "is excellent", ... }
It is obvious that using the token-ngram scheme where each piece of information is a word the sliding window equals 1. Why i.e you didn;t choose token ngrams of length 3 with a sliding window 2? Is there some logc there?thank you

Re: simhash

mat kelcey — Mon, 06 Feb 2012 13:15:34 -0000

Can't think why not. It's a larger hash size (SHAs start at 160 bits) which I saw improved accuracy but on the other hand SHAs aren't cheap to calculate.

Re: simhash

mat kelcey — Mon, 06 Feb 2012 13:08:53 -0000

for small documents where i've been interested in differences such as punctation or character normalisation (ie is "Bobs cafe" the same as "Bob's café" ?) i've used character ngrams {"Bo", "ob", "bs", "s ", ... }

for larger documents where i've been more interested in general content i've used token ngrams eg { "Bobs cafe", "cafe is", "is excellent", ... }

varying the ngram length give different results depending on the overlap of content and the overall document length. there is no magic number.

Re: simhash

lalakis — Mon, 06 Feb 2012 04:19:13 -0000

Also can you give some more information about the way you construct the shingles? Is there any particular way to construct the shingles or you group words together choosing the just it's size?

Re: simhash

lalakis — Mon, 06 Feb 2012 04:12:01 -0000

Yes it is clear.And can you use any kind of crypto-hash functions like SHA? Or there is a special one with particular features?

Re: simhash

lalakis — Mon, 06 Feb 2012 04:09:23 -0000

Yes it is clear.And can you use any kind of crypto-hash functions like SHA? Or there is a special one with particular features?

Re: simhash

mat kelcey — Fri, 03 Feb 2012 12:22:01 -0000

It might also help to think of it as a form of dimensionality reduction where the hash functions are used to decide where documents are located in the lower dimensional space.

The important thing is that the hash functions produces the same result for the same input.

There is some information lost in the sense you can't go backwards from the hash values to the shingles that generated them but that doesn't matter, all the comparisons are done with the hashed values, not the original shingles.

Does this help clarify things a bit?

Re: simhash

lalakis — Fri, 03 Feb 2012 12:02:04 -0000

I can't understand how simhash works with a good approximation in similarity checking since the initial shingles are hashed using a crypto hash function.
That means that there is no correlation between the produced digest and the input. ???

Re: brain of mat kelcey

mat kelcey — Sun, 08 Jan 2012 18:58:48 -0000

Hey,
I've been working out the best way to make something like
this publicly available,
I'll let you know when it's up.
Mat

Re: brain of mat kelcey

Manikandan Paneerselvam — Sun, 08 Jan 2012 06:42:33 -0000

Hi Mat,

Can you share the 3TB Results with us ?

Re: brain of mat kelcey

mat kelcey — Mon, 02 Jan 2012 19:18:32 -0000

I think you mean "The first link in Surface Water Sports i now Skurfing but wasn't at the time you wrote this blog post"

Re: brain of mat kelcey

mat kelcey — Mon, 02 Jan 2012 19:10:41 -0000

yes; not in parentheses, not in italics
https://github.com/matpalm/wik...

Re: brain of mat kelcey

semininja — Mon, 02 Jan 2012 19:07:50 -0000

The first link in Surface Water Sports is Skurfing.

Re: brain of mat kelcey

semininja — Mon, 02 Jan 2012 19:01:43 -0000

The alt-text from this comic (http://xkcd.com/903/) says: "[I]f you take any article, click on the first link in the article text not in parentheses or italics, and then repeat, you will eventually end up at 'Philosophy'."