25 9 / 2012

Before I begin let me make one thing clear, I was just playing around and the code you are about to see is just something that came to me out of random,it’s not meant for production but its an idea I want to shootout. For me Redis can turn out to be an excellent inverted index storage system due to its available data-structures. I won’t be implementing anything from scratch or anything; but just to kick off and get some more minds into circle, I took whoosh (no particular reason) and tried to modify its storage for Redis, so that it becomes a semi equivalent of RamStorage. So without further delay here is the gist for my code: 

Just to try it I’ve also made a python script to index the Piratebay’s data dump [Complete Gist is here]. Now just to mention I’ve indexed the Piratebay’s dump before in Lucene and it was disk stored index. I am not going to compare them today as I am not satisfied with the speed yet and I am planning to write my own IndexWrite and IndexReader classes. Let’s see if somebody can pull similar solution for Lucene in the mean time.

  1. maxpert posted this