Take the 2-minute tour ×
Stack Apps is a question and answer site for apps, scripts, and development with the Stack Exchange API. It's 100% free, no registration required.

Screenshots

Stackdump home

Stackdump search results

Stackdump question view

About

Stackdump was conceived for those who work in environments that do not have easy access to the StackExchange family of websites. It allows you to host a read-only instance of the StackExchange sites locally, accessible via a web browser.

Stackdump comprises of two components – the search indexer (Apache Solr) and the web application (written in Python). It uses the StackExchange Data Dumps as its source of data.

License

Stackdump (the application, not the content) is licensed under the MIT License. The content is obviously licensed under the cc-wiki license.

System requirements

Stackdump was written in Python (2.5 or later but not 3) and leverages Apache Solr, which requires Java (6 or later). It was written and tested on CentOS, but should work on other Linux distributions too. It should also work on Windows and OSX, but the start scripts will need some tweaking, particularly on Windows.

Having 3GB of RAM, at least 20-30GB of space, and around 10 hours is recommended if you're planning on importing the largest StackExchange site, Stack Overflow from August 2012. The other sites require much fewer resources, but 3GB of RAM is still recommended.

For the September 2013 data dump of Stack Overflow, it took just over 23 hours with a VM with roughly the same resources, using Stackdump v1.2.

Download

https://bitbucket.org/samuel.lai/stackdump/downloads

Besides Python 2.5+, Java 6 and 7-zip (for extracting the data dumps), all other dependencies are bundled up in the download for easier deployment.

Installation and usage instructions

https://bitbucket.org/samuel.lai/stackdump/overview

Code

https://bitbucket.org/samuel.lai/stackdump/src

Contact

Stackdump was written by Samuel Lai; feel free to contact me via Bitbucket.

share|improve this question
    
That's just amazing. Thank you! (Be informed that the standard had changed a bit since v1.0 - the names of the XMLs are now capitalized e.g. comments.xml -> Comments.xml) –  Rely Apr 20 '13 at 10:58
    
@Rely thanks for that report; finally got a chance to fix it. v1.1 is now out (available at bitbucket) with that fix and a few other minor things. –  Sam Oct 22 '13 at 2:41
    
FYI - Having this running over the current SO database (a few GBs) took much longer than I could let it, so I had to stop it at some point... I'm not sure why it's so slow and I didn't have the time to check this out, but maybe you could use some performance enhancements in the part that commits changes (a decent option may allow recontinuing an paused operation). –  Rely Nov 15 '13 at 14:59
    
The SO dataset has grown significantly since I first created this (mid-2011), and even then it took around 10 hours to import. I'll take a look at the commit interval and see if that can be tweaked. The problem with picking up from where it stopped is that there is no easy way to tell the XML parser to start from point X besides parsing and dumping everything until point X is reached (in which case, you might as well start again). –  Sam Nov 15 '13 at 15:17
1  
I wrote a local web app to host the data dump, and query it quickly! github.com/peterburk/stackoverflowlocal It can search the 4.6GB StackOverflow data dump while compressed, and compress it further down to only 2.32GB. Stackdump requires decompressing it to 30GB, but I don't have the disk space for that. For those looking for an iPhone version, try a competitor's app, StackStash. But that takes up 3.6GB, and doesn't work on my laptop. itunes.apple.com/us/app/stackstash-stackoverflow-offline/… Peter –  user1196179 Feb 10 at 13:23
    
Sam, this is amazing! Thank you so much for making this. I can't believe how professional this product feels, from the instructions to the tools to the website. Import of SO took 21.5 hours on my 2.3GHz Haswell i7 Macbook — slightly annoying, but tolerable if you do it on your internal drive, since you can put your computer to sleep and seamlessly resume later. (Strangely, row processing seemed to slow down towards the end of the import.) Out of curiosity, how long did this take to make? –  Archagon Apr 24 at 4:29
    
I have a hunch that you could speed up the program significantly by utilizing multiple cores for SQL and Solr access since those appear to be the bottlenecks, but so far I haven't been able to figure it out. (I'm not very familiar with SQL, Solr, or Python multiprocessing, unfortunately.) Have you looked at all at Solr's DataImportHandler? They claim to be able to import Wikipedia's 40GB XML dump in about 50 minutes, which might be useful for speeding up Stackdump. –  Archagon Apr 24 at 4:29
    
@Archagon I'd guess that it gets slower during the import as the database and index grow larger. Solr actually runs as a separate process, so it is able to use multiple cores. The SQL runs in-process though. I've considered using the DataImportHandler, but because the data is manipulated before it is imported (the posts in the XML are grouped up into questions) it doesn't work out-of-the-box. Because of this grouping behaviour, I'm also not sure if parallel processing would be much benefit. I'm not sure if writing the import code in Java would be faster; open to suggestions though! –  Sam Apr 24 at 13:14
    
@Archagon Turns out I've actually thought about this a bit more a while back - bitbucket.org/samuel.lai/stackdump/issue/5. Feel free to add suggestions there on how it could be faster. –  Sam Apr 24 at 13:18
    
I added a comment on the BitBucket issue. By the way, I was wrong about my total run time, since I didn't take standby into consideration. I think the total active import time was actually closer to 7-10 hours. –  Archagon Apr 24 at 20:18

You must log in to answer this question.

Browse other questions tagged .