Take the 2-minute tour ×
Stack Apps is a question and answer site for apps, scripts, and development with the Stack Exchange API. It's 100% free, no registration required.

Screenshots

Stackdump home

Stackdump search results

Stackdump question view

About

Stackdump was conceived for those who work in environments that do not have easy access to the StackExchange family of websites. It allows you to host a read-only instance of the StackExchange sites locally, accessible via a web browser.

Stackdump comprises of two components – the search indexer (Apache Solr) and the web application (written in Python). It uses the StackExchange Data Dumps as its source of data.

License

Stackdump (the application, not the content) is licensed under the MIT License. The content is obviously licensed under the cc-wiki license.

System requirements

Stackdump was written in Python (2.5 or later but not 3) and leverages Apache Solr, which requires Java (6 or later). It was written and tested on CentOS, but should work on other Linux distributions too. It should also work on Windows and OSX, but the start scripts will need some tweaking, particularly on Windows.

Having 3GB of RAM, at least 20-30GB of space, and around 10 hours is recommended if you're planning on importing the largest StackExchange site, Stack Overflow from August 2012. The other sites require much fewer resources, but 3GB of RAM is still recommended.

For the September 2013 data dump of Stack Overflow, it took just over 23 hours with a VM with roughly the same resources, using Stackdump v1.2.

Download

https://bitbucket.org/samuel.lai/stackdump/downloads

Besides Python 2.5+, Java 6 and 7-zip (for extracting the data dumps), all other dependencies are bundled up in the download for easier deployment.

Installation and usage instructions

https://bitbucket.org/samuel.lai/stackdump/overview

Code

https://bitbucket.org/samuel.lai/stackdump/src

Contact

Stackdump was written by Samuel Lai; feel free to contact me via Bitbucket.

share|improve this question
    
That's just amazing. Thank you! (Be informed that the standard had changed a bit since v1.0 - the names of the XMLs are now capitalized e.g. comments.xml -> Comments.xml) –  Rely Apr 20 '13 at 10:58
    
@Rely thanks for that report; finally got a chance to fix it. v1.1 is now out (available at bitbucket) with that fix and a few other minor things. –  Sam Oct 22 '13 at 2:41
    
FYI - Having this running over the current SO database (a few GBs) took much longer than I could let it, so I had to stop it at some point... I'm not sure why it's so slow and I didn't have the time to check this out, but maybe you could use some performance enhancements in the part that commits changes (a decent option may allow recontinuing an paused operation). –  Rely Nov 15 '13 at 14:59
    
The SO dataset has grown significantly since I first created this (mid-2011), and even then it took around 10 hours to import. I'll take a look at the commit interval and see if that can be tweaked. The problem with picking up from where it stopped is that there is no easy way to tell the XML parser to start from point X besides parsing and dumping everything until point X is reached (in which case, you might as well start again). –  Sam Nov 15 '13 at 15:17
    
I wrote a local web app to host the data dump, and query it quickly! github.com/peterburk/stackoverflowlocal It can search the 4.6GB StackOverflow data dump while compressed, and compress it further down to only 2.32GB. Stackdump requires decompressing it to 30GB, but I don't have the disk space for that. For those looking for an iPhone version, try a competitor's app, StackStash. But that takes up 3.6GB, and doesn't work on my laptop. itunes.apple.com/us/app/stackstash-stackoverflow-offline/… Peter –  user1196179 Feb 10 at 13:23
add comment

You must log in to answer this question.

Browse other questions tagged .