48

Screenshots

Stackdump home
(source: edgylogic.com)

Stackdump search results
(source: edgylogic.com)

Stackdump question view
(source: edgylogic.com)

About

Stackdump was conceived for those who work in environments that do not have easy access to the StackExchange family of websites. It allows you to host a read-only instance of the StackExchange sites locally, accessible via a web browser.

Stackdump comprises of two components – the search indexer (Apache Solr) and the web application (written in Python). It uses the StackExchange Data Dumps as its source of data.

License

Stackdump (the application, not the content) is licensed under the MIT License. The content is obviously licensed under the cc-wiki license.

System requirements

Stackdump was written in Python (2.5 or later but not 3) and leverages Apache Solr, which requires Java (6 or later). It was written and tested on CentOS, but should work on other Linux distributions too. It should also work on Windows and OSX, but the start scripts will need some tweaking, particularly on Windows.

Having 3GB of RAM, at least 20-30GB of space, and around 10 hours is recommended if you're planning on importing the largest StackExchange site, Stack Overflow from August 2012. The other sites require much fewer resources, but 3GB of RAM is still recommended.

For the September 2013 data dump of Stack Overflow, it took just over 23 hours with a VM with roughly the same resources, using Stackdump v1.2.

Download

https://bitbucket.org/samuel.lai/stackdump/downloads

Besides Python 2.5+, Java 6 and 7-zip (for extracting the data dumps), all other dependencies are bundled up in the download for easier deployment.

Installation and usage instructions

https://bitbucket.org/samuel.lai/stackdump/overview

Code

https://bitbucket.org/samuel.lai/stackdump/src

Contact

Stackdump was written by Samuel Lai; feel free to contact me via Bitbucket.

Improve this question
12
  • 1
    That's just amazing. Thank you! (Be informed that the standard had changed a bit since v1.0 - the names of the XMLs are now capitalized e.g. comments.xml -> Comments.xml)
    – Omer Perry
    Commented Apr 20, 2013 at 10:58
  • @Rely thanks for that report; finally got a chance to fix it. v1.1 is now out (available at bitbucket) with that fix and a few other minor things.
    – Sam
    Commented Oct 22, 2013 at 2:41
  • FYI - Having this running over the current SO database (a few GBs) took much longer than I could let it, so I had to stop it at some point... I'm not sure why it's so slow and I didn't have the time to check this out, but maybe you could use some performance enhancements in the part that commits changes (a decent option may allow recontinuing an paused operation).
    – Omer Perry
    Commented Nov 15, 2013 at 14:59
  • The SO dataset has grown significantly since I first created this (mid-2011), and even then it took around 10 hours to import. I'll take a look at the commit interval and see if that can be tweaked. The problem with picking up from where it stopped is that there is no easy way to tell the XML parser to start from point X besides parsing and dumping everything until point X is reached (in which case, you might as well start again).
    – Sam
    Commented Nov 15, 2013 at 15:17
  • 1
    I wrote a local web app to host the data dump, and query it quickly! github.com/peterburk/stackoverflowlocal It can search the 4.6GB StackOverflow data dump while compressed, and compress it further down to only 2.32GB. Stackdump requires decompressing it to 30GB, but I don't have the disk space for that. For those looking for an iPhone version, try a competitor's app, StackStash. But that takes up 3.6GB, and doesn't work on my laptop. itunes.apple.com/us/app/stackstash-stackoverflow-offline/… Peter
    – peterburk
    Commented Feb 10, 2014 at 13:23
  • Sam, this is amazing! Thank you so much for making this. I can't believe how professional this product feels, from the instructions to the tools to the website. Import of SO took 21.5 hours on my 2.3GHz Haswell i7 Macbook — slightly annoying, but tolerable if you do it on your internal drive, since you can put your computer to sleep and seamlessly resume later. (Strangely, row processing seemed to slow down towards the end of the import.) Out of curiosity, how long did this take to make?
    – Archagon
    Commented Apr 24, 2014 at 4:29
  • I have a hunch that you could speed up the program significantly by utilizing multiple cores for SQL and Solr access since those appear to be the bottlenecks, but so far I haven't been able to figure it out. (I'm not very familiar with SQL, Solr, or Python multiprocessing, unfortunately.) Have you looked at all at Solr's DataImportHandler? They claim to be able to import Wikipedia's 40GB XML dump in about 50 minutes, which might be useful for speeding up Stackdump.
    – Archagon
    Commented Apr 24, 2014 at 4:29
  • 1
    @Archagon I'd guess that it gets slower during the import as the database and index grow larger. Solr actually runs as a separate process, so it is able to use multiple cores. The SQL runs in-process though. I've considered using the DataImportHandler, but because the data is manipulated before it is imported (the posts in the XML are grouped up into questions) it doesn't work out-of-the-box. Because of this grouping behaviour, I'm also not sure if parallel processing would be much benefit. I'm not sure if writing the import code in Java would be faster; open to suggestions though!
    – Sam
    Commented Apr 24, 2014 at 13:14
  • 1
    @Archagon Turns out I've actually thought about this a bit more a while back - bitbucket.org/samuel.lai/stackdump/issue/5. Feel free to add suggestions there on how it could be faster.
    – Sam
    Commented Apr 24, 2014 at 13:18
  • 2
    I added a comment on the BitBucket issue. By the way, I was wrong about my total run time, since I didn't take standby into consideration. I think the total active import time was actually closer to 7-10 hours.
    – Archagon
    Commented Apr 24, 2014 at 20:18
  • I have been trying to figure out how to exactly run the software program the import sites is the tricky part, where i can't seem to solve the configuring. i first execute the stackdump_dir/start_solr.sh command, to run the search indexer after the import: stackdump_dir/mange.sh import_site --base-url opensource.stackexchange.com --dump-date "August 2012" /Users/saif/Desktop/StackOverFlow/stackexchange all i get in return is :-bash: stackdump_dir/mange.sh: No such file or directory so my question is how do i import, is there any step i have been missing?
    – Sam
    Commented Aug 15, 2016 at 9:29
  • I have friends who live on a country with very limited and expensive access to internet, they expend most of the time asking questions and looking for answer on stackoverflow. I am trying to make this running to give it to them as an special gift. How often the database get updated?. Are the updates cumulatives or we have to download the whole database each time?. What is the aproximate size of each database update in case it is cummulative?
    – Noel Carcases
    Commented Dec 8, 2016 at 14:32

1 Answer 1

Reset to default
2

I've been searching for a Stack Exchange offline version, for research purposes at my University, then I found this.

I have been trying to figure out how to exactly run the software program on my Mac OS X Yosemite. I have downloaded the entire data dump and everything that it requires:

  • Python, version 2.7.12
  • Java, version 1.8.0_71
  • Stackdump
  • the StackExchange
  • Data Dump
  • 7-Zip

The import sites is the tricky part, where I can't seem to configure it. I first executed the stackdump_dir/start_solr.sh command to run the search indexer.

Output:

Saifs-MBP:~ saif$ cd /Users/saif/Desktop/StackOverFlow
Saifs-MBP:StackOverFlow saif$ stackdump_dir/start_solr.sh

That all goes fine, until I have to execute the import process:

stackdump_dir/mange.sh import_site --base-url opensource.stackexchange.com --dump-date "August 2012" /Users/saif/Desktop/StackOverFlow/stackexchange

All I get in return is: -bash: stackdump_dir/mange.sh: No such file or directory

So my question is how do I import it? Is there any step I missed?

Improve this answer
1
  • 3
    There's a typo in your command - should be manage.sh
    – Sam
    Commented Aug 20, 2016 at 23:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .