Timeline for Stackdump, an offline browser for StackExchange sites
Current License: CC BY-SA 4.0
21 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Dec 28, 2020 at 23:25 | history | edited | Glorfindel♦ | CC BY-SA 4.0 |
broken images fixed (click 'side-by-side' to see the difference; images retrieved via Wayback Machine); for more info, see https://gist.github.com/Glorfindel83/9d954d34385d2ac2597bbe864466259f; broken link fixed
|
| Dec 8, 2016 at 14:32 | comment | added | Noel Carcases | I have friends who live on a country with very limited and expensive access to internet, they expend most of the time asking questions and looking for answer on stackoverflow. I am trying to make this running to give it to them as an special gift. How often the database get updated?. Are the updates cumulatives or we have to download the whole database each time?. What is the aproximate size of each database update in case it is cummulative? | |
| Oct 13, 2016 at 9:56 | history | bumped | CommunityBot | This question has answers that may be good or bad; the system has marked it active so that they can be reviewed. | |
| Sep 13, 2016 at 9:04 | history | bumped | CommunityBot | This question has answers that may be good or bad; the system has marked it active so that they can be reviewed. | |
| Aug 15, 2016 at 9:29 | comment | added | Sam | I have been trying to figure out how to exactly run the software program the import sites is the tricky part, where i can't seem to solve the configuring. i first execute the stackdump_dir/start_solr.sh command, to run the search indexer after the import: stackdump_dir/mange.sh import_site --base-url opensource.stackexchange.com --dump-date "August 2012" /Users/saif/Desktop/StackOverFlow/stackexchange all i get in return is :-bash: stackdump_dir/mange.sh: No such file or directory so my question is how do i import, is there any step i have been missing? | |
| Aug 14, 2016 at 8:33 | answer | added | Sam | timeline score: 2 | |
| Apr 24, 2014 at 20:18 | comment | added | Archagon | I added a comment on the BitBucket issue. By the way, I was wrong about my total run time, since I didn't take standby into consideration. I think the total active import time was actually closer to 7-10 hours. | |
| Apr 24, 2014 at 13:18 | comment | added | Sam | @Archagon Turns out I've actually thought about this a bit more a while back - bitbucket.org/samuel.lai/stackdump/issue/5. Feel free to add suggestions there on how it could be faster. | |
| Apr 24, 2014 at 13:14 | comment | added | Sam | @Archagon I'd guess that it gets slower during the import as the database and index grow larger. Solr actually runs as a separate process, so it is able to use multiple cores. The SQL runs in-process though. I've considered using the DataImportHandler, but because the data is manipulated before it is imported (the posts in the XML are grouped up into questions) it doesn't work out-of-the-box. Because of this grouping behaviour, I'm also not sure if parallel processing would be much benefit. I'm not sure if writing the import code in Java would be faster; open to suggestions though! | |
| Apr 24, 2014 at 4:29 | comment | added | Archagon | I have a hunch that you could speed up the program significantly by utilizing multiple cores for SQL and Solr access since those appear to be the bottlenecks, but so far I haven't been able to figure it out. (I'm not very familiar with SQL, Solr, or Python multiprocessing, unfortunately.) Have you looked at all at Solr's DataImportHandler? They claim to be able to import Wikipedia's 40GB XML dump in about 50 minutes, which might be useful for speeding up Stackdump. | |
| Apr 24, 2014 at 4:29 | comment | added | Archagon | Sam, this is amazing! Thank you so much for making this. I can't believe how professional this product feels, from the instructions to the tools to the website. Import of SO took 21.5 hours on my 2.3GHz Haswell i7 Macbook — slightly annoying, but tolerable if you do it on your internal drive, since you can put your computer to sleep and seamlessly resume later. (Strangely, row processing seemed to slow down towards the end of the import.) Out of curiosity, how long did this take to make? | |
| Mar 4, 2014 at 4:10 | history | edited | Sam | CC BY-SA 3.0 |
Minor tweaks for v1.3 release.
|
| Feb 10, 2014 at 13:23 | comment | added | peterburk | I wrote a local web app to host the data dump, and query it quickly! github.com/peterburk/stackoverflowlocal It can search the 4.6GB StackOverflow data dump while compressed, and compress it further down to only 2.32GB. Stackdump requires decompressing it to 30GB, but I don't have the disk space for that. For those looking for an iPhone version, try a competitor's app, StackStash. But that takes up 3.6GB, and doesn't work on my laptop. itunes.apple.com/us/app/stackstash-stackoverflow-offline/… Peter | |
| Nov 30, 2013 at 11:38 | history | edited | Sam | CC BY-SA 3.0 |
Added details on v1.2 and importing the latest Stack Overflow data dump.
|
| Nov 15, 2013 at 15:17 | comment | added | Sam | The SO dataset has grown significantly since I first created this (mid-2011), and even then it took around 10 hours to import. I'll take a look at the commit interval and see if that can be tweaked. The problem with picking up from where it stopped is that there is no easy way to tell the XML parser to start from point X besides parsing and dumping everything until point X is reached (in which case, you might as well start again). | |
| Nov 15, 2013 at 14:59 | comment | added | Omer Perry | FYI - Having this running over the current SO database (a few GBs) took much longer than I could let it, so I had to stop it at some point... I'm not sure why it's so slow and I didn't have the time to check this out, but maybe you could use some performance enhancements in the part that commits changes (a decent option may allow recontinuing an paused operation). | |
| Oct 22, 2013 at 2:41 | comment | added | Sam | @Rely thanks for that report; finally got a chance to fix it. v1.1 is now out (available at bitbucket) with that fix and a few other minor things. | |
| Apr 20, 2013 at 10:58 | comment | added | Omer Perry |
That's just amazing. Thank you! (Be informed that the standard had changed a bit since v1.0 - the names of the XMLs are now capitalized e.g. comments.xml -> Comments.xml)
|
|
| Nov 30, 2012 at 6:31 | history | edited | Sam | CC BY-SA 3.0 |
Added in links and screenshots, now that I have enough rep!
|
| Sep 13, 2012 at 22:03 | review | First posts | |||
| Sep 19, 2012 at 11:32 | |||||
| Aug 20, 2012 at 11:27 | history | asked | Sam | CC BY-SA 3.0 |