341 reputation
125
bio website edgylogic.com
location Australia
age
visits member for 1 year, 11 months
seen Apr 25 at 2:31

Software developer who dabbles in system administration on Linux and Windows. Also moonlight as a scripting monkey, automating anything and everything. I code in a variety of languages, including Python, C#, Java, C, JavaScript, PowerShell... whatever works best for the job.


Apr
24
comment Stackdump, an offline browser for StackExchange sites
@Archagon Turns out I've actually thought about this a bit more a while back - bitbucket.org/samuel.lai/stackdump/issue/5. Feel free to add suggestions there on how it could be faster.
Apr
24
comment Stackdump, an offline browser for StackExchange sites
@Archagon I'd guess that it gets slower during the import as the database and index grow larger. Solr actually runs as a separate process, so it is able to use multiple cores. The SQL runs in-process though. I've considered using the DataImportHandler, but because the data is manipulated before it is imported (the posts in the XML are grouped up into questions) it doesn't work out-of-the-box. Because of this grouping behaviour, I'm also not sure if parallel processing would be much benefit. I'm not sure if writing the import code in Java would be faster; open to suggestions though!
Mar
4
revised Stackdump, an offline browser for StackExchange sites
Minor tweaks for v1.3 release.
Feb
18
awarded  Famous Question
Dec
17
awarded  Notable Question
Dec
17
awarded  Nice Question
Dec
17
awarded  Yearling
Nov
30
revised Stackdump, an offline browser for StackExchange sites
Added details on v1.2 and importing the latest Stack Overflow data dump.
Nov
15
comment Stackdump, an offline browser for StackExchange sites
The SO dataset has grown significantly since I first created this (mid-2011), and even then it took around 10 hours to import. I'll take a look at the commit interval and see if that can be tweaked. The problem with picking up from where it stopped is that there is no easy way to tell the XML parser to start from point X besides parsing and dumping everything until point X is reached (in which case, you might as well start again).
Nov
8
awarded  Popular Question
Oct
22
comment Stackdump, an offline browser for StackExchange sites
@Rely thanks for that report; finally got a chance to fix it. v1.1 is now out (available at bitbucket) with that fix and a few other minor things.
Nov
30
awarded  Student
Nov
30
awarded  Editor
Nov
30
revised Stackdump, an offline browser for StackExchange sites
Added in links and screenshots, now that I have enough rep!
Aug
27
awarded  Tumbleweed
Aug
20
asked Stackdump, an offline browser for StackExchange sites