Skip to main content
Archagon's user avatar
Archagon's user avatar
Archagon's user avatar
Archagon
  • Member for 10 years, 7 months
  • Last seen more than 4 years ago
comment
Stackdump, an offline browser for StackExchange sites
I added a comment on the BitBucket issue. By the way, I was wrong about my total run time, since I didn't take standby into consideration. I think the total active import time was actually closer to 7-10 hours.
comment
Stackdump, an offline browser for StackExchange sites
I have a hunch that you could speed up the program significantly by utilizing multiple cores for SQL and Solr access since those appear to be the bottlenecks, but so far I haven't been able to figure it out. (I'm not very familiar with SQL, Solr, or Python multiprocessing, unfortunately.) Have you looked at all at Solr's DataImportHandler? They claim to be able to import Wikipedia's 40GB XML dump in about 50 minutes, which might be useful for speeding up Stackdump.
comment
Stackdump, an offline browser for StackExchange sites
Sam, this is amazing! Thank you so much for making this. I can't believe how professional this product feels, from the instructions to the tools to the website. Import of SO took 21.5 hours on my 2.3GHz Haswell i7 Macbook — slightly annoying, but tolerable if you do it on your internal drive, since you can put your computer to sleep and seamlessly resume later. (Strangely, row processing seemed to slow down towards the end of the import.) Out of curiosity, how long did this take to make?
awarded