Take the 2-minute tour ×
Stack Apps is a question and answer site for apps, scripts, and development with the Stack Exchange API. It's 100% free, no registration required.

Screenshots

About

These books contain the top questions from a selection of the top tags on Stack Overflow. The top questions include those with a score of 10 or greater (except in the case of a high volume tag like c#).

License

The content of the ebooks is licensed under the same Creative Commons license as Stack Overflow itself.

Download

Stack Overflow ebooks

Platform

The ebooks are in Mobipocket format. They have been tested with the Kindle and are likely to work on other devices. They can be converted to other ebook formats with a tool such as Calibre.

Contact

These books were created by Greg Hewgill.

Code

Source is on Github. It's sort of a hodgepodge of Python, Java, and XSLT (applying the best tool for each job).

share|improve this question
    
+1 for Python. Maybe consider doing this for some other SE sites? –  Nathan Osman Feb 15 '11 at 19:14
    
@Greg it gives me org.xml.sax.SAXParseException: character not allowed running the xt script on the latest datadump, any hint? –  systempuntoout Apr 19 '11 at 20:22
    
@systempuntoout: You may need to manually remove the BOM from the top of the users.xml file. (I think that's the file in question, I'm not in a position to check at the moment.) –  Greg Hewgill Apr 19 '11 at 22:07
    
@Greg damned BOM, that made the trick. Now I get File "unify.py", line 28, in <module> Posts = mmap.mmap(f.fileno(), size, access=mmap.ACCESS_READ) OverflowError: cannot fit 'long' into an index-sized integer –  systempuntoout Apr 20 '11 at 14:33
    
@systempuntoout: These tools require a 64-bit operating system (and a corresponding 64-bit build of Python) because unify.py wants to mmap() the entire XML dump into memory. The input posts.xml file is currently 5.2 GB. –  Greg Hewgill Apr 20 '11 at 20:33
    
@Greg oh I see, I'm still on Leopard. Maybe I will look into your code to tweak it a little. –  systempuntoout Apr 20 '11 at 20:37

1 Answer 1

This is very cool, but suffers from the obvious problem of:

  • being limited to the tag combinations you've pre-selected
  • being limited to the SE sites you choose to support
  • having to be manually kept up-to-date by you

I'm interested in adapting your work to a Google App Engine service that generates these on-the-fly for a user-specified site and set of tags, and then e-mails them to the Kindle's delivery address so that they're downloaded automatically over the air. Then just make a particularly Kindle-friendly frontend for this site, so that the request can be made straight from the Kindle's browser, and this becomes much more useful.

Just thought I'd put this out there :)

share|improve this answer
    
@Adrian I don't think Google App Engine could generate this on-the-fly, in 30 seconds you have plenty of things to do like fetching the data from SE, fetching the images, formatting the data to mobi etc.etc. . –  systempuntoout Feb 16 '11 at 7:56
    
@systempuntoout Correct me if I'm wrong, but I think tasks on a task queue can add more tasks to the task queue. So as long as each individual step can be completed in 30 seconds I can just have a pipeline of tasks until the whole thing is assembled. In fact, if even they can't, you can still get the same effect by creating all the task queues at the same time and having the later stages fail until the output from the previous stages is present in the Datastore. I guess my point is I don't think the 30-second limit is too much of a limitation in this case. –  Adrian Petrescu Feb 16 '11 at 8:21
    
Actually, I just checked the relevant docs, and the timeout for Tasks in a Task Queue in the 1.4.0 API is ten minutes, not 30 seconds (which is the limit for a request to return). Since sending the document through e-mail can be done asynchronously, this is totally acceptable. –  Adrian Petrescu Feb 16 '11 at 8:26
    
@Adrian of course you can and each task on the task queue is not limited to 30 seconds but 10 minutes. I think that Channel API would help in this case: 1. User clicks on the Id question he/she wants 2. The process is started in the background using the taskqueue 3. User is asynchronously notified by the channel API that the process is complete. The first thing to check is that all the Python libraries used by Greg could be adopted on GAE considering that GAE does not allow write to file-system for example. –  systempuntoout Feb 16 '11 at 8:31
    
@systempuntoout I don't even think there's much use in notifying the user that the process is complete, since there's still an nondeterministic amount of time left until the file is actually sent to the device by Amazon through the e-mail address. I think the request can just automatically return "Your book will appear on the device shortly" and if this fails somewhere for some reason, we can send an error message/trace as the document instead. As you say, the hard part will be adapting his code to work on GAE :) I will look into that tomorrow, hopefully. I'm pretty optimistic though! –  Adrian Petrescu Feb 16 '11 at 8:42
1  
@Adrian let me know if you need a hand –  systempuntoout Feb 16 '11 at 9:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .