Scrape the site list, show sites sorted by answer ratio

Question

About

Display the list of Stack Exchange sites, with the number of questions and answers on each site, and the answer ratio (number of answers divided by the number of questions). The list is sorted by answer ratio. The answer ratio computed this way is close, but not equal to the Area 51 statistic for beta sites, as Area 51 ignores closed or very recent questions and my crude script doesn't.

This script works by scraping the site statistics page, since as far as I know you can't obtain these statistics through the API without querying each site separately. It can serve as an example of scraping the site directory.

License

CC BY-SA, same as the posts on Stack Exchange.

Requirements

Python, BeautifulSoup

Code

#! /usr/bin/env python
import re, urllib2
from BeautifulSoup import BeautifulSoup
qa_re = re.compile('^([0-9,]+) total (questions|answers)')

def generate_site_qa(soup):
    for site in soup('div', {'class' : 'itemNarrow'}):
        site_a = site('a', {'class' : 'site-summary-name'})
        site_url = site_a[0]['href']
        site_name = site_a[0].decodeContents()
        questions = None
        answers = None
        for stat in site.findNextSibling()('span'):
            m = qa_re.match(stat.get('title', ''))
            if m:
                n = int(m.group(1).replace(',', ''))
                if m.group(2)[0] == 'a': answers = n
                else: questions = n
        yield {'name' : site_name,
               'url' : site_url,
               'q' : questions,
               'a' : answers,
               'r' : float(answers) / questions}

def show_sites_by_answer_ratio(sites):
    sites.sort(key=lambda s:s['r'])
    for site in sites:
        print (' * <a href="%(url)s">%(name)s</a>: ' +
               '%(q)d questions, %(a)d answers, ratio = %(r).1f') % site

if __name__ == '__main__':
    url = 'http://stackexchange.com/sites?expand=true'
    soup = BeautifulSoup(urllib2.urlopen(url))
    show_sites_by_answer_ratio(list(generate_site_qa(soup)))