133

UPDATE: 2.x support is now mainline! Please read the wiki page for important information about the update.

A warm welcome to you, traveller. You have arrived at the home of Py-StackExchange, the library definitively proven to be the best library for using the SE API from Python. If you are still interested (and by golly, you should be) after glancing at the masterpiece below, please check the wiki on Github.

† Ahem.

About

So, what is Py-StackExchange? Well, I'm glad you asked.

It is a Python library for querying the StackExchange API from your Python applications. Integration, ahoy!

So why should you use it? After all, the SE API is sooo simple that you might think it'd be quicker to just write your own, and that it'd be faster and you wouldn't have to look at all that documentation and do all that thinking... well:

Let's start with the API coverage - what can the API do? And, more importantly, what can the library do?

  • Access any StackExchange site, with just its URL! Even those that aren't online yet!
  • If you just can't decide which one to use, you can use StackAuth to look up the full list of sites.
  • Once you're online, you can view everything about users, questions, answers, badges, comments and tags.
  • You can even go back in time by playing with post revisions.
  • Stalk Generate a detailed profile of a user's life Help users by looking up every StackExchange account they have. Every single one.
  • And, on any of those sites, peruse a detailed history of everything they've ever done - every edit, every comment, every time they were awarded a badge... Watch StackOverflow become the new Facebook overnight with the timeline feature.
  • See how well an SE site is doing; obsessively check its site statistics.
  • Search the questions of StackExchange sites.

So, why not write your own classes to consume said pure, concentrated brilliance?

  • Let someone else deal with all that laborious HTTP request business... you know you want to...
  • All the little idiosyncratic potholes on your road to API happiness have been filled in for you. We have little elves which jump into your code and parse your JSON and your dates and your lists until every response is itself a little baby python.
  • URLs change 99.9% more often than the interface of this module. Fact.
  • It's faster than Michael Palin on a broken bicycle. It also knows about request throttling, so when it gets too fast for its own good, it applies the brakes just enough to restore order.
  • It loads lazily information that would take another request to fetch, meaning you never use more of your limit than you need to.
  • It caches requests automatically, so you need to care slightly less about writing efficient code! (new in 1.1)

Now, onto the religious advantages:

  • Documentation? Bah, we have naming conventions. (This feature was inspired by Rails.)
    Pssst - don't tell anyone, but there is documentation too, if that's your style. (README/Wiki)
  • Naming conventions? Who needs them? We have an interactive program that writes your code for you while you look around the StackExchange site of your choice. (This feature was inspired by Jon Skeet.)
  • Almost-sentient, artificially intelligent programming programs? Ugh, how 20th century. There are metric heaps of example code available in the source repo, a small excerpt of which is presented below for your viewing pleasure.

Please note: This is not an official product of Stack Overflow Internet Services, Inc.

Code Snippet

The wiki has details of all the example code in the code repository. In fact, here's a small taster from the Narcissism demo.

#!/usr/bin/env python

# a hack so you can run it 'python demo/stats.py'
import sys
from stackauth import StackAuth
from stackexchange import Site, StackOverflow

user_id = 41981 if len(sys.argv) < 2 else int(sys.argv[1])
print 'StackOverflow user %d\'s accounts:' % user_id

stack_auth = StackAuth()
so = Site(StackOverflow)
accounts = stack_auth.associated(so, user_id)
reputation = {}

for account in accounts:
    print '  %s: %s / %d reputation' % (account.display_name, account.on_site.name, account.reputation)

    reputation[account.reputation] = account.on_site.name

print 'Most reputation on: %s' % reputation[max(reputation)]

Or how about a scrolling list of questions?

import stackexchange
so = stackexchange.StackOverflow()

for q in so.questions(pagesize=50):
    print q.title

About

This is a Python library/wrapper around the StackExchange and StackAuth APIs. It provides a clean, object-oriented API for accessing the various sites.

License

The script is licensed under the Simplified BSD license. You can find the full text of the license here, but the gist of it is that:

  • You need to give attribution when you distribute (compiled or in source form) the library - not your application unless you include the library files `in the box'.
  • The standard "NO WARRANTY" (in caps!) is provided.

Other than that you can more or less do what you like!

Download

In the bad old days (i.e. about 2 hours before I wrote this), you had to manually install Py-StackExchange after cloning the Git repository.

You can still do this: http://github.com/lucjon/Py-StackExchange. You can also download a ZIP or TGZ file from there.

However, there is a new and improved way to get Py-StackExchange: you can install it straight from the PyPI! Just type:

~$ easy_install py-stackexchange

Also, distutils gives me fantastical benefits on the side, such as a completely original Windows installer with an all-new design. You can also find a stable source distribution on the downloads page @ Github.

Platform

The library is written in standard Python 2.6, with, as far as I am aware, no specific platform dependency. As long as your Python install has the full standard library available, it should work fine.

Python 2.6 is required for the json module. (EDIT: @ADB in the comments has noted that the SimpleJson library can be used instead. This means it works on Python 2.5 and also on the Google App Engine.)

Python 3.x is also supported.

Contact

The library is being written by Lucas Jones (lucasjones.co.uk / SO). If you want to contact me, send me some mail at lucas @ lucasjones.co.uk.

18
  • @Lucas: What would be really useful to me is a list of the methods exposed by py-SE, and a list of parameters for each, like the list at api.stackoverflow.com/1.1/usage for the SEAPI. (Bonus: a mapping between the SEAPI and py-SE's methods.)
    – LarsH
    May 18, 2011 at 16:49
  • @LarsH: That's a good idea; it'll also encourage me to keep as close to 100% coverage as is humanly possible! May 18, 2011 at 17:57
  • 2
    @Lucas great work, any plan to support also the V2.0 Api? Apr 12, 2012 at 10:27
  • @systempuntoot: Thanks for the comment; it arrived in my inbox and reminded me I've not finished updating it yet! There is a branch on Github I started a while ago with some basic things working... I'll try and get that finished as soon as I can. Apr 12, 2012 at 11:20
  • @LucasJones I see there hasn't been any activity for a year or so. How complete is your 2.0 coverage, and do you have any plans for updating to 2.1? I just discovered the API, and love Python, so this library is an obvious choice, but I don't want to screw anything up with the wrong API calls and whatnot. Thanks!
    – MattDMo
    Oct 21, 2013 at 16:16
  • @MattDMo: Regrettably, the 2.0 branch is not entirely stable, mainly due to differences in the way filters work, and the coverage is certainly not complete. As you have deduced, there has not been much meaningful effort dedicated to improving it for a significant period of time; I currently have no explicit plans to update it for v2.1. The existing v1.x code remains entirely functional if you don't require any of the new API calls. Oct 21, 2013 at 20:01
  • @LucasJones I just got a notice that API v1.x will be shut down 3 months after the v2.2 release, but no earlier than April 1, 2014. Do you have any plans to support v2.x before the v1.x shutdown? Jan 31, 2014 at 18:13
  • @Eric Leichtenschlag: While I'm sure I'm overlooking something obvious, I can't find a source for your exact date for a v1.x shutdown. The v2.x branch of the repository has improved somewhat since my last comment here, passing its (small) set of unit tests. The Python API remains almost identical. Certainly if there were such a date it would be a useful motivator; while I can't make any guarantees, I imagine the finishing touches could be put on before then. Feb 3, 2014 at 22:56
  • I couldn't find it online anywhere either - I happened to get an advanced notification email about it. I suspect they'll announce it as part of their their v2.2 release announcement. Anyways, I'll stay tuned for an exact date and what the future plans of this library are. Thanks! Feb 3, 2014 at 23:18
  • 1
    So the shutdown of v1.x is official now. I see you've made some changes to support v2.0. Has it moved completely to 2.0 yet? Apr 3, 2014 at 18:36
  • @Eric Leichtenschlag: Thanks for letting me know; my apologies for not replying to you at the v2.2 is now the primary API version targeted by the library (the v2.x branch has been merged, etc.). May 1, 2014 at 21:37
  • @LucasJones, with the update to support 2.2, does this also mean it supports the ability to add data to SE (ie. add flags, comments, questions, etc)?
    – Andy
    May 2, 2014 at 21:36
  • @Andy: Not currently, but I'm certainly having a look at it. May 3, 2014 at 13:39
  • 2
    @apnorton: I've added support for it just now; you'll need to use the latest version in the repository (github.com/lucjon/Py-StackExchange). If you call Site#question or Site#answer with the filter= optional parameter pointing to a filter set to return the body_markdown field, then the returned object will have a body_markdown attribute. Apr 25, 2015 at 17:52
  • 1
    @Tim: It is indeed possible to get the Markdown; you can request the body_markdown attributes on questions and answers using an appropriate filter. See my responses to apnorton on Apr 25 for more detail. May 26, 2015 at 16:07

30 Answers 30

20

You can get around the json/simplejson easily in your wrapper. Just change your import line from

import urllib2, json, httplib, datetime, operator

to

import urllib2, httplib, datetime, operator
try:
    import json
except ImportError:
    import simplejson as json

That way you can just use json in the rest of your code, but client pythons will load whichever library is available on that system.

Keep up the good work, I'll probably be using your library assuming I can dream up something cool to do with it.

3
  • This is done in the latest revision. May 28, 2010 at 16:06
  • Fyi, this doesn't work in Python 2.5. Since I'm using Django, I modified the above to try and import from django as well (from django.utils import simplejson as json). I don't know if you should add this to the wrapper or not, but if anyone is using Django, just use that line instead.
    – Edan Maor
    Jun 16, 2010 at 8:08
  • @Edan: I think it might be too specialised a case, but I'll definitely include it in the FAQ. Jun 16, 2010 at 15:09
12

I am using Py-StackExchange in our Stack2Blog application.

How do I retrieve a list of answers of a specific user using this wrapper?

After retrieving a user object, the .answers member variable returns an empty list while I know the user does have posts.

2
  • 2
    Managed to figure it out. The list is populated only after the list's .fetch() method is called. Jun 11, 2010 at 15:52
  • Yes, sorry - this is by-design to limit the number of API calls the app makes. That looks like an awesome app, though! Jun 12, 2010 at 21:41
8

Edit:

My problem below has been "solved", and it's a problem with the fact that I'm getting back gzip-compressed data from stackapps. See the SO Answer. I'm still unsure why this happens only on my computer (possible reason: routers in my network adding content-headers), but I'm guessing this should be fixed in the wrapper itself.

Come to think of it, the wrapper should probably be requesting gzip-compress data in the first place, to save download time.

Original question

I'm having problems using the library. I'm doing the following:

site = stackexchange.Site(stackexchange.StackOverflow)
user = site.user(userid)

And I'm getting the following exception:

ValueError at /answers
No JSON object could be decoded

After some debugging, I see that the line which throws the error is: dump = json.load(conn) (around line 410). When I try printing the urllib object I get back (conn), I get the following:

\x1f\x8b\x08\x00\x00\x00\x00\x00\x04\x00\xed\xbd\x07`\x1cI\x96%&/m\xca{\x7fJ\xf5J\xd7\xe0t\xa1\x08\x80`\x ...

I don't have any experience with urllib, so I'm not sure if that's a good output, although it doesn't look right to me (by contrast, opening http://www.google.com gives back actual meaningful text).

Any ideas as to what the problem could be?

Edit: Just running the code snippet in your question in IDLE is causing a problem.

5
7

Total results vary with page size

The idea of producing a transparent iterator (getting the next page as needed) is cool, but it just isn't working; the page size affects the total number of results returned (apparently to a multiple of the page size)...:-(

My code needs to look at all Qs asked for a given tag in a given month:

def qstats(site, tag, month, year=None):
    fromdate, todate = month_range(month, year)

    qs = site.questions(pagesize=100,
        sort='votes',
        fromdate=ts(*fromdate),
        todate=ts(*todate),
        tagged=tag)

    good_qs = 0
    for i, q in enumerate(qs, start=1):
        good, answered = qtype(q)
        if good:
            good_qs += 1
    return i, good_qs

With a page size of 100, as here, I see, from a loop calling this for different months on a certain tag:

2014/11: 242 good Qs out of 600 (40%)
2014/12: 242 good Qs out of 500 (48%)
2015/1: 235 good Qs out of 500 (47%)

but e.g with an arbitrary page size of 37, it's instead

2014/11: 179 good Qs out of 518 (35%)
2014/12: 179 good Qs out of 518 (35%)
2015/1: 172 good Qs out of 518 (33%)

(518 is 37 * 14, whence my hypothesis that the returned number of items is somehow constrained to be a multiple of the page size -- but clearly it's not just that, as the page size of 100 gave 600 questions for the tag in Nov'14, but the page size of 37 still gives up at 518). I guess this is connected with the already reported bug of fetch_next returning nothing at unpredictable times, even though here it's buried in the iteration.

But I don't understand at all the code of the next method of StackExchangeResultset in the core.py file...: it starts...:

def next(self):
    for obj in self.items:
        yield obj

    current = self
    while current.has_more:
        for obj in current.items:
            yield obj

won't this yield each item in self.items twice? Once from self.items, and then again from current.items after setting current = self?! I just don't understand the logic of this snippet. I'm going to instrument my code to check for duplicates, which should be present if my doubts are well-founded, and report on that check...

EDIT: yep, confirmed, the duplicates appear exactly as I had thought they would -- I've added a set of ids and return prematurely after seeing a duplicate, and the number of total results reported for each and every month is exactly the page size -- confirming that each question is being yielded again right after all items on the first page have been yielded, exactly as the code I show above appears to say they would be.

So, any suggested workaround so I can examine all Qs meeting the constraints, and, only once each?-)

EDIT AGAIN: so I partly fixed def next(): in the result-set class (there's still a bug I can't yet fathom -- shows duplicates with a page size of 37 -- but at least it seems to work with a page size of 100, with no duplicates nor truncation to a multiple of 100). To make python3 setup.py install work I also had to change setup.py since (at the github master) it was missing some modules (?).

The resulting branch is at https://github.com/aleaxit/Py-StackExchange/tree/betteriter and of course I've also sent a pull request for it. But the gist is to change the above-quoted brokenness into

def next(self):
    current = self
    while True:
        for obj in current.items:
            yield obj
        if not current.has_more:
            return

(no further changes below this).

1
  • Thanks very much for sorting this out; I'll respond in more detail on Github. Mar 16, 2015 at 1:44
7

I am trying to use the Search option to get the list of results from SO.

I have tried using pagesize=50 and I still get large number of titles returned.

Is there another option to limit that search? Sorry if I am posting this the wrong way or in the wrong area but I am new to Stack Overflow and to programming as well :)

This is the part of the code I am referring to:

qs = so.search(intitle=term, pagesize = 50, sort='votes')

Thank you

4
  • If the problem concern the Py-StackExchange library (and it seems like it might), then this is the place to post. Sep 2, 2015 at 19:06
  • 1
    Like most of the Site methods, search returns an iterator over all the items (in this case, questions) in the result, no matter how many pages it takes. That is, it fetches the first page of results, gives them to your program, then fetches the next page, and so on, transparently. The pagesize= argument just specifies how many it fetches at a time from Stack Exchange. To get just the first 50 questions, use so.search(...)[:50]. You don't necessarily need the pagesize = 50 argument here either, but if you know you'll look at all of the 50 it is faster to request them all at once. Sep 2, 2015 at 21:14
  • Please get back to me if this is unclear :) Sep 2, 2015 at 21:14
  • It is perfectly clear. Thank you very much for such a fast response :) Sep 2, 2015 at 21:36
5

OK, I'm probably missing the obvious, but how do I get the body of an answer?

For example, in your code snippet:

>>> so = stackexchange.Site('api.stackoverflow.com')
>>> me = so.user(41981)
>>> me.answers.fetch()
>>> me.answers[0].body
Traceback (most recent call last):
  File "<pyshell#45>", line 1, in <module>
    me.answers[0].body
AttributeError: 'Answer' object has no attribute 'body'
6
  • 1
    Ah, sorry. Due to a change in the API (I think - or perhaps me just being stupid and not reading the docs close enough!) question and answer bodies need to be explicitly requested. You can do this through so.be_inclusive(), which you should call before your first request. Jun 13, 2010 at 15:36
  • Yeah just figured out the be_inclusive() bit. Is there any way to specify it per-request? (I assume most people won't want to get all the bodies and all the comments from every request, just from specific requests). I can see that there are places where you check for a "body" keyword param, but I'm not sure where I can send one in the snippet above...
    – Edan Maor
    Jun 13, 2010 at 15:38
  • I've also updated the FAQ with this, as it's not documented very clearly. Jun 13, 2010 at 15:57
  • @Edan: Not in your specific case right now (see the FAQ - link in question - for those that are covered), as I'm not sure how best to implement it. Do you think calling me.answers.fetch(body='true') would be best? Jun 13, 2010 at 15:59
  • 1
    @Edan: Done now in the latest revision, with the same syntax. fetch_page (but not other overloads right now) can do the same, too. Jun 15, 2010 at 17:18
  • @Edan: Darn comment edit time limits. It's actually user_id not userid in the library code. Jun 15, 2010 at 17:24
5

Possible bug in the latest build? Found this (excellent) library via a Bryce Boe post and dropped in Py-Stackoverflow via easy_install.

Running the script triggered the following, however:

sog@bishop:~/Dropbox/Code/StackOverflow$ ./minestack.py 
Traceback (most recent call last):
  File "./minestack.py", line 3, in <module>
    from stackexchange import Site, StackOverflow
  File "/usr/local/lib/python2.6/dist-packages/py_stackexchange-1.1-py2.6.egg/stackexchange.py", line 6, in <module>
    from stacksites import *

As it's a problem with the library, I thought that pulling it down from GitHub and recompiling might solve the problem. Running the build.sh script in the repo, however, eventually returned this:

[...]
  File "/usr/lib/python2.6/httplib.py", line 904, in endheaders
    self._send_output()
  File "/usr/lib/python2.6/httplib.py", line 776, in _send_output
    self.send(msg)
  File "/usr/lib/python2.6/httplib.py", line 735, in send
    self.connect()
  File "/usr/lib/python2.6/httplib.py", line 716, in connect
    self.timeout)
RuntimeError: maximum recursion depth exceeded

Any ideas as to where I went wrong?

2
  • Aha. I should probably rename build.sh to release.sh - it's the script I use for publishing new releases to PyPI. I'll write a README or something too... anyway, it should just work, from Github, out-of-the-box without running the script. If it doesn't, let me know! :) Feb 23, 2011 at 22:07
  • Updated version on Github, and the egg should work now. Feb 23, 2011 at 23:50
5

When I try running the code with python 3.4 I always get this error:

AttributeError: 'HTTPMessage' object has no attribute 'getheader'

Here is the full traceback. I'd be very happy if any of you could tell me what the problem is. Is it the fact that I'm using python3? I converted the example files using 2to3 and it seems to me that it should work fine...

Traceback (most recent call last):
  File ".\question.py", line 11, in <module>
    question = site.question(id)
  File "C:\Python34\lib\site-packages\py_stackexchange-1.1_4-py3.4.egg\stackexchange\__init__.py", line 677, in question
    q, = self.questions((nid,), **kw)
  File "C:\Python34\lib\site-packages\py_stackexchange-1.1_4-py3.4.egg\stackexchange\__init__.py", line 473, in __call__
    return self.site._get(Question, ids, 'questions', kw)
  File "C:\Python34\lib\site-packages\py_stackexchange-1.1_4-py3.4.egg\stackexchange\__init__.py", line 623, in _get
    return self.build(root, typ, coll, kw)
  File "C:\Python34\lib\site-packages\py_stackexchange-1.1_4-py3.4.egg\stackexchange\__init__.py", line 600, in build
    json = self._request(url, kw)
  File "C:\Python34\lib\site-packages\py_stackexchange-1.1_4-py3.4.egg\stackexchange\__init__.py", line 572, in _request
    json, info = request_mgr.json_request(url, new_params)
  File "C:\Python34\lib\site-packages\py_stackexchange-1.1_4-py3.4.egg\stackexchange\web.py", line 119, in json_request
    req = self.request(to, params)
  File "C:\Python34\lib\site-packages\py_stackexchange-1.1_4-py3.4.egg\stackexchange\web.py", line 98, in request
    if conn.info().getheader('Content-Encoding') == 'gzip':
AttributeError: 'HTTPMessage' object has no attribute 'getheader'
1
  • 1
    This is indeed a Python 3 issue; I have fixed this and a few other compatibility problems just now in the latest version of the library available on Github. The master branch targets v2.x of the StackExchange API, as will the latest PyPI version in the near future. If you want to continue using v1.x (e.g. you have an API key registered), the compatibility fixes have been backported to the v1.1 branch on Github. Jun 26, 2014 at 22:13
4

Based on the demo *recent_questions.py* example, I am using the following code to retrieve the 10 most recent questions at the command line:

so = stackexchange.Site(stackexchange.StackOverflow)
questions = so.recent_questions(pagesize=10)
for q in questions:
    print "question retrieved:", q.title.encode('ascii', 'ignore'), "\t", q.id
    ...

As you can see I am displaying the title and id attributes of each question object.

My question has 2 parts:

  1. How can I get a list of attributes to each method (such as recent_questions)? I simply guessed correctly that id would work, but it does not seem to be listed when I refer to >>> help(stackexchange)
  2. I would like to extend my script to mark as 'starred' ('favourite') some questions that grab my interest. Is this a reasonable prospect? Is there a method/attribute for this, and in which class? Most importantly, is authentication even possible via the API? This would presumably be necessary in order to 'star' questions.
1
  • Currently, keyword arguments are effectively passed verbatim to the API (albeit with a bit of processing to make them work in a URL). The best documentation I can offer in that regard are the official API docs. Regarding your second question: unfortunately, the current version of the API (v1.1) and the next planned version (v2.0) are read-only; you won't be able to star questions, although I think v2.0 will provide authentication functionality. Jan 7, 2012 at 0:34
4

I'm trying to access different attributes of the Answer class but I get this error:

AttributeError: 'Answer' object has no attribute 'accepted'

I looked into your init.py file and I can see that you have a tuple there that lists the attributes:

transfer = ('accepted', 'locked_date', 'question_id', 'up_vote_count', 'down_vote_count', 'view_count', 'score','community_owned', 'title', 'body')

My intuition is that those attributes should be accessible in some way but it looks that I am doing something wrong. Could you please enlighten me?

1
  • Thanks for letting me know about this, it's a 2.x change that slipped through. The field is now called is_accepted; I've pushed a fix to the Github repository. Jul 2, 2014 at 14:14
4

Lucas,

Something must have changed in JSON data format sent from the Stack Exchange API because yesterday for no reason I started getting:

    ValueError("Unexpected UTF-8 BOM (decode using utf-8-sig)")

(I searched through many forums, reinstalled python a couple of times etc etc)

As the error says, there is a problem with the encoding. Basically all I had to do is change the encoding type

in the line 157 of the web.py file

from:

    parsed_result = json.loads(req.data.decode('utf8'))

to:

    parsed_result = json.loads(req.data.decode('utf-8-sig'))

and then reinstall the library.

Hope that helps anyone with the same issue...

2
  • Thanks for letting me know about this; however, I've not been able to reproduce it myself. Is there anything unusual about your setup (in particular, are there any proxies between you and Stack Exchange?) which might get in the way? Either way, if changing the 'encoding' is as harmless as it seems to be from my initial look at the documentation, I may well just change it. Jul 4, 2014 at 0:56
  • I'm not sure what caused the problem. It seems to be something quite unusual that appears when you combine things like windows 8.1, cygwin and python3 altogether.
    – Tomazz
    Jul 10, 2014 at 14:02
3

Quick bug:

Doing this:

anss = so.answers(user_id=175645, pagesize=100, body=True)
ans1 = anss[0]
ans1.owner

Produces this stacktrace:

Traceback (most recent call last):
  File "<pyshell#46>", line 1, in <module>
    a.owner
  File "E:\Projects\Web\Stack2Blog\repos\stack2blog-src-main\stack2blog\stack2blogapp\stackexchange.py", line 218, in <lambda>
    owner = property(lambda self: self._owner if self._owner is not None else self._get_user(self.owner_id), _set_user)
  File "E:\Projects\Web\Stack2Blog\repos\stack2blog-src-main\stack2blog\stack2blogapp\stackexchange.py", line 207, in _get_user
    s._owner = self.site.user(id)
NameError: global name 'self' is not defined
2
  • D'oh. Fixing... Jun 16, 2010 at 15:04
  • 1
    Done in latest revision. Jun 16, 2010 at 15:05
3

Hi Lucas,

I'm running into an issue finding a user's accept rate. I hope to find it across all the stack exchange sites, but when I pull ii.get_user().unaccepted_questions, it is always an empty list.

    accounts = STACK_AUTH.associated(site, owner_id)
    for ii in accounts:
       questions, unaccepted_questions, site = \
            ii.get_user().questions.count,  \
            ii.get_user().unaccepted_questions.count,  \
            ii.on_site.name

Am I doing something wrong?

1
  • I'll double-check this, but unaccepted_questions is probably one of the fields that is not fetched automatically; instead use ii.get_user().unaccepted_questions.fetch().count. I know this is a little clunky, but otherwise it would need to pull in tonnes of data each request. May 25, 2011 at 20:37
3

The library currently does not yet support the use of access_token obtained through OAuth 2.0, does it?

1
  • This is, unfortunately, correct. Dec 20, 2012 at 18:35
3
import stackexchange so =
stackexchange.Site(stackexchange.StackOverflow)
comment = so.comment(valid_comment_id)

the above code returns the following error in init.py:

    --->166         self.post_type = PostType.from_string(json.post_type)
        167 
        168         def _get_post(self):

AttributeError: DictObject instance has no attribute 'post_type'

I get the valid comment id returned from Firehose JSON streaming event service

1
  • 1
    Thanks for bringing this to my attention, I've fixed the problem in the latest revision on the Github repository (github.com/lucjon/Py-StackExchange). Your code should now work if you use the version of the library from there. Aug 28, 2014 at 16:58
2

Using the most recent version of the library as of yesterday. I needed to get some information on users answers and the question itself. However this didn't work. Every time I tried to access another page of answers I got ()

answers = me.answers.fetch()

answerDetails = []

while True:
    for answer in answers:
        answerDetails.append({
            'question_id'       :answer.question_id,
            'up_vote_count'     :answer.up_vote_count,
            'down_vote_count'   :answer.down_vote_count,
            'accepted'          :answer.accepted
            'tags'              :so.question(answer['question_id']).tags
        })

    answers = answers.fetch_next() #returns ()
    if answers == None or len(answers) == 0:
        break

I had to move the requests to get the tags outside of the loop that got the answers:

#[code cut]

    for answer in answers:
        answerDetails.append({
            'question_id'       :answer.question_id,
            'up_vote_count'     :answer.up_vote_count,
            'down_vote_count'   :answer.down_vote_count,
            'accepted'          :answer.accepted
            #'tags'              :so.question(answer['question_id']).tags
        })

#[code cut]

for answer in answerDetails:
    answer['tags'] = so.question(answer['question_id']).tags

While the latter is for various reasons a better solution, it was quite a surprise that it didn't do what I expected.

A better code sample to reproduce is below:

Doesn't work as expected:

so = stackexchange.Site(stackexchange.StackOverflow)
me = so.user(118145)
answers = me.answers.fetch()
so.question(answers[0].question_id).tags
print answers.fetch_next() #prints ()

Works as expected:

so = stackexchange.Site(stackexchange.StackOverflow)
me = so.user(118145)
answers = me.answers.fetch()
print answers.fetch_next() #prints the correct array
1
2

I was willing yo use your lib to do some statistics on the questions from SO (just for fun, nothing really serious, though it can become serious if I like the results :P).

For this I'd have to get the data from I whole bunch of questions. As an example, I tried to get all questions from roughly 1 month (2e6 seconds). I tried this:

import time
from numpy import *
from stackexchange import *
from stackauth import *

currentDate = floor(time.time())
aWhileAgo = currentDate - 2000000

print currentDate, aWhileAgo


stack_auth = StackAuth()
so = Site(StackOverflow)

questions = list(so.questions(sort=Sort.Creation, order=DESC,\
                    fromdate=aWhileAgo, todate=currentDate) )
for q in questions:
    print q

But this gives me just around 20 questions... is this a limitation of the API? Is there a way around this?

1
  • Unfortunately, about all you can do is pass pagesize=100 to Site.questions. You can then use the methods on the returned resultset to advance through the pages. You won't be able to get anything bigger than 100 at a time, though. There are a couple of other options: you could try and export results from data.stackexchange.com, but those will probably be size-limited too. 'Worst case', you could download the data dump, but it's rather large (multi-gigabyte range) and hard to deal with. Jun 9, 2011 at 23:38
2

I am running py-stackexchange v1.1-4 and I'm hitting a strange issue after not touching this library for perhaps eight months or so... My queries used to be relatively fast... I could get results in 30 seconds.

Now my queries literally take hours with the same code... example query:

def filtered_question_dict(sesite, mytags, pagesize=100,
    max_timedelta_hours=24):
    ## Get questions updated in the last 24 hours and not created more than 2 weeks ago

    all = dict()
    now = datetime.now()
    MAX_LAST_DELTA = timedelta(hours=max_timedelta_hours)
    MAX_CREATE_DELTA = timedelta(weeks=2)
    for ii in [sesite.questions.no_answers, sesite.recent_questions,
        sesite.questions.unanswered]:
        ## Iterate over questions, no answers sent, no body sent...
        for qq in ii(pagesize=pagesize, answers=False, body=False):
            qcreate = datetime.fromtimestamp(qq.json['creation_date'])
            qlast = datetime.fromtimestamp(qq.json['last_activity_date'])
            qtags = set(qq.json['tags'])
            if ((now-qlast)>MAX_LAST_DELTA) or \
                ((now-qcreate)>MAX_CREATE_DELTA) or \
                (len(mytags.intersection(qtags))==0):
                continue
            elif not accepted(qq.json):
                all[qtitle] = (qq, mytags, qcreate)
    return all

Just iterating over sesite.questions.no_answers ran for six hours last night before I hit control-C to stop it. I was also sniffing the session... I had pulled over 4000 'pages' in this query...

23467.909060 172.16.1.52 -> 64.34.119.12 HTTP GET /1.1/questions/no-answers?body=false&pagesize=100&answers=false&comments=false&key=REDACTED&page=4688 HTTP/1.1

Maybe Stack Exchange has changed their servers to rate-limit things, but if there is no server-side filtering it seems like the library would be unusable at this rate.

Is there something I should be doing differently?

3
  • I have a feeling I may have fixed this bug in the latest version of the code, but that I haven't pushed it to PyPI. I'll try and push the latest code up; in the meantime you can get it straight from the repository at github.com/lucjon/Py-StackExchange. May 7, 2012 at 18:06
  • OK. The behaviour of the iterators changed recently. Site.questions.no_answers, Site.questions.recent_questions and Site.questions.unanswered will go on forever. The pagesize= argument actually tells the API how many items to return on each iteration - it will just keep iterating until it's returned all the pages. I agree this is not the most useful behaviour for this case, and I'm now planning to add a cleaner interface. For now, though, change your loop to for qq in ii(...).items, which will only iterate through the questions on the first page, of size pagesize. May 11, 2012 at 14:31
  • @LucasJones, I will try that. Thank you for following up May 11, 2012 at 15:18
2

The API was working but now I receive an error when I try to run the demo examples.
Sample error dump:

/usr/bin/python2.7 /home/donbeo/PycharmProjects/stack_overflow/experience.py
StackOverflow user 41981's experience:
Traceback (most recent call last):
  File "/home/donbeo/PycharmProjects/stack_overflow/experience.py", line 13, in <module>
    user = so.user(user_id)
  File "/home/donbeo/Py-StackExchange-master/stackexchange/__init__.py", line 699, in user
    u, = self.users((nid,), **kw)
  File "/home/donbeo/Py-StackExchange-master/stackexchange/__init__.py", line 704, in users
    return self._get(User, ids, 'users', kw)
  File "/home/donbeo/Py-StackExchange-master/stackexchange/__init__.py", line 694, in _get
    return self.build(root, typ, coll, kw)
  File "/home/donbeo/Py-StackExchange-master/stackexchange/__init__.py", line 671, in build
    json = self._request(url, kw)
  File "/home/donbeo/Py-StackExchange-master/stackexchange/__init__.py", line 642, in _request
    json, info = request_mgr.json_request(url, new_params)
  File "/home/donbeo/Py-StackExchange-master/stackexchange/web.py", line 139, in json_request
    req = self.request(to, params)
  File "/home/donbeo/Py-StackExchange-master/stackexchange/web.py", line 111, in request
    conn = req_open.open(request)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 400: Bad Request

Process finished with exit code 1

How can I solve this?

1
  • Thanks for getting in touch. The demo works for me locally; could I ask you to do the following: (1) ensure that you are using the latest version of the library from the Github repository; (2) if the problem persists, enable debug mode by adding the line stackexchange.web.WebRequestManager.debug = True to the top of your script. May 27, 2014 at 22:40
1

Two of the demo scripts currently do not work because there is no 'answers' field in the json :(

It might be an ugly workaround, but for now this diff does the trick:

badp@delta:~/Py-StackExchange$ git diff 0a5c60f71310757a4c1de6547cf113c7d6834e85
diff --git a/stackexchange.py b/stackexchange.py
index 959cf3e..4a17b10 100644
--- a/stackexchange.py
+++ b/stackexchange.py
@@ -492,10 +492,12 @@ unlike on the actual site, you will receive an error rathe

        def questions(self, ids, **kw):
                """Retrieves a set of the comments with the IDs specified in the
+               kw["answers"] = "true"
                return self._get(Question, ids, 'questions', kw)

        def recent_questions(self, **kw):
                """Returns the set of the most recent questions on the site, by 
+               kw["answers"] = "true"
                return self.build('questions', Question, 'questions', kw)

        def users_with_badge(self, bid, **kw):

However, this fails to take account of the good points behind the change.

3
  • Thanks - I'll add the diff for now, to keep compatibility. I may factor it into the be_inclusive() method later, though. Jun 12, 2010 at 21:43
  • I've added the patch manually (it's only small :D), but I can't figure out how to do it with Git - git apply didn't work. Just for future reference, do you know how I'd do that? Jun 12, 2010 at 21:47
  • Done in the latest revision. Jun 12, 2010 at 21:57
1

Feature Request

A little feature I think could come in handy.

Right now, there is no way to get the url of a Question/Answer from the Answer object (I'm talking about the actual URL on the site, e.g. stackoverflow.com/answers/id). This is also not returned by the api itself (see this answer).

Ideally, I think the wrapper should include a method that builds the url for you, i.e. the Answer object will have a getUrl method which will build up the url based on which site you're querying.

If not, another good idea would be to provide the Site object with a method that gets the url of the site. When you build a Site object you send in a constant like "api.stackoverflow.com", so the object should have a method which strips out the api part.

1
  • Done in latest revision. site.root_domain and {user|question|answer}.url. Beware that this is hard-coded and relatively hacky! :) Jun 14, 2010 at 18:50
1

Another quick question:

Is there any way to get a user's answers without going through the user object? I want to save a fetch of a user's answers, and since I already have his id, I could go to: http://api.stackoverflow.com/0.8/users/id/answers?body=true&pagesize=100.

I'm just not sure how to do such a thing using the wrapper.

If it doesn't exist, it would probably make sense to be able to do:

# so is my stackexchange object
so.answers(userid=id, other_keywords...)

Also, for bonus points, when getting a user's answers, you get back info on the user as well (e.g. you get the user's display name). So such a call could automatically create the user object that's linked to each answer.

4
  • 1
    Finished in latest revision. Jun 15, 2010 at 17:17
  • Looking good so far. One quick question: right now, doing site.answers(user_id=some_id_that_does_not_exist) just gives back an empty list. I don't know about this, but would it be better to raise a "User does not exist" exception? You could also raise this in other places that have the same "problem". Not sure if this would actually be better, just an idea.
    – Edan Maor
    Jun 16, 2010 at 6:50
  • I was thinking about that when I accidentally requested page 2 of what I expected to be a multi-page list (it was only 14 items!). Some distinction could be useful... Jun 16, 2010 at 15:07
  • Note to future readers: a nicer API is supported in v1.1: site.questions.by_user(uid). May 18, 2011 at 15:27
1

There's a bug with deleted users. Attempting to call question.comments.fetch() on a question that has comments by a deleted users, such as this, results in:

  File "stackcloud.py", line 60, in <module>
    for comment in question.comments.fetch():
  File "stackcore.py", line 151, in fetch
    res = self.site.build(self.url, self.m_type, self.collection, kw)
  File "stackexchange.py", line 341, in build
    return JSONMangler.json_to_resultset(self, json, typ, collection, (self, url, typ, collection, kw))
  File "stackcore.py", line 188, in json_to_resultset
    return cls.paginated_to_resultset(site, json, typ, collection, params)
  File "stackcore.py", line 174, in paginated_to_resultset
    items.append(typ(json_item, site))
  File "stackcore.py", line 19, in __init__
    self._extend(self.json_ob, site)
  File "stackexchange.py", line 105, in _extend
    self.owner_id = json.owner['owner_id'] if 'owner_id' in json.owner else json.owner['user_id']
AttributeError: DictObject instance has no attribute 'owner'

Here's a fix proposal.

2
  • Thanks! I really should have tested that... thanks again, though. I'll incorporate that just now. :) Jan 20, 2011 at 18:16
  • 1
    Committed and pushed! Jan 20, 2011 at 19:16
1

I'm building a Django app, and I'l like to be able to search for users' SO account based on their name. I've got the following, which works from a python shell, but this doesn't work when called from within a Django view. Any ideas?

def so_user_search(full_name):
    so = Site(StackOverflow, MY_API_KEY)
    results = so.users([], **{'filter':full_name})
    results = results.fetch()

    result_list = []
    for item in results:
        d = { 
            'id':item.id, 
            'display_name':item.display_name, 
            'url':item.url, 
            'website':item.website_url, 
            'email_hash':item.email_hash
        }   
        result_list.append(d)
    return result_list
1
  • 1
    Sorry for the delay - OpenID/hosting problems... Anyway, doing some testing, I noticed that Django view parameters are Unicode strings. I hadn't thought to have handled Unicode explicitly in the library, and it turned out that the urllib module encodes it as UTF-16 by default, which the API (understandably) doesn't like. I've changed it to UTF-8, which now works under Django for me. Summary: update to the latest version in the Git repo. (Or, I can push a new release of the egg out if necessary.) May 4, 2011 at 21:00
1

I'm running a program using py-stackexchange that gets an error,

HTTPError: HTTP Error 500: Internal Server Error

Do you have any debugging suggestions? If I could turn on a debugging flag, that would cause py-SE to print the exact URL that was requested, that would help... Then I could visit that URL myself in the browser and see if a more detailed error message was shown, such as "your parameter X was invalid" or "you've exceeded your API key limit" or something.

I could modify __init__.py myself to print such debugging messages, but I'm not up on how to recompile python functions within an egg and redeploy them.

Thanks...

Update:

P.S. It wasn't my intention to ask you to find the actual problem, but heck, I'd be just as happy to have that answer as to have the debugging tool described above, and it might be easier to supply. So here's my code:

currentDate = floor(time.time())
aWhileAgo = currentDate - 10

# E.g. fromdate=1293840000&todate=1294444800

questions = so.questions(sort=Sort.Creation, order=DESC, fromdate=aWhileAgo, \
    todate=currentDate)

The last line is where I get the 500 error.

But calling this URL directly (which is what I think py-SE should be using under the hood):

http://api.stackoverflow.com/1.1/questions?fromdate=1305580368&order=desc&sort=creation&todate=1305580478

is successful.

P.P.S.

If I take away the fromdate and todate keywords, it runs successfully:

questions = so.questions(sort=Sort.Creation, order=DESC)

So I guess the problem has to do with those keywords. Probably I'm using Py-SE incorrectly (as opposed to trying to get the SE API to do something it's not designed to do).

SOLVED.

I figured it out: I'm supposed to pass from_date and to_date, not fromdate and todate, despite the SE API parameter names.

I figured that out by guessing. Next, I would like to know how I was supposed to find that out? Naming conventions? If so, where are they documented? Just by example?

4
  • 1
    Hello; sorry for the delay, glad you solved your problem. To answer your first question, you can turn on debug printing of URLs by setting stackexchange.web.WebRequestManager.debug = True. I agree that the documentation in that area is lacking; a wiki page is coming up. I'll look into that inconsistency; it seems quite pointless... I apologise on behalf of my younger self. May 18, 2011 at 15:05
  • While I'm here, I'll try putting in some more descriptive exceptions, too. May 18, 2011 at 15:13
  • Doing some testing... strange: with from_date=aWhileAgo and to_date=currentDate it works, as it does with fromdate=int(aWhileAgo) and todate. But using a float as fromdate (as aWhileAgo was), which is formatted with a trailing .0 fails with error 500. So both work with integers; this might be an API inconsistency. May 18, 2011 at 15:21
  • Ahem. Revisited this, doing browser testing. The reason from_date and to_date work is that they are ignored. Compare the total field on api.stackoverflow.com/1.1/… to api.stackoverflow.com/1.1/…. Either way, you can pass in a float if you want now; in the latest revision it'll be appropriately converted behind-the-scenes. Jun 10, 2011 at 0:46
1

How can I access the q in the questions method?

In the Stack Exchange API, Usage of /search/advanced:

q - a free form text parameter, will match all question properties based on an undocumented algorithm.

1
  • Sorry for the delay in responding. You should be able to simply pass a q parameter to the Site#search_advanced method, e.g. Site('stackoverflow.com').search_advanced(q = 'python'). Let me know if you have any more difficulties. Sep 12, 2016 at 22:15
1

Seems very unreliable to me. Using the GitHub number of answers example results in a NeedsAwoken error, while the scrolling list of questions makes a StackExchangeError.

List of errors (community can add to this):

  • NeedsAwokenError
  • StackExchangeError

How do I solve this problem?

5
  • I've fixed the example in the readme (this was caused by a change in the default set of fields returned by the user API call), thanks for pointing out the error. I can't reproduce the issue with the questions example, could you give me some more detail? Dec 26, 2016 at 18:13
  • @LucasJones thanks for your quick response! I'll try it again and if it works I will remove this answer.
    – Mark Yisri
    Dec 26, 2016 at 21:35
  • @LucasJones thanks for your help! Everything works now
    – Mark Yisri
    Dec 29, 2016 at 12:21
  • @LucasJones or maybe not. This is what I get with the example in README.md now: stackexchange.core.StackExchangeError: 502 [throttle_violation]: Violation of backoff parameter
    – Mark Yisri
    Dec 29, 2016 at 12:23
  • That error is raised by the API if you make what it deems to be 'too many' requests in a short time period. This should go away shortly. If you are not passing in an API key when you create a Site object, the limits might be smaller than you expect, see api.stackexchange.com/docs/throttle. Jan 3, 2017 at 22:11
1

I can't figure out how to fetch the text body of the answers and comments to a question given the question id. Also I can't figure out what are the supported sites(variable Name) in the method name stackexchange.site(Name,api_key) . Can anybody provide me with the list of supported sites of stackexchange and what exactly should be provided in the variable Name during method invocation? Here's the code snippet I've written so far:

import stackexchange site = stackexchange.Site(stackexchange.StackOverflow, app_key=None) site.be_inclusive() for e in url_list: if stack_id(e): question = site.question(stack_id(e)) print('--- %s ---' % question.title) print(question.body) print() print(question.answers) Please can somebody help me figure this out? I'm using Python 3.4.3 and latest version of Py-StackExchange.

0

First things first: Thank you very much for your efforts in building this Python package. It saves me lots of hours for sure. Keep up the good work.

Nevertheless, I have a small issue when fetching data from the Stackoverflow site. I try to load a specific user object and would like to get reputation as well as up vote count, down vote count and view count. Fetching reputation works well, but when I try to access the other attributes, such as up_vote_count, I fail.

Here is my code so far:

so = stackexchange.Site(stackexchange.StackOverflow)
so_user_obj = so.user(so_user_id)
so_user_obj.badges.fetch()
so_user_obj.questions.fetch()
so_user_obj.answers.fetch()
user_dict['reputation'] = so_user_obj.reputation
user_dict['questions_asked'] = len(so_user_obj.questions)
user_dict['answers_posted'] = len(so_user_obj.answers)
user_dict['badges_earned'] = len(so_user_obj.badges)
user_dict['up_votes_received'] = so_user_obj.up_vote_count
user_dict['down_votes_received'] = so_user_obj.down_vote_count
user_dict['profile_views'] = so_user_obj.view_count

When running the code, I can fetch reputation successfully, but get this error for the other attribute:

Traceback (most recent call last):
  File "./stacktrack.py", line 405, in <module>
    user_data = so_get_user_obj_as_dict(so_user_obj)
  File "./stacktrack.py", line 193, in so_get_user_obj_as_dict
    user_dict['up_votes_received'] = so_user_obj.up_vote_count
AttributeError: 'User' object has no attribute 'up_vote_count'

I actually thought, that up_vote_count, down_vote_count and view_count are part of the response, because of this code in models.py (~ line 342):

class User(JSONModel):
    """Describes a user on a StackExchange site."""
    transfer = ('display_name', 'profile_image', 'age', 'website_url',
        'location', 'about_me', 'view_count', 'up_vote_count',
        'down_vote_count', 'account_id', 'profile_image',
        ('creation_date', UNIXTimestamp),
        ('last_access_date', UNIXTimestamp),
        ('reputation', FormattedReputation),
        ...

What am I doing wrong here?

I use Python 3.4.3 with py-stackexchange 2.2.7 (installed via pip3).

Any help highly appreciated!

Thank you André

3
0

I have been experiencing some throttle issues when trying to fetch timeline data for some users. I get a: stackexchange.core.StackExchangeError: 502 [throttle_violation]: Violation of backoff parameter exception after the api handles some requests...

The weird thing is I'm using:

so.impose_throttling = True
so.throttle_stop = False

So I really don't understand why py-stackexchange does not handle the overloading requests and make my application wait until it can make more requests... What am I doing wrong?

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .