Skip to main content
Stack Apps

Return to Answer

added 834 characters in body
Source Link
Alex Martelli
Alex Martelli
  • 171
  • 1
  • 3

EDIT: yep, confirmed, the duplicates appear exactly as I had thought they would -- I've added a set of ids and return prepamaturelyprematurely after seeing a duplicate, and the number of total results reported for each and every month is exactly the page size -- confirming that each question is being yielded again right after all items on the first page have been yielded, exactly as the code I show above appears to say they would be.

So, any suggested workaround so I can examine all Qs meeting the constraints, and, only once each?-)

EDIT AGAIN: so I partly fixed def next(): in the result-set class (there's still a bug I can't yet fathom -- shows duplicates with a page size of 37 -- but at least it seems to work with a page size of 100, with no duplicates nor truncation to a multiple of 100). To make python3 setup.py install work I also had to change setup.py since (at the github master) it was missing some modules (?).

The resulting branch is at https://github.com/aleaxit/Py-StackExchange/tree/betteriter and of course I've also sent a pull request for it. But the gist is to change the above-quoted brokenness into

def next(self):
    current = self
    while True:
        for obj in current.items:
            yield obj
        if not current.has_more:
            return

(no further changes below this).

EDIT: yep, confirmed, the duplicates appear exactly as I had thought they would -- I've added a set of ids and return prepamaturely after seeing a duplicate, and the number of total results reported for each and every month is exactly the page size -- confirming that each question is being yielded again right after all items on the first page have been yielded, exactly as the code I show above appears to say they would be.

So, any suggested workaround so I can examine all Qs meeting the constraints, and, only once each?-)

EDIT: yep, confirmed, the duplicates appear exactly as I had thought they would -- I've added a set of ids and return prematurely after seeing a duplicate, and the number of total results reported for each and every month is exactly the page size -- confirming that each question is being yielded again right after all items on the first page have been yielded, exactly as the code I show above appears to say they would be.

So, any suggested workaround so I can examine all Qs meeting the constraints, and, only once each?-)

EDIT AGAIN: so I partly fixed def next(): in the result-set class (there's still a bug I can't yet fathom -- shows duplicates with a page size of 37 -- but at least it seems to work with a page size of 100, with no duplicates nor truncation to a multiple of 100). To make python3 setup.py install work I also had to change setup.py since (at the github master) it was missing some modules (?).

The resulting branch is at https://github.com/aleaxit/Py-StackExchange/tree/betteriter and of course I've also sent a pull request for it. But the gist is to change the above-quoted brokenness into

def next(self):
    current = self
    while True:
        for obj in current.items:
            yield obj
        if not current.has_more:
            return

(no further changes below this).

added 556 characters in body
Source Link
Alex Martelli
Alex Martelli
  • 171
  • 1
  • 3

EDIT: yep, confirmed, the duplicates appear exactly as I had thought they would -- I've added a set of ids and return prepamaturely after seeing a duplicate, and the number of total results reported for each and every month is exactly the page size -- confirming that each question is being yielded again right after all items on the first page have been yielded, exactly as the code I show above appears to say they would be.

So, any suggested workaround so I can examine all Qs meeting the constraints, and, only once each?-)

EDIT: yep, confirmed, the duplicates appear exactly as I had thought they would -- I've added a set of ids and return prepamaturely after seeing a duplicate, and the number of total results reported for each and every month is exactly the page size -- confirming that each question is being yielded again right after all items on the first page have been yielded, exactly as the code I show above appears to say they would be.

So, any suggested workaround so I can examine all Qs meeting the constraints, and, only once each?-)

Source Link
Alex Martelli
Alex Martelli
  • 171
  • 1
  • 3

Total results vary with page size

The idea of producing a transparent iterator (getting the next page as needed) is cool, but it just isn't working; the page size affects the total number of results returned (apparently to a multiple of the page size)...:-(

My code needs to look at all Qs asked for a given tag in a given month:

def qstats(site, tag, month, year=None):
    fromdate, todate = month_range(month, year)

    qs = site.questions(pagesize=100,
        sort='votes',
        fromdate=ts(*fromdate),
        todate=ts(*todate),
        tagged=tag)

    good_qs = 0
    for i, q in enumerate(qs, start=1):
        good, answered = qtype(q)
        if good:
            good_qs += 1
    return i, good_qs

With a page size of 100, as here, I see, from a loop calling this for different months on a certain tag:

2014/11: 242 good Qs out of 600 (40%)
2014/12: 242 good Qs out of 500 (48%)
2015/1: 235 good Qs out of 500 (47%)

but e.g with an arbitrary page size of 37, it's instead

2014/11: 179 good Qs out of 518 (35%)
2014/12: 179 good Qs out of 518 (35%)
2015/1: 172 good Qs out of 518 (33%)

(518 is 37 * 14, whence my hypothesis that the returned number of items is somehow constrained to be a multiple of the page size -- but clearly it's not just that, as the page size of 100 gave 600 questions for the tag in Nov'14, but the page size of 37 still gives up at 518). I guess this is connected with the already reported bug of fetch_next returning nothing at unpredictable times, even though here it's buried in the iteration.

But I don't understand at all the code of the next method of StackExchangeResultset in the core.py file...: it starts...:

def next(self):
    for obj in self.items:
        yield obj

    current = self
    while current.has_more:
        for obj in current.items:
            yield obj

won't this yield each item in self.items twice? Once from self.items, and then again from current.items after setting current = self?! I just don't understand the logic of this snippet. I'm going to instrument my code to check for duplicates, which should be present if my doubts are well-founded, and report on that check...