I'll give you the context of what I was doing in case anyone wants to offer suggestions for the best route I might take to achieve my goal, which is to have easy access to do queries on the newish stachexchange data.
I first looked for the data dump but found it isn't very new (Aug 2012).
I then found the online interface to enter sql queries and I was happy that I could get results pretty easily. Here's the query I did:
with T as (
select top 100 Users.Id UserId, count(Posts.Id) NumPosts
from Users
inner join Posts on Users.Id = Posts.OwnerUserId
group by Users.Id, Users.Reputation
order by Reputation desc
)
select top 100 percent *
from T, Users
where Users.Id = T.UserId
order by Users.Reputation desc
However, I saw nothing in the FAQ this was an API.
Finally, I found the OData endpoint and gave that a try but found that a query to count posts with a given OwnerUserId randomly throws an exception that only says "see the server logs for more info". Since it was random I wrote some retry logic.
class Row
{
public int Id { get; set; }
public string Name { get; set; }
public int Points { get; set; }
public int PostCount { get; set; }
}
[TestMethod]
public void GetRows()
{
var context = new Entities(new Uri("http://data.stackexchange.com/stackoverflow/atom"));
var topUsers = context.Users.OrderByDescending(c => c.Reputation).Take(50);
Func<int, int> postCount = (id) =>
{
Thread.Sleep(1000);
try
{
return context.Posts.Where(c => c.OwnerUserId == id).Count();
}
catch
{
return -1;
}
};
var topUsersWithPostCount =
from u in topUsers.AsEnumerable().Where(c => c.Id != 157882)
select new Row
{
Id = u.Id,
Name = u.DisplayName,
Points = u.Reputation ?? 0,
PostCount = postCount(u.Id)
};
var ok = topUsersWithPostCount.ToLookup(c => c.PostCount != -1);
int tries = 1;
int maxTries = 5;
while (true)
{
Console.WriteLine("retrying..");
var retry = from item in ok[false]
select new Row
{
Id = item.Id,
Name = item.Name,
Points = item.Points,
PostCount = postCount(item.Id)
};
var result = ok[true].Concat(retry);
if (result.Any(c => c.PostCount == -1) && tries++ < maxTries)
continue;
result.ToList().ForEach(c => Console.WriteLine("{0},{1},{2}", c.Id, c.Name, c.PostCount));
break;
}
}
Without the 1sec pause it goes past max number of retries.
Is there a limit to number of requests in a time period? Alternately, is there a way I could have achieved the same result with fewer requests? I tried using .Any but it returned a "OData must be version 3" error.