OBSOLETE - Soapi.CS : A fully relational fluent .NET Stack Exchange API client library

Question

Soapi.CS for .Net / Silverlight / Windows Phone 7 / Mono

as easy as breathing...:

var context = new ApiContext(apiKey).Initialize(false); 

Question thisPost = context.Official
    .StackApps
    .Questions.ById(386)
    .WithComments(true)
    .First();
    
Console.WriteLine(thisPost.Title);

thisPost
  .Owner
  .Questions
  .PageSize(5)
  .Sort(PostSort.Votes)
  .ToList()
  .ForEach(q=>
    {
        Console.WriteLine("\t" + q.Score + "\t" + q.Title);
        
        q.Timeline.ToList().ForEach(t=>
          Console.WriteLine("\t\t" +  
             t.TimelineType + "\t" + t.Owner.DisplayName));
              
        Console.WriteLine();
    });

// if you can think it, you can get it.

Output

Soapi.CS : A fully relational fluent .NET Stack Exchange API client library
  21  Soapi.CS : A fully relational fluent .NET Stack Exchange API client library
    Revision  code poet
    Revision  code poet
    Votes  code poet
    Votes  code poet
    Revision  code poet
    Revision  code poet
    Revision  code poet
    Votes  code poet
    Votes  code poet
    Votes  code poet
    Revision  code poet
    Revision  code poet
    Revision  code poet
    Revision  code poet
    Revision  code poet
    Revision  code poet
    Revision  code poet
    Revision  code poet
    Revision  code poet
    Revision  code poet
    Votes  code poet
    Comment  code poet
    Revision  code poet
    Votes  code poet
    Revision  code poet
    Revision  code poet
    Revision  code poet
    Answer  code poet
    Revision  code poet
    Revision  code poet

  14  SOAPI-WATCH: A realtime service that notifies subscribers via twitter when the API changes in any way.
    Votes  code poet
    Revision  code poet
    Votes  code poet
    Comment  code poet
    Comment  code poet
    Comment  code poet
    Votes  lfoust
    Votes  code poet
    Comment  code poet
    Comment  code poet
    Comment  code poet
    Comment  code poet
    Revision  code poet
    Comment  lfoust
    Votes  code poet
    Revision  code poet
    Votes  code poet
    Votes  lfoust
    Votes  code poet
    Revision  code poet
    Comment  Dave DeLong
    Revision  code poet
    Revision  code poet
    Votes  code poet
    Comment  lfoust
    Comment  Dave DeLong
    Comment  lfoust
    Comment  lfoust
    Comment  Dave DeLong
    Revision  code poet

  11  SOAPI-EXPLORE: Self-updating single page JavaSript API test harness
    Votes  code poet
    Votes  code poet
    Votes  code poet
    Votes  code poet
    Votes  code poet
    Comment  code poet
    Revision  code poet
    Votes  code poet
    Revision  code poet
    Revision  code poet
    Revision  code poet
    Comment  code poet
    Revision  code poet
    Votes  code poet
    Comment  code poet
    Question  code poet
    Votes  code poet

  11  Soapi.JS V1.0: fluent JavaScript wrapper for the StackOverflow API
    Comment  George Edison
    Comment  George Edison
    Comment  George Edison
    Comment  George Edison
    Comment  George Edison
    Comment  George Edison
    Answer  George Edison
    Votes  code poet
    Votes  code poet
    Votes  code poet
    Votes  code poet
    Revision  code poet
    Revision  code poet
    Answer  code poet
    Comment  code poet
    Revision  code poet
    Comment  code poet
    Comment  code poet
    Comment  code poet
    Revision  code poet
    Revision  code poet
    Votes  code poet
    Votes  code poet
    Votes  code poet
    Votes  code poet
    Comment  code poet
    Comment  code poet
    Comment  code poet
    Comment  code poet
    Comment  code poet

  9  SOAPI-DIFF: Your app broke? Check SOAPI-DIFF to find out what changed in the API
    Votes  code poet
    Revision  code poet
    Comment  Dennis Williamson
    Answer  Dennis Williamson
    Votes  code poet
    Votes  Dennis Williamson
    Comment  code poet
    Question  code poet
    Votes  code poet

---

About

A robust, fully relational, easy to use, strongly typed, end-to-end StackOverflow API Client Library.

Out of the box, Soapi provides you with a robust client library that abstracts away most all of the messy details of consuming the API and lets you concentrate on implementing your ideas.

A few features include:

• A fully relational model of the API data set exposed via a fully 'dot navigable' IEnumerable (LINQ) implementation. Simply tell Soapi what you want and it will get it for you.

  e.g. "On my first question, from the author of the first comment, get the first page of comments by that person on any post"

  my.Questions.First().Comments.First().Owner.Comments.ToList();

  _{(yes this is a real expression that returns the data as expressed!)}

• Full coverage of the API, all routes and all parameters with an intuitive syntax.

• Strongly typed Domain Data Objects for all API data structures.

• Eager and Lazy Loading of 'stub' objects. Eager\Lazy loading may be disabled.

• When finer grained control of requests is desired, the core RouteMap objects may be leveraged to request data from any of the API paths using all available parameters as documented on the help pages.

• A rich Asynchronous implementation.

• A configurable request cache to reduce unnecessary network traffic and to simplify your usage logic. There is no need to go out of your way to be frugal. You may set a distinct cache duration for any particular route.

• A configurable request throttle to ensure compliance with the api terms of usage and to simplify your code in that you do not have to worry about and respond to 50X errors.

  The RequestCache and Throttled Queue are thread-safe, so can make as many requests as you like from as many threads as you like as fast as you like and not worry about abusing the api or having to write reams of management/compensation code.

• Configurable retry threshold that will, by default, make up to 3 attempts to retrieve a request before failing. Every request made by Soapi is properly formed and directed so most any http error will be the result of a timeout or other network infrastructure. A retry buffer provides a level of fault tolerance that you can rely on.

• An almost identical javascript library, Soapi.JS, and it's full figured big brother, Soapi.JS2, that will enable you to leverage your server cycles and bandwidth for only those tasks that require it and offload things like status updates to the client's browser.

License

Licensed GPL Version 2 license.

• Why is Soapi.CS GPL?
• Can I get an LGPL license for Soapi.CS? _{(hint: probably)}

Platforms

• .NET 3.5
• .NET 4.0
• Silverlight 3
• Silverlight 4
• Windows Phone 7
• Mono

Download

Source code lives @ http://soapics.codeplex.com. Binary releases are forthcoming.

codeplex is acting up again.

get the source and binaries @ http://bitbucket.org/bitpusher/soapi.cs/downloads

The source is C# 3.5. and includes projects and solutions for the following IDEs

• Visual Studio 2008
• Visual Studio 2010
• ModoDevelop 2.4

Documentation

Full documentation is available at http://soapi.info/help/cs/index.aspx

Sample Code / Usage Examples

Sample code and usage examples will be added as answers to this question.

• Full API Coverage all API routes are covered
• Full Parameter Parity If the API exposes it, Soapi giftwraps it for you.
• Building a simple app with Soapi.CS - a simple app that gathers all traces of a user in the whole stackiverse.
• Fluent Configuration - Setting up a Soapi.ApiContext could not be easier
• Bulk Data Import - A tiny app that quickly loads a SQLite data file with all users in the stackiverse.
• Paged Results - Soapi.CS transparently handles multi-page operations.
• Asynchronous Requests - Soapi.CS provides a rich asynchronous model that is especially useful when writing api apps in Silverlight or Windows Phone 7.
• Caching and Throttling - how and why

Apps that use Soapi.CS

• Soapi.FindUser - .net utility for locating a user anywhere in the stackiverse
• Soapi.Explore - The entire API at your command
• Soapi.LastSeen - List users by last access time
• Add your app/site here - I know you are out there ;-) if you are not comfortable editing this post, simply add a comment and I will add it.

The CS/SL/WP7/MONO libraries all compile the same code and with the exception of environmental considerations of Silverlight, the code samples are valid for all libraries.

You may also find guidance in the test suites.

More information on the SOAPI eco-system.

Contact

This library is currently the effort of me, Sky Sanders (code poet) and can be reached at gmail - sky.sanders

Any who are interested in improving this library are welcome.

Support Soapi

You can help support this project by voting for Soapi's Open Source Ad post

For more information about the origins of Soapi.CS and the rest of the Soapi eco-system see What is Soapi and why should I care?

---

Answer 1

Asynchronous Requests

Async requests can be made using the Site's RouteFactory.

A salient use case for this functionality is using Soapi in Silverlight.

[TestMethod, Asynchronous]
public void AnswersByIdTestValid()
{

    // The thing to be aware of is that the IEnumerable, lazyloading
    // model is synchronous, and since we are on the UI thread here
    // we need to start with an async request using the site's RouteFactory.

    StackOverflow
        .RouteFactory
        .AnswersById(AnswersByIdValidId)
        .BeginGetResponse(a =>
        {
            // we are off the UI thread here so we can use the full power of
            // Soapi.

            var answers = a.Result.Items;

            var answerersOtherAnswers = answers[0].Owner.Answers.ToList();

            foreach (var otherAnswer in answerersOtherAnswers)
            {
                var qTitle = otherAnswer.Question.Title;
                var aBody = otherAnswer.Body;

            }
            EnqueueTestComplete();
        });
}

Requests

http://api.stackoverflow.com/clientaccesspolicy.xml
http://api.stackoverflow.com/1.0/answers/2917525?key=qgAq_KfDu0KYzlNG-qaTuw
http://api.stackoverflow.com/1.0/users/242897?key=qgAq_KfDu0KYzlNG-qaTuw
http://api.stackoverflow.com/1.0/users/242897/answers?key=qgAq_KfDu0KYzlNG-qaTuw
http://api.stackoverflow.com/1.0/questions/287903;1890582;1937805;1939446;1988022;2010405;2403263;2952788;2991978;3163678;3261144;3287656;3287957;3298685;3299236;3299480;3299964;3300051;3300386;3301925;3307832;3312226;3312541;3313771;3315860;3326820;3329645;3330028;3333304;3399501?answers=true&body=true&comments=true&key=qgAq_KfDu0KYzlNG-qaTuw
http://api.stackoverflow.com/1.0/users/4333;4639;5309;6068;9382;10039;26160;77775;85040;85606;99256;102937;109676;119396;134430;135731;145682;146857;163728;177333;231821;248616;327528;338195;371434;388364;397880;398084;410007?key=qgAq_KfDu0KYzlNG-qaTuw

Answer 2

Soapi.CS Fluent Configuration

var ctx = new ApiContext(ApiKey).Options
    // obviates a call to stackauth
    .SerializedSites("{ 'api_sites': [ {'name': 'Stack....")
    // can also be accomplished by seeding the cache
    .SeedCache(new RequestCacheItem
        {
            Uri = new Uri("http://stackauth.com/1.0/sites"),
            ResponseText = "{ 'api_sites': [ {'name': 'Stack....",
            Expiration = DateTime.Now.AddDays(1)
        })
    .UserAgent("SOAPI.APP")
    // 'stub' objects are fetch individually upon access
    .LazyLoadingEnabled(true)
    // all 'stub' objects for a batch are requests as a batch
    // and pre populated
    .EagerLoadingEnabled(true) //
    .Context // fluent terminator
    .Initialize();

Answer 3

Soapi.CS Full API Coverage

// instantiate a context

ApiContext ctx = new ApiContext("qgAq_KfDu0KYzlNG-qaTuw");
ctx.Initialize().AsyncWaitHandle.WaitOne();


// get a site
Site sofu = ctx.Official.StackOverflow;

// or 

Site stackapps = ctx.Sites.First(s=>s.ApiEndpoint == "http://api.stackapps.com");


---



// StackAuth Routes

// /sites                          
ctx.Sites.ToList();

// /users/{id}/associated          
sofu.Users.ById(new Guid("d6b99e8e-0f6c-4f68-92d0-6a050feea1fc")).Associated.ToList();




// /answers/{id}                   
sofu.Answers.ById(2917525).ToList();

// /answers/{id}/comments          
sofu.Answers.ById(2907216).Comments.ToList();


// /badges                         
sofu.Badges.ToList();

// /badges/{id}                    
sofu.Badges.ById(9).ToList();

// /badges/name                    

// /badges/tags                    
sofu.Badges.Tags.ToList();


// /comments/{id}                  
sofu.Comments(2917319).ToList();


// /posts/{id}/comments            
sofu.PostComments(2912300).ToList();


// /questions                      
sofu.Questions.ToList();

// /questions/{id}                 
sofu.Questions.ById(2912300).ToList();

// /questions/{id}/answers         
sofu.Questions.ById(2912300).Answers.ToList();

// /questions/{id}/comments        
sofu.Questions.ById(2912300).Comments.ToList();

// /questions/{id}/timeline        
sofu.Questions.ById(2912300).Timeline.ToList();

// /questions/unanswered           
sofu.Questions.Unanswered.ToList();

// /revisions/{id}                 
sofu.Revisions.ById(2350874).ToList();

// /revisions/{id}/{revisionguid}  
sofu.Revisions.ById(2350874).ByRevisionGuid(new Guid("d6b99e8e-0f6c-4f68-92d0-6a050feea1fc")).ToList();

// /search                         
sofu.Search.InTitle("sqlite").ToList();

// /stats                          
sofu.Statistics;

// /tags                           
sofu.Tags.ToList();

// /users                          
sofu.Users.ToList();

// /users/{id}                     
sofu.Users.ById(242897).ToList();

// /users/{id}/answers             
sofu.Users.ById(242897).Answers.ToList();

// /users/{id}/badges              
sofu.Users.ById(242897).Badges.ToList();

// /users/{id}/comments            
sofu.Users.ById(242897).Comments.ToList();

// /users/{id}/comments/{toid}     
sofu.Users.ById(242897).Comments.To(160173).ToList();

// /users/{id}/favorites           
sofu.Users.ById(242897).Favorites.ToList();

// /users/{id}/mentioned           
sofu.Users.ById(242897).Mentioned.ToList();

// /users/{id}/questions           
sofu.Users.ById(242897).Questions.ToList();

// /users/{id}/reputation          
sofu.Users.ById(242897).Reputation.ToList();

// /users/{id}/tags                
sofu.Users.ById(242897).Tags.ToList();

// /users/{id}/timeline            
sofu.Users.ById(242897).Timeline.ToList();

// /users/moderators               
sofu.Users.Moderators.ToList();



// from the perspective of a user

sofu.User(242897).Answers.ToList();
sofu.User(242897).Badges.ToList();
sofu.User(242897).Comments.ToList();
sofu.User(242897).CommentsTo(160173).ToList();
sofu.User(242897).Favorites.ToList();
sofu.User(242897).Mentioned.ToList();
sofu.User(242897).Questions.ToList();
sofu.User(242897).RepChanges.ToList();
sofu.User(242897).Tags.ToList();
sofu.User(242897).Timeline.ToList();

Answer 4

Soapi.CS - Full Parameter Parity

Soapi provides full API coverage including parameters.

[Test]
public void info_param()
{
    // /questions
    SO.Questions
        // answers (optional)   When "true", the answers to a question will be returned
        .WithAnswers(true)
        // body (optional)      When "true", a post's body will be included in the response.
        .WithBody(true)
        // comments (optional)  When "true", any comments on a post will be included in the response.
        .WithComments(true)
        // fromdate (optional)  Unix timestamp of the minimum creation date on a returned item.
        .FromDate("january 1, 2001") // can parse dates
        // max (optional)       Maximum of the range to include in the response according to the current sort.
        .Max(2)
        // min (optional)       Minimum of the range to include in the response according to the current sort.
        .Min(1)
        // order (optional)     How the current sort should be ordered.
        .Order(SortOrder.Asc)
        // page (optional)      The pagination offset for the current collection. Affected by the specified pagesize.
        .Page(1)
        // pagesize (optional)  The number of collection results to display during pagination. Should be between 1 and 100 inclusive.
        .PageSize(100)
        // sort (optional)      How a collection should be sorted.
        .Sort(QuestionSort.Votes)
        // tagged (optional)    Semicolon delimited list of tags questions must have
        .Tagged("a", "b") // accepts param aary
        // todate (optional)    Unix timestamp of the maximum creation date on a returned item.
        .ToDate(new DateTime(2020, 1, 1))

        .PageCount(10) // fetch multiple pages .PageCount(-1) to get all pages
        .ToList();
}

Request that is issued: (bogus request, so no paging occurs)

http://api.stackoverflow.com/1.0/questions?answers=true&body=true&comments=true&fromdate=978307200&max=2&min=1&order=Asc&page=1&pagesize=100&sort=Votes&tagged=a;b&todate=1577836800

Answer 5

Building a simple app with Soapi.CS

Suppose you would like to build a complete dataset containing all of your content and activity for all of your accounts on all Stack Exchange sites.

You would need to:

1. Get a list of your accounts from stackauth /users/{id}/associated
2. For each account
   1. Get all of your questions
   2. Get all of your answers
   3. Get all of your reputation events
   4. Get all of your timeline events

The following class does that with Soapi. It uses a separate thread for each site. The throttle and cache are thread-safe.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using Soapi;
using Soapi.Domain;

namespace Samples
{
    /// <summary>
    /// Builds a complete stack exchange dataset of a single user
    /// </summary>
    public class AllMyStuff
    {

        /// <summary>
        /// Contains all your data on all sites
        /// </summary>
        public List<UserData> Accounts
        {
            get { return _accounts; }
        }


        private const string ApiKey = "Your Key";
        private readonly List<UserData> _accounts;
        private readonly ApiContext _context;

        /// <summary>
        /// Provide a reference user from which to get all 
        /// associated accounts.
        /// </summary>
        /// <param name="endpoint"></param>
        /// <param name="userId"></param>
        public AllMyStuff(string endpoint, int userId)
        {
            _context = new ApiContext(ApiKey);
            _context.Options.LazyLoadingEnabled(true);
            _context.Initialize(false);
            _accounts = _context.Sites.First(s => s.ApiEndpoint.StartsWith(endpoint))
                .User(userId).Associated.Select(u => new UserData(u)).ToList();

            var initEvent = new AutoResetEvent(false);

            foreach (var item in _accounts)
            {
                UserData userData = item;
                ThreadPool.QueueUserWorkItem(o =>
                    {
                        User account = userData.User;

                        userData.Questions = account.Questions
                            .PageCount(-1)
                            .FromDate(new DateTime(1980, 1, 1))
                            .ToDate(new DateTime(2020, 1, 1))
                            .ToList();

                        userData.Answers = account.Answers
                            .PageCount(-1)
                            .FromDate(new DateTime(1980, 1, 1))
                            .ToDate(new DateTime(2020, 1, 1))
                            .ToList();

                        userData.Reputation = account
                            .RepChanges
                            .PageCount(-1)
                            .FromDate(new DateTime(1980, 1, 1))
                            .ToDate(new DateTime(2020, 1, 1))
                            .ToList();

                        userData.Timeline = account
                            .Timeline
                            .PageCount(-1)
                            .FromDate(new DateTime(1980, 1, 1))
                            .ToDate(new DateTime(2020, 1, 1))
                            .ToList();

                        userData.Initialized = true;

                        lock (_accounts)
                            if (_accounts.All(u => u.Initialized))
                                initEvent.Set();
                    });
            }
            initEvent.WaitOne();
        }

 
    }

    public class UserData
    {
        public UserData(User user)
        {
            User = user;
            ApiEndpoint = user.Site.ApiEndpoint;
        }

        public List<Answer> Answers { get; set; }
        public string ApiEndpoint { get; set; }
        public bool Initialized { get; set; }
        public List<Question> Questions { get; set; }
        public List<RepChange> Reputation { get; set; }
        public List<UserTimeline> Timeline { get; set; }
        public User User { get; set; }
    }
}

Answer 6

Soapi.CS Paged Results

Pay particular attention to the third example; thats all 2800+ stack apps users in less than 5 seconds without breaking any rules. not too shabby.

// Soapi elegantly handles multi-page requests.
// Multi-paged requests are issued in parallel making data retrieval quick

// get the default, 1 page of 30 using default sort

var questions = SO.Questions.ToList();
// --> 5:05:57 AM    http://api.stackoverflow.com/1.0/questions?key=qgAq_KfDu0KYzlNG-qaTuw

// get pages 3-4

var pages3to4 = SO.Questions.Page(3).PageCount(2).ToList();

// -->5:05:58 AM    http://api.stackoverflow.com/1.0/questions?page=3&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:05:59 AM    http://api.stackoverflow.com/1.0/questions?page=4&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw

// get all pages of a set

var stackAppsUsers = CTX.Official.StackApps.Users.PageCount(-1).ToList();


// -->5:06:01 AM    http://api.stackapps.com/1.0/users?pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:01 AM    http://api.stackapps.com/1.0/users?page=1&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:02 AM    http://api.stackapps.com/1.0/users?page=2&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:02 AM    http://api.stackapps.com/1.0/users?page=3&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:02 AM    http://api.stackapps.com/1.0/users?page=4&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:02 AM    http://api.stackapps.com/1.0/users?page=5&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:02 AM    http://api.stackapps.com/1.0/users?page=6&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:02 AM    http://api.stackapps.com/1.0/users?page=7&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:02 AM    http://api.stackapps.com/1.0/users?page=8&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:02 AM    http://api.stackapps.com/1.0/users?page=9&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:02 AM    http://api.stackapps.com/1.0/users?page=10&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:02 AM    http://api.stackapps.com/1.0/users?page=11&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:02 AM    http://api.stackapps.com/1.0/users?page=12&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:02 AM    http://api.stackapps.com/1.0/users?page=13&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:02 AM    http://api.stackapps.com/1.0/users?page=14&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:02 AM    http://api.stackapps.com/1.0/users?page=15&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:02 AM    http://api.stackapps.com/1.0/users?page=16&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:02 AM    http://api.stackapps.com/1.0/users?page=17&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:02 AM    http://api.stackapps.com/1.0/users?page=18&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:03 AM    http://api.stackapps.com/1.0/users?page=19&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:03 AM    http://api.stackapps.com/1.0/users?page=20&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:03 AM    http://api.stackapps.com/1.0/users?page=21&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:03 AM    http://api.stackapps.com/1.0/users?page=22&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:04 AM    http://api.stackapps.com/1.0/users?page=23&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:04 AM    http://api.stackapps.com/1.0/users?page=24&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:04 AM    http://api.stackapps.com/1.0/users?page=25&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:04 AM    http://api.stackapps.com/1.0/users?page=26&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:05 AM    http://api.stackapps.com/1.0/users?page=27&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:05 AM    http://api.stackapps.com/1.0/users?page=28&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw
// -->5:06:05 AM    http://api.stackapps.com/1.0/users?page=29&pagesize=100&key=qgAq_KfDu0KYzlNG-qaTuw

Answer 7

Bulk Data Import

This is a simple example of using Soapi.CS for bulk data importation.

This sample creates/updates a SQLite database with the the complete profile of all Stack Exchange users with reputation >= 100 and assigns a rank according to order of appearance.

Soapi processes the ~1000 requests required in less than 5 minutes.

You may run this code against the same database and the records are updated in place.

This code requires references to System.Data.SQLite and Soapi.dll.

Code very similar to this is the core of the SO-REP service.

Usage:

new SoapiUserIndexer()
    .Index("data source=users.db;new=true", apikey);

Code:

using System;
using System.Collections.Generic;
using System.Data;
using System.Data.SQLite;
using System.Linq;
using Soapi.Domain;

namespace Soapi.Data
{

    public class SoapiUserIndexer
    {

        /// <summary>
        /// 
        /// Provides you with a local database of all stack exchange users with 
        /// rep >=100, 90,000 records, in less than 5 minutes using Soapi.CS
        /// 
        /// It is not recommended to decrease the minimum rep value as the numbers
        /// increase drastically with questionable benefit.
        /// 
        /// This is the core of the SO-REP Service, which ranks provides 
        /// ranking data.
        /// 
        /// Requires System.Data.SQLite
        /// http://sqlite.phxsoftware.com/
        /// 
        /// </summary>
        /// <param name="connectionString"></param>
        /// <param name="apiKey"></param>
        public void Index(string connectionString, string apiKey)
        {
            var ctx = new ApiContext(apiKey).Initialize(false);

            var lastModified = DateTime.Now;
            const int batchSize = 50;
            const int minRep = 100;

            using (var connection = new SQLiteConnection(connectionString))
            {
                connection.Open();
                var cmd = connection.CreateCommand();
                cmd.CommandText = CreateUsersTable;
                cmd.ExecuteNonQuery();


                foreach (Site site in ctx.Sites.Where(s => s.ApiEndpoint.IndexOf("stackauth") == -1))
                {

                    // get the count of users with Rep >= 100 and div by 100 to get a page count
                    var pageCount = (int)(Math.Ceiling((double)site.Users.Min(minRep).Total / 100) + 1);

                    var batchCount = Math.Ceiling((double)pageCount / batchSize);
                    for (int i = 0; i < batchCount; i++)
                    {
                        var startPage = i*batchSize + 1;
                        var rank = (batchCount*i) + 1;
                        var users = site.Users.PageCount(batchSize).Page(startPage).Min(minRep).ToList();

                        using (var txn = connection.BeginTransaction())
                        {
                            cmd = BuildInsertCommand();
                            cmd.Connection = connection;
                            foreach (User item in users)
                            {
                                cmd.Parameters["@ApiEndpoint"].Value = item.Site.ApiEndpoint;
                                cmd.Parameters["@UserId"].Value = item.UserId;
                                cmd.Parameters["@AssociationId"].Value = item.AssociationId;
                                cmd.Parameters["@UserType"].Value = item.UserType.ToString();
                                cmd.Parameters["@DisplayName"].Value = item.DisplayName ?? "";
                                cmd.Parameters["@AboutMe"].Value = item.AboutMe ?? "";
                                cmd.Parameters["@Location"].Value = item.Location ?? "";
                                cmd.Parameters["@WebsiteUrl"].Value = item.WebsiteUrl ?? "";
                                cmd.Parameters["@EmailHash"].Value = item.EmailHash ?? "";
                                cmd.Parameters["@CreationDate"].Value = item.CreationDate;
                                cmd.Parameters["@LastAccessDate"].Value = item.LastAccessDate;
                                cmd.Parameters["@TimedPenaltyDate"].Value = item.TimedPenaltyDate;
                                cmd.Parameters["@AcceptRate"].Value = item.AcceptRate;
                                cmd.Parameters["@Age"].Value = item.Age;
                                cmd.Parameters["@AnswerCount"].Value = item.AnswerCount;
                                cmd.Parameters["@DownVoteCount"].Value = item.DownVoteCount;
                                cmd.Parameters["@QuestionCount"].Value = item.QuestionCount;
                                cmd.Parameters["@Reputation"].Value = item.Reputation;
                                cmd.Parameters["@UpVoteCount"].Value = item.UpVoteCount;
                                cmd.Parameters["@ViewCount"].Value = item.ViewCount;
                                cmd.Parameters["@BadgesGold"].Value = item.BadgeCounts.Gold;
                                cmd.Parameters["@BadgesSilver"].Value = item.BadgeCounts.Silver;
                                cmd.Parameters["@BadgesBronze"].Value = item.BadgeCounts.Bronze;
                                cmd.Parameters["@Rank"].Value = rank++;
                                cmd.Parameters["@LastModifiedDate"].Value = lastModified;
                                cmd.ExecuteNonQuery();
                            }
                            txn.Commit();
                        }
                    }
                }

                connection.Close();
            }
        }

        private static SQLiteCommand BuildInsertCommand()
        {
            var cmd = new SQLiteCommand(SqlUerInsertOrUpdate)
                {
                    CommandText = SqlUerInsertOrUpdate
                };

            foreach (var item in new List<object[]>
                {
                    new object[]{"@ApiEndpoint", DbType.String,128},
                    new object[]{"@UserId", DbType.Int32},
                    new object[]{"@AssociationId", DbType.Guid},
                    new object[]{"@UserType", DbType.String,20},
                    new object[]{"@DisplayName", DbType.String,50},
                    new object[]{"@AboutMe", DbType.String,4000},
                    new object[]{"@Location", DbType.String,128},
                    new object[]{"@WebsiteUrl", DbType.String,256},
                    new object[]{"@EmailHash", DbType.String,64},
                    new object[]{"@CreationDate", DbType.DateTime},
                    new object[]{"@LastAccessDate", DbType.DateTime},
                    new object[]{"@TimedPenaltyDate", DbType.DateTime},
                    new object[]{"@AcceptRate", DbType.Int32},
                    new object[]{"@Age", DbType.Int32},
                    new object[]{"@AnswerCount", DbType.Int32},
                    new object[]{"@DownVoteCount", DbType.Int32},
                    new object[]{"@QuestionCount", DbType.Int32},
                    new object[]{"@Reputation", DbType.Int32},
                    new object[]{"@UpVoteCount", DbType.Int32},
                    new object[]{"@ViewCount", DbType.Int32},
                    new object[]{"@BadgesGold", DbType.Int32},
                    new object[]{"@BadgesSilver", DbType.Int32},
                    new object[]{"@BadgesBronze", DbType.Int32},
                    new object[]{"@Rank", DbType.Int32},
                    new object[]{"@LastModifiedDate",DbType.DateTime}
                })
            {
                var p = new SQLiteParameter { ParameterName = (string)item[0], DbType = (DbType)item[1] };
                if (p.DbType == DbType.String)
                {
                    p.Size = (int)item[2];
                }
                cmd.Parameters.Add(p);
            }
            return cmd;
        }


        private const string SqlUerInsertOrUpdate = @"
               INSERT OR REPLACE INTO [Users] ([ApiEndpoint],[UserId],[AssociationId],[UserType],[DisplayName],[AboutMe],[Location]
               ,[WebsiteUrl],[EmailHash],[CreationDate],[LastAccessDate],[TimedPenaltyDate],[AcceptRate],[Age],[AnswerCount],[DownVoteCount]
               ,[QuestionCount],[Reputation],[UpVoteCount],[ViewCount],[BadgesGold],[BadgesSilver],[BadgesBronze],[Rank],[LastModifiedDate]) 
               VALUES (@ApiEndpoint,@UserId,@AssociationId,@UserType,@DisplayName,@AboutMe,@Location,@WebsiteUrl,@EmailHash,@CreationDate
               ,@LastAccessDate,@TimedPenaltyDate,@AcceptRate,@Age,@AnswerCount,@DownVoteCount,@QuestionCount,@Reputation,@UpVoteCount
               ,@ViewCount,@BadgesGold,@BadgesSilver,@BadgesBronze,@Rank,@LastModifiedDate);";

        private const string CreateUsersTable = @"
            CREATE TABLE IF NOT EXISTS [Users] (
                [RowId] integer PRIMARY KEY AUTOINCREMENT NOT NULL,
                [ApiEndpoint] varchar(128) NOT NULL,
                [UserId] int NOT NULL,
                [AssociationId] guid NOT NULL,
                [UserType] varchar(20) NOT NULL,
                [DisplayName] nvarchar(50) NOT NULL,
                [AboutMe] nvarchar(4000) NOT NULL,
                [Location] nvarchar(128) NOT NULL,
                [WebsiteUrl] nvarchar(256) NOT NULL,
                [EmailHash] varchar(64) NOT NULL,
                [CreationDate] datetime NOT NULL,
                [LastAccessDate] datetime NOT NULL,
                [TimedPenaltyDate] datetime NOT NULL,
                [AcceptRate] int NOT NULL,
                [Age] int NOT NULL,
                [AnswerCount] int NOT NULL,
                [DownVoteCount] int NOT NULL,
                [QuestionCount] int NOT NULL,
                [Reputation] int NOT NULL,
                [UpVoteCount] int NOT NULL,
                [ViewCount] int NOT NULL,
                [BadgesGold] int NOT NULL,
                [BadgesSilver] int NOT NULL,
                [BadgesBronze] int NOT NULL,
                [Rank] int NOT NULL,
                [LastModifiedDate] datetime NOT NULL
            );

            CREATE UNIQUE INDEX  IF NOT EXISTS [IX_ApiEndpoint_UserId] ON [Users] ([ApiEndpoint], [UserId]);";
    }
    
}

Answer 8

Soapi Caching and Throttling

Throttle

All API requests made by Soapi are routed first through Soapi.Net.RequestCache and failing a cache hit, they are then routed through Soapi.Net.RequestThrottle.

The throttle, which is described in detail here, works on a sliding window, allowing maximum request rate up to the prescribed limits, currently 30 per 5 seconds, and subsequently blocking requests, in the order received, to maintain the prescribed rate.

An additional measure, that compensates for what seems to be an undocumented rate restriction, once a request passes the rate throttle, it enters the active request queue which limits the number of active requests to a configurable number, currently 10.

Ultimately this means that short bursts of request that are characteristic of an end user discovery or research tool will be serviced as quickly as they are received, improving user experience and long running processes that may involved thousands of requests can be trusted to run reliably to completion as quickly as allowed.

In testing, I have pumped 5000 requests into Soapi as fast as my machine with 50 threads will go and received all responses successfully.

Cache

Implementing the cache proved to be a challenge. My initial attempts were quite naive, simply caching successful results for a specific period of time.

Casual testing showed this to improve performance and reduce API bandwidth usage as expected.

It was when I implemented lazy loading, e.g stub hydration, that the shortcomings of this approach were exposed.

It became clear that many identical request could be pushed into the cache/throttle in immediate succession resulting in multiple identical requests being made simultaneously.

The obvious problem is that each request checks the cache before proceeding and since the initial request has not completed and cached the results for that url, multiple identical request are issued. This is a no-no for several reasons.

Ultimately, the solution is to implement smart cache items that know when they are pending, valid and expired. Thus when the initial request is recieved, an item is placed in the cache with a state of 'pending' until the request completes whereupon the cache item is populated with data and flagged as valid.

Subsequent requests that are received for that url while the request is pending register as 'waiting'. When the original request completes and signals, the waiting requests simply grab the cached data from memory and carry on.

Implementing this was an interesting challenge in thread management, especially when the Silverlight and Windows Phone environments are factored in, but ultimately, the implementation appears successful.

Fault tolerance

When the complimentary services provided by the throttle and cache are combined in a real world scenario, request failures become a big issue.

If a request is pending for a piece of data that 10 other requests are waiting for and it fails, the failure cascades.

Now, if the failure is a hard error, such as might result from bad parameters or repeatable unrecoverable error, all of the requests are going to fail in any case.

But consider transient network problems or the spurious 500 errors that the server throws occasionally. These types of failures are recoverable.

With this in mind, the requests made by Soapi, upon failure simply rest a moment and retry up to a configurable amount of times. Currently 3 is the default retry count.

This capability spans the throttle and the cache providing an impressive degree of reliability and fault tolerance.

Answer 9

MakerOfThings7:

Why is SOAPICS GPL? That really limits it's usage. Can you change it to LGPL?

http://soapics.codeplex.com/license

My application is proprietary, and the GPL license requires that I distribute source of my app if I simply link to this library. That's a no-go for me.

code poet:

There are many reasons that this, and other libraries, are GPL as opposed to LGPL.

You can find a really succinct summary at here.

The altruistic line is that the more free (as in speech as well as beer) software we publish under FLOSS licenses now, the more there will be in the future.

The pragmatic line, which is the one I tend to defer to, is that I am willing to pay forward the hundreds of hours and countless headaches (and those to come) that went into developing Soapi.CS. Anyone who is writing FLOSS, thus paying back and forward, is welcome to use the code without cost or restriction.

Proprietary, or closed source, software, is not 'free as in speech' and typically not 'free as in beer', and implies that the free (as in speech, definitely not as in beer) work that has been put into the library is being taken out of with nothing put back in.

But back to the pragmatic part:

It is my feeling that if someone wishes to use my work in proprietary or commercial software that, depending on the particular scenario, some type and degree of recompense is in order and that is where an LGPL exception comes into play.

Whether that be a simple thank you, a prominent attribution, a one time fee or a maintenance agreement etc all depends on the particular situation.

I encourage you (and anyone else) to email me directly at sky.sanders at gmail.com with salient details regarding your project and I am confident that satisfactory arrangements can be made.

_{Note: this is an issue of substance and should be discussed to the satisfaction of all and the form factor here, e.g. main post and cramped comments, does not lend itself well to that, so I will just carry on the conversation with you here in the main body, if you don't mind.}

MakerOfThings7:

In my situation, this code will probably benefit [email protected] more than it will benefit myself. Basically, I'd like to permit users to log into my site and grant them additional permissions if they have a high SO rating. This will in turn get you more users through an "Affiliate" and sell more advertising. I'll have the benefit of weeding out spammers and non-technical people.

If you believe it's worthwhile to get payment for the library in this use case, perhaps you should sell it to [email protected]

code poet:

What I infer from your last sentence is that you feel that I am asking too much from you in providing free code for free usage but requiring an explicit agreement for taking it out of FLOSS.

Do please correct me if I am reading you wrong, and in any case let me iterate some things and re-iterate others. ;-)

1. stack overflow does not need a client library for their own codebase
2. code poet != stack overflow (and apparently never will be)
3. what is worthwile for for Soapi/code poet != what is worthwhile for stack overflow.
4. i (hopefully) made it clear that for non FLOSS, some sort of agreement would need to be arrived at and provided some possibilities, including some that involve no cost what-so-ever.
5. yes, i feel that my work is of a caliber that deserves compensation in situations that indicate compensation is due. this might just be my ego on overtime, who knows?

With that said, again, I encourage you to email me with salient details regarding your proposed usage of Soapi in a proprietary setting and it is highly likely that satisfactory arrangements can be made but a blanket license shift to LGPL is not in the cards for Soapi.CS.

MakerOfThings7:

I totally appreciate your work and hope those ideas help you cross promote. I think SO should pay for your library if it drives traffic to their site, since they are getting increased advertising traffic from that. Maybe you can get compensation from both sides (SO and the consumer).

When I get closer to actually needing the library, and can find a user based on email hash (has that been implemented?) then I will send you more info at your email addy.

code poet

Regarding the 'users by hash' question, I know the issue has been raised in several posts, here, and in great detail in a meta post that I cannot find, but in short, the only way to do this is to maintain a complete list of users which you index yourself. This is a task that, if handled properly, is not unreasonable from both the perspective of maintenance and that of API usage. The utility of such a dataset goes far beyond simply finding users by email. I will post a CW dev-tip regarding this.

MakerOfThings7:

Not sure what a CW dev tip is, but I'll look for it when it's ready.

Answer 10

So many documentations are missing... Please correct these.

Answer 11

I'm getting an error when using this code:

Soapi.ApiContext context = new Soapi.ApiContext("...").Initialize(false);

// Now we're going to make an attempt to retrieve some questions
// from the StackExchange API.
Soapi.Domain.Question thisPost = context.Official
    .StackApps
    .Questions.ById(386)
    .GetEnumerator().Current;

// Now print the question title
Console.WriteLine(thisPost.Title);

Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object at Thing.ThingClass.Main (System.String[] args) [0x0001b] in Main.cs:18

I'm using Mono 2.4.