8

Update:

All of our grumblings and the unbearable beauty of our workarounds resulted in this issue being fixed last week.

You can now send 100 id in one request and receive 100 responses.

See Url length limits have been increased

This code could be implemented such to output fixed batches of 100 items instead of being constrained on string length, for when you are gathering id's from a possibly very large source and it would simplify your life greatly.

I will create another implementation in C# and JS.

Problem

When batching up id requests, e.g. /questions/{id} the responsibility is on you to ensure that the length of your path is less than ??? otherwise a 400 Bad Request is thrown.

A path is the part of the url that is before any ?. e.g.

http://api.stackapps.com/0.9/method/0;1;2;/foo

Use Case

It is always best, for the performance of your app as well as reducing requests to the api, to batch up multiple requests, when it makes sense to do so.

For instance, if you have a list of questions that have previously been pulled and you simply want to update them if they have had activity.

You could simply repull each one individually. This is the worst solution. It slows down your app, wastes bandwidth both net and cpu and exhausts your api key.

So you want to batch them, which means constructing a 'vectorized' or semi-colon delimited string containing the ids of the questions you wish to query.

Something similar to...

http://api.stackapps.com/0.9/questions/95;96;97;98;99;100;101;102;103;104;105;106?sort=activity&min=1277476364&max=1593095563&order=asc

Which gets these questions if they have had activity, e.g. created, answered, edited, between the dates specified in min/max.

The task is to manage the length of the path with the least amount of complexity.

One Solution

The code presented is in C# but is quite general and can be applied in most any programming language. I will provide a JavaScript implementation at some later date, others may wish to contribute by providing implementations in their language of choice.

One solution is to create a class that allows you to stuff as many values into it as you like and at your request, batch them up into groups that satisfy your current length requirements which includes then length of your current path without the id component.

Test First

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using NUnit.Framework;

namespace StackApps.Tips
{
    [TestFixture]
    public class VectorizedIdListFixture
    {
        [Test]
        public void Test()
        {
            // according to the only semi-answer as to the max length of a
            // path you need to take into account the length of the path, 
            // e.g.
            // http://api.foooverflow.com/0.X/foo/{dont count id}/bar
            // to ensure your path is less that 260.

            // this test shows that ~290 is the max path including protocol
            // but it seems to waver quite a bit so I am guessing that 
            // urlencoding of the semicolons is being counted on the other end.

            // Creating a usable constant from a variable amount of semis is 
            // a losing fight, so I would suggest 240 as a safe max
            // path length.

            const int maxPathLength = 240; 

            const string yourAPiKey = "";

            const string myApiRequestPath 
                = "http://api.stackapps.com/0.9/questions/{0}";

            int urlLength = myApiRequestPath.Length - "{0}".Length;

            var list = new VectorizedIdList<int>(maxPathLength - urlLength);

            const int questionCount = 200;

            for (int i = 0; i < questionCount; i++)
            {
                list.Add(i);
            }

            // get the batches as List<int> for testing
            var batches = list.GetBatches();

            // all our id's are accounted for
            Assert
                .AreEqual(questionCount, batches.Sum(b => b.Count()));


            // get the batches as vectorized for direct consumption
            var vectorizedBatches = list.GetBatchesVectorized();

            // none of our batches exceed the max length
            Assert
                .LessOrEqual(vectorizedBatches.Max(b => b.Length), maxPathLength);


            // build a list of URL
            List<string> batchedUrls = vectorizedBatches
                .Select(b => string.Format(myApiRequestPath, b) + "?key=" + yourAPiKey).ToList();

            Console.WriteLine(maxPathLength);

            // fetch all of the urls
            batchedUrls.ForEach(u =>
                {
                    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(u);
                    request.UserAgent = "soapi-tips testing path length";
                    request.AutomaticDecompression = DecompressionMethods.GZip;

                    Console.WriteLine(u);
                    var json =
                        new StreamReader(request
                            .GetResponse()
                            .GetResponseStream())
                            .ReadToEnd();
                    // no need to write it. a 400 would have failed the test
                    // Console.WriteLine(json);
                });
        }
    }
}

Results:

240

http://api.stackapps.com/0.9/questions/0;1;2;3;4;5;6;7;8;9;10;11;12;13;14;15;16;17;18;19;20;21;22;23;24;25;26;27;28;29;30;31;32;33;34;35;36;37;38;39;40;41;42;43;44;45;46;47;48;49;50;51;52;53;54;55;56;57;58;59;60;61;62;63;64;65;66;67;68;69?key=
http://api.stackapps.com/0.9/questions/70;71;72;73;74;75;76;77;78;79;80;81;82;83;84;85;86;87;88;89;90;91;92;93;94;95;96;97;98;99;100;101;102;103;104;105;106;107;108;109;110;111;112;113;114;115;116;117;118;119;120;121;122;123;124;125;126?key=
http://api.stackapps.com/0.9/questions/127;128;129;130;131;132;133;134;135;136;137;138;139;140;141;142;143;144;145;146;147;148;149;150;151;152;153;154;155;156;157;158;159;160;161;162;163;164;165;166;167;168;169;170;171;172;173;174;175;176?key=
http://api.stackapps.com/0.9/questions/177;178;179;180;181;182;183;184;185;186;187;188;189;190;191;192;193;194;195;196;197;198;199?key=

The Code:

This code is taken from Soapi.CS and is free to use with no restrictions.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace StackApps.Tips
{
    /// <summary>
    ///   A class for batching up API requests without hitting
    ///   the max path limit and generating (400) Bad Request errors
    /// </summary>
    /// <typeparam name = "T"></typeparam>
    public class VectorizedIdList<T>
    {
        public VectorizedIdList(string id, int maxBatchLength)
            : this(maxBatchLength)
        {
            Id = id;
        }

        public VectorizedIdList(int maxBatchLength)
        {
            MaxBatchLength = maxBatchLength;
            List = new List<T>();
        }

        public List<T> List { get; private set; }

        /// <summary>
        ///   Maximum string length of batches
        /// </summary>
        public int MaxBatchLength { get; set; }

        /// <summary>
        ///   To make managing instances of IdList easier
        /// </summary>
        public string Id { get; private set; }

        public void AddRange(IEnumerable<T> values)
        {
            foreach (T value in values)
            {
                Add(value);
            }
        }    

        public void Add(T item)
        {
            if (!List.Contains(item))
            {
                List.Add(item);
            }
        }

        public void Remove(T item)
        {
            if (List.Contains(item))
            {
                List.Remove(item);
            }
        }


        public List<string> GetBatchesVectorized()
        {
            return GetBatchesVectorized(false);
        }

        public List<string> GetBatchesVectorized(bool clear)
        {
            var batches = GetBatches(clear);

            var result = batches.Select(batch =>
                                        string.Join(";", batch.Select(i => i.ToString()).ToArray())
                                            .TrimEnd(';'));
            return result.ToList();
        }

        public List<List<T>> GetBatches()
        {
            return GetBatches(false);
        }

        public List<List<T>> GetBatches(bool clear)
        {
            List<List<T>> result = new List<List<T>>();

            List<T> batch = new List<T>();
            // to keep track of the net length of the batch
            StringBuilder batchCounter = new StringBuilder();

            foreach (T item in List)
            {
                string token = String.Format("{0};", item);

                if (batchCounter.Length + token.Length >= MaxBatchLength)
                {
                    result.Add(batch);
                    batch = new List<T> {item};
                    batchCounter = new StringBuilder(token);
                }
                else
                {
                    batch.Add(item);
                    batchCounter.Append(token);
                }
            }

            // pick up the get the crumbs
            if (batch.Count > 0)
            {
                result.Add(batch);
            }

            if (clear)
            {
                List.Clear();
            }

            return result;
        }
    }
}

More implementations in the language of your choice are welcome

Improve this question

1 Answer 1

Reset to default
1

JavaScript implementation

This code is taken from Soapi.JS and is free to use without restriction of any kind.

Documented with JsDoc and VS Xml Docs, take your pick of either or remove both. Not my business. ;-)

The Code:

/**
* Creates a new VectorizedIdList.
* @class A class for batching up API requests without hitting the max path limit and generating (400) Bad Request errors
* @param {Number} maxBatchLength Maximum string length of batches
*/
var VectorizedIdList = function(maxBatchLength) {
    /// <summary>
    /// A class for batching up API requests without hitting
    /// the max path limit and generating (400) Bad Request errors
    /// <summary>
    /// <param name="maxBatchLength" type="Number">Maximum string length of batches</param>
    /// <field name="maxBatchLength" type="Number">Maximum string length of batches</field>
    /// <returns type="Soapi.Utilities.VectorizedIdList"/>

    /**
    * Maximum string length of batches
    * @field
    * @type Number 
    */
    this.maxBatchLength = maxBatchLength;


    var list = [];
    var that = this;


    this.indexOf = function(value) {
        for (var i = 0; i < list.length; i++) {
            if (list[i] == value) {
                return i;
            };
        };
        return -1;
    };


    /**
    * Adds an array of values to the batch, each if it does not already exist
    * @function
    * @param {Array[Number|String]} values An array of values to add to the batch
    */
    this.addRange = function(values) {
        /// <summary>Adds an array of values to the batch, each if it does not already exist</summary>
        /// <param name="values" type="Array[Number|String]">An array of values to add to the batch</param>
        list = list.concat(values);
    };

    /**
    * Adds a value to the batch, if it does not already exist
    * @function
    * @param {Number|String} value A value to add to the batch
    */
    this.add = function(value) {
        /// <summary>Adds a value to the batch, if it does not already exist</summary>
        /// <param name="value" type="Number|String"></param>
        if (this.indexOf(value) == -1) {
            list.push(value);
        }
    };

    /**
    * Removes a value from the batch, if it exists
    * @function
    * @param {Number|String} remove A value to remove from the batch
    */
    this.remove = function(value) {
        /// <summary>Removes a value from the batch, if it exists</summary>
        /// <param name="value" type="Number|String">A value to remove from the batch</param>

        var index = this.indexOf(value);
        if (index > -1) {
            list.splice(index, 1);
        };
    };

    /**
    * Gets an array of vectorized list strings, each length being equal or less than maxBatchLength
    * @function
    * @param {Boolean} clear whether to clear the inner list after batch generation
    * @returns {Array[String]} An array of vectorized id lists
    */
    this.getBatchesVectorized = function(clear) {
        /// <summary>Gets an array of vectorized list strings, each length being equal or less than maxBatchLength</summary>
        /// <param name="clear" type="Boolean">whether to clear the inner list after batch generation</param>
        /// <returns type="Array"/>
        var batches = this.getBatches(clear);
        for (var i = 0; i < batches.length; i++) {
            batches[i] = batches[i].join(";");
        };
        return batches;
    };

    /**
    * Gets an array of arrays containing batched values
    * @function
    * @param {Boolean} clear whether to clear the inner list after batch generation
    * @returns {Array[Array[Number]]} An array of array, each representing a viable batch
    */
    this.getBatches = function(clear) {
        /// <summary>Gets an array of arrays containing batched values</summary>
        /// <param name="clear" type="Boolean">whether to clear the inner list after batch generation</param>
        /// <returns type="Array"/>

        var batches = [];
        var batch = [];
        var batchString = "";
        for (var i = 0; i < list.length; i++) {
            var token = list[i].toString() + ";";
            if ((batchString + token).length >= that.maxBatchLength) {
                batches.push(batch);
                batch = [];
                batchString = "";
                batch.push(list[i]);

            } else {
                batchString += token;
                batch.push(list[i]);
            };

        };

        if (batch.length > 0) {
            batches.push(batch);
        }
        if (clear) {
            list = [];
        };
        return batches;
    };
};

The qunit test:

module("vectorizedIdList");
test("", function()
{
    var list = new VectorizedIdList(200);
    for (var i = 0; i < 200; i++)
    {
        list.add(i);
    };
    var batches = list.getBatches();
    var total = 0;
    for (var i = 0; i < batches.length; i++)
    {
        total += batches[i].length;
    }
    ok(total == 200, "item count");

    batches = list.getBatchesVectorized();
    var passes = true;
    for (var i = 0; i < batches.length; i++)
    {
        if (batches[i].toString().length > 200)
        {
            passes = false;
            break;
        };
    }

    ok(passes, "vlist length");
});

More implementations in other languages welcome.

Improve this answer

Not the answer you're looking for? Browse other questions tagged .