This is a data dump importer written in C# that works with current versions of the data dump. It's written using Winforms, NET 4.5 and only supports SQL/Server so if you want to import a full Stack Overflow dump you'll need a full version of SQL/Server rather than the Express edition due to the 10GB limit.
I imported the full Electrical Engineering dump into an Express instance in under ten minutes. I'm still waiting for the full SO dump to download to try it but don't expect any problems although the import will likely take hours to complete because I've kept the code short rather than efficient.
How to use it and what it does:
If you have an SQL Server Express LocalDB instance installed you can use the default connection string otherwise press the ellipsis (...) button to build a connection string. Under "select or enter a database name" you can either select an existing database or type in a new name and it will be created automatically.
Change the import directory to the folder containing the XML data dump files which will typically be Badges, Comments, PostLinks, Post, Tags, Users and Votes.XML. They should all be in the same directory - it doesn't matter if any you don't want are missing but ensure that no additional XML files are present because it will attempt to import them to a table that doesn't exist.
The table prefix is added to the start of each table so you might like to add something like "EE_" if you'll be importing dumps from multiple sites into the same database, otherwise it may be left blank.
Once you press the begin import button the following operations will take place:
The specified database will be created if it doesn't exist.
Schema information will be fetched from Stack Exchange Data Explorer and corresponding tables will be created, even if they are ones not present in the data dump.
Data for each XML file in the import directory will be imported and a primary key added on the
Idcolumn.If the file
ImportReferenceTables.SQLis present in the same directory as the executable it will run to import some reference tables not present in the data dump, such asPostTypes. I manually extracted this data using the export to CSV featured of the data explorer so they may need updating from time to time.
To build the source you will need the following:
- Visual Studio 2015.
- The DataConnectionDialog package available via NuGet.
- The HtmlAgilityPack package available via NuGet.
If you don't want to build from source the bin\debug directory contains the executables and DLLs that should be enough to run it as long as you have .NET 4.5 installed. Feel free to do whatever you like with the source, I suspect it wouldn't be too hard to get it working with MySQL for anyone interested in doing that. The files may be downloaded as a zip from:
