About

With Stack Exchange, Inc.'s recent data dump restrictions, the official SE data dump on archive.org is dead, and single-click downloads of the entire data dump are too.

This program is meant to auto-download and auto-transform the data dump into one of (hopefully) several more convenient formats, and to contribute to data archival efforts.

Download

Download from GitHub. Instructions for setting up and running are listed in the README

Platform

Downloader: Theoretically anything able to use Python 3.10 and newer; Verified with Linux Mint 21.3 with Python 3.10.12 and Windows 10 with some version of Python newer than 3.10

Transformer: Theoretically anything able to compile C++20; It's auto-built on Linux (Ubuntu), Windows, and macOS for good measure, but not tested outside my distro at this time. Docker is also supported, expanding support to anything able to support Docker.

Contact

Use GitHub to report any issues. General questions can be posted on GitHub Discussions.

License

The code is under the MIT license; see the LICENSE file.

The data downloaded and produced is under various versions of CC-By-SA, as per Stack Exchange's licensing rules, possibly in addition to whatever extra rules they try to impose on the data dump.

Code

The code is available on GitHub, and is split in two components:

1. The downloader: Python-based. This component automatically downloads the data dumps from the entire network. It's currently fully functional, and will actually download the data dumps as the feature becomes available to more users.
2. The transformer: C++-based. Takes the data dump as input, and outputs it in an alternate format. Only JSON and SQLite have been implemented so far. This component is also (theoretically) compatible with most previously uploaded data dumps - however, only the recent data dump is actively supported, as they're the ones that see the most use.

About

With Stack Exchange, Inc.'s recent data dump restrictions, the official SE data dump on archive.org is dead, and single-click downloads of the entire data dump are too.

This program is meant to auto-download and auto-transform the data dump into one of (hopefully) several more convenient formats, and to contribute to data archival efforts.

Download

Download from GitHub. Instructions for setting up and running are listed in the README

Platform

Downloader: Theoretically anything able to use Python 3.10 and newer; Verified with Linux Mint 21.3 with Python 3.10.12 and Windows 10 with some version of Python newer than 3.10

Transformer: Theoretically anything able to compile C++20; It's auto-built on Linux (Ubuntu), Windows, and macOS for good measure, but not tested outside my distro at this time. Docker is also supported, expanding support to anything able to support Docker.

Contact

Use GitHub to report any issues. General questions can be posted on GitHub Discussions.

License

The code is under the MIT license; see the LICENSE file.

The data downloaded and produced is under various versions of CC-By-SA, as per Stack Exchange's licensing rules, possibly in addition to whatever extra rules they try to impose on the data dump.

Code

The code is available on GitHub, and is split in two components:

1. The downloader: Python-based. This component automatically downloads the data dumps from the entire network. It's currently fully functional, and will actually download the data dumps as the feature becomes available to more users.
2. The transformer: C++-based. Takes the data dump as input, and outputs it in an alternate format. Only JSON and SQLite have been implemented so far. This component is also (theoretically) compatible with most previously uploaded data dumps - however, only the recent data dump is actively supported, as they're the ones that see the most use.

About

With Stack Exchange, Inc.'s recent data dump restrictions, the official SE data dump on archive.org is dead, and single-click downloads of the entire data dump are too.

This program is meant to auto-download and auto-transform the data dump into one of (hopefully) several more convenient formats, and to contribute to data archival efforts.

Download

Download from GitHub. Instructions for setting up and running are listed in the README

Platform

Downloader: Theoretically anything able to use Python 3.10 and newer; only verified on Linux Mint 21.3 with Python 3.10.12 at the time of writing

Transformer: Theoretically anything able to compile C++20; It's auto-built on Linux (Ubuntu), Windows, and macOS for good measure, but not tested outside my distro at this time. Docker is also supported, expanding support to anything able to support Docker.

Contact

Use GitHub to report any issues. General questions can be posted on GitHub Discussions.

License

The code is under the MIT license; see the LICENSE file.

The data downloaded and produced is under various versions of CC-By-SA, as per Stack Exchange's licensing rules, possibly in addition to whatever extra rules they try to impose on the data dump.

Code

The code is available on GitHub, and is split in two components:

1. The downloader: Python-based. This component automatically downloads the data dumps from the entire network. It's currently fully functional, and will actually download the data dumps as the feature becomes available to more users.
2. The transformer: C++-based. Takes the data dump as input, and outputs it in an alternate format. Only JSON and SQLite have been implemented so far. This component is also (theoretically) compatible with most previously uploaded data dumps - however, only the recent data dump is actively supported, as they're the ones that see the most use.

About

With Stack Exchange, Inc.'s recent data dump restrictions, the official SE data dump on archive.org is dead, and single-click downloads of the entire data dump are too.

This program is meant to auto-download and auto-transform the data dump into one of (hopefully) several more convenient formats, and to contribute to data archival efforts.

Download

Download from GitHub. Instructions for setting up and running are listed in the README

Platform

Downloader: Theoretically anything able to use Python 3.10 and newer; Verified with Linux Mint 21.3 with Python 3.10.12 and Windows 10 with some version of Python newer than 3.10

Transformer: Theoretically anything able to compile C++20; It's auto-built on Linux (Ubuntu), Windows, and macOS for good measure, but not tested outside my distro at this time. Docker is also supported, expanding support to anything able to support Docker.

Contact

Use GitHub to report any issues. General questions can be posted on GitHub Discussions.

License

The code is under the MIT license; see the LICENSE file.

The data downloaded and produced is under various versions of CC-By-SA, as per Stack Exchange's licensing rules, possibly in addition to whatever extra rules they try to impose on the data dump.

Code

The code is available on GitHub, and is split in two components:

1. The downloader: Python-based. This component automatically downloads the data dumps from the entire network. It's currently fully functional, and will actually download the data dumps as the feature becomes available to more users.
2. The transformer: C++-based. Takes the data dump as input, and outputs it in an alternate format. Only JSON and SQLite have been implemented so far. This component is also (theoretically) compatible with most previously uploaded data dumps - however, only the recent data dump is actively supported, as they're the ones that see the most use.

NOTE: This app will remain a placeholder with general scaffolding for downloading and processing until the new data dump format is out. It's not possible to work on the exact downloader and transformer until the new data dump format and process are out.

At the time of writing, it's still unclear whether or not the data dump is exactly the same as it is today, just in a new place, or if it's both in a different place and formatted differently. This particularly applies to Stack Overflow. On archive.org, Stack Overflow's data dump is split across several .7z files (with MSO still being a monolith), but the current system seems to suggest there's only one download for MSO + SO. Whether that's breaking compared to today's data dump is unclear and something we'll have to find out as we go. Parts of the system do, however, assume that .7z and XML files are still used to minimise the amount of time between the release of the data dump, and the restoration of the community's ability to effectively archive our own data.

About

With Stack Exchange, Inc.'s recent data dump restrictions, the official SE data dump on archive.org is dead, and single-click downloads of the entire data dump are too.

This program is meant to auto-download and auto-transform the data dump into one of (hopefully) several more convenient formats, and to contribute to data archival efforts.

Download

Download from GitHub. Instructions for setting up and running are listed in the README

Platform

Downloader: Theoretically anything able to use Python 3.10 and newer; only verified on Linux Mint 21.3 with Python 3.10.12 at the time of writing

Transformer: Theoretically anything able to compile C++20; Docker will be supported in the future to simplify the dependencies, which means anything able to run Docker will also be able to run the transformer. It's auto-built on Linux (Ubuntu), Windows, and macOS for good measure, but not tested outside my distro at this time

Contact

Use GitHub to report any issues. General questions can be posted on GitHub Discussions.

License

The code is under the MIT license; see the LICENSE file.

The data downloaded and produced is under various versions of CC-By-SA, as per Stack Exchange's licensing rules, possibly in addition to whatever extra rules they try to impose on the data dump.

Code

The code is available on GitHub, and is split in two components:

1. The downloader: Python-based. When the data dump is out, this will deal with account creation and data dump download on a per-site basis.
2. The transformer: C++-based. Takes the data dump as input, and outputs it in an alternate format. Only JSON and SQLite have been implemented so far. This component is also (theoretically) compatible with most previously uploaded data dumps - however, only the recent data dump is actively supported, as they're the ones that see the most use.

About

With Stack Exchange, Inc.'s recent data dump restrictions, the official SE data dump on archive.org is dead, and single-click downloads of the entire data dump are too.

This program is meant to auto-download and auto-transform the data dump into one of (hopefully) several more convenient formats, and to contribute to data archival efforts.

Download

Download from GitHub. Instructions for setting up and running are listed in the README

Platform

Downloader: Theoretically anything able to use Python 3.10 and newer; only verified on Linux Mint 21.3 with Python 3.10.12 at the time of writing

Transformer: Theoretically anything able to compile C++20; It's auto-built on Linux (Ubuntu), Windows, and macOS for good measure, but not tested outside my distro at this time. Docker is also supported, expanding support to anything able to support Docker.

Contact

Use GitHub to report any issues. General questions can be posted on GitHub Discussions.

License

The code is under the MIT license; see the LICENSE file.

The data downloaded and produced is under various versions of CC-By-SA, as per Stack Exchange's licensing rules, possibly in addition to whatever extra rules they try to impose on the data dump.

Code

The code is available on GitHub, and is split in two components:

1. The downloader: Python-based. This component automatically downloads the data dumps from the entire network. It's currently fully functional, and will actually download the data dumps as the feature becomes available to more users.
2. The transformer: C++-based. Takes the data dump as input, and outputs it in an alternate format. Only JSON and SQLite have been implemented so far. This component is also (theoretically) compatible with most previously uploaded data dumps - however, only the recent data dump is actively supported, as they're the ones that see the most use.