Work continues on a WordPress.org mirror

Recent days have seen lots of work on the WordPress.org mirror. From the API and the plugins to the downloading of the WordPress plugin library, we’re making progress.

AspirePress Updater Plugin

The AspirePress Updater plugin has been completed in its initial phase. We still need to add an admin section that allows for greater configuration, but all told the plugin is designed and implemented.

Configuration remains an issue. In order to use the plugin you have to edit wp-config.php which is a barrier for non-technical website owners. We’ll add admin section configuration soon.

You can check the plugin out on GitHub.

CDN And API

The CDN/API work has initially been started. For starters, we wrote something that simply proxies the WP.org site. We’ll then start filling in our own API calls, to allow for replacement of those APIs that we want to use.

The CDI/API component is built in PHP with a Postgres backend. It’s easy to install and set up with Docker. We intend to deploy it on cloud hosting in containers.

This effort has been stalled a bit by the need to have actual data and plugins to work with. You can view current progress on GitHub.

AssetGrabber

The AssetGrabber plugin has seen most effort over the last couple of days. We need a way to obtain all the plugins from WordPress that are available, so that we can mirror them. We also need a way to update the list, and provide updated plugins to users that use our endpoints.

We were successful in downloading all the plugin metadata today. There are over 100,000 plugins in the WordPress SVN repository; however, many of them are “closed” or unavailable. This will reduce the number of plugins that we have to download and store, which will in turn improve the scalability and reduce the cost of hosting.

We have an innovative approach to identifying the latest revision, using SVN, and then updating the plugins.

Work still needs to be completed on pulling WordPress themes and core. You can see progress on GitHub.

What’s next?

Over the next day we anticipate making an initial download of the plugins and themes, and establishing those components in our database. We’ll then start moving them into S3-compatible storage and configuring a CDN for delivery. We also need to finish building the basic API, and update the plugin for configuration in the UI. There’s likely a punchlist to be done after that before launch, including issuing API keys for users of the AspirePress mirror.

We welcome your feedback and, if you wish, your involvement! Please feel free to fork and commit code to the repositories on GitHub.