Introduction

As requested in ticket 269, post commit hooks are needed to improve the integration of Indefero with other tools (continuous integration, etc.). A web hook is basically generating an HTTP request after a given action, the action can be post commit, but one can imagine a hook after the creation of an account (to register to a mailing list in the background) or for other "events".

To keep the interface responsive, this means that the hooks must be run asynchronously, also an HTTP request may temporarily fail and one may need to retry several times.

So, we need:

flexible hook system to accommodate post commit but also other events hooks;
queue system to manage the requests.

Flexible Hook System

Some prerequisites:

the hook may run or not. Indefero will try its best to get it through but no guarantee;
the hook may run several time, it will be up to the receiver to deduplicate the calls (each notification may have a unique id to help);
the notifications may not come in order.

Hooks are always going to generate a JSON payload with MD5-HMAC authentication. The MD5-HMAC authentication will be done with a key provided by the project admin or for the forge level hook, by the forge admin.

Queue System to Store the Notifications

A simple table with the following fields:

id
project (can be null if forge level)
status (sent, pending, inprogress)
retries (number of retries, max 5)
lasttry_dtime (to manage the retry intervals)
payload (json)
completed hooks (one queue item can trigger many hooks, if one out of 3 fails, the next run must not perform the 2 already completed hooks)
creation_dtime
type (webhook, whatever)

The last try time combined with the retries count allow the cron job managing the notifications to wait the correct delay (5min 30min 1h 2h 8h).

The status is to mark the notifications in progress (should run every 5 min or so), for the hosted forges a daemon doing continuous ping is maybe the right approach.

Maintenance Queue

On a regular basis, some operations need to be performed asynchronously to keep a responsive interface, for example:

Evaluate the disk usage of a project (ticket 403);
Automatically backup the repository after a commit;
Automatically backup the uploaded files after an upload;
Send some emails;
Run the post commit hooks.

The webhook queue could be use to store what needs to be done as it takes into account the requirements like retries etc.

When are Queue Items Created

The simplest way is to create the new queue items for most of the elements directly in the given notify method. This way, one can branch depending of several conditions and it is easy to create the post commit hook.

Git post-update Hook

The hook should be something like:

echo "php /path/to/hook/script/git.php $GIT_DIR" | batch

From $GIT_DIR, the script can find, which project is updated and then run a simple IDF_Scm::syncTimeline($project); to get the timeline updated.

When the timeline is put in sync, the notify call is performed, when performed, it will add the items in the queue for the webhooks.

Subversion post-commit Hook

echo "php /path/to/hook/script/svn.php $1" | batch

Note that the use of batch is not an obligation, but it has some advantages:

returns immediately.
run the command only when the load is low.

Indefero