MIRRORING-PORTS(7)

NAME
mirroring-ports - how to build a mirror for ports distfiles

DESCRIPTION
The OpenBSD Ports Collection offers some powerful tools to mirror software sources. The ports infrastructure provides a mirror-maker target that can be used to build a Makefile to facilitate mirroring distfiles. This target builds ${DISTDIR}/Makefile, which can be used on almost any Unix-like machine to mirror OpenBSD distfiles. Optionally, port dependencies may also be recorded in the Makefile, which greatly increases the execution time of mirror-maker. The variable RECURSIVE_FETCH_LIST can be set to ‘Yes’ to request that this is done.

A sample Makefile entry is formatted like this:

    all:: audio/tracker/tracker-5.3
    .PHONY: audio/tracker/tracker-5.3
    audio/tracker/tracker-5.3:  tracker-5.3.tgz
    tracker-5.3.tgz: $F
            @MAINTAINER="espie@openbsd.org" \
             SITES="ftp://ftp.uni-trier.de/pub/unix/audio/tracker/ " \
             CIPHER="sha1" CKSUM="b0973d6a9c363caebd3a71547412f42b0681f323" \
             exec ${FETCH} "$@"

This Makefile is usually invoked by the user from the directory where they wish to mirror distfiles, with variables FETCH and F set on the command line, e.g.,

    cd mirror && make -k -j 5 -f path_to_makefile FETCH=fetch_script

Targets are set up so that each port is referenced by its full name, and retrieves all its distfiles. The all target can be used to retrieve all distfiles. Targets ftp and cdrom can be used to retrieve all distfiles necessary to build ftp and CD-ROM packages, respectively. Default is ftp plus cdrom. Actual fetching is usually invoked with a parallel-make option, so that several retrievals through ftp can proceed simultaneously.

The F variable can be set to a dummy name, or a recent filename, to force re-fetching of anything which is older than the filename. Its intended use is to force re-fetching existing files, or to checksum all files.

The ${FETCH} script should be supplied by the user, and will download and verify the archive file. It must obey the following variables:

MAINTAINER
    Port maintainer, used to report errors,

ERROR
    Some ports problems can be detected while building the Makefile, in which case this variable will be set to a proper error message.

DIST_SUBDIR
    See ports(7) for more details. The ${FETCH} script is responsible for creating this subdirectory and cd'ing to it before performing the actual fetch.

SITES
    A list of sites to try for fetching the distribution file.

CIPHER
    The checksumming utility to use for verifying the distribution file. It will normally be set to sha1(1) unless you tinker with PREFERRED_CIPHER while building the mirroring Makefile.

CKSUM
    The corresponding checksum. If neither CIPHER nor CKSUM nor ERROR are set, the distribution file needs not be checked.

A standard fetch strategy is to try all sites in order: whenever the distribution file is found, download it; verify the checksum; erase the file and try the next site if it doesn't match.

Mirroring sites should update their master Makefile fairly often. Activities a proper mirror should offer (in order of decreasing importance):

Mirror new files
    Use a proper fetch script to download missing files,

Run ${PORTSDIR}/infrastructure/fetch/link-checksums
    This script creates permanent hardlinks that preserve distfiles against checksum changes.

Verify all checksums
    All checksums should be verified from time to time, and maintainers notified of persistent discrepancies,

Check mastersites liveliness
    Use a tool such as ‘mirror’ to check that the master sites haven't fallen off the Earth. Even though the first site in the site list is the most important site, good mirrors will scan all sites and report all problems,

Remove old files
    To gain room this, the mirror should maintain a list of ‘active’ files (easy enough, just provide a fetch script that just lists the file names), and remove files that are no longer active. Since OpenBSD releases happen every six months, this delay should be longer than that.
Sample scripts are provided in the ${PORTSDIR}/infrastructure/fetch directory.

FILES
${DISTDIR}/Makefile
    Main mirroring Makefile
${PORTSDIR}/infrastructure/fetch/fetch-all
    Sample script usable to retrieve distfiles.
${PORTSDIR}/infrastructure/fetch/check-all
    Sample script to check all distfiles checksums.
${PORTSDIR}/infrastructure/fetch/link-checksums
    Populating checksums subdirectories with links, to guard against shifting checksums.

SEE ALSO
ports(7)

HISTORY
This infrastructure was introduced for OpenBSD 2.7 by Marc Espie, with feedback from Bob Beck, Todd Fries, Camiel Dobbelaar, and a few other people.

CAVEATS
Changing checksums is a recurring problem that is outside the direct control of the OpenBSD Project. Some software distributors change distribution files without warning, without changing the file name proper. Once the problem has been identified, the port maintainer should usually contact the software author to fix the problem, or, if the software author is unresponsive, the maintainer should use DIST_SUBDIR to provide some state to guard against shifting checksums.

However, a more robust approach is also needed, so that ports users can depend on distfiles mirrors to carry what they need irrespective of those synchronization issues. The link-checksums script creates another access to the distfiles, indexed through the actual checksums that the files should match. Provided mirroring is run sufficiently often, together with link-checksums, two versions of the same distfile with respective checksums cksum1 and cksum2 will be available under the names ${DISTFILES}/sha1/cksum1/distfile and ${DISTFILES}/sha1/cksum2/distfile.

Starting revision 1.281, if REFETCH is set to true, bsd.port.mk will try to retrieve files under that naming scheme as a last resort.