distributed version control: version and help | local and remote repository | working directory | track and commit | branch and merge | history | push and pulll | configuration files | external repositories | packaging | integrity check and garbage collection

version control: svn | cvs | rcs

archive and patch tools: diff | cpio | diff3 | ar | tar | patch | zip | diffstat | jar | rsync | colordiff

Metasyntactic Variables

Help

show version

list subcommands

get help for subcommand

list topic guides

get help for topic

Local and Remote Repository

When a set of files is edited, we can view the set of files as a sequence of versions or revisions which are ordered in time. A repository is a record of the versions of a set of files; it permits recovering the set files as they were at any of the recorded points in time.

A working directory (q.v the working directory of a process) can contain a copy of one of the file set versions; it can also be edited to create a new version of the file set. A commit is the act of recording the new version of the file set in the repository. This is done by comparing the working directory with the previous version or an empty file set in the case of an initial commit. The commit can also be regarded as the difference—that is, the output of diff -r—between the previous and the new version. This difference is also called the changeset.

When calculating the changeset, the version control system may ignore files in the working directory which are not tracked files.

A set of files and directories under version control is called a repository.

git

A file or directory under version control has one or more versions. One adds new versions to the repository by making a commit. The set of all files and directories in the repository can also be seen as having versions; these versions are called commits; they consist of at most one version of each file or directory in the repository.

hg

A file or directory under version control has one or more revisions. One adds new revisions to the repository by making a commit. The set of files and directories in the repository can also be seen as having revisions; these revisions are called changesets.

Working Directory

check out version

git:

These two commands appear to do the same thing:

$ git checkout -f COMMIT
$ git reset --hard COMMIT

list modified files

ignore file

git

man gitignore

A list of file patterns, one per line. The patterns specify files that git status and git add should ignore. Shell glob syntax (i.e. the asterisk: *) can be used.

A .gitignore can be placed in any directory in the repository. The rules in a given .giitignore file will only apply to the current directory and the directories beneath it.

Lines starting with a pound sign: # are ignored.

A pattern starting with an exclamation point: ! will negate a pattern. This can be used to include files that were excluded by a pattern higher in the file matching a broader set of files.

hg

Unlike .gitignore, an .hgignore file must be in the root of the working directory.

The format is one Perl regular expression per line. All files which match the regular expression will be ignored.

Comments start with the pound sign: #

It is also possible to use glob syntax:

# regexp to ignore twiddle files:
~$

# glob to ignore compiled python files:
syntax: glob
*.pyc

# additional patterns will use regexp format:
syntax: regexp

Track and Commit

git

Git keeps copies of all versions of files and directories that have been committed, as well as the commits themselves, in the directory .git/objects. All objects are identified by their 40 character SHA-1 checksum called the hash. There are three types of git objects in this directory: a blob, which is the contents of a file. A tree, which corresponds to file system directory; it contains the file system name of the objects, which can be blobs (regular files) or trees (directories) and their hashes. Finally, a commit contains the top level tree for the commit and the parents of the commit. There will be zero parents for the initial commit and more than one parent for a commit which was created by a merge. Git stores a separate, albeit compressed, copy of each version of a file, tree, or commit in the .git/objects directory.

The git cat-file -p HASH command, though not needed for day-to-day use, provides a way to inspect a git object. It shows the additional information stored in trees and commits which we have not mentioned here.

hg

Mercurial uses a storage format called a revlog to store the versions of a file. Most revlogs are kept in .hg/store/data. A revlog usually consists of two files: one with an .i suffix and another with a .d suffix. If the file is small and has little or no history, the revlog might consist of only a .i file. A revlog which tracks the history of a file is called a filelog. When the file is first committed, it is written to the filelog. Each time a commit is made which alters it, a delta describing the change is appended to the file. Thus, to fetch the current version of a file, all the deltas must be applied in order to the original version of the file. As a performance optimization, Mercurial will sometimes append the full version of the file to a filelog. Thus, when reconstructing the current version, one need only apply delta starting from the last time the full version was stored.

Mercurial stores a manifest for each revision of the repository. A reviion of the repository is called a changeset. The manifest is list of the pathnames relative to the root of all files in the changeset. Rather than store the manifests in separate files, all the manifests for the repository are stored in a revlog in .hg/store. Each time a new changeset is added to a repository by a push, pull, or commit command, it is assigned a local revision number which is the order in which it was appended to the local manifest revlog. If the changeset was pulled from a different repository, the local revision numbers might not match.

Information about changesets is also stored in the changelog, which is another type of revlog. The changelog has a pointer to manifest revision, pointers to parents of the changeset, and information about the committer.

git

Git has three types of objects: commits, trees, and blobs. Each is assigned a unique hash ID which is a 40 digit hex string. The identifier is called the hash, SHA1, object name, or object identifier with no difference in meaning. When the underlying object is a commit or tree it is also called a tree-ish.

Commit hashes are the hashes the user most commonly sees and needs to reference. Only as many of the digits that are necessary to uniquely identify an object in the object database need to be provided to a git command; usually the first 6 or 7 is sufficient.

HEAD is a special name which refers to the most recent commit of the current branch. It is stored in .git/HEAD. The previous commit is HEAD^ and the commit before that is HEAD^^. The is also numerical notation: HEAD~4 is 4 commits ahead of HEAD. If HEAD is the result of a merge, then the antecedents can be be referenced with HEAD^1 and HEAD^2.

hg

In Mercurial, every commit is assigned two identifiers: a local revision number and a universal changeset identifier. The local revision number is a small integer that is unique only to the local repository. The first local revision number issued is zero, and it increments with each local commit. The changeset identifier is a twelve digit hex number which is unique across all repositories.

The -r option is used to pass a mercurial commit identifier to a command. The argument can be a local revision number, a changeset identifier, or both separated by a colon.

move file in working directory and next commit

It is desirable for a version control system to track file name changes. Otherwise commands like blame and log when used on a single path will not show activity before the name change. If the version control system is aware of a name change, it can correctly handle the case when merging where the name was changed on one branch and edited on the other.

git

Although Git provides a git mv subcommand, it does not actually track name changes. Instead, it will assume that a name change occurred during a commit when one file disappeared, another appeared, and they have similar contents. Hence, even if the user uses git rm, a Unix command mv, and git add, Git will preserve the history for the file.

hg

Mercurial keeps track of the name a file had in each revision of a filelog. The hg rename subcommand must be used to preserve history.

move file

commit changes in staging area

In the case of Git, the staging area is the index.

In the case of Mercurial, the staging area is the files in the working directory which are tracked.

commit identifiers

mercurial:

null refers to an empty revision which is the parent of revision 0.

Branch and Merge

git

Git has a low level feature called a ref which it uses to implement branches and tags. A ref is a file in .git/refs which contains the hash of a commit. Branches are in .git/refs/heads and tags are in .git/refs/tags. Whenever a commit is made, the value in .git/refs/heads/BRANCH is updated where BRANCH is the current branch. The values in .git/refs/tags/TAG do not change.

The name of the branch which is currently checked out is stored in .git/HEAD. It is stored as the relative path refs/head/NAME.

Git also stores remote branches and tags in .git/refs/remotes/REPO. The git branch -r command can be used to list remote branches. Remote branches have names of the form REPO/BRANCH, and each remote branch will usually have a tracking branch, which is a local branch named BRANCH. This will be the case for any branches which were copied when a repository is created via git clone. A tracking branch can also be created when a remote repository is added using git remote -t BRANCH REPO URL. git fetch will only update remote branches. git pull will update remote branches and merge them with their tracking branches.

The default branch is called master. It is created by git init, and is the branch that is copied by git clone if no branch is explicitly specified.

Commits have zero or more parent commits. git commit creates a commit with one parent, except in the case of the initial commit. git merge creates a commit with two or more parent commits. If the commit has three or more parents, the merge is called an octopus merge.

staging numbers:

To perform a merge Git gets the tree contained in the common ancestor and puts its items into the staging area with staging number 1. It puts the current branch tree items in the staging area with staging number 2. It puts the tree items of the other branches in the staging area with staging number 3 or higher.

fast-forward commits aren't actually commits:

Suppose that bar is a branch of foo. If commits have subsequently been made to foo but not to bar, then running the following when bar is the current branch will perform a fast-forward:

git merge foo

In a fast-forward no merge commit is created. Instead the head of bar is simply moved to point to the same commit as the head of foo.

hg

A Mercurial branch is a name which is stored in a changeset. When a commit is made, the new changeset inherits the branch name of the previous changeset, unless a different name was specified before the commit with hg branch. To switch to a new branch one must make a commit.

Mercurial branches differ from Git branches in that:

• every commit belongs to a single branch
• a branch can have multiple heads

Mercurial tags are names for changesets. They are stored in the .hgtags file at the repository root. Creating a tag requires making a commit.

Mercurial does not support octopus merges. Thus changesets have at most two parents. A changeset created by hg merge sets the branch of the new changeset to be the branch of the first argument.

Changesets can have no branch specified. This is also called the default branch.

bookmarks:

Mercurial bookmarks work like Git branches, with the exception that Mercurial does not have the equivalent of Git tracking branches.

list "dedicated commit" branches

"dedicated commit" branches is the term used in this sheet to distinguish Mercurial-style branches from Git-style branches. Mercurial bookmarks are equivalent to Git-style branches.

In Git, branches are names which contain the hash of a commit. This commit is the head of the branch, and because each commit knows its parents, the entire history of the branch can be constructed. Git commits can belong to multiple branches, or they can belong to none at all. If a Git commit is not reachable from any branch or tag, it is at risk of being garbage collected.

In Mercurial, each changeset has a branch name associated with it. Thus, each changeset must belong to exactly one branch. Also, Mercurial branches can have multiple tips, whereas a Git branch always has a single head.

Mercurial does not provide a mechanism for renaming or deleting branches. The recommended way to get rid of unwanted branches is to rename the repository and then clone it to the original name with:

$ hg clone -r REV

History

Push and Pull

git

The basic command for getting changes from a remote repository origin is:

$ git fetch

Which branches are fetched is controlled by the fetch key in the remote section of .git/config. If the local repository was created by git clone, here is a likely value:

[remote "origin"]
        fetch = +refs/heads/*:refs/remotes/origin/*

In this case, git fetch origin connects to the remote repository and copies all of the remote branches to refs/remotes/origin. Then it adds all remote objects referred to by the remote branches to the local objects database. It also puts the remote HEAD into FETCH_HEAD. The + indicates that local branches should be updated even if the commits are not fast-forwards.

The basic command for sending changes back to the remote repository origin is:

$ git push

Which branches are pushed is controlled by the push key in the remote section of .git/config. Here is an example entry which pushes commits on the master branch, and fails if the commits are not fast-forwards:

[remote "origin"]
        push = refs/heads/master:refs/heads/master

A git pull is a git fetch followed by a git merge FETCH_HEAD, which git fetch sets to whatever was in HEAD on the remote repository.

hg

hg pull pulls changesets for all the remote branches that are also local branches unless branches are listed explicitly with the -b flag. hg pull -u is equivalent to hg pull followed by hg update. Pulling can create local branches with multiple heads, in which case an hg update will fail. An hg merge is used to merge the two heads, or an hg commit --close-branch is used to mark one of them as closed.

hg push pushes changsets for all local branches that are also remote branches unless branches are listed explicitly with the -b flag. A push which would create a branch with multiple heads will fail unless the -f flag is used. The --new-branch flag must be used to create a new branch.

Configuration Files

External Repositories

Packaging

Integrity Check and Garbage Collection

integrity check

garbage collection

Version Control

svn | cvs | rcs | sccs

sccs (1972)

• CSSC Documentation CSSC is the GNU implementation of SCCS
• The Source Code Control System Rochkind 1975

In his 1975 paper Rochkind describes SCCS as a "radical departure from conventional methods for controlling source code". SCCS was initially implemented in 1972 on the IBM 370. The implementation language was SNOBOL. Rochkind was an employee of Bell Laboratories and SCCS was soon ported to Unix where it became a cornerstone of the "Programmer's Workbench", a suite of software distributed with early Unix.

The radical departure of SCCS appears to be the decision to store every version of each file under source control. This is done in a space efficient manner by means of deltas: the original file is stored with a delta for each change. To get the most recent version of the file all of the deltas must be applied to the original file. Also stored with each delta is the name of the user who made the change, the date and time of the change, and a user supplied comment explaining the change.

SCCS introduces a file format so that the original file, the deltas, and the meta-information can all be stored in a single history file. If the original file was foo.c, a common early convention was for the history file to be named s.foo.c. In the original Unix implementation the SCCS commands were standalone Unix commands. Starting with the version of SCCS which Allman wrote for BSD Unix in 1980 the SCCS commands became arguments or subcommands to a sccs executable.

Here is an sample SCCS session. The file foo.txt is put under source control. It is then checked out, edited, and the change committed. Finally a non-editable copy of the most recent version is checked out.

$ echo "foo" > foo.txt
$ sccs admin -ifoo.txt s.foo.txt
$ rm foo.txt
$ sccs get -e s.foo.txt
$ vi foo.txt
$ sccs delta s.foo.txt
$ sccs sccsdiff -r1.1 -r1.2 s.foo.txt
$ sccs get -p s.foo.txt > foo.txt

The SCCS history file format consists of fields separated by the Ctrl-A (ASCII 1) characters. The fields are divided into headers, which contain the meta-information, and the body, which contains the original file and the deltas. The original file is given revision number 1, and the number is incremented with each change.

The body consists of the original file interspersed with nested insert blocks and delete blocks. The format for an insert block is

^AI REV
added line one
added line two
...
^AE REV

where REV is the revision number for which the lines were added. Similarly the format for a delete block is

^AD REV
deleted line one
deleted line two
...
^AE REV

When extracting a version of the file, the desired version is compared with each block. Insert blocks are ignored if they have a higher number than the desired version and delete blocks are ignored if they have a lower or equal number than the desired version.

rcs (1982)

In RCS, the history file is always identified with a ,v suffix; the history file for foo.txt is foo.txt,v. Because this convention is used consistently, RCS commands can take the original file as an argument instead of the history file like in SCCS.

One can keep the ,v files in a separate directory. RCS has built in support for using a subdirectory named RCS in the same directory as the source. When this convention is used, it is not necessary to specify both the ,v file and the source file when using rcs ci and rcs co. If the source code tree has subdirectories, each subdirectory should contain an RCS subdirectory.

RCS supports multiline commit messages and it adds the log command for getting all the commit messages for a file.

example session

Here is an example work session using RCS. It is equivalent to the SCCS work session in the previous section.

$ echo "foo" > foo.txt
$ rcs ci -i foo.txt
$ rcs co -l foo.txt
$ vi foo.txt
$ rcs ci foo.txt
$ rcs co foo.txt

make has a built-in rule for creating a file from its ,v file. The file will be checked out as read-only.

rcs file format

• man rcsfile The RCS history file format

An RCS history file has four sections:

• head
• deltas
• description
• deltatexts

The head contains the revision number of the current version. If any of the revision numbers have been assigned symbolic names (i.e tags), they are listed here. If there

There is a delta section for each revision. It contains the time the revision was added to the history file and the author.

The description is a string describing the file. Strings are delimited by ampersands @. An ampersand in the string is escaped by doubling it.

There is a deltatext section for each revision. It contains a log, which is the commit message, and the text), which is either the full text of the revision or an ed style edit describing how to generate the revision from another revision. Both the log and the text are ampersand delimited strings with ampersands escaped by doubling.

Here is an example of a text which adds two lines after line 6:

@a6 2
added line one
added line two
@

Here is an example o a text which deletes two lines after line 6:

@d6 2
@

revision numbers and branching

RCS revision numbers consist of 2n positive integers joined by decimals. n is itself positive. By default the revision number given to a file when it is first placed under version control is 1.1.

RCS revision numbers with 2 integers refer to revisions on the trunk. RCS revision numbers with 4 integers refer to branches off the trunk. The first two integers indicate the trunk revision that is the root of the branch. RCS revision numbers with 6 integers refer to branches off of a branch of the trunk.

When a commit is made, by default the last integer of the revision is incremented.

When a branch is created off of a revision that does not have a branch, the revision number of the new revision is created from the revision number of the root of the branch by appending .1.1 to it. If the root revision already has a branch off of it, the new branch revision number will have a .2.1 appended to it.

cvs (1990)

CVS uses RCS to manage the history of each file under version control.

How to set up a CVS repository:

$ mkdir cvsroot
$ export CVSROOT=/PATH/TO/cvsroot
$ cvs init

For CVS commands to work, either the CVSROOT environment variable must be set or the location of the repository root must be passed to the command with each invocation using the -d option.

It is possible to set up a server on the repository host which listens on port 2041. The server uses ssh authentication, so the client must have an account on the host. To use the server, the client sets the CVSROOT environment variable to something like this:

$ export CVSROOT=:pserver:foo.com:/PATH/TO/cvsroot

How to create a project in the repository:

$ mkdir foo
$ cd foo
$ touch README
$ cvs import foo FOO_CORP V1

The 2nd and 3rd argument to cvs import are required and are used to create tags.

Here is an example of how to check out the foo project but name the working directory bar:

$ cvs checkout -d bar foo

Files only needed to only needed to be added once, like Mercurial and unlike Git:

$ cd bar
$ vim Makefile
$ cvs status
$ cvs diff
$ cvs add Makefile
$ cvs commit -m 'adding a Makefile'
$ cvs log

Branching is a three step process: (1) tag the commit you are branching from, (2) create the branch, and (3) update the working directory to the branch.

$ cvs tag WUMPUS_ROOT
$ cvs tag -r WUMPUS_ROOT -b WUMPUS
$ cvs update -r WUMPUS

How to merge

svn (2000)

• Beautiful Code, Ch. 2: Subversion's Delta Editor

How to set up an SVN server:

$ svnadmin create /PATH/TO/svn
$ vim /PATH/TO/svn/conf/svnserve.conf
$ vim /PATH/TO/svn/conf/passwd
$ mkdir /tmp/empty
$ svn import /tmp/empty file:///PATH/TO/svn/NAME
$ svnserve -d  -r /PATH/TO/svn --log-file /PATH/TO/svn/svnserve.log
$ svn co svn://localhost/NAME

When editing svn/conf/svnserve.conf, add these lines to the [general] section:

anon-access = none
auth-access = write
password-db = passwd

In svn/conf/passwd, create a username and password:

joe = passwerd123

To have svnserver start automatically at boot on an Ubuntu server, put this script in /etc/init.d. Change the value of DAEMON_ARGS from -d -r /usr/local/svn/repos to -d -r /PATH/TO/svn —log-file /PATH/TO/svnserve.log.

Archive and Patch Tools

diff | cpio | diff3 | ar | tar | patch | zip | diffstat | jar | rsync | colordiff

diff (1974)

• An Algorithm for Differential File Comparison Hunt & McIlroy 1976
• man diff

To implement an efficient version control system one needs to find a minimal delta or difference between two similar text files. The problem led to the development of the Unix diff utility. Regarding a file as a sequence of lines, the problem can be treated as an example of the longest common subsequence problem. The standard solution to this problem has O(nm) performance in both time and space, where n and m are the lengths of the two files. To facilitate quick comparison of lines, each line is replaced with a hash code. When implementing diff McIlroy developed an algorithm that was more efficient than the standard solution in most cases.

The standard diff notation prefixes lines with < and > to indicate whether the line originated in the first or second file. It also uses the letters a, c, and d to indicate lines being added, changed, or deleted:

$ echo "foo" > foo.txt

$ echo "bar" > bar.txt

$ diff foo.txt bar.txt 
1c1
< foo
---
> bar

$ diff foo.txt /dev/null
1d0
< foo

$ diff /dev/null foo.txt 
0a1
> foo

These letters used in diff notation are also ed commands. In fact, diff -e will output an ed script which can be used to convert the first file into the second:

$ diff -e foo.txt bar.txt > diff.ed

$ ( cat diff.ed ; echo "w" ) | ed foo.txt

The version of diff released with BSD 2.8 in 1981 added the -c option to show three lines of context around each change. This is called the context format.

The BSD 2.8 diff also added an -r option to perform a recursive diff on directories.

In 1990 the -u option was added, which gives a diff inunified format. In the context format, if a line is changed, the context is repeated: once around the old version of the line and once around the new. The uniformed format puts both version of the line in the same context, reducing the size of the diff file.

The -C NUM and -U NUM options are like the -c} and {{-u options, except that they show NUM lines of context.

normal format:

$ sed s/^root:/ROOT:/ /etc/passwd > /tmp/passwd

$ diff /etc/passwd /tmp/passwd
12c12
< root:*:0:0:System Administrator:/var/root:/bin/sh
---
> ROOT:*:0:0:System Administrator:/var/root:/bin/sh

ed script format:

$ sed s/^root:/ROOT:/ /etc/passwd > /tmp/passwd

$ diff -e /etc/passwd /tmp/passwd
12c
ROOT:*:0:0:System Administrator:/var/root:/bin/sh
.

context format:

$ sed s/^root:/ROOT:/ /etc/passwd > /tmp/passwd

$ diff -c /etc/passwd /tmp/passwd
*** /etc/passwd    2013-10-24 17:38:39.000000000 -0700
--- /tmp/passwd    2014-04-26 12:57:57.000000000 -0700
***************
*** 9,15 ****
  # Open Directory.
  ##
  nobody:*:-2:-2:Unprivileged User:/var/empty:/usr/bin/false
! root:*:0:0:System Administrator:/var/root:/bin/sh
  daemon:*:1:1:System Services:/var/root:/usr/bin/false
  _uucp:*:4:4:Unix to Unix Copy Protocol:/var/spool/uucp:/usr/sbin/uucico
  _taskgated:*:13:13:Task Gate Daemon:/var/empty:/usr/bin/false
--- 9,15 ----
  # Open Directory.
  ##
  nobody:*:-2:-2:Unprivileged User:/var/empty:/usr/bin/false
! ROOT:*:0:0:System Administrator:/var/root:/bin/sh
  daemon:*:1:1:System Services:/var/root:/usr/bin/false
  _uucp:*:4:4:Unix to Unix Copy Protocol:/var/spool/uucp:/usr/sbin/uucico
  _taskgated:*:13:13:Task Gate Daemon:/var/empty:/usr/bin/false

unified format:

$ sed s/^root:/ROOT:/ /etc/passwd > /tmp/passwd

$ diff -u /etc/passwd /tmp/passwd
--- /etc/passwd    2013-10-24 17:38:39.000000000 -0700
+++ /tmp/passwd    2014-04-26 12:57:57.000000000 -0700
@@ -9,7 +9,7 @@
 # Open Directory.
 ##
 nobody:*:-2:-2:Unprivileged User:/var/empty:/usr/bin/false
-root:*:0:0:System Administrator:/var/root:/bin/sh
+ROOT:*:0:0:System Administrator:/var/root:/bin/sh
 daemon:*:1:1:System Services:/var/root:/usr/bin/false
 _uucp:*:4:4:Unix to Unix Copy Protocol:/var/spool/uucp:/usr/sbin/uucico
 _taskgated:*:13:13:Task Gate Daemon:/var/empty:/usr/bin/false

recursive format:

$ mkdir /tmp/a /tmp/b

$ cp /etc/passwd /tmp/a

$ $ sed s/^root:/ROOT:/ /etc/passwd > /tmp/b/passwd

$ diff -r /tmp/a /tmp/b
diff -r /tmp/a/passwd /tmp/b/passwd
12c12
< root:*:0:0:System Administrator:/var/root:/bin/sh
---
> ROOT:*:0:0:System Administrator:/var/root:/bin/sh

cpio (1977)

• man cpio
• cpio file format (POSIX 2008)

An ancient and to most people unfamiliar Unix archiving tool which is roughly equivalent to tar. The suffix .cpio is often used for cpio archive files.

The format is used by RPM packages, though RPM 5.0 and later also support the xar format. The Linux kernel since version 2.6 has a cpio archive called initramfs which it uses during the boot process. cpio is also used by the Mac OS X .pkg format.

The cpio file format is similar to the tar file format in that for each file which is added to an archive, a header and the file contents are appended to the archive file. In the case of cpio the header is smaller (76 bytes vs 512 bytes). This is in part because the header only contains the file name length; the actual file name is appended to the archive file between the header and the file contents. By contrast the tar format stores the name in fixed length fields, putting a limit on the possible path length. Another different is the cpio format lacks a checksum.

Another difference between tar and cpio is that whereas tar takes the files to be archived on the command line, recursively descending any arguments which are directories, cpio when used with the -o flag takes its list of files to be archived from standard input. cpio was designed to be used with the find command. Similarly when using the i flag cpio reads the files to be extracted from an archive from standard input.

diff3 (1979)

• man diff3

diff3 displays the differences between three versions of the same file.

The three way diff is the foundation of branch merging. A two way diff is insufficient for merging because deleting a line in one branch looks like adding a line in the other branch. Only by comparing both branches with the original can these two cases be distinguished.

diff3 has three basic invocations:

diff3 EDIT1 ORIG EDIT2
diff3 -e EDIT1 ORIG EDIT2
diff3 -m EDIT1 ORIG EDIT2

The first invocation writes a description of the three-way diff to standard out.

The second invocation writes an ed script to standard out which will merge the changes in EDIT2 to EDIT1.

The third invocation performs the merge. It writes a version of the file with changes from both EDIT1 and EDIT2 to standard out.

Here is an example of the output format used by the first invocation:

$ cat /tmp/orig.txt 
a
b
c
d
e

$ cat /tmp/edit1.txt 
a
b1
c
d
e
f

$ cat /tmp/edit2.txt 
a
b
c
d1
e

$ diff3 /tmp/edit1.txt /tmp/orig.txt /tmp/edit2.txt
====1
1:2c
  b1
2:2c
3:2c
  b
====3
1:4c
2:4c
  d
3:4c
  d1
====1
1:6c
  f
2:5a
3:5a

Each hunk of the diff3 output starts with four hyphens. All of the hunks in the example above are two-way hunks, meaning that two of the three files are the same. In this case the number of the differing file as it appears in the diff3 arguments is placed after the hyphens.

Here is an example of a three-way hunk, where all three files differ and no number is placed after the hyphens:

$ cat /tmp/orig.txt 
a

$ cat /tmp/edit1.txt                               
a1

$ cat /tmp/edit2.txt 
a2

$ diff3 /tmp/edit1.txt /tmp/orig.txt /tmp/edit2.txt
====
1:1c
  a1
2:1c
  a
3:1c
  a2

ar (1979)

• man ar

A tool on Unix systems to create static libraries from compiled objects. In other words, to create a .a file from a set of .o files. The format is understood by the linker—which these days is usually built into the compiler—and the loader ld.

The command line interface is broadly similar to tar. Here is how to create an archive; remove files from an archive; list the archive contents; extract files from an archive:

ar -c NAME.a FILE ...
ar -d ARCHIVE FILE ...
ar -t ARCHIVE
ar -x ARCHIVE FILE ...

The ar file format is not standardized and may differ between systems.

The file format used by GNU ar on Linux starts with the new line terminated string "!<arch>".

Each file starts with a 60 bytes header, followed by the file contents. The header has the following fixed-width fields:

The space allocated for the file name in the header is quite short. GNU ar actually stores a special file named "//" in the archive with a new line separated list of file names. A header can reference a name in this special file by storing "/" and a the decimal offset in the "//" file of the file name. When file names are stored directly in the header, a "/" is used to mark the end of the file and the rest of the field is space padded. This supports spaces in the file name.

GNU ar also stores a special file named "/" is the archive for a symbol table. The format is

• a 32-bit integer containing the number of symbols
• a list of 32-bit integers, one for each symbol, containing the offset of the header in the archive for the file containing the symbol
• a list of null terminated strings, in the same order as the previous list, containing the symbol names

tar (1979)

• man tar

The more portable twin of ar. Originally used for creating and using magnetic tape archives.

How to create a tar file; list the contents of a tar file; compare a tar file with the file system; and extract the contents of a tar file:

tar [-]cf NAME.tar DIR
tar [-]tf TARFILE
tar [-]df TARFILE [DIR]
tar [-]xf TARFILE

The -v option can be used with -c or -x to write the files being added or extracted to standard error.

Tar files store the files in sequential order. Each file is precede by a 512 byte header. The file itself is null byte padded to a multiple of 512 bytes.

Tar can write to and read from stdout. The following two invocations behave identically:

tar cf - . | (cd DIR ; tar xf -)
tar cf . - | tar xf - -C DIR

Tar can append data to an existing tar file. These commands append the contents of a directory to a tar file; append the contents of the directory which are newer than what is already on a tarfile; append subsequent tar files to the first tar file:

tar [-]rf TARFILE DIR
tar [-]uf TARFILE DIR
tar [-]Af TARFILE1 TARFILE2 ...

How to create a compressed tar file:

tar [-]czf NAME.tar.gz
tar [-]cjf NAME.tar.bz2
tar [-]cJf NAME.tar.xz

In 1988 POSIX extended the format of the header block in a backwardly compatible way. Additional header type flags were added in 2001.

patch (1985)

• man patch

The patch command can apply the output of diff to the file that was the first argument of diff to recover the file that was the second argument of diff. patch reads the output of diff from standard input:

$ echo "foo" > foo.txt
$ echo "bar" > bar.txt 
$ diff foo.txt bar.txt > foo.patch
$ patch foo.txt < foo.patch 
patching file foo.txt
$ cat foo.txt 
bar

The above is only a slight improvement over what could have been achieved with diff -e and ed. The novelty of patch is its ability to apply a patch file to an entire directory:

$ mkdir old
$ echo "bar" > old/bar.txt
$ echo "baz" > old/baz.txt
$ cp -R old new
$ echo "qux" > new/bar.txt
$ diff -Naur old new > foo.patch
$ rm -rf fnew
$ patch -Np0 < foo.patch
patching file old/bar.txt
$ cat old/bar.txt 
qux

This is a good way to create a patch file:

diff -Naur OLD NEW

When creating the patch file with diff, the -u or -c flags seem to be necessary so that patch has the file names. The -N flag is necessary if files are added or removed. The -a flag prevents diff from skipping files which it thinks are binary.

If the diff was performed outside of the directories, then the patch should be performed outside of the directory to be patched with the -p0 flag. Optionally the patch can be performed inside the directory to be patched with the -p1 flag. The -N flag instructs patch to not make a change if the patch appears to be reversed or already applied.

zip (1989)

• man zip

zip combines file compression and archiving. It is a better choice for sharing files with Windows hosts than tar, which most Windows hosts don't have installed.

zip [-r] [-0] ARCHIVE FILE ...
zip -d ARCHIVE FILE ...
zip -u ARCHIVE [FILE ...]

unzip -l ARCHIVE
unzip ARCHIVE [FILE ...]

Compression is the DEFLATE algorithm, or no compression if the -0 flag is used.

zip stores the file name, file size, and last modification time of the file. The information is in a header which precedes the file itself and in the "central directory" at the end of the file.

By default zip does not recursively descend directories, adding their contents to the archive. Use the -r flag to get this behavior.

diffstat (1992)

Summarizes a recursive diff on two directories. It lists the files that were modified, added, or deleted with the number of lines which changed.

jar (1995)

• man jar

jar supports some of the tar commands:

jar cf NAME.jar DIR
jar tf JARFILE
jar xf JARFILE
jar uf JARFILE DIR

jar can write to and read from stdout; the syntax is different from tar:

jar c . | (cd DIR ; jar x)
jar c . | jar x -C DIR

Use jar -e to make a jar file runnable by java. The argument to -e is a class with a main routine which will be used as the entry point.

$ mkdir

$ cat > foo/A.java
package foo;

public class A {
    public static void main(String[] args) {
        System.out.println("A");
    }
}

$ sed s/A/B/ foo/A.java > foo/B.java

$ javac foo/*.java

$ jar cef foo.A foo.jar foo

$ java -jar foo.jar        
A

A jar file is a zip file; unzip can also be used to extract the contents. jar stores extra information about the jar file in META-INF/MANIFEST.MF:

$ unzip foo.jar

$ cat META-INF/MANIFEST.MF 
Manifest-Version: 1.0
Created-By: 1.6.0_26 (Sun Microsystems Inc.)
Main-Class: foo.A

rsync (1996)

• man rsync
• the rsync algorithm

A tool for copying files and directories between hosts. Usually it uses ssh. It is faster than scp when some of the files are already on the destination or when copying files that have been modified.

Here is the usage for putting and getting:

  rysnc -a PATH ... HOST:PATH
  rsync -a HOST:'PATH ...' PATH

The -a flag is equivalent to the flags -rptoglD which (1) recursively copy the contents of directories, (2) copy file permissions, (3) copy file times, (4) copy owner, (5) copy group, (6) copy symlinks, and (7) copy special devices.

Other useful flags are -v for verbose mode and --exclude which takes a file glob pattern to specify files to skip.

If the source and target paths have the same basename, then rsync will copy the contents of the source into the contents of the target. If the basenames are different, rsync will create a directory with the same name as the source inside the target. This behavior can be suppressed by putting a trailing slash / on the end of the source.

rsync can be used to backup a directory on a remote host. With the --backup flag, files which are already on the destination but have been modified on the source will be copied into a separate incremental directory with a tilde (~) suffix. The --backup-dir flag can be used to specify a different incremental directory.

colordiff (2002)

A version of diff which colorizes the output. It takes the same options as diff.