# Version control for collaboration (with word-level diffs)?

Most papers are now written collaboratively, and collaborators are often located in different places. I have always used version control systems for my documents and code, and have also found version control critical for collaborative software projects, but it seems many researchers in theory avoid their use for writing joint papers. To convince my collaborators that version control (revision control) is a good idea for working together, there seem to be some prerequisites. It is not possible to force everyone to worry about a specific set of conventions for line breaks and paragraphs, or to avoid tab/space conversions.

Does someone offer free hosting of small shared document repositories, with text-document-friendly version control that can handle word-level diffs (not line-based)?

If not, then I would welcome other suggestions that are based on experience (let's avoid speculation, please).

I was thinking of Git, Subversion, Mercurial, darcs, or Bazaar, set up to handle word-level differences with wdiff, together with a simple way of setting up access secured by public keys (for instance via ssh). However, none of the version control providers that I looked at seem to offer anything like this. For scientific collaboration the "enterprise" features stressed by many of these companies are not very important (lots of branches, integration with trac, auditing by third parties, hierarchical project teams). But word-level diffs seem critical yet unsupported. In my experience, with line-level diffs for text files, everyone has to avoid reformatting paragraphs and editors that change tabs to spaces or vice versa cause problems; there also seem to be many spurious edit conflicts. I have used wdiff and latexdiff quite successfully to help with manual merging of changes previously, so I am hoping a word-level diff would reduce such problems.

See related question at MO about tools for collaboration, and related questions over at TeX.SE, about version control for LaTeX documents and LaTeX packages for version control. See also the SVN Hosting Comparison Review Chart for a large list of hosting providers, for just one of the main version control systems.

Edit: Jukka Suomela's answer to the TeX.SE question "Best LaTeX-aware diff and merge tools for subversion" seems to be the best suggestion so far, covering how to interpret the deltas on a word level. Moreover, Jukka has explained how the differences between successive versions on the repository end are separate from the user-level differences used for conflict detection and merging of changes. Jukka's answer at TeX.SE explicitly excludes simultaneous edits and merging, relying instead on the traditional atomic edit token to avoid edit conflicts. Clarifying (and modifying) my original question, is there a way to ensure that edit conflicts can be resolved on a word difference basis, rather than on a line difference basis? In other words, can wdiff or similar tools be integrated into the conflict detection part of the version control tools, similar to the way end-of-line differences and differences in whitespace can be ignored?

• I don't quite understand the question. For example, in SVN, diffs displayed to a user are generated by the client, and it depends on your SVN client (and its configuration) whether you get word-based diffs or line-based diffs. The company that hosts your SVN repository doesn't affect this at all. Nov 12, 2010 at 18:41
• @suresh If you're editing (written) text documents, it's often a pain to have to scan a whole line in a diff to see that someone changed one comma. The correct behavior usually is to show the minimal unit of change. Or, consider the behavior if someone doesn't use line breaks. Then changing a single word will cause the whole paragraph to show up in the diff for you to find the tiny change. Nov 12, 2010 at 19:10
• I don't use hard line breaks to wrap lines. In my Latex source code, a physical line of text is usually a full paragraph of text. The editor can word-wrap it for display, depending on the current window width. It simplifies things a lot; there is never need to worry about things like should I re-word-wrap a paragraph, or to agree on the "right" line width with your co-authors. However, you will need a word-level diff tool to see the changes quickly. Nov 12, 2010 at 19:53
• @Andras My point was that the VC system only needs to be able to reconstruct the two revisions on the client side, and not surprisingly all VC systems can do that. What you then need is a word-level three-way merge utility, but I don't know of any. (For example, TortoiseMerge and kdiff3 are both line-based.) Once you have such a utility, then any VC system that allows you to specify an external merging utility will suffice. (That includes svn, bzr, git, hg...) Nov 14, 2010 at 11:28
• One source of confusion here is that there is a built-in binary diff algorithm (that operates on the level of individual bytes) that is used by SVN in the communication between the server and client, and also internally by the server to keep the repository compact. This is merely an optimisation; it is not visible to the user and the same binary diff algorithm can be applied to any kind of file. All user-visible things (human-readable diffs, merging, conflict resolution...) happen on the client side. Nov 14, 2010 at 11:56

I've used git to collaborate on some documents written in latex. You'll have to adhere to some rules:

• Start each sentence on a new line, latex ignores these newlines as long as there is no blank line
• Use the same configuration for formatting (tab/spaces/max text width)
• For best results, create a .gitattributes file in your repository and add the line *.tex diff=tex. This makes diff aware of the tex syntax and leads to more meaningful output.

You can then use git diff --color-words and gitk --color-words to see word differences (also see this article Word-by-word diffs in Git on how to configure git to always use word-diff algorithm for displaying the git diff/git log).

To reduce the manual merges, I can recommend using separate files for sections and subsections (depending on the size of your document).

• I'll consider doing this for my own documents, it seems to be an easy way to achieve most of my aims. But not everyone is keen to work this way... Nov 14, 2010 at 18:48
• For people hesitant to work this way, you can use TortoiseGit if they don't like the git command line. If it is about the each sentence on a new line part, well as long as there is no maximum text width forced, this isn't that important. (I have worked on some projects without that rule) Nov 14, 2010 at 19:32
• On the whole, I agree that git is a good choice. But why can separate files for (sub)sections reduce the number of manual merges? I also wonder how starting each sentence on a new line helps (sometimes sentences mix in the process of editing).
– dd1
Feb 19, 2013 at 10:59
• regarding the separating files: at that time, I did not understand the exact details of git merging, so that is actually unneeded, but still advisable for other reasons. The sentence on a new line is very important, since most tools around git always show line changes, if you then use another strategy, say let the editor do linebreaks, every time someone changes 1 word in a paragraph, you will have to hunt were it happend, and in case of automatic merging: no way. Feb 20, 2013 at 10:04

I suggest taking a look at:

and the related paper:

• Unfortunately the "best practices" in these documents are precisely the kinds of things one cannot force on collaborators. Nov 13, 2010 at 19:03

I really want to echo others and suggest that you sit down and work out a nice SVN strategy. I use SVN to host my entire "research" structure:

• JabRef reference managing
• Articles

It's great because it contains everything, and of course provides a history. The caveat being you need your own server. But if you have some existing Windows machine (or whatever you a comfortable with) you can install it simply through VisualSVN Server. You then create appropriate accounts for collaborators, and give them access to an appropriate area (i.e. perhaps read-access to your JabRef bibtex file, and read/write to a shared 'in-progress' article area).

TortiseSVN can be used as the Windows client for interacting with SVN. You need to be careful about moving/deleting files and copying folders (SVN will store metadata inside hidden folders in each of your folders, so you must execute the delete command from within SVN to get rid of it, it takes a bit of getting used to, but is worth the investment).

Then, when working with a collaborator, they clearly must also use SVN. But, again, the investment in learning is not worthless. And via some thought, you can also have it so you have read-only access to their jabref file (perhaps via the 'external' facility in svn).

In this way, with a bit of thought and a little bit of effort, you can be in a situation where you are editing documents as per normal, commiting changes nightly, updating in the morning and resolving all conflicts easily.

I really recommend it. The more people that set up their own SVNs the better, as it will only improve collaboration options in the future (though, of course, it would be beneficial if perhaps there was a 'standard' way of setting up a scientific repository).

-- Edit: Infact, I've written up such a proposal here: Strategy for Scientific Collaboration with LaTeX and SVN. It proposes to make use of the svn externals feature to allow easy collaboration between people with a similar setup. Let me know if it needs changing or is simply not appropriate.

While reading your great post and looking around for a solution myself I stumbled into the option to colorize changes at word level in gitk. The gitk parameter seems to be a new and/or undocumented feature since the auto-completion does not offer it and the gitk man page does not list it.
Here are the options the I found:

gitk --word-diff=plain
gitk --word-diff=porcelain
gitk --word-diff=color


You can find several discussions on that topic searching for "diff --color-words" gitk.

Edit:
This is what is looks like ...

I understand the problem very well. I have started using Kaleidoscope for diffs with git. It is Mac-only but its comparisons work better than wdiff, and it also has an interface and live updates.

• To me it seems that Kaleidoscope is just a line-based diff tool which, additionally, highlight changes inside each line. It is not a replacement for wdiff and friends. Kaleidoscope produces unreadable diffs if you, e.g., just take a paragraph of text and change some line breaks. Wdiff-based tools simply ignore changes in line breaks. Nov 12, 2010 at 20:33