Add pairwise distance computation in condensed form for symmetric metrics.

[lucabrivio]

This patch adds support for computing pairwise distances (for semimetrics) in a condensed form, similar to how pdist works in several packages available for MATLAB/Octave, R, Python, etc. (although on columns).
At the moment I have added no tests for the new methods, nor have I written functions to convert between the condensed form and the redundant one.

[nalimilan]

Thanks, but I'm not sure we want to provide this kind of function. Julia supports very powerful AbstractMatrix representations which can behave like a standard Matrix while storing only half of the entries. Base provides Symmetric for this, though currently it is represented as a standard Matrix whose upper/lower triangle is ignored because it generally allows more efficient computations and "only" uses twice the needed memory. See this Discourse thread and this one.

So instead of returning a vector, which is arguably a hack due to the limitations of other languages, we should use Symmetric or another similar type. We could also allow passing the expected output type as the first argument to pairwise.

What the most interesting feature for you in this PR? Saving RAM? Making subsequent computations faster?

[lucabrivio]

Thank you for your answer. The most interesting feature for me is indeed making subsequent computations faster, for instance when comparing a large number of such matrices... (In my case it also matches the form I intended to store the results in.) I am fairly new to Julia actually and could not find the best way to represent the result matrix, however I agree that we should use something different!

[nalimilan]

If you mainly care about the computation speed, then returning a Symmetric matrix should be enough, as comparisons will only check one half of the tables.

I think you could just add a pairwise(metric::PreMetric, a::AbstractMatrix) method which would create a Symmetric matrix rather than a standard Matrix before passing it to pairwise!. Then you would need to add a pairwise!(r::Symmetric, metric::SemiMetric, a::AbstractMatrix) method which would call pairwise(r.data, metric, a). That would be slightly wasteful since it would fill the whole matrix, so you could imagine also passing r.uplo as an additional argument to decide which triangle needs to be filled (but that would probably not make a big difference compared with the time required to compute distances).

[yakir12]

I too agree that pairwise(metric::PreMetric, a::AbstractMatrix) and pairwise(metric::SemiMetric, a::AbstractMatrix, b::AbstractMatrix) should return a Symmetric (came here looking for it).

[juliohm]

What is the status on this? I also think that the output for semi-metrics should be symmetric. Also, besides saving computation, we could have a method to flatten a symmetric matrix (the upper or lower part of it as a vector). Many times that is what is needed in distance calculations on a huge point cloud.

[juliohm]

Re-stating what was already stated in the previous comments. The main advantage of storing half of the matrix entries in a flattened vector is that we can perform subsequent calculations more efficiently. Simple statistics like histograms, means, variances are easy to apply. Even if pairwise returns a matrix type that is symmetric, its zero entries would compromise the summaries.