MKL support out-of-the-box?

[luc-j-bourhis]

As far as x86_64 is concerned, MKL provides one of the fastest, if not the fastest, dense and sparse matrix operations, and it would be great if there was a turnkey install of SciRuby nmatrix with MKL. One important point is that Intel MKL has been distributed under an open-source licence allowing redistribution for several years. Therefore by turnkey, I have in mind shipping MKL with nmatrix: ideally MKL support out-of-the-box with just gem install nmatrix. The Anaconda distribution of numpy [*] has been doing exactly that [**] since 2016. There was even a numpy-mkl available with pip at some point.

[*] one of the recommended distro by the Scipy people themselves, supporting Linux, MacOS and Windows
[**] technically speaking, Anaconda has two packages, mkl and numpy-mkl, with the latter depending on the former

[translunar]

I think it would be lovely to support the Intel Math Kernel Library, but it might be an entire GSOC project in and of itself. How would you feel about mentoring such a project next year? :)

That said, we'd be happy to accept patches from eager non-students. ;)

[wlevine]

At one point, we supported the MKL unofficially thru the nmatrix-lapacke package. There is a comment here discussing how to use it. I suspect that method still works, or if it doesn't could be fixed fairly easily.

@luc-j-bourhis I know this is not exactly what you are asking for, but it would probably be a good starting point.

By the way, the MKL is not open-source. It's free to redistribute, but only in binary form without modifications.

[luc-j-bourhis]

@MohawkJohn My knowledge of the inner working of nmatrix is basically zero: I am just a user right now but I was indeed planning to learn enough about it to try to put together a pull request. The question I did not phrase clearly enough perhaps is: would you support committing the intel libraries for MacOS/Linux/Windows to the repo, and build a gem installing them alongside nmatrix?

@wlevine Reading that comment in extconf.rb at face value, what I proposed seems relatively easy to achieve. What am I missing here? (I should not have used "open source" indeed)

[translunar]

I'd be glad to include a plugin linking MKL to NMatrix, or just to update the lapacke plugin to support it.

[wlevine]

@luc-j-bourhis I don't see any big issues. Let me give you a little more context. The nmatrix-lapacke should (in theory) support any library (like MKL) that implements LAPACK and BLAS functions. I tested it briefly with the MKL (which is where that comment comes from), but didn't put any effort into officially supporting it. At the time I remember that the Intel software was very difficult to download and install, which ran counter to the goal of making nmatrix easier to install. But apparently that has changed, and certainly we would welcome any effort to provide better support for MKL.

Making a turnkey solution may be a bit more difficult. Making a numpy-mkl package might just involve duplicating all the code from numpy-lapacke, which would be sub-optimal. On the other hand, making MKL the default would raise other issues (e.g. do we really want to depend on non-open-source library by default? and what about our friends running AMD processors?).