Reduce populateKinship.r peak memor

[ankurjuneja]

Rationale

The nightly kinship calculation runs as a child of the web server, so its peak is charged against the JVM's memory budget and spikes cloud servers every night.
Processing one family at a time and streaming rows to disk bounds the peak by the largest family rather than the whole colony.

Related Pull Requests

Changes

[bbimber]

@ankurjuneja and @labkey-martyp: have you confirmed this produces identical results using the real data from production NPRCs?

[ankurjuneja]

@ankurjuneja and @labkey-martyp: have you confirmed this produces identical results using the real data from production NPRCs?

Yes, tested with data from three centers and output is identical.

[bbimber]

@ankurjuneja and @labkey-martyp: have you confirmed this produces identical results using the real data from production NPRCs?

Yes, tested with data from three centers and output is identical.

OK, thanks for confirming. All of this is quite old code and it would not surprise me if there was big room for efficiency both here, and at import time.

[labkey-martyp]

@ankurjuneja and @labkey-martyp: have you confirmed this produces identical results using the real data from production NPRCs?

Yes, tested with data from three centers and output is identical.

@bbimber if you want to send us study.Pedigree export and your kinship.txt, we can test yours as well.

[bbimber]

@ankurjuneja and @labkey-martyp: have you confirmed this produces identical results using the real data from production NPRCs?

Yes, tested with data from three centers and output is identical.

@bbimber if you want to send us study.Pedigree export and your kinship.txt, we can test yours as well.

@ankurjuneja wrote that this was tested on three centers - was ONPRC not one of them?

[bbimber]

If you're touching this code, the first message I got from R was that kinship2 is deprecated in favor of this, which seems like it might be a drop-in replacement: https://louislenezet.github.io/Pedixplorer/