Rcpp multithreading surprise


I did a bit more work on the fasqtc package after investigating multithreading.

  1. I found that I could simply multithread an Rcpp function by using the parallel package and the mclapply() function. I suppose this makes sense but given the sometime painful complexity of doing this in C (or C++) code I just didn’t expect it to work.
  2. I was extra surprised that you can use mclapply() to split a list of fastq files to be handled by different processes and this gives a considerable speed up (~1/3rd faster with just 2 cores). Even though the processes are essentially reading from the same disk. Again I expected that reading from the disk would be a bottleneck that negated any multithreading gain. Apparently not.
  3. I read somewhere else (I cannot recall now) that if you set mclapply() to use more processes than you actually have cores this would be optimal. Again this sounded weird but it’s true!  mc.cores=detectCores()+1  ##FTW.

Oh yes and I finally ran build check and it all worked!

# This is the C++ workhorse using Rcpp, Armadillo and especially seqan
# The resulting object contains all the necessary data for other functions.
# If mc=TRUE the the function will attempt parallel processing
# If sampled = FALSE all the fastq file wil be read, if sampled= TRUE the first 200K
fastqc= function(files, mc=TRUE, numreads= 200000)
  # for a single file
  if (length(files)==1)
    res= fastqCpp(files, numreads)

  # multi files multi core
  if (length(files) > 1 & mc==TRUE)
    res= mclapply(X=files, FUN=fastqCpp, numreads=numreads, mc.preschedule=TRUE, mc.cores=detectCores()+1)  

  # multi files single core
  if (length(files) > 1 & mc==FALSE)
    res= lapply(X=files, FUN=fastqCpp, numreads=numreads)

  #.Call("fastqCpp", file, package="fastqc")  

Created by Pretty R at inside-R.org


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s