r - Why is as.Date slow on a character vector?

Question

I started using data.table package in R to boost performance of my code. I am using the following code:

sp500 <- read.csv('../rawdata/GMTSP.csv')
days <- c("Monday","Tuesday","Wednesday","Thursday","Friday","Saturday","Sunday")

# Using data.table to get the things much much faster
sp500 <- data.table(sp500, key="Date")
sp500 <- sp500[,Date:=as.Date(Date, "%m/%d/%Y")]
sp500 <- sp500[,Weekday:=factor(weekdays(sp500[,Date]), levels=days, ordered=T)]
sp500 <- sp500[,Year:=(as.POSIXlt(Date)$year+1900)]
sp500 <- sp500[,Month:=(as.POSIXlt(Date)$mon+1)]

I noticed that the conversion done by as.Date function is very slow, when compared to other functions that create weekdays, etc. Why is that? Is there a better/faster solution, how to convert into date-format? (If you would ask whether I really need the date format, probably yes, because then use ggplot2 to make plots, which work like a charm with this type of data.)

To be more precise

> system.time(sp500 <- sp500[,Date:=as.Date(Date, "%m/%d/%Y")])
   user  system elapsed 
 92.603   0.289  93.014 
> system.time(sp500 <- sp500[,Weekday:=factor(weekdays(sp500[,Date]), levels=days, ordered=T)])
   user  system elapsed 
  1.938   0.062   2.001 
> system.time(sp500 <- sp500[,Year:=(as.POSIXlt(Date)$year+1900)])
   user  system elapsed 
  0.304   0.001   0.305 

On MacAir i5 with slightly less then 3000000 observations.

Answer 1

As others mentioned, strptime (converting from character to POSIXlt) is the bottleneck here. Another simple solution uses the lubridate package and its fast_strptime method instead.

Here's what it looks like on my data:

> tables()
     NAME      NROW  MB COLS                                     
[1,] pp   3,718,339 126 session_id,date,user_id,path,num_sessions
     KEY         
[1,] user_id,date
Total: 126MB

> pp[, 2]
               date
      1: 2013-09-25
      2: 2013-09-25
      3: 2013-09-25
      4: 2013-09-25
      5: 2013-09-25
     ---           
3718335: 2013-09-25
3718336: 2013-09-25
3718337: 2013-09-25
3718338: 2013-10-11
3718339: 2013-10-11

> system.time(pp[, date := as.Date(fast_strptime(date, "%Y-%m-%d"))])
   user  system elapsed 
  0.315   0.026   0.344  

For comparison:

> system.time(pp[, date := as.Date(date, "%Y-%m-%d")])
   user  system elapsed 
108.193   0.399 108.844 

That's ~316 times faster!