I have 500GB of .csv data which include these three (and other) variables: 1. where a passenger gets on a train, 2. where they get off and 3. The time it takes.
I need to make box plots of the time it takes based on where they got on and where they got off in an interactive R-shiny app - this is straight forward. But first I need to minimise the size of the file as reading 500GB in an R shiny app is prohibitive. Is there a way to store these variables in such a way that it makes this possible?
Even with vroom it takes too long and I don't think {Disk.frame} would work either. Any thoughts?
