java - Parsing different types of data format from a CSV file

Question

I am still a beginner with Java programming so I apologise in advance if I am over-complicating my problem.

What is my program? I am building a GUI based program. The goal of the program is to load a CSV, XML or JSON file and for the program to then store the data into an Array. The data will then be displayed in a text box. Ultimately, the program will have the ability to plot data to a graph.

GUI Details:

• 3 Radio buttons - Allows the user to select CSV, XML OR JSON
• Load File Button
• Display Button - Displays the data into the textArea
• Display Graph Button
• Text Area

Problem: I am having trouble storing the data into an Array. I believe this is because of the format of the data. So for example, this is the first 3 lines of the CSV file:

millis,stamp,datetime,light,temp,vcc
1000, 1273010254, 2010/5/4 21:57:34, 333, 78.32, 3.54
2000, 1273010255, 2010/5/4 21:57:35, 333, 78.32, 3.92
3000, 1273010256, 2010/5/4 21:57:36, 344, 78.32, 3.95

(Note - there are 52789000 lines of data in the CSV/XML/JSON files)

The CSV-Reader Class contains the method for reading through the data, storing it into an array and then storing it to a dataList.

As you can see from the above example, some of the data types are much different. I am having particular trouble with splitting/parsing the time and date variables.

Here is what my CSV-Reader Class code looks like at the moment (Again, I apologise for noob code).

import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

public class CSVReader {

//create a class that will hold arraylist which will have objects representing all lines of the file

private List<Data> dataList = new ArrayList<Data>();
private String path;

public List<Data> getDataList() {
    return dataList;
}

public String getPath() {
    return path;
}
public void setPath(String path) {
    this.path = path;
}

//Create a method to read through the csv stored in the path
//Create the list of data and store in the dataList

public void readCSV() throws IOException{

    //i will create connection with the file, in the path
    BufferedReader in  = new BufferedReader(new FileReader(path));  

    String line = null;
    line = in.readLine();

    while((line = in.readLine())!=null){

        //I need to split and store in the temporary variable and create an object


        String[] splits = line.split("\s*(=>|,|\s)\s*");

        long millis = Long.parseLong(splits[0].trim());
        long stamp = Long.parseLong(splits[1].trim());
        DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy/M/d HH:mm:ss");
        System.out.println(splits[2].trim());
        LocalDateTime dateTime = LocalDateTime.parse(splits[2].trim(), formatter);
        LocalDate dateTime = dateTime.toLocalDate();
        LocalTime time = dateTime.toLocalTime();
        int light = Integer.parseInt(splits[3].trim());
        double temp = Double.parseDouble(splits[4].trim());
        double vcc = Double.parseDouble(splits[5].trim());

        Data d = new Data(millis,stamp,datetime,light,temp,vcc);//uses constructor


        //final job is to add this object 'd' onto the dataList
        dataList.add(d);

    }//end of while loop

}

Any help would be greatly appreciated!

Edit 1 - I thought that date and time were seperate CSV headers. They were not. Therefore the time variable has been deleted from the program. It has been replaced with the datetime variable.

Edit 2 - My program is now reading the CSV file up until line 15 of the csv

27000, 1273010280, 2010/5/4 21:58:0, 288, 77.74, 3.88

CONSOLE ERRORS

Exception in thread "AWT-EventQueue-0" 
java.time.format.DateTimeParseException: Text **'2010/5/4 21:58:0'** could not 
be parsed at index 15
at java.time.format.DateTimeFormatter.parseResolved0(Unknown Source)
at java.time.format.DateTimeFormatter.parse(Unknown Source)
at java.time.LocalDateTime.parse(Unknown Source)
at CSVReader.readCSV(CSVReader.java:55)
at GUI$2.actionPerformed(GUI.java:85)
at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
at javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
at javax.swing.DefaultButtonModel.setPressed(Unknown Source)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source)
at java.awt.Component.processMouseEvent(Unknown Source)
at javax.swing.JComponent.processMouseEvent(Unknown Source)
at java.awt.Component.processEvent(Unknown Source)
at java.awt.Container.processEvent(Unknown Source)
at java.awt.Component.dispatchEventImpl(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Window.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
at java.awt.EventQueue.access$500(Unknown Source)
at java.awt.EventQueue$3.run(Unknown Source)
at java.awt.EventQueue$3.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
at java.awt.EventQueue$4.run(Unknown Source)
at java.awt.EventQueue$4.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
at java.awt.EventQueue.dispatchEvent(Unknown Source)
at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.run(Unknown Source)

Answer 1

Since your date column is a string, you need to parse it into a date object. This depends on your version of Java; if you're using Java 8, you could use the new date classes. Maybe this answer could help you along.

It looks like you have two separate columns, date and time in the CSV header, but the lines have the date and time in one column. This could be part of your problem. You need to determine if they are two columns or one, but they can be parsed with the Java libraries in any case.

Also, it would probably be a good idea to use a battle-tested CSV library like Apache Commons CSV, instead of rolling your own CSV parser. You might get by with your own in a limited case, but CSV is not as simple as it first appears.