Java reads files and gets the leading BOM [ï »]

I was reading the file containing keywords line by line and found a strange problem

sony
sony

Only the first one is being processed The problem is that Java does not see them as equal

INFO: [,s,o,n,y]
INFO: [s,y]

My code is as follows. What's the problem?

FileReader fileReader = new FileReader("some_file.txt");
    BufferedReader bufferedReader = new BufferedReader(fileReader);
    String prevLine = "";
    String strLine
    while ((strLine = bufferedReader.readLine()) != null) {
        logger.info(Arrays.toString(strLine.tocharArray()));
        if(strLine.contentEquals(prevLine)){
            logger.info("Skipping the duplicate lines " + strLine);
            continue;
        }
        prevLine = strLine;
    }

to update:

There seems to be a space in the first line, but it's not, and the pruning method doesn't work for me They are different:

INFO: [,y]
INFO: [,y]

I don't know what the first char added by Java is

Solved: the problem has been solved. Thanks for pointing out its BOM problem, which helps me find a solution quickly

Solution

Try trimming spaces at the beginning and end of the read line Simply replace your with the following:

while ((strLine = bufferedReader.readLine()) != null) {
        strLine = strLine.trim();
        logger.info(Arrays.toString(strLine.tocharArray()));
    if(strLine.contentEquals(prevLine)){
        logger.info("Skipping the duplicate lines " + strLine);
        continue;
    }
    prevLine = strLine;
}