Remove file Syncronously cant find file

I have this function that make a directory have same files as the other one, so one directory is the source and the other is the one that should look like it so it removes files found in the second one that doesnt exist in the src. Anyway the thing is in the function below if I replaced fs.rmdirSyncwith fs.rmdir it works but if I dont it returns the error shown below.

function (srcDir, similarDir){

        srcDir = path.resolve(process.cwd(),srcDir);
        similarDir = path.resolve(process.cwd(),similarDir);
        var info = {deletedFiles : 0, deletedDir : 0};
        dive(similarDir,function(err, file){
            if(err){
                console.log(err.toString())
            }
            else
            {
                var baseFile = path.basename(file,'.css');
                var srcFileDir = path.dirname(file).replace(similarDir,srcDir)
                var srcFilePath = path.resolve(srcFileDir , baseFile);

                if(!fs.existsSync(srcFilePath)){
                    console.log("before");
                    try {
                        fs.rmdirSync(file)
                    }
                    catch(error){
                        console.log(error)
                    }
                    console.log("after");
                    info.deletedFiles++;
                }

            }
        })

This is the error that occurs when I use fs.rmdirsync although the file does exist and If i changed it to fs.rmdir it works ( on windows os )

{ [Error: ENOENT, no such file or directory 'D:\angularjs\build\css\components\test\test\test.css']
  errno: 34,
  code: 'ENOENT',
  path: 'D:\\angularjs\\build\\css\\components\\test\\test\\test.css',
  syscall: 'rmdir' }