Sorry, I was away from the office for a couple of days due to illness. In the meantime, the problem can be regarded as solved, I think. But let me take the chance to summarize the events and our findings:
Around January 31st, there were a coupld of people reporting that Windows Defender had killed and deletetd their mAirList installation, claiming that it contained one of several malwares, in particular “Trojan:Script/Wacatac.B!ml”, "Program:Win32/Wacapew.C!ml”, or others.
Apparently, it was not only the latest version 7.2/7.3 that was affected (so it couldn’t be pinned down to any changes we made in the code recently, but we also received reports from people who where running older versions like 7.1, and hadn’t updated their playout in months.
We had this situation before, and it was obvious that this must be a “false positive” (antivirus software detecting malware where there is none actually). So the first thing we did was uploading the current mAirList.exe (7.2.7 at that time) to virustotal.com, a website that checks a file with 60+ different antivirus products at the same time. This is the report we were getting:
As you can see, all other antivirus products thought that the file was safe, only Defender said it was a threat. So clearly a false positive. This happens every now and then, and we are not the only ones affected, it happens with other applications as well (even open source software). Some examples:
https://www.reddit.com/r/SteamDeck/comments/19fdp06/windows_defender_thought_proton_was_a_trojan/
https://www.reddit.com/r/sysadmin/comments/1acbp0v/anyone_else_getting_bombarded_with_defender/
https://www.reddit.com/r/mountandblade/comments/1aepfes/warband_script_extender_2_v1115_flagged_as_trojan/
https://www.reddit.com/r/WEPES/comments/1aewicr/pesedit_2013_marked_as_a_trojan/
Usually, there is one “bad” signature file (Antivirus update), and Microsoft fixes it quickly (Defender signatures are updated multiple times a day). So while we could not do much about it, we were confident that it would “heal itself” after a few hours. By the way, you can look up the current Antivirus version (= definition files) in the Windows Security settings → Settings → About:
Another thing we found out during research was that Defender is “caching” its results, which means, if a file (identified by its MD5 checksum) is regarded as malware one, it will not be checked again soon, even if it wasn’t malware anymore according to an update definition file. So in order to force Defender re-checking our executable, we quickly released an update, version 7.2.8. Other than bumping the version number, the only change in that version was that we started to sign the mAirList.exe file with our code signing certificate, hoping that that would make a difference to Defender. In earlier versions, only the installer was digitally signed, not the actual executable.
Unfortunately, the next morning, we were still getting reports of mAirList being deleted, even with the latest Defender updates. Interestingly, when I used virustotal.com again to re-check the 7.2.7 executable, it was “all green” now:
So there must have been some kind of magic happening in Windows Defender in the background. Luckily, we received remote access to one of the customer machines that were affected, so we were able to dig deeper into the Windows Event Log, and found this:
Apparently, Defender leveraged some kind of “cloud protection” to check the file, and the result was that it had to be deleted. There is very little information on the web regarding this cloud protection thing, what it does and how it does it. Apparently it is controlled by these switches in the Windows Security settings:
And we found this (older) article by Microsoft that says that the aim of “cloud protection” is to identify and eliminate threats before they even happen, with the help of machine learning (AI): Windows Defender Antivirus cloud protection service: Advanced real-time defense against never-before-seen malware | Microsoft Security Blog
So judging by the evidence we have found, it is most likely that the following happened:
-
Some part of our code, that has been around for months or years, suddently triggered Windows Defender to think that
mAirList.exeis subject to further checking by “cloud-delivered protection”, and upload the file to Microsoft servers for further analysis. -
On these servers, some artificial intelligence concluded that our software is malware, and instructed the local Windows Defender to to delete it. The “ml” suffix in the alleged malware name was a hint that some kind of machine learning played a role here.
-
This happened independently from the actual Antivirus version, and it could not be reproduced by a manual check of the file on the affected machine.
We have absolutely no idea what part of our code triggered Defender to use it’s cloud protection. As I said, even versions that were several months old were affected, so the code must have been around for a very long time, but Defender did not care about it until that day (and also not anymore after it). One possible thing is the new “update check” mechanism we introduced in mAirList 7, based on the popular WinSparkle library, which, of course, makes some HTTP requests every now and then to check for updates. But this is just a guess. On the other hand, maybe not, because WinSparkle is in its own DLL, and the threat was supposed to be in the main executable.
We also don’t know what exactly happens on the server side of the cloud protection, in other words, what made that AI thinking that our software was a virus. This is a total black box and absolutely not under our control. I find it very questionable that Microsoft uses techniques likes this, which can lead to the instant deletion of legit software.
All I know is that, while everybody claims that AI will make lives easier for the future generations - it made our lives definitely harder that week.
What are our conclusions and countermeasures?
First of all, as we do not know which part of our code triggered all this, there is not much we can do about it on our side. I already mentioned that we started signing our executable, not just the installer, and we will definitely look into getting an even better (and more expensive) “extended validation” (EV) certificate for that, which will also stop the annoying attempts by Windows SmartScreen to keep people from downloading and installing our application. Maybe the EV certificate will also help with Windows Defender false positives, who knows.
If you are using Windows Defender, and you want to make sure that it does not delete mAirList without asking you, we can only recommend one of the following:
- Turn off cloud protection.
- Add an exception rule for the mAirList program folder.
- Or switch to a different security software altogether.
On the other hand, you can just hope that this was a single event, and will not happen again soon.