Security Vendor Qihoo 360 Censured By AV‐Comparatives, AV-TEST & Virus Bulletin

[Corrine]

Security Vendor Qihoo 360 Censured By AV‐Comparatives, AV-TEST & Virus Bulletin

Several days ago, AV-Comparatives reported that a product submitted for testing by an un-named vendor had been specifically engineered for the major testing labs but public availability of the version was limited. A collaborative investigation by AV‐Comparatives, AV-TEST and Virus Bulletin was conducted in which it was learned that Qihoo 360 submitted products for comparative and certification testing which behaved significantly differently from those made available to its users and customers.

All three testing organizations are striking any awards and certifications to from the start of 2015. In addition, AV‐Comparatives, AV-TEST and Virus Bulletin will be imposing stricter demands on test participants to avoid this from occurring in the future.

It is also noted in the PDF report (http://www.av-comparatives.org/weblog/wp-content/uploads/2015/04/VB-AVC-AVT-press-release.pdf) that questions were raised about Baidu and Tencent:

As part of the investigation into Qihoo 360, counter‐accusations were levelled by the company against
two fellow Chinese security firms, Baidu and Tencent. Analysis of products submitted for testing by
these companies turned up some unexpected flags within their products, marked with the names of
several test labs and implying some difference in product behavior depending on the environment they
were run in – similar flags were also found in Qihoo products. However, no evidence could be found that
this gave any significant advantage to either product, and in some cases it even seemed to put them at a
disadvantage. Both firms were able to provide good reasons for including these flags in their products.

 

[Aura]

Re: Security Vendor Qihoo 360 Censured By AV‐Comparatives, AV-TEST & Virus Bulletin

Looks like we had good reasons to be suspicious of Qihoo products. Now to see if this will catch back Baidu and Tencent in the future, if they take the road Qihoo did.

 

[Corrine]

Re: Security Vendor Qihoo 360 Censured By AV‐Comparatives, AV-TEST & Virus Bulletin

Via Facebook update by AV-Comparatives:

Statement regarding Tencent products in recent Windows tests

After in-depth investigations by AV-Test, certain optimizations have been identified in Tencent products which are clearly designed to improve their ratings in AV-TEST's performance testing. These optimizations, which have been found in all recent public versions of the products, provide minimal benefit to normal users and could even degrade the level of protection offered by the products.

All three testing labs involved in these investigations - AV-TEST, AV-Comparatives and Virus Bulletin - expect participants in their tests to behave in an open and ethical manner at all times, and consider this sort of "gaming" of tests to be unhelpful to both developers and users. The labs will be imposing stricter controls on participants to reduce opportunities for such actions, and will revoke all affected certifications and awards granted so far in 2015.

 

[Aura]

Re: Security Vendor Qihoo 360 Censured By AV‐Comparatives, AV-TEST & Virus Bulletin

Is Tencent going to be punished for it, or is it just some kind of "warning"?

 

[satrow]

Re: Security Vendor Qihoo 360 Censured By AV‐Comparatives, AV-TEST & Virus Bulletin

There's also a response from Qihoo re. the original report:

April 30th, Qihoo 360 received comments from its industry partners with allegation of inappropriate behaviour on the benchmarking processes in test labs. We regret that this behaviour has resulted into such comments from these labs, who we recognize as reference for security benchmarking. However, we hereby offer our perspective to the alleged comments.

The allegation highlights that the default configuration of the product available for the public, differs from the configuration used by the labs for testing. This configuration was explicitly declared upon submission of the tests, and was thereafter confirmed by the test labs.

In the public version, 3rd party engine is off by default, in the consideration that the majority of our users, are running on lower computing power. To satisfy lab conditions, the consideration of power constraint was therefore discarded. In any case, no alleged comment indicates that the level of protection from the product, is lower than the records achieved during the testing sessions.

Qihoo is committed to provide free security solutions, in order to convert security a commodity for all PC users. In addition, our product offers multiple engines, and regardless of the initial configuration, it is the user who has the complete freedom to choose how many of them should be activated at any time. It is with such understanding that AV-Test, one of the three testing labs involved, has stated that further investigation is ongoing, and will provide further updates on this topic.

 

[Corrine]

Re: Security Vendor Qihoo 360 Censured By AV‐Comparatives, AV-TEST & Virus Bulletin

Is Tencent going to be punished for it, or is it just some kind of "warning"?

All affected certifications and awards granted so far in 2015 will be revoked by the three test labs.

 

[Aura]

Re: Security Vendor Qihoo 360 Censured By AV‐Comparatives, AV-TEST & Virus Bulletin

I wonder if there will be a scandal around Baidu soon as well, since when AV-Comparatives implied that one company wasn't playing by the rules, Tencent and Baidu were accusing each other, and now Tencent got caught. So I wonder if Baidu is next.