Since founding Edgemesh five years ago, our motto has remained: “accelerate your website and accelerate your business.” Half a decade later, I can confidently say that we’ve brought cutting-edge performance improvements to billions of page views a month across our fantastic customer base.
When we designed the Edgemesh Server platform, we realized the opportunity provided by becoming the “front door” of the website goes well beyond performance. With over 100 million requests hitting Edgemesh Server each day, we’re seeing the impact that bots, hackers and generally nefarious actors has against our customer’s stores.
So today, we’re taking the first step to expand our offerings beyond just “accelerating your website” and moving toward more specific solutions to continue to push for “accelerating your business” by integrating seamless protection systems to keep our customer websites and their wallets safe from a major threat—ad fraud.
Edgemesh Server’s new AdFraud Analysis and Protection System helps ensure the most costly automated traffic—namely paid traffic—is identified and prevented to dramatically reduce the wasted advertising spend lost to ad fraud. In this post, we’ll talk about how it does this.
Identifying costly clicks
First, let’s start at the beginning and define ad fraud. Ad fraud, also known as invalid traffic, or IVT, is the practice of fraudulently misrepresenting online ad impressions, clicks, traffic, conversions, or data to generate revenue. In 2016, for every $3 spent on digital ads, ad fraud took $1, which led to an estimated loss of $7.2 billion globally. At a high level, this comes from two sources:
- Automated clicking of ad links via “bots”
- Human-based repetitive clicks of ad links
In both cases that effect is the same: A user clicks an ad link thereby generating a charge from the ad provider (as nearly all ad platforms are pay per click). This happens regardless of whether the customer actually viewed the website. This is an important distinction, as the ad platform charges customers based on the link being clicked, not if the page itself is actually viewed—or if the customer actually engages with the site. Once that link is clicked, the cost is already sunk—so “blocking” bots from the page itself has no material impact on the actual wasted ad spend.
Bot detection
That said, identifying the amount of traffic that is automated bot traffic is incredibly important to customers and marketing partners. So starting today, Edgemesh Server has integrated bot-detection systems—including the award-winning Cloudfire Bot Management system —into every page request … with zero performance impact on the page load. But of course, not all bots are bad bots —crawlers like the GoogleBot are critical to site performance, and Edgemesh Server differentiates between verified bots (good bots) and non-verified bots (bad bots).
Introducing BTAG (bot tagging)—a new approach to bot protection
The problem with blocking bots? There’s a chance you might end up blocking real user traffic. Compounding this is the fact that, for the purposes of ad fraud prevention, blocking the bot has no impact on your wallet either. So blocking these requests at the Edgemesh Server layer was a non-starter.
Instead, we’ve implemented a solution we call BTAG or bot tagging. BTAG is the core feature of our new AdFraud Analysis and Protection System. Here’s how it works: Unlike blocking, BTAG uses the bot information to seamlessly redirect the request to a special subpath called /em-cgi/btag. Since Edgemesh Server can return pages on any route, this allows the originator of the request to receive the page they requested—yet do so on an identifiable subpath.
Best of all, this page now runs every marketing tag enabled on the site, ensuring that every ad platform you use today collects information about this user having visited the BTAG version of the site. This provides a simple way to identify bad actors and exclude them from future targeting—a feature we call de-targetting (see more below).
Since the BTAG requests are all specific, you can see the impact directly in Google Analytics using a segment available here. Like all Google Analytics segments, you can see the subset of bot-based users across every view, including the acquisition channel!
Bad people detection
Bot detection is a fully automated solution, and can be done online—meaning, in the moment. But detecting if someone is routinely clicking on an ad—when they are in fact a real person—is a lot more difficult.
Edgemesh Server includes a robust method for determining if a visitor is “harvesting” ad clicks—even if they’re doing so by hand. By analyzing the network used (VPN, datacenter, etc.), combined with the history of that user’s action, Edgemesh Server automatically detects users who are continually clicking on your ads but NOT generating conversions. Edgemesh Server keeps track of the ad-click rate per visitor, and once crossing a threshold, will tag that user and IP address as it’s generating fraudulent clicks.
In the example below, we can see a single device clicking on 7 distinct advertising links: 6 from Google and 1 from a Facebook campaign. Although the bot score shows this user is not a bot, the short session times (less than 10 seconds) combined with additional network information has flagged this activity as likely fraudulent.
Taking a look at the details of this IP address, we can see that it is a known source of nefarious activity.
Over the next few weeks, we’ll begin integrating this IP-based information with the online BTAG system—allowing customers to identify not only bots, but also known IP sources of fraud activity.
By combining these two detection methods, we are able to cover the two major sources of ad fraud—bad bots and bad people. The IP reputation information is continually updated, and combined with our internal heuristics on click rates and lack of engagement (e.g., users who continually click ads, but have no cart or product activity). Below are some examples of these two main metrics put side by side.
Putting it all together: How to prevent bot traffic and bad traffic from ending up as bought traffic
With this new system in place, the question is—can we generate enough fleetwide intelligence to develop a known database of places where not to show ads and/or use the BTAG system to automatically adjust our targeting? With more than 100 million requests hitting the front door across our customer stores, we now have a meaningful level of credible data to use.
Best of all, by unifying data across the fleet, fraud actions committed against one customer will power future protection for all customers—providing a phalanx of protection for partners across the fleet.
Defending with IP exclusion lists
As the fraud-based IP addresses are identified, each day, Edgemesh will provide a unified set of IP addresses to add to IP exclusion lists. The only way to truly combat ad fraud is to stop showing ads to people who commit fraud. IP exclusion lists provide some force majeure protection against fraud by restricting where your ads can be displayed.
The fleet-wide IP exclusion list is updated each day at midnight UTC. Today, this data set can be manually entered into advertising platforms that support IP-based exclusions (Google and Bing). In the near future, we hope to automate this update, as the platforms have defined hard limits for the total number of IP addresses that can be added (500 for Google, and 100 for Bing). The process for Google is available here, but at the end of the day IP exclusions lists are limited due to their small size.
Real-time defenses with de-targeting
At the crux of most effective advertising platforms is a robust retargeting system. Most systems today are pixel-based—and through the use of cookies and/or historical IP information retargeting provides a complete platform to show ads to anonymous visitors who have triggered some action (such as visiting a specific page).
One of the major advantages of pixel-based retargeting is that it’s timely—retargeting systems are notoriously fast (seconds) as they form the basis for recapturing lost opportunities and can be specific to a particular page on your site, and behavior-based.
Finally, retargeting is a major component of audience building—allowing marketeers to craft specific subsets of users to whom to show specific advertising messages.
Edgemesh’s BTAG system allows you to craft exceptions to audiences, campaigns and even display ads by leveraging this vast retargeting apparatus. By adding an exception to your audiences that do not show ads to customers who have visited store.com/em-cgi/btag/* —you can effectively use the ad platform's own retargeting smarts to remove, or de-target visitors who have displayed fraudulent behavior.
De-targeting provides the following major advantages over IP exclusions lists:
- It’s trivial to operate; a one-time configuration per platform adds exclusions for audiences
- It works across any platform that supports retargeting (Google, Bing, Facebook, Adroll etc.)
- BTAG ensures all the pixels used by your site fire on the subpath, automatically
- It actively moves ad dollars away from wasted/fraudulent visitors and back toward the rest of the audience
- It improves automatically over time, as more bad visitors are redirected to BTAG subpaths
Starting today, Edgemesh customers can add exceptions to their audiences for all BTAG-based views. This will only exclude the most egregious bot-based actions.
In the coming months, we’ll be merging the IP exclusion logic into our BTAG system—giving you the best of both worlds.
