Web Analytics and Bots: the long missing data point

TL;DR

Drop our invisible.js on your page and get bot scores on your analytics events.

No user interaction needed.

What percentage of your visitors are actually bots? You probably have no mechanism to answer that question, and with the rise of stealth automated browsers and bespoke captcha solvers in recent years, even planting a disruptive challenge on your site’s door is not a reliable measure anymore.

You hear about other sites being overrun by automated browsers and agents that scan, click and perform actions, but can’t tell whether that’s your problem or not.

Understanding your traffic

To gauge how many genuine visitors you have and what they do on your site, the initial (and naive) approach is to start reviewing your server logs,

but you quickly realize a kind of white-noise continuous scanning of URLs like /wp-content/themes/admin.php, along with suspicious spikes from hundreds of IPs that simply make it impossible to get a clear view of what your real visitors do.

The next approach is to generate analytics events from your web pages, sending a pageview event on page load either manually or by embedding a third-party JS tag. It improves the situation by filtering out those simple scanners that never really load a page, but modern crawlers and agents are stealthy, automated browsers that actually execute your JS and continue polluting your logs while doing what they please on your site.

Better analytics with invisible bot detection

Truesign extracts hundreds of data points from each visit and is able to identify even the most stealth scraping services and automated browsers, with no user interaction and no tracking needed. One of the ways customers start to use Truesign is by creating non-blocking site rules to simply enrich the information they get about their users:

They obtain a token transparently for each visitor and attach it to the events’ payload, then on the server extract bot and anonymity scores from the token and enrich the analytics logs with those values.

<script src="https://edge.truesign.ai/v2/invisible.js" async></script>
<script>
  async function sendPageViewEvent() {
    // your visitor obtains a token from truesign
    if (!window.truesignReady) {
      await new Promise(res => (window.truesignReady = res));
    }
    const truesignResult = await window.truesignGetToken(12345);

    // gather visitor information and send the event (sample endpoint)
    await fetch('/analytics/pageview', {
      method:'POST',
      body: JSON.stringify({
        token: truesignResult.token,
        ...visitorInfo
        })
    });
  }

  document.addEventListener('DOMContentLoaded', sendPageViewEvent);
</script>

Once they get an idea of what’s happening on their site thanks to this extra information, they may decide to disallow suspicious visitors from taking certain actions, show them stale data or enqueue their requests for further ML or human review.

Advantage over alternatives

Small details often make the difference:

• our bot detection doesn’t require challenges or user interactions
• it’s available immediately on page load, unlike other solutions that require a stream of scroll/click/mouse events to perform an assessment
• no PoW, it doesn’t waste CPU resources computing unnecessary stuff
• no tracking, you receive the “what” not the “who”

The real world

A customer, owner of an e-commerce site, wanted to understand the extent of automation and suspicious activity occurring on their site without annoying users with challenges.

A few days after embeddeding invisible.js on their site, logs clearly showed continuous bot activity on their site, scraping their inventory and prices, with spikes of bot activity right after certain business events. They went on to block bot traffic from reading prices and making purchases, which resulted in:

Depending on the nature of their business, our clients typically observe that between 20% to 60% of their traffic is suspicious, changing their perspective from “I may have some fake users” to “I have an infestation I couldn’t see”.

The aftermath

Today’s automation is stealthy and affordable, even open source tools can nowadays evade most detection methods and gain free access to “protected” sites. At Truesign we have developed one of the most sophisticated detection technologies available, enabling us to uncover a wide range of abuse vectors like automation, IP anonymizers, fake email addresses and spikes of suspicious activity.

With Truesign, it becomes possible to understand what your visitors are, transparently without disrupting the user experience. Content and APIs will continue to be scraped, forms will get abused, but now you have a tool to help tackle these challenges.