[Q4 2014]
akamai.com
= bots, spiders & scrapers
• Prevalence of third-party content bots is increasing • Used to collect data from other websites
• Poorly coded bots can impact site performance • May resemble a denial of service (DoS) attack • Secure web presence requires a comprehensive understanding of how they affect performance [Download the Q4 2014 Global DDoS Attack Report for supporting data and analysis]
2 / [The State of the Internet] / Security (Q4 2014)
= purposes of bots and scrapers
Akamai has seen bots and scrapers used for many purposes, such as: •
Setting up fraudulent sites
•
Reuse of consumer price indices
•
Analysis of corporate financial statements
•
Search and metasearch engines
•
Data mashups
•
Analysis of stock portfolios
•
Competitive intelligence
•
Location tracking
3 / [The State of the Internet] / Security (Q4 2014)
= example of a bot targeting a financial aggregator
In this example, the bot targeted the financial aggregator to scrape a large amount of data quickly.
Figure 1: Bot targeting financial aggregator to scrape large amount of data quickly
4 / [The State of the Internet] / Security (Q4 2014)
= four categories of bots and scrapers
Figure 2: Desirability is based on how much the site owner wants to host the bot. Aggressiveness the rate of requests from the bot and its impact on site availability.
5 / [The State of the Internet] / Security (Q4 2014)
= triage and categorization
• Mitigation techniques vary depending on bot classification • Volume of requests can help determine platform • The sequence and pages a bot targets can reveal intent • The user-agent header can sometimes provide a unique and identifiable user agent • Additionally, Whois can sometimes identify bot owners
6 / [The State of the Internet] / Security (Q4 2014)
= corresponding mitigation strategies
Figure 3: Mitigation strategies are based on bot desirability and aggressiveness
7 / [The State of the Internet] / Security (Q4 2014)
= moving forward
• Bots and scrapers will continue to affect organizations in all industries • Every website should develop a strategy to address and mitigate undesirable bot behavior • Each organization will also need to a framework to evaluate which bots to allow access to it site • Companies will also need to analyze and modify security policies to keep them current with the changing times
8 / [The State of the Internet] / Security (Q4 2014)
= Q 4 2014 global attack report • Download the Q4 2014 State of the Internet Security Report • The Q4 2014 report covers: / Analysis of DDoS attack trends / Breakdown of average Gbps/Mbps statistics / Year-over-year and quarter-by-quarter analysis / Types and frequency of application-layer attacks / Types and frequency of infrastructure attacks / Trends in attack frequency, size and sources / Where and when DDoSers launch attacks / Case study and analysis
9 / [The State of the Internet] / Security (Q4 2014)
= about stateoftheinternet.com
• StateoftheInternet.com, brought to you by Akamai, serves as the home for content and information intended to provide an informed view into online connectivity and cybersecurity trends as well as related metrics, including Internet connection speeds, broadband adoption, mobile usage, outages, and cyber-attacks and threats. •
Visitors to www.stateoftheinternet.com can find current and archived versions of Akamai’s State of the Internet (Connectivity and Security) reports, the company’s data visualizations, and other resources designed to put context around the ever-changing Internet landscape.
10 / [The State of the Internet] / Security (Q4 2014)