Downloaded 12 times
Uploaded bySylvain Zimmer
Ranking the Web with Spark
This document discusses ranking web pages using Apache Spark. It begins by introducing the speaker and their background. It then provides an overview of how search engines traditionally work, including crawling, indexing, and ranking pages. It discusses using static features like URL depth and dynamic features like click-through rates to calculate page scores. The document proposes using Spark to perform learning to rank by training models on features and user data to improve results. It also demonstrates calculating PageRank on the Common Crawl dataset using GraphFrames in SparkSQL. Finally, it provides links to learn more about the Common Search open source project.
More Related Content
Similar to Ranking the Web with Spark
![[fr] Introduction et Live-code Backbone.js à DevoxxFR 2013](https://cdn.slidesharecdn.com/ss_thumbnails/backbone-devoxxfr-130327123943-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
Ranking the Web with Spark
- 1. Ranking the Webwith Spark Apache Big Data Europe 2016 [email protected] @sylvinus
- 2. /usr/bin/whoami • Jamendo (Founder& CTO, 2004-2011) • TEDxParis (Co-founder, 2009-2012) • dotConferences (Founder, 2012-) • Pricing Assistant (Co-founder & CTO, 2012-)
- 3.
- 5.
- 6.
- 7.
- 8. Disclaimer: IANASRE (I AmNot A Search Relevance Engineer)
- 9. What's in ascore score = fn( doc, query, language, user, time )
- 10. What's in ascore score = fn( doc, query )
- 11. What's in ascore score = fn( static_score, dynamic_score ( query ))
- 12.
- 13. Static features • Scopes: •Page: URL depth, markup stats, ... • Domain: Age, page count, blacklists, ... • WebGraph: PageRank, ...
- 14. http://infolab.stanford.edu/~backrub/google.html The Anatomy ofa Large-Scale Hypertextual Web Search Engine (1998) Crawler Indexer Database SearcherRanker
- 15.
- 16. Dynamic features • Textmatch: TF-IDF, BM25, proximity, topic, ... • Query-level: number of words, popularity, ... • Usage: clicks, dwell time, reformulations, ... • Time
- 17.
- 18. Users Database Elasticsearch Indexer Python, Spark Data sources CommonCrawl, Alexa top 1M, ... words, static score query top 10 docs, final scores Offline Online Searcher Go
- 20.
- 21. Issues with thisarchitecture • Static & dynamic scoring are in different codebases • No control over result diversity • Hard to optimize • Very dependent on Elasticsearch
- 22.
- 23. Users Database Indexer words, static score,features query Searcher top 1k docs, features Rescorer final 10 docs
- 24. Issues with rescoring •Latency • Pagination • Harder to explain
- 25.
- 26. LTR Model • Features •Training dataset • Evaluation: NDCG, ERR, ... • Algorithms: AdaRank, ListNet, LambdaMART, ... • Learning with Spark!
- 27. The right questions •What do users expect? • What features? • How to evaluate and fine-tune in the real world?
- 28.
- 30.
- 31.
- 32. Common Search Pipeline Docsources Common Crawl, WARC files, URLs ... Filter plugins Document parsing Output plugins Data output Database, file, HDFS, S3, ...
- 33.
- 34. Dumping the webgraph
- 35.
- 36.
- 37.
- 38.
- 39.
- 40.
- 42.
- 44.
- 46. Spamdexing • Keyword stuffing,hidden text • Scraper sites, Mirrors • Link farms • Splogs, Comment spam • Domaining • Cloaking • Bombing
- 47.