Twitter's Recommendation Algorithm
What is the project about?
The project is Twitter's Recommendation Algorithm, a system of services and jobs that generate feeds of Tweets and other content for various Twitter product surfaces like the "For You" Timeline, Search, Explore, and Notifications.
What problem does it solve?
The algorithm solves the problem of surfacing relevant and engaging content to users from the vast amount of content generated on Twitter daily. It helps users discover Tweets, topics, and accounts they might be interested in, even if they don't directly follow the content creators.
What are the features of the project?
- Candidate Sourcing: Retrieves Tweets from various sources, including in-network (people you follow) and out-of-network sources.
- Ranking: Uses machine learning models (light and heavy rankers) to score and rank candidate Tweets based on relevance and user engagement predictions.
- Mixing and Filtering: Combines Tweets from different sources, applies visibility filters for safety and compliance, and constructs the final timeline.
- User and Tweet Representations: Utilizes various embeddings (SimClusters, TwHIN) and graph features to understand user interests and Tweet content.
- Real-time and Batch Processing: Processes user actions and updates recommendations in real-time, while also using batch jobs for tasks like calculating user reputation.
- Recommended Notifications: surfaces recommendations to our users via notifications.
What are the technologies used in the project?
- Programming Languages: Scala, Java, Python, Rust.
- Machine Learning Frameworks: TensorFlow v1 (legacy), custom frameworks.
- Serving Frameworks: Navi (Rust-based), Product Mixer.
- Data Storage and Processing: Tweetypie (Tweet service), Unified User Actions (real-time stream), User Signal Service, GraphJet.
- Graph Technologies: Real Graph (interaction prediction), Tweepcred (PageRank), User-Tweet-Entity-Graph (UTEG).
- Build System: Bazel.
What are the benefits of the project?
- Improved User Experience: Provides users with personalized and engaging content, increasing their satisfaction and time spent on the platform.
- Content Discovery: Helps users discover new content and accounts they might not have found otherwise.
- Increased Engagement: Drives user interactions (likes, replies, retweets) by surfacing relevant content.
- Platform Health: Filters out harmful or unwanted content, contributing to a safer and more positive user experience.
What are the use cases of the project?
- For You Timeline: The primary use case is generating the personalized "For You" timeline, which is the main feed users see when they open Twitter.
- Search and Explore: Enhances search results and the Explore tab by recommending relevant Tweets and topics.
- Notifications: Powers personalized notifications, alerting users to content they might be interested in.
- Other Product Surfaces: Can be adapted to provide recommendations in other areas of the Twitter platform.
