How Pinterest Built an Experimentation Data Engineering Pipeline That Scales to 500 Million Users

Components

The first two applications the data flows though are Apache Flink and Kafka. They communicate with each other using a read committed mode, which reads only events that are checked to be valid. If the value is not committed due to Flink malfunctioning, the events will not be read, and Kafka will wait until Flink recovers. This process ensures Exactly-Once-Semantic processing is met across the whole system.

There are three other technologies used in the system:

• AWS S3: S3 is a highly scalable and reliable file store. The alternative is to store the data locally but AWS offers many benefits like automatic encryption of the data, automatic scaling and reliable uptime. Using the cloud enables consistent access and management of the data across the whole organisation.
• HBase: HBase is designed for big data so has extremely fast read/write operations and supports regular overwriting and inputting of data. The alternative is Hadoop Distributed File System(HDFS) which has a more rigid structure and doesn’t support dynamic storage. Because we want to run different experiments on the data at different times, it’s not suitable.
• Spark Workflow: Compared to Hadoop MapReduce, which reads and writes data from disk, Spark executes jobs much quicker by caching data in memory across multiple parallel operations. This gives Spark the edge in parallelism and better CPU utilisation.
• Airflow: A great workflow orchestration tool allowing you to easily author, monitor and schedule data pipeline workflows. An alternative, which Pinterest uses, is Pinball. Pinball is built in Python2 and was last updated 2 years ago compared to Airflow, which runs on Python3 and gets daily updates. Airflow has a huge since being backed by the Apache foundation so has become the industry standard for orchestrating workflows.
• Prometheus: Prometheus can ingest exported metrics from many different systems. It has inbuilt monitoring and allows querying of data for real-time analytics. Originally built by SoundCloud, it is now open-source, low-latency, robust and highly scalable - capable of serving millions of metrics every second.
• Grafana: Grafana can easily edit and create new dashboards for any metrics exported by Prometheus, allowing flexibility in what is monitored by the system. Grafana is an excellent tool for visualising data, it is well-equipped to make sense of complicated data using visualisations like heatmaps, histograms and geo maps. You can build multiple dashboards customised to your liking to serve both the batch and real-time processing layers. It is constantly being upgraded easily integrates with Prometheus.
• AWS RDS: RDS provides excellent scalability for the system, storage resources can be increased with just the click of a button or an API call. It is built on highly available infrastructure from Amazon Web Services and its speed can support even the most demanding applications. Using IAM Roles on AWS the resource can be controlled to give only the right users access to the system and all data is highly encrypted keeping user data safe.
• Fast API: Can currently support up to 7501 API requests calls per second, making it one of the best Python API frameworks. It’s high performance, intuitive to use and robust enough to produce production-quality code. Another option would be to use Flask but Fast API has some key advantages. For example, Fast API automatically produces documentation as you’re building an API, speeding up the development process. Fast API also integrates data validation through the use of the Python Pydantic library which creates more robust APIs.
• Kafka: Kafka is built with real-time processing, fault tolerance and scalability in mind. It can handle terabytes of data with little overhead. Kafka replicates data so can be used to send data to both the batch processing pipeline and the real-time pipeline.  
• Spark Streaming: Spark Streaming supports processing huge amounts of data with super low latency. It is the industry standard for streaming data in real-time. Spark streaming has many benefits including load balancing, fast recovery from failure, interactive querying and integration with advanced processing libraries such as SQL, machine learning libraries and graph processing.

Data Flow

1. Messages are logged from the Pinterest application servers and API and logged to Kafka. Kafka then streams the data to both the Batch Processing layer(3) and the Real-time streaming layer(2).
2. Kafka serves the data to Spark streaming which processes it in real-time and inserts the metric data into an RDS database. The RDS data is then exported to Prometheus which serves the Grafana dashboard. From the dashboard, data is monitored in real-time so the user is alerted to any failures in the pipeline such as inconsistencies.
3. Kafka sends the raw logs to S3. The nightly experimental workflows orchestrated by Airflow and performed by Spark process the data to determine experiment user groups and metrics. The output data from Spark processing is then inserted into HBase ready to be served to the analytics dashboard, Grafana, via Prometheus.  

Summary

This pipeline design meets the three main criteria; being able to serve metrics in real-time/batch, for a huge and rapidly growing userbase, and with the ability to extend the system.

The addition of Spark and its in-memory processing capabilities allowed Pinterest to reduce the time taken for experiments to run, from an average of four hours to well under two hours. This improvement in processing time allows teams to make decisions faster and run more experiments as a result.

The implementation of batch and real-time processing allows them to make quick and effective decisions on both historical data and the real-time incoming data. This leads to a more robust, versatile and easier to debug pipeline.

References

1. Wang, C, 2017, Scalable A/B experiments at Pinterest