QCon San Francisco Day 2 Recap

Day 1 was a great start to the conference. Here we are on the second day of the conference.

Polyglot Persistence

The first talk I went to was Polyglot Persistence from Roopa Tangirala from Netflix. She covered 5 use cases of data persistence at Netflix, the challenges and the approaches to the data problem.

One of the promises of microservice architecture is that you’re no longer bound to a single or homogenous data store. Each component in the microservice stack can choose their own data store according to their use case and what Roopa has demonstrated here was a prime example of such promise.

Data stores

The data stores she talked about in the talk were:

  • Elasticsearch: For search analyze and visualize in near real time
  • EVCache: Distributed in-memory caching solution based on memcached
  • Cassandra: Distributed nosql database to handle large datasets providing high availability
  • Dynamite: Distributed dynamo layer for different storage engines and protocols supporting Redis, memcached, RocksDB
  • TitanDB: Scalable graph database optimized for storing and querying graph datasets

CDN URL service

requirements: - high availability - very low r/w latency (less than 1ms) - high throughtput per node

EVCache is chosen. In-memory so it provides very low latency.

Playback Manifest service

requirements: - quick incident resolution - interactive dashboards - near realtime search - adhoc queries

Elasticsearch is chosen. Powerful search & analytics and interactive dashboards.

Viewing History

requirements: - time series dataset - support high writes - cross region replication - large dataset

Cassandra is chosen. Multi-datacentre, multi-directional replication and HA and highly scalable.

Digital Asset Management

requirements: - one backend plane for all asset metadata - storage of relationships/connected data - searchable

TitanDB. Distributed graph db and support for vaious storage backends.

Distributed delayed queues

requirements: - distributed - highly concurrent - at-least-once delivery semantics - delayed queue - priorities within the shard

Dynamite. Pluggable data store supporting Redis and multi-datacentre replication. They tried to use cassandra as a queue but it’s not great at being a backend for queues.

Challenges & Approaches

  • Capacity planning
  • Maintenance
  • Monitoring


  • Subject matter experts (2 or 3). Work with microservice team to come up with requirements.
  • CDE service (a self-serving service to provision data stores)
    • thresholds/SLAs
    • cluster metadata
    • self service
  • Proactive maintenance
    • don’t wait for amazon to terminate the instances
  • Upgrade
    • Use NdBench to build up confidence
    • Open source and have pluggable backends

$200 Self-Driving cars with RasPi and Tensor Flow

This talk has less relevance to my day-to-day job but more of a curiosity of mine. However, I didn’t get too much out of it. The project itself is pretty cool and it sounds like something you can DIY because all the components are low-cost and instructions are freely available.

Istio - Weaving the service mesh

Service mesh seems to be the next battle ground after the container orchestration layer. Since Kubernetes is “winning” the orchestration war, the battle has gone up the stack.

Attitude determines altitude

comments powered by Disqus