Puppet Camp Paris: Improving Operations Efficiency with Puppet

  • HOME
  • >
  • NEWS
  • >
  • Puppet Camp Paris: Improving Operations Efficiency with Puppet
Published: , Last Updated:
May 4, 2020

During Puppet Camp Paris, I got the privilege to present the Continuous Delivery Workflow of TubeMogul's Operations Engineering Team. In few years, we went from few servers to over two thousands nodes fully managed by Puppet. In our presentation, we went over the challenges we faced as well as the implementation of our workflow to improve our day to day operation while still moving fast.

With our operations continuous delivery workflow using Git, Gerrit, and Jenkins, we have been able to manage over 10,000 Puppet deployment in 2014.

TubeMogul

  • Enterprise software company for digital branding
  • Over 27 Billion Ads Served in 2014
  • Over 30 Billion Ad Auctions per day
  • Bid processed in less than 50 ms
  • Bid served in less than 80 ms (include network round trip)
  • 5 PB of monthly video traffic served
  • 1.1 EB of data stored

Operation Engineering

  • Ensure the smooth day to day operation of the platform
  • Provide a cost effective and cutting edge infrastructure
  • Team composed of SREs, SEs, and DBAs
  • Managing over 2,500 servers (virtual and physical)

Five Years Of Puppets!

  • 2008 - 2010: Use SVN, Bash scripts and custom templates.
  • 2010: Managing about 250 instances. Start looking at Puppet.
  • 2011: Started with Puppet 0.25 then upgraded to 2.7 by EOY on 400 servers with 2 contributors.
  • 2012: 800 servers managed by Puppet. 4 contributors.
  • 2013: 1,000 servers managed by Puppet. 6 contributors.
  • 2014: 1,500 servers managed by Puppet. Workflow using Git, Gerrit and Jenkins. 9 contributors. Start migration to 3.7.
  • 2015: 2,000 servers managed by Puppet. 13 contributors.
Related news posts that you may like
What's New in Focal Fossa Ubuntu 20.04 LTS?
Ubuntu 20.04 LTS is available for download. Find out some of the major security and performance improvements from this new distribution, including the introduction of WireGuard VPN, upgraded suite of software packages, faster boot time, etc.
What's New in GNU Bash 5?
Bash version 5 is generally available and comes with some important improvements and new features like BASH_ARGV0, EPOCHSECONDS, and EPOCHREALTIME.
Top 5 Machine Learning and Self-Healing Techniques used by SRE
Applying Machine Learning and Self-Healing techniques to the day operations of a production system has become common practices for most SREs. This post cover some real production use cases like automated failover, forecasting, anomalies detection, risk classification and so on.