Monitoring Manifesto

Software development keeps on being pushed everyday – Agile, DevOps, Kanban, TDD, BDD and so many other practices keep on evolving to let us create applications faster and better than previously – could you really imagine the current scale in the 1990s? No one did and when we compare it with established engineering professions like architecture or mechanical it is just a baby.

So we keep on improving on how we establish our models in this emerging class of engineering, we create applications and we make them in an incremental and sustainable way – or do we?

We dropped the waterfall model a long time ago, redefined how to acquire requirements and broke down tasks in ways that can be scheduled and rethought but in reality we’re still stuck in the big upfront calculations.

The system is specified to death (1) and tasks are decided based on personal experience from developers or management building a theoretical model.

But is it really possible to build a theoretical system upfront for software? Looking into biology and real life systems we take into consideration that all the elements that interact with what is being studied and then try to build a model that represents what can be observed – we model its features and representation around what we log.

This means we’re bypassing the scientific method (2), we do not take into consideration real world observations and the methods involved in it which will always be better than any theoretical model – Can you really second guess every physical issue that a disk has, all the servers you’re using, all the application deploys, all the sick days that the development team takes? Because that is what we’re now doing on several levels.

None of this is new, we see management keeping tabs on deadlines or Ops keeping track of server performance.

But somehow development is fixed on what cosmologists call the Goldilocks Effect (3) where ‘everything is just right’ where every part of the system works perfectly and is designed for it.

But if we had the data we could use it for benchmarking, performance testing and even research (4) – We are less likely to be surprised if we can see how events accumulate into dynamic patterns of behaviour (5).

(1) The Pragmatic Programmer: From Journeyman to Master by Andrew Hunt, David Thomas

(2) http://en.wikipedia.org/wiki/Scientific_method

(3) A Short History Of Nearly Everything by Bill Bryson

(4) The Art of Capacity Planning: Scaling Web Resources by John Allspaw

(5) Thinking in Systems: A Primer by Meadows. Donella