End-to-End Monitoring

How do you bridge the gap between IT and Business?


Personal experience confirms that it is very common for large IT infrastructure teams to work in silos, and yet manage to support business operations in a stumbled, finger-pointing driven, best effort manner. “Eppur si muove”!

Although, somehow, this IT management model is able to support the business at the end of the day, the inefficiency of its operation oftentimes leads to frustration on all sides, and the company that funds both IT and Business Operations ends up being ultimately at a loss. 

If only there would be a way to bridge the gap between the business and the isolated support groups (network, OS, storage, DBAs, and even HelpDesk) to allow for an orchestrated collaboration into supporting the IT infrastructure and its consumer: the business. Many companies are willing to invest in technologies that can produce quantifiable benefits, and can transform a hap hazarded operational model into a professional, disciplined and organized structure. The End-to-End Monitoring paradigm is, in essence, the key to such organizational dream.

Lack of information is far too common nowadays, yet the paradox of living in an information-rich era. Lack of information is what creates chaos in any organization.  As an example, many IT operations stumble from time to time with persistent or generalized performance issues, and at the pressure of IT management, IT staff is often hard press to take a decision over the likely root cause of a performance issue, and consequently create compelling business cases in order to obtain the funds necessary to alleviate performance issues by innefectivelly throwing more hardware at what, most of the time, is a software issue. An End-to-end monitoring solution is aimed at extracting raw information from beneath the surface of an IT infrastructure swamp. A well architected solution should be able to correlate information accross tiers and functional domains, and process the information into a meaningful data – I cannot stress this enough! The resulting information provides value by allowing for informed decisions to be taken at both business and IT levels. A large component of ROI is achieved into this manner.  This is also what, ultimately, ties the teams together, and all effort becomes information driven, and more organized.

At True Consulting, LLC, I’ve worked for one of the largest global restaurant chains in the world, architecting such an End-to-End Monitoring solution. The field, which falls under the Application Performance Management (APM) domain, is now mature, and there are a plethora of puzzle components to play with to architect a competent yet cost-effective solution. The design took into consideration all five dimensions of a complete and effective APM solution:

1. End-User Experience Monitoring (from browser-level)
2. End-User Transaction Profiling and Tracing (cross-tier correlation from web, application and database tiers)
3. Component Modeling and Analysis (a key component for rapid fault domain isolation, especially useful in complex infrastructure deployments)
4. Drill-Down visibility into virtual run-time environments (for low level insight, from heap behavior, to JMS queues and encoded functions)
5. Cross-Tier Correlation (to make sense of all the data and be able to take appropriate action in a timely manner)

It is my encouragement to all future or current APM architects to pay close attention to these 5 important dimensions, as, in practice, the lack of each of these can severely cripple the effectiveness of the solution architected.