Comments (0)

By Jason Linehan, April 23, 2013 6:14 pm

New techniques using Pathway Systems Blueprints and Amazon EC2

Growth of cloud-based IT infrastructure has accelerated in recent years, and the unique capabilities of these platforms continues to drive adoption. Amazon Elastic Compute Cloud (EC2), Google Compute Engine, and Microsoft Windows Azure Services Platform are just a few of the services at the leading edge of these developments. The advantages of cloud services — rapid deployment, dynamic scale, and distributed infrastructure — makes them ideal for building failure-tolerant systems to support critical applications.

Such critical applications require appropriately rigorous documentation, as their integration and maintenance are often highly prioritized. One of the challenges in managing cloud infrastructure is building a seamless and complete picture of these remote resources so that they can be incorporated into the existing context of an organization’s information architecture.

This challenge is in two parts. The first is that the resources are not only remote from the maintainer, they may exist in an entirely virtualized environment, spanning many physical locations. The second and greater complication is that the dynamic nature of cloud systems (“elastic,” to use the Amazon jargon) means that static documentation of these resources rapidly becomes obsolete. The inevitability of its decay tends to encourage half-hearted documentation, because the maintainers know how it will end, and don’t enjoy wasting their time.

Amazon EC2 micro-instance automatically synchronized using Blueprints

Rapid and constant changes are a feature of the cloud environment, not a bug. The Blueprints platform helps to smooth away the challenges, and turn them to the advantage of the maintainer. By interfacing with Amazon Web Services, Blueprints can automatically build a dependency map of the resources associated with an Amazon EC2 account. This dependency structure can be continuously synchronized to reflect the reported real-time status of your cloud components, essentially creating the documentation for you. Synchronized remote components such as EC2 resources can be seamlessly integrated with an existing dependency model, and co-exist with hand-maintained objects and relations in the usual way.

Of course, when the environment changes too rapidly, it can become difficult to get a sense of the evolution of the system over time. Maintainers see the latest configuration, and perhaps remember the previous one, but what might have happened in between, when they weren’t watching? This is one of the reasons that Blueprints maintains a complete history of changes to the model. Maintainers are able to call up the state of the model at any time in the past, or inspect changes over an arbitrary time span. These features apply to any resource in a Blueprints model, but are especially suited to automated information, which can change constantly, often in small but crucial ways.

The currents and trends of dynamic deployment and distributed IaaS present real challenges for traditional methods of documentation and maintenance, but the unique architecture of the Blueprints platform keeps you comfortably ahead of the curve, providing consistent, rigorous visual documentation that blends automatic and manual knowledge to help chart the changing seas of modern IT.

Comments (0)

Comments Off

By Daniel Evenson, January 30, 2012 8:43 pm

Dependency mapping is useful not just for the basic IT configuration items such as servers and services, but for all components supporting your critical IT services.

Here we’ve modeled some of the dependencies of how we host videos on this blog and the Pathway Systems website.

Since video streaming uses a lot of bandwidth, we host our videos using Amazon’s S3 cloud based storage service. The videos appear to come from our website, but really they’re streamed from Amazon so we don’t saturate our network connection with video traffic.

This amalgamation of disparate resources brought together to solve a technical problem is quite common especially for web-based solutions using cloud services. One pitfall however, is that it’s easy to lose sight of how this solution is architected.

On the cloud, out of sight is out of mind.

So we must ensure we have this solution documented. On the left, we built a quick dependency map of the major components involved. Folks reviewing this down the road will quickly understand the solution.

Starting at the top of the diagram, we see the website pathwaysystems.com depends upon a service we’re calling bulk-hosted-video. It’s not an actual service that runs on a server, it’s just a conceptual service that conveniently describes the overall service.

Below that, we see bulk-hosted-video needs dir video which is a sub directory of dir pathwaysystems which is hosted by us-east-1c and depends upon the service s3.

We’ve chosen to abstract Amazon’s us-east-1c availability zone as a server since that’s sort of how it appears to us as an end user. If we wanted to, we could create specific object types particular to Amazon’s nomenclature: AMIs, instances, regions etc… But for now, we’ll keep it simple.

The point of this article is to demonstrate how we don’t stop modeling the dependencies at the IT boundary. Instead we keep going and include all the big components that could cause this system to fail.

Our ability to use Amazon’s cloud resources requires a valid AWS account so we see that dependency too.
A specific of dependency of AWS is that it needs to be funded with money, so we’ve included the “debits” relationship as a blue dotted line to the credit card that’s used for auto-payment and the bank account from which the funds are pulled.

If you’ve got a business to run, it’s nice to be able to call up such a diagram on the fly when you’re considering such things as follows:

What service will be impacted if a particular credit card is canceled or expires?

What’s impacted if Amazon reports problems with the us-east-1c availability zone?

By modeling more than just IT specific components you can see the bigger picture, and do a better job of managing that service and ultimately your mission.

Comments Off

Comments (1)

By Daniel Evenson, April 7, 2011 5:50 pm

One of the more popular goals of dependency mapping is to map out all your applications’ dependencies. I guess they don’t call it application dependency mapping for nothing! Certainly that’s not the only thing we do with dependency mapping but it’s a great place to start.

Why? Because it gives us our first visualization of the complexity of the IT environment we manage. The results can be amazing at first, like a blind man gaining sight for the first time.

See the following set for an example:

Application Dependency Mapping Blueprints

Why do dependency maps shed more light than the typical IT systems or network diagrams we’ve been using all along? Let’s consider that for a moment. What is a typical IT systems diagram? I don’t think there is one. I’ve worked in IT for about 20 years, and still don’t understand half the works of art I see. I know I’m not alone. UML (Unified Modeling Language) is an effort to standardize IT modeling babel into a common pictorial language that we can all speak. But it appears to have jumped the shark–it’s overkill for most IT folks who are not in large professional development teams.

That’s where dependency mapping comes in. It’s ridiculously simple, but produces grand results. It’s a meta relationship between IT objects.

A needs B.

We don’t say why A needs B.

This is the secret to why this works so well; there’s danger in getting too detailed with the information you document.

If you want to see the big picture, you must stay high above the landscape.

For example, perhaps your timekeeping app needs database tracker001. Maybe timekeeping writes data there. Maybe it reads data instead. Maybe one object just uses another, or hosts it. There can be many reasons for a dependency but by keeping most of that lower level information out of the dependency map, it keeps our model high-level and consistent. IT configurations are diverse, and difficult to model in part because innovation is constantly changing the archetypes we use to represent our systems.

Fortunately, the notion of dependency between resources is timeless and relevant to everything from people to processes, to hardware and software. Tell me what you rely on, and who relies on you and I’ll understand your job, your risks, and how you fit into the big picture, whether you are a person, a database, or a SaaS service.

Model the needs and quickly move on.

As you apply this documentation method across the board, your systems will come into focus. You’ll see why some applications (or IT services) fail more than others. Why one costs more, or why one is so hard to shutdown, replace, upgrade, consolidate, explain, etc…

In other words, the visual application dependency map supports much of the routine work we do in IT, whether we’re troubleshooting a problem, moving datacenters, funding upgrades, or analyzing impact of natural disaster scenarios.

If the dependency maps are indexed with references to each object in a map, you can use the maps from top-down or bottom-up.

Did you just inherit responsibility to support a legacy app? Refer to the applications dependency map to gauge the work ahead. Planning hardware maintenance on a server? Look up that server in the index, and see what depends upon it. Hurricane expected in Tampa? Look up your dependencies upon Tampa.

The value derived from dependency mapping your IT environment produces immediate visibility and understanding, but the best part is that it pays off over time if you make this sort of meta-documentation a central artifact of your operations. By maintaining and using such models, you re-enforce attention on tracking your resources and their critical dependencies, reducing risks that an obscure dependency will catch you off guard when something fails.

Having worked for years as a Unix consultant, I sure wished more of my customers had dependency maps available before I dove into many uncertain projects booby trapped with dangerous, unseen dependencies. But it was that market need that drove the development of our Blueprints product which helps you build and use dependency maps.

No matter how you do it, dependency mapping is a great starting point for gaining control of what you currently manage and establishes effective systems documentation for moving forward.

Comments (1)

Comments Off

By Daniel Evenson, July 28, 2010 1:29 pm

In this post I’d like to go over a point made in a previous post, that dependency maps are not dataflow diagrams.

The most typical IT diagram you find is some variation of the dataflow diagram. How information flows within a system is typically shown with something like the following.

Here we see data flows as email is delivered. It’s routed from the external mail server, through the spam filter, and into the internal mail server. IT staff envision their world like this. Its a conceptual model that facilitates troubleshooting of email problems. But it doesn’t clearly show how email services might be impacted by various system failures. By adding dependency relations it will.

We’ve added a high level object to represent all email service, and also two intermediate objects to distinguish internal functionality from external. We can now see that an external email server failure will impact only external email functionality leaving internal services unaffected. Lose the internal email server however, and both external and internal email functionality is lost.

To simplify the diagram, remove the dataflow information to get the following:

Now we have a dependency map of our email service which shows more clearly than a dataflow diagram the result of various system failures. Since a dependency map shows only dependencies, it’s a simple diagram that’s easy to understand by anyone involved in IT: managers, sysadmins, application specialists and even users. It’s not meant to replace the dataflow diagram’s use within IT, but to better document IT knowledge to support business continuity and disaster recovery for a broader audience.

Comments Off

Comments Off

By Daniel Evenson, July 13, 2010 1:05 am

This post explains the proper way to describe the dependencies for a simple database backed application. What better example to use than this very blog!

Here’s how we might show the dependencies:

In the above diagram, the only relations we show are dependencies (the green dashed line). The service-to-server relation is also shown as a dashed green line. But there’s a better way to visualize the hosting dependency, so the following diagram improves the visual aspects of the model by showing all hosting dependencies as one object contained within another. It’s the same info, but much more readable.

Let’s go over why we describe the dependencies like we do. The highest level object in this diagram is the application dependencymapping.com. Here we use the term application to describe a conglomeration of IT components which together provide a very high level service to our users, in this case the blog site itself. We use service objects to describe lower level components which run on physical or virtual servers.

The application dependencymapping.com directly needs 2 things, the Wordpress service and the wp_depmap database. Notice how we don’t say Wordpress needs wp_depmap. Even though the two are associated with one another, there’s no direct dependency between them. We can conclude that by asking, “If Wordpress failed, would wp_depmap fail?”, or the converse. In both cases, the answer is no, neither object would cause a failure of the other, but it would cause the higher level blog application to fail.

Let’s take a quick look at how useful this diagram is. As simple as it is, it tells us a lot. We easily see there are two physical servers involved, and one virtual server. Its obvious how complicated this application deployment is (not very). And we can see how a failure of any component, or perhaps botched maintenance like an upgrade could cause an outage. This simple diagram should be readable by both technical and managerial audiences, and can support many important IT operational decisions.

Comments Off

Comments (2)

By Daniel Evenson, July 12, 2010 1:59 pm

This is a basic dependency map of a couple of business processes mapped to the supporting IT systems. The dashed green line in the diagram denotes a needs relation between two objects.

In this example we see just two business processes, but for a typical company there could be a hundred or more. It can be a big task to accomplish so here’s some tips to keep in mind as you get going.

Don’t get too detailed early on

A common mistake we see when our customers embark on mapping their dependencies is to put too much information in initially. They get twisted up on the complex details of things such as virtualization, clusters, and load-balanced configurations. In general, research and map out the high level stuff initially, like process to application or application to service. Just like making a geographic map, make the large scale first, then fill in the details later. It takes a bit of experience to know the best way to represent dependencies, and that experience will come the more you do.

Focus on what you know first

Get the hang of modeling dependencies with the information you already have in your head before you get others involved. Odds are you’ll revise your own models a few times before you get the hang of it, so it’s better not to confuse others as you’re learning yourself. Most of our customers come from an IT background and are usually an expert on some applications or systems, and can start there.

Don’t go all BPMN on anybody (or have them do it to you)

Business process modeling notation (BPMN) is typically used for detailed modeling of business processes to better study the business for process re-engineering. If you’re looking to map dependencies to support impact analysis, or change management, or support business continuity and disaster recovery, you don’t need to get as detailed as BPMN would have you do. Sometimes the subject matter experts (SMEs) you speak with will be well versed in BPMN and may have models available. Explain that you’re concerned with just the high level dependencies initially and use the BPMN models to identify them.

Beware the data-flow diagrams

Speak with IT centric folks and they’ll give you a Vulcan mind-meld of data-flow information, describing the intricate details of how data flows within a system. Like a BPMN model this can make your head spin, but again it provides the basis for much of the dependencies. Use their expertise to distill the critical dependencies, and record them in your model. Use the simple interrogative of “If this were to fail, would it cause that to fail?” It’s a simple question that helps identify the existence of a dependency, and its direction.

Integrate with operations like change management or production acceptance

To really benefit from dependency mapping, the knowledge recorded must be published and used for a higher purpose than simply to keep you busy, or scratch someones itch for such a project. The reason for tracking, recording, and publishing dependency information is usually to make to some lasting improvements in systems uptime, maintenance, or recoverability. To do this, dependency knowledge must be available at strategic decision points. The most frequent bang-for-the buck comes from reviewing dependency models during change management and production acceptance meetings. In those meeting, the dependency maps clearly show what might be impacted by changes and who should be involved in approving a change or acceptance request. This regular use of dependency information also reminds participants to contribute to the body of knowledge.

Comments (2)