How to create a "dependency graph" for IT assets

Question

One of my customers is trying to create an interactive "matrix" of interdependencies for the various applications used in their company (it's a travel&leisure company with around 2500 employees).

The idea (still at the prototype stage) is to create a sort of Map, based on Visio or similar tool, which traces the communication and interdependencies between all the IT assets in the company, so that when someone asks for a change they can get an overview of the impacts.

This was mentioned in a casual setting and it will not be my responsability to directly work on this, but I did contribute the little I know already in terms of vaguely related methodologies (Zachman Framework).

I'd like to hear from the people in here if they know of methodologies, or tools, that may help this kind of effort, and if they have any specific experience to contribute. I'll digest the answers and send the result back to my customer, hoping this may be of some help with their task (which I consider a bit visionary and prone to all the pitfalls of any documentation project, but still well worth pursuing).

N.B.: The question is not "I have all this data I collected about the IT assets and I am unable to afford Visio or Google for Graphviz or convert it to a MindMapping tool or create a custom navigator using Jgraph etc.etc.". The problem is "how do I collect the relevant/useful information, and how should I organize these, considering I may have to update the data regularly, due to interface, version, and package changes?"

This is not much of a Visualization problem, or not yet. We have to get to a good start with data collection and organization first. If you want to suggest a tool, it must also include the data collection and management part (e.g.: Rational System Architect). But in that case, please suggest it if you have some actual experience, or if you are pretty sure it's quite niche and not very well know (I can Google, thank you). If you want to suggest some books/methodologies this is helpful too (I know only of the Zachman Framework, and not sure it's really a good fit).

"Just create an Excel file" or "You could use SmartDraw, man!!!" don't help much, I am afraid.

---

Found Iteraplan which looks a very good fit!

Answer 1

Have you tried using Graphviz?

It can draw out a graph based on dependencies in a text file. Simple!

You may want to start with the Graphviz Gallery for basic examples and resulting diagrams.

Answer 2

Visual Studio 2010 has a nifty new feature called "Architect Explorer" that allows you to generate graphs showing dependencies of different .net classes.

It may help if you tell what technologies you're using.

Answer 3

I am not sure if you need to capture just the relationship, or also the role (depends on vs. has as a dependent, as in a parent-child relationship). The following seems to do everything you need:

Answer 4

Correct me if I'm wrong... You're looking to use a diagram as a tool to trace the dependencies of a system.

If that's the case the what you'd use to graphically map the layout would probably be a Deployment Diagram in UML.

You map out the systems/servers/physical assets as box objects, then inside of those you map out the various applications, databases, components, and their inter-relationships with one-another.

The problem with using UML is, programmers focus mainly on class diagrams (because of their direct relationship to data modelling within the software), so it's difficult to find 'good' resources and examples on the non-class UML diagrams.

I have used this once in the past to map a relatively complex cross-system cross-application implementation and get my ideas down in a manner that could be shared with other developers while designing the system. In short, it was simple and it did its job well.

To create the diagram I used dia. If I had to do it all over again I would definitely use Visio because it's a lot easier to find pre-made stencil/template packages to diagram with online, and if those don't have what you need it's very easy to roll your own stencils in Visio.

Note: I have a lot (hundreds of hours) of experience in Visio doing electrical designs so I would consider myself familiar enough with the tools to give an objective comparison.

The downside to adapting a well-specified format is:

• they're usually overly-complex
• they usually have a strict format or set of rules
• it's easy to find yourself trying to mould the system to the diagram, it should be the other way around
• diagrams can easily grow to be too complex to comprehend/understand by anyone other than the person who created them

My suggestions are:

• make the diagrams as simple as possible
• don't be afraid to break the rules
• make the diagrams as simple as possible
• do everything in your ability to keep the diagrams as straight-forward and simple as possible
• ... you get the point.

If a developer who has never seen the diagram can't start to decipher the layout and meanings within the first few minutes of looking at it, it will probably do more harm than good.

I would consider the Zachman Framework to be an bad option because:

• it's difficult to understand the 'point' the diagrams are trying to get across
• the diagram formats are too complex
• the rules are in an inflexible format that limit, not complement the system's design

While you're researching options ask yourself this. Do you understand the point of the diagram?

The problem with diagramming a design is, if done wrong it will create more headaches then its worth and nobody will use it. No design document is usually better than a bad design document.

I hope that helps.

Answer 5

Maybe good solution to store your data is to write simple custom web application. If you have requirements pointed clearly it should be a piece of cake. It should be cheaper than buying some customizable huge tool but may take some time to develop. From my experience sometimes is better to do something that suits your needs exactly as you want rather than learning about some new, general-purposes tool that was designed to do everything and nothing.

About visualization: I checked various tools for UML and related diagrams and I found 2 that are very good for most of tasks (very intuitive, easy to use and time-saving).

1. VisualParadigm - corporate stuff, a lot of diagrams, powerful with quite easy interface recommended for your case I think. Might also have support for maintaining data you deal with.
2. UMLet - my favourite one:) very simple and can do most of things that big players can.

I not advice Visio for this case, I found it is useful for small diagram but your case doesn't sound like a small one.

Answer 6

If your customer wants to understand impact, you need to model what the artifacts are, what information flows between them, and how the parts of company interact with the artifacts.

You might consider building an SADT model. Boxes in SADT models represent processes. Labelled data input arcs show what [possibly compound] information/resources is fed into a process; labelled output arcs show data/resources produced by it. Control arcs indicate "large" signals that control the processing. Resource/mechanism arcs show what resources are required to carry off the process (e.g., hardware systems, networks, ...).

For your task, you would treat applications and company activities as SADT processes (the boxes). Data in/out and control arcs connect applications (SADT boxes) to other SADT boxes, or to external data sources and sinks (internal departments, staff, sales, shipping, e.g, corporate stakeholders). Thus you can model the information flows through the company via the various applications and what information they consume/process/produce/use, and what agents produce/use the data. (Doing all this is called "Structured Analysis").

[For sophisticated SADT models, each process box can be usefully recursively decomposed into sub SADT diagrams. I don't think you need this to model just the application dependencies; you don't need to know how the applications work inside unless they are truly complex and separating dataflows matter.]

Any change to informaiton input/output, deletion of an application would then have an obvious correspondence in the SADT diagram, and thus would lead to a better understanding of what the consequences are.

Its a big undertaking to this this right, and you'll have to work to keep it up to date, which is likely the place it will fail unless everybody is signed up for the long run.

For those of you that have not tried using SADT, it is a remarkably simple system to grasp (this matches other answer's dictum keep it simple), and remarkably effective at breaking up complex processing tasks into chunks where you can see (and actually communicate) essentially everything, even to managers! A key to making SADT work is to avoid being sloppy; define the arcs and nodes carefully and do not skip information sources or sinks. If you do that, SADT pays handsomely. [ Most whiteboard box and arrow diagrams are awful: you can't tell what is really an action, what is really data, or if all the information is actually shown and who uses it].

IMHO it is interesting to note that SADT models capture the intuition behind colored Petri nets, which model arbitrarily complex asynchronous computations, a generalizaton of Petri nets, which are a generalization of finite state automata.

Answer 7

I don't know the scope of this problem exactly, or how much effort it would be worth investing in, but it seems to me that one of the most obvious ways of documenting how the people and systems interact, would be to start with a workflow. There are tons of workflow programs out there ranging from really functional to more documentation based. The hey would be to figure out what all the players are in the workflow. Some with be people/job titles, while others would be systems.

Through a workflow, even a rough one, it should be fairly straightforward to analyze what people interact with what systems through what actions. Not that this is a simple task, of course. To describe the tasks of each job is definitely an undertaking. The good thing with doing a workflow, though, is that it is easily understood. It could be distributed throughout the company, and be much more easily kept up to date, because it is something that any manager should be capable of maintaining for themselves and their team.

When considering the workflow software, just make sure that it is capable of describing the workflow with adequate detail, and also that the resulting data is easily analyzed through a program you could write and translate into a dependency graph.

It would also serve as an excellent piece of documentation in its own right.

Answer 8

I know at least two customers, two big financial institutions, that implemented a custom webapp allowing to achieve the same goal (finding dependencies to do impact analysis) and I humbly think that you don't need a map to "visualize" things.

Basically, store machines, services (app server, database, etc), applications (languages, functional domain, etc), and dependencies between applications and implement a query module allowing to find (optionally transitive) dependencies for a given app (depends on/depends from) and to print a report.

I would implement this using a rapid CRUD application development framework like RoR, Grails, etc which is what they (the financial institutions above) did.

Answer 9

@p.marino the diagramming part of EAI/ESB products is usually not worth it for any moderately complex system. Biggest issue is that I still haven't seen one that can show the big picture and meaningful slices of the system without excessive manual tweaking.

You might want to check OS registry/repository thingies like Mule Galaxy, WSO2 Registyry or JBoss's similar product.

I have experience on doing something along these lines with Mule Galaxy. You can define entities with attributes and dependencies of different type. Each change is audit-tracked and can be promoted according to a lifecycle. The entity can be abstract or a file (e.g. application's main configuration file). Also each entity or domain can have access permissions.

This allows you to describe the logical structure of your system in maintainable way. For visualization, you would need to roll it yourself. You can suck up the data via simple REST-style API.

Answer 10

Even if the question is "closed" and the bounty has been paid, the person who will work on the actual issue has found a tool which looks a very good fit, so I am including this for everyone interested (or people who may look for the question in the future):

https://www.iteraplan.de/en

Answer 11

Your first problem is how to build a conceptual framework to hold all this information (NB: I'm not talking about the format of the data, but rather how you attach meaning to it). This is harder than it looks, but luckily there's been quite a bit of work done on this already (see Figure 1) so you don't need to start from a blank sheet of paper.

Then you need to collect the information. Thankfully the client isn't too big so you might be able to succeed at that, but for large organizations it is often the case that they have no idea what the real dependencies between different pieces of their IT infrastructure really are. (They may well think that they know, and will persist with that illusion up until they change something and the law of unexpected consequences bites.) I wish I was able to recommend some products (free or commercial) that could help with this, but all I've really got is a mix of war-stories and lack of satisfaction. In particular, a lot of the more traditional tools for this sort of thing don't seem to cope well with virtualized servers. If there's something open-source for this sort of thing, I'd love to hear about it!

Finally, you need to present the information. That's the easiest bit which most of the other answers here address. My only comment here is that the overall (conceptual) graph of information is probably going to be too complex to display in its entirety and remain understandable, so whatever you do you're going to be having to think about how to hide information so that only partial views are ever presented.