How to Evaluate a Vendor

I never trusted the vendor-provided survey/ROI reports. They're useful to turn in after you've chosen them, but I always tried to get funding enough to take one of the vendor's low-level training classes. That's a lot cheaper than buying a product and then finding out it didn't hold up (why does any company think it's a good idea to just buy and find out?). The training class method is good, because, when they're training you on a product, you usually have someone who knows the product inside and out. This gives you a direct channel to ask better questions and get the real answers - instead of the sales-guy spiel. Might be an option?

Gathering requirements is a difficult task. There are always some general requirements that fit with nearly every single-sourcing project.

These kinds of requirements break up into the following categories:

General System Requirements

Requirements for all applications and vendors

Additional XML Authoring Tool Requirements

Additional Workflow Tool Requirements

Performance Requirements

Data Object Requirements

Network Requirements

Benchmarks for Requirements Verification

These are the underlying requirements that define any good system: the ability to control document source, the ability to create and produce documentation products for customers. The goal of a good requirements document is to capture and define requirements for a new system above and beyond these basic requirements. A good requirements document includes these basic, best practices requirements, in addition to the problem-specific requirements that can only be defined in terms of your business processes and needs.

These are the kinds of requirements will help to ensure a system that encourcages user participation. It helps you to design a system that can provide real benefits to the user and to the company. A well-designed system will provide many opportunities for small efficiencies in addition to any benefits due strictly to upgrading to new technology.

It is my intention here, then, to lay out some best practice requirements, to get requirements documents writers started. This is a jumping off point. Not an end point. The real end point includes all those intangibles that only you know about as the in-house expert, the requirements document writer.

General System Requirements

Can keep data local to task

Minimizes overhead

Fits into existing business practices

Optimizes, improves, or otherwise streamlines existing business practices

Does things right

Does not compromise customer-driven or business-driven requirments

Avoids unnecessary structure for the sake of structure

Can customize metadata/structure to reflect real business requirements

Guarantees functional quality with required output systems

Guarantees informational integrity

Improves quality

Improves accuracy

Improves flexibility

Guarantees consistent presentation

Supports authoring

Supports editing

Supports document management/configuration management

Supports workflow/tracking

Supports production

Requirements for all applications and vendors

Is currently supported

Uses current technology

Has plan for incorporating new technology

Has clear path for upgrades

Has clear plan for product deliveries

Is customizable

Is extensible

Has clear, defined plan for upgrading customization code

Supports automation through customization or extension

Application itself can be automated

Designed to scale to meet new customers and future company requirements

Additional XML Authoring Tool Requirements

Compatible with graphics output (e.g., SVG, CGM, EPS, TIFF, GIF, JPG, PNG, BMP) output

Compatible with graphics input (may be same as or a subset of those formats listed above)

Vendor has lifecycle policy

Produces documents that are of comparable quality

Supports content reuse

Supports custom-defined chunking

Supports smaller than single-document chunking

Additional Workflow Tool Requirements

Has control mechanisms

Has approval mechanisms

Has guaranteed audit trail

Records transactional historical data

Tracks user tasks and events

Records: state transistions, approvals, times/dates, routing activities, user

Can automate the creation of ancilliary documents

Can interact with other external systems

Can fire off external events

Can be advanced through external mechanisms (e.g., API)

Can store additional metadata about stored data objects

Uses Lock-Modfiy-Unlock or Copy-Modify-Merge (Specify which)

Can define roles

Can define access requirements

Can use role-based access to guarantee entitlement

Integrates with LDAP

Supports execution of custom, exernal, automation tools

Performance Requirements

Side-by-Side Performance Test

Check-in/Check-out Directory

Check-in/Check-out Single-File

Check-out of documents for edit

Check-out of documents for read-only

Level 1 Workflow testing (edit, route for review, annotated, dispatch)

Level 2 Workflow testing (Level 1 + re-edit, 2nd draft, route, dispatch, approval)

Level 3 Workflow testing (include full production run)

Testing with multiple clients/multiple users

Tangible improved level of functionality achieved

Tangible improved level of performance achieved

Tangible improved level of integration achieved

Data Object Requirements

Binary Document types (MS Office, FrameMaker, PageMaker, Interleaf, etc)

SGML Files

XML Files

Metadata extraction from SGML/XML file (CMS integration)

XML/SGML file internal link management

Binary graphics file type: EPS

Binary graphics file type: TIFF

Binary graphics file type: GIF

Binary graphics file type: JPG

Binary graphics file type: BMP

Binary graphics file type: WMF

Binary graphics file type: CGM

Network Requirements

Directory structuring

Permission structuring

Roles-based permission entitlement

Benchmark Test Requirements

Task: Authoring Application Launch

Task: CMS Launch

Task: Check-out directory

Task: Check-in directory

Task: Create new document

Task: Delete document

Task: Grant user permissions/access

Task: Change user permissions/access

Task: Look at object properties/metadata

Task: Look at object history/audit trail

Task: Retrieve old version of document

Task: Check-out single-file-document

Task: Check-in single-file-document

Task: Search for document

Task: Change user ownership

Task: Generate object report

Task: Check-out SGML/XML-document

Task: Check-in SGML/XML-document

Task: View Tasks/Jobs

Task: Acknowledge/Claim assigned task

Task: Review Task

Task: Add Reviewer

Task: Approve

Task: Dispatch

Task: Create revision/tag/branch

Task: Create release

Technorati Tag: single-sourcing

Avoiding Vendor Lock-in

The goal of object-oriented tool development is to avoid vendor lock-in. Lock-in is not determined by the cost of the application but by the method of its implementation. The level of integrated customization determines the level of vendor lock-in.
One-of-a-kind software (fully custom or “one-off” software) is both desirable and necessary when no other solution is available. Unique software is custom-built for one specific purpose; it cannot be used by multiple customers and cannot be rented.
Customizable software (tightly integrated software like Vulcan) was a natural development of productized custom software. Customizable software was an attempt by vendors to create generic applications that could be set up according to individual business requirements, specifications, and needs. In a very real sense, customized software is harder to upgrade, expand, or replace than completely unique software: customized applications have as much single-purpose code as they have COTS application software, but the single-purpose software is tied to both the particular COTS application as well as the particular COTS application version. This means that all of the customizations must be upgraded whenever the application upgrades, if it can be upgraded at all.
Open standards software development (API tool development) is a direct response to the customizable software development over the last 20 years. API tool development focuses on isolating the impact of an application-specific code. For example, rather than implementing customizations entirely inside a particular vendor’s customization language, API developers create the small custom tools required to implement specific business requirements. They isolate required functionality in a small custom code components and then build an API layer to communicate with a specific vendor’s application. This way, when the application required upgrade, developers only need to upgrade the API layer: the customization layer does not significantly retard the ability to upgrade or replace any particular vendor product at any time.
That is what vendor lock-in means.
NGES-MS is facing this situation with Vulcan today. Vulcan is tightly integrating the functionality of Epic with Visual Source Safe. As a result, NGES-MS is tied to Epic 4.3.1. Although Arbortext charges an annual maintenance fee, we cannot upgrade Epic to take advantage of new features, Because Vulcan is customized COTS software, upgrading Epic will not occur until Vulcan upgrades all of their 4.3.1-specific code.
In addition, NGES-MS is facing a deadline: it is highly probably that Interleaf will stop functioning with the next release of Windows (Windows Vista). Interleaf has been in use at NGES-MS since 1995. Interleaf might have still retain long-term prospects, if other factors were not at work to place the documents and data it supports at extreme risk. NGES-MS discovered that with the release of Windows XP, RDM became inoperable. Windows XP is scheduled to reach end of life in 2006. Extended support will end 5 years after mainstream support ends, in 2011. Documents still in Interleaf after 2011 will likely be unrecoverable.
Today, 78 percent of NGES-MS Launcher documentation is not supported in the Vulcan/CAE data model. Given the laundry list of features currently facing the Vulcan development team, it’s unlikely that Vulcan will be able to support the additional document types in time and we would simply be exchanging one highly-customized, proprietary system for another.
On the other hand, S1000D is written with interoperability in mind, but without limiting this to the lowest technical application. Although it may appear rigid or involved with tracking irrelevancies, the specification defines application behaviors; it requires vendors to provide data interoperability (a “way out”). If a product claims S1000D-conformance, developers supporting this format will never find themselves held captive. If a vendor is compliant, support staff will always have at significantly less expense options than if they were dealing with a proprietary or highly customized environment.
S1000D is not a tool. By its own definition,
S1000D is an international specification for the procurement and production of technical publications.. for [use in] the support of any type of equipment, including both military and civil products. (Chap 1.1, Page 1)
For the first time, interoperability is considered important from more than the technical (software engineering) perspective: a usability perspective is also involved. The uniform presentation of content increases both author and user efficiency. If every manual is created in accordance with the same specification, and content is always presented the same format, the learning curve for new personnel can be drastically reduced. In fact, a novice user becomes productive much more quickly without interference from the manual, in order to complete the job at hand. The majority of the S1000D specification addresses this very issue. By thoroughly defining how manuals should be constructed, the specification improves overall quality for every single person who uses the manual.
Besides defining the schema and enforcing interoperability, the real revolution S1000D offers is that it is completely modular. A data module is a data module is a data module, and output is done via assembly. S1000D will force users to change both the way they work and they way the “have always done things.” New policy decisions will be required. S1000D requires changes because, it demands following best practices. The end results, the benefits, really do outweigh the costs.
Can projects using S1000D be tailored, expanded, and customized? Yes.
Do you have to use their process? Yes.
Do you have to change your processes? Perhaps
The best part of the S1000D specification is that, once agreement occurs between user, customer, supplier, and other internal groups, the implementation can still be tailored business requirements.
The "known standard" benefits are self-evident.
As with other documentation formats, S1000D reduces support costs, facilitates modularity and reuse, and allows users to view electronic documentation via a common Web browser or other interactive electronic technical manual (IETM) viewer.
S1000D indicates that it is flexible system. In contrast, most of the commercial world is adopting IBM’s Darwin Information Type Architecture (DITA) format. Many early DITA users have struggled with its overly generic character.
This was a conscious choice by the DITA architects: DITA legislates only the framework (topics, tasks, references, et al), but requires users to flesh out the details through "specialization." Those who have used DITA out of the box have customized it and are now looking for something better.
S1000D is better. While it has many similarities with DITA, it goes even further. S1000D represents an emerging technology. This means that the tools are not fully developed and that vendors have not yet fully embraced it. However, because many organizations are adopting it, vendor support will grow. Working with this standard, we can leverage the same benefits that we would get by using any other.
S1000D is specific to air, land, and sea systems. S1000D captures more information (for documents supporting these products) than any documentation format to date. The specification is essentially complete. S1000D is ready to use today. It has more leverage for information authored and for tools that serve documents, on a smaller-than-document-level basis. In addition, S1000D has significant COTS vendor support and a wide community of practice to draw upon.
Because the DOD has specified that new documents be S1000D-compatible, and because migration is inevitable, going to the new system now is most cost effective. S1000D is XML, and XML transformation tools are immediately available. S1000D is the full-blown schema; all that remains are refinements of the data that the S1000D format contains.
We see several valuable side effects of this implementation. First, we believe that S1000D provides a neutral response to the DOD. This gives the entire SP community adequate opportunity to say: “S1000D is being investigated.” Second, the trial implementation provides a genuinely independent evaluation: NGES-MS has no interest in one system over another and can evaluate S1000D without impacting other deliverables.
Third, because we will be supporting Vulcan output as well as IETM and pdf, as a bonus, we will be able to provide the roadmap for Vulcan-S1000D transformations without delaying current deliverables, or diverting attention away from current development efforts. If S1000D/Vulcan interoperability ever becomes desirable, the SP community will know the requirements, how much work will be involved, and a starting point (our transformation code). This information becomes available without derailing current deliverables or distracting the Vulcan implementation team from their primary focus.


Technorati Tag: single-sourcing