Part 5: High-Level Solution Design (HLD) Documents: What Is It and When Do You Need One

All large IT projects will require a High-Level Solution Design, also known as High-Level Design or HLD, an artifact instrumental to the Software Development Lifecycle analysis phase. The HLD gives the stakeholders a bird’s eye view of the system once the implementation is completed. At its core, the proposed solution architecture is designed to solve a business problem and generate customer value. Architecting a solution generates different concepts and selects an optimal candidate that satisfies budgetary and planning constraints while mitigating project risk. This article examines all aspects of a High-Level Design for a project. It will give software practitioners an idea of when, why, and how to create a valuable HLD. A few notes are due before we proceed:

• Development vs Integration — High-level designs (HLD) work best for system integration projects where a solution is assembled from mature technology and products. The components of this solution and the interfaces joining them might need to be customized for every implementation. However, no substantial development effort is expected in such undertakings, in contrast to what we see in green field assignments.
• Designing an IT solution — We recommend reading the Solution Design article for context and background. A design exercise uses an iterative approach to determine the best technical solution, ensuring that functional requirements and non-functional constraints are adequately understood, documented, and satisfied.
• High vs Low-Level Design — The HLD should not be confused with the Low-Level Design (or LLD), whose primary role is to decompose the systems defined in the HLD into subsystems, subcomponents and, most importantly, select optimal values for key design and operations parameters laid out in the architecture.
• Agile Architecture and Design — Can a design or architecture be Agile? The answer is yes, and the two topics of solution architecture and software architecture have been discussed thoroughly. Depending on the context, a top-to-bottom or a bottom-up approach can be used for building large software systems. Requirement volatility compels you to use Agile methodologies, while Waterfall is appropriate for familiar and classical implementations. More on that later.

2. Pre-requisites

• If this is your first time working on a solution design document, we recommend looking at Solution Design — Part 1: Introduction and First Principles.

Secondo —

• We recommend reading From Abstract Concepts to Tangible Value: Solution Architecture in Modern IT Systems for advanced concepts on architecting IT solutions.

Terzo —

• Once the High-Level Design (HLD) is approved, ushering in the work commencement, you might need to generate a low-level design, another document we describe in Part 4: Solution Design Documents — What You Need to Know.

The remainder of the articles can be accessed from the below:

Solution Design: First Principles

What is design, and why is it vital for software solutions?

Design of Modern IT Systems

Design process of modern IT systems and large software integration projects.

Agile Design: Solution design under conditions of uncertainty.

Solution Design Documents

Generating detailed solution design documents (SDD).

High-Level Solution Design Documents (HLD)

How to create a high-level system design document or HLD.

Component-Based Software Engineering (CBSE)

A discussion of CBSE and its development process.

3. FAQ

Solution design is creating a comprehensive plan for developing software systems that satisfy specific requirements. This involves designing the software’s architecture, components, modules, interfaces, and data structures.

Solution design principles in software include modularity, abstraction, separation of concerns, simplicity, scalability, and maintainability. These principles provide guidance for creating software systems that are easy to understand, extend, and modify, while also being adaptable to changing requirements over time.

Solution architecture principles are a set of rules and best practices that govern the design and development of software systems. These principles include flexibility, agility, security, reliability, and interoperability, ensuring that software systems are developed with the necessary quality attributes to meet the needs of the business and its users.

A solution design should include a detailed description of the problem to be solved, the requirements of the solution, the proposed architecture and design, the software components and modules, the interfaces and data flows, the testing and deployment plan, and any other relevant information.

A solution design should include a clear understanding of the business problem, the goals and objectives of the solution, the user requirements, the technical requirements, the system architecture, the software modules and components, the data and interfaces, and the testing and deployment plan.

A solution design diagram is a visual representation of the software system that shows the modules and components, their interactions and dependencies, the data flows, and the interfaces between the system and its environment. These diagrams help communicate the design to stakeholders and ensure a common understanding of the system.

The steps of solution design include understanding the problem and requirements, defining the system architecture and design, identifying the modules and components of the system, specifying the interfaces and data flows, testing and validating the design, and deploying the solution.

To start a solution design, you should first understand the problem and its context, then identify the business goals and user requirements. After that, you should define the system architecture and design, identify the software modules and components, specify the interfaces and data flows, and test and validate the design before deployment.

The role of a solution designer is to work with stakeholders to understand the problem and requirements and to design and develop software solutions that meet those needs. This involves defining the system architecture and design, identifying the software modules and components, specifying the interfaces and data flows, testing and validating the design, and deploying the solution. The solution designer must ensure that the solution meets the business and technical requirements, is scalable and maintainable, and is delivered on time and within budget.

4. High-Level Design (HLD)

4.1 What Is a High-Level Design (HLD) Document

A High-Level Design (HLD) is a technical document for a (generally) non-technical audience. A High-level Design aims to provide all relevant stakeholders with a bird’s eye view of the solution architecture and design after implementation (or integration).

It’s mainly comprised of the following items:

• A brief description of the solution — While keeping the discussion fairly simple, the HLD describes the primary features of the system. It is best to list all the elements for clarity without going near the implementation details, as these are not for this particular audience.
• Referenceto requirements — A reference to the functional requirements covering user functionality, such as business features, and non-functional requirements revolving around reliability, performance, high availability, security, and compliance. Business requirements are documented outside the HLD in a Business Requirements Document (BRD).
• A high-level network and architecture diagram —Architectural diagrams describe the major components of a software system, such as platforms, applications, and interfaces. Network diagrams describe servers, network devices, communication links, and specialized hardware. Make sure you use consistent system modelling language across all your diagrams.
• A logical breakdown of the solution is necessary if it is complicated, adds new modules, or updates existing ones. This section details whether new modules, skillsets, licenses, or operational requirements are needed for this project.
• The business processes, use cases, and user stories — IT projects typically add or modify business use cases, such as user interfaces or functionality, besides pure compliance or maintenance. These must be suitably documented as they might impact operations or how users interact with the system.
• Data flowdiagrams describe the input and output data and its path in the system. Information capture, processing, storage, replication, validation, and reporting are all aspects of data flow that must be documented.
• Other forms and representations — Besides the popular decomposition methods presented above, an IT solution can be represented in a few different views: behavioural, temporal flow, and states and modes. Illustrating solutions from different angles helps reduce their complexity and enhance their assimilation.

4.2 Why We Need an HLD

The infographic below shows the main advantages of having a High-Level Design.

A high-level design offers the following advantages:

• It forms the basis of the stakeholders’ acceptance of the proposed solution, a baseline from which implementation can begin.
• The HLD is generally provided with the Functional Specs to support the Statement of Work (SOW), a binding document between the software vendor and the client.
• High-level estimates can be derived from the HLD, allowing project planners to calculate the project’s cost and assess project risk.
• Like any technical artifact, the HLD is an excellent communication and collaboration tool. It’s a vehicle for bouncing off ideas on a complex and potentially cognitively overloading topic.

Professionals espousing Agile values consider extensive documentation an effort-intensive, low-value activity and overhead that can be abandoned without risk. That may be true for small projects or standard, off-the-shelf offerings.

We examined Agile extensively in other articles and believe Agile is sometimes promoted as a context-free answer to a context-specific question. In other words, there are instances where it applies and others where it doesn’t.

The effort to produce an HLD can be easily justified by understanding when and why it’s used. Part 2 of this series provided a detailed exploration.

4.3 When Is a High-Level Design Required

An HLD is usually prepared at the early stages of a software project, typically during the Analysis phase of the SDLC.

The HLD is one of the documents accompanying the SoW in the Analysis phase. It clarifies the solution’s high-level concepts so stakeholders can decide whether the design satisfies their requirements.

4.4 Why Not Go Straight to an LLD

Preparing a Low-Level Design is a multi-iteration, customized, bespoke, and expensive process involving a broad audience of technical staff (business analysts, developers, testers, and DevOps engineers). This investment cannot be justified at the early stages of a project as the stakeholders have yet to sign off on the requirements, Statement of Work, and scope. Stakeholders could even ask for significant design changes or have alternative options considered before deciding on a specific path. A document highlighting the fundamental concepts of a proposed solution, allowing them to make an informed decision on the budget, schedule, and project risk, is required during this stage. The HLD, Functional Specifications and SOW should provide precisely this information.

4.5 High-Level Design (HLD) Document Ownership

The Solution Architect usually owns the high-level design (HLD) document. People might cringe when they hear the word “architect”. They imagine someone somehow detached from real-world implementation realities, but that does not have to be the case. So, what is a solution designer/architect?

A solution architect is a highly skilled technical resource who is proficient in the following areas:

• The industry and the business —Architects thoroughly understand the industry and the end-to-end business requirements. They are often senior people familiar with the organisational culture and the significant business and technical decisions that shape the organization’s value proposition.
• The state of the current systems — The architect understands the system’s strengths and weaknesses and can offer insights on how to address them. They also have a thorough understanding of the interfaces and dependencies between the different components and can quickly gauge the impact of any change.
• The technology stack with which the system was built — Architects have a fair knowledge of the technology’s pros and cons, challenges and strengths, and how these compare to alternatives on the market. They also offer insights and best practices on system integration issues.
• Any compliance or regulatory requirements — The long expertise in the industry exposes architects to the different regulations and compliance requirements that must be satisfied before a system can run in production.

5. Architecture Documents and High-Level Design

Let’s briefly examine architecture and how its documents are relevant to High-Level Design.

5.1 What Is Solution Architecture

Solution architecture assembles technological assets (applications, products, servers) in an IT solution that satisfies specific business needs (functional requirements).

The subject has been explored from all angles in the series of instalments (of which this article is the fifth and final one), and, therefore, we invite the reader to look at Parts 1 and 2 for first principles and details.

Solution architecture is relevant to this discussion, and the high-level design captures its broad features.

5.2 What Is Software Architecture

Software architecture is an entirely different topic, which is not relevant here. First, here is a brief definition.

People have spent much time attempting to articulate a rigorous definition of architecture in software applications. In property development, architecture determines a resident’s user experience. This analogy, however, is of less value in software. For now, we use Martin Fowler’s definition, which is quite lucid (see the related link below for an in-depth analysis of the topic):

“Software architecture is those technical decisions that are both important and hard to change. This means it includes things like the choice of programming language, something architects sometimes gloss over or dismiss. Both aspects land squarely on the economics of software development.” — Martin Fowler

So, where does that leave us as regards the HLD? At this stage of the SDLC, the low-level, highly technical issues are not yet relevant. The SOW has not been signed, the scope is not yet locked, and developers are yet to be engaged.

6. Design Documents and Agile

Detailed documentation, top-down overarching design, and extensive planning are hallmarks of Waterfall, practices entirely left behind by purist Agile practitioners.

So, how does a high-level design (HLD) document fit into Agile frameworks? It doesn’t! Agile responds to the context-specific question of dealing with unarticulated needs and requirement volatility on a small scale. Agile performs poorly under the following circumstances:

• Massive projects where budgetary and scheduling constraints are paramount to stakeholders. Organizations cannot manage their strategic planning on two-week sprints. In this sense, Agile applies poorly outside semi-isolated development teams.
• Large-scale systems integration projects typically involve several platforms, suppliers, and technical and business teams. Synchronizing their cadence on two-week sprints is impractical.
• Projects with little or no novelty where all the details have been figured out are delivered more efficiently with Waterfall. The overhead incurred with Agile cannot be quickly justified.

In addition to all the advantages described in this article, planning and top-down design approaches for large projects involving many stakeholders require serious investment in design efforts, hence the need for high-level design and low-level design documents.

7. High vs Low-Level Design

diagram is a fundamental component of a high-level design (HLD) document. This diagram shows all the network components involved and what interfaces facilitate their communications." width="1024" height="939" />

The below table provides a comparison between the two design documents.

8.2.4 Architectural Diagrams

If you’re familiar with the C4 model for architectural visualization (or M2LP), this corresponds to Level 1: System Context Diagrams (or Views in M2LP). Its job is to lay out the whole ecosystem of the IT solution. At this stage, the nitty-gritty details of the implementation are not required. What is needed is a mapping between the high-level design decisions and the solution’s essential requirements. Once you have completed the description of the major components, it’s time to focus on the application level. The next topic you must include is application-level details in the high-level design. An architectural diagram looks like this:

• Core system componentsprovide basic IT or business functions. For example, an Enterprise Message Bus is a typical architectural pattern in many online transaction-processing systems. For the HLD, keep the non-business components at a minimum, as they may not be particularly interesting to a business audience, and focus instead on the business-centric elements.
• User interfacestell the reader how users interact with the system or a particular component. This information signals changes on the operational level. Such changes are usually followed by training, user manual updates, and user education, and it is essential to communicate with the customer well ahead of development.
• The database layeris where all the data resides. Although not fascinating, data storage and management are a pathway leading to discussions on data backup, disaster recovery, redundancy, and high availability, all of which are critical for uptime and business continuity.
• The business logicmodules deliver the solution’s business functionality. For example, a retail business can have billing, e-commerce, shipping, warehousing, and business intelligence components. It is best to depict business components in a separate diagram from IT components such as databases or web servers.
• Licensed components can be interesting for a business audience since they must be budgeted and sourced, impacting cost and schedule.
• Application Programming Interfaces (APIs)are essential to call out so that inter-platform dependencies are vividly illustrated. Interface design and management are vital, especially in large systems integration projects.

8.2.5 Data Flow and Use Cases

The data flow and use cases section documents all necessary interactions between the user and the system. Software solutions can be very complex and typically involve many stakeholders. List and define the user profiles interacting with the system, such as online users, system admins, or application admins. This section captures the user’s expectations of how they intend to operate the system. These expectations will drive the development of the necessary UI components, reports, and internal software functionality to make that happen. A typical business process flow diagram can look like this, with swimlanes delineating the boundaries between different business units:

The processes must be organized for the system to work, primarily if they depend heavily on one another. Designing performing processes is a topic in itself, and there are rules and best practices that can help you achieve that; see production processes for a complete discussion. Ensure this section of your high-level design satisfies the following:

• List all the parties, users, and modules involved.
• Define all the process steps while establishing ownership and responsibilities (the swimlane diagram can be beneficial).
• Make sure the criteria for movement between the steps is clear as well (inputs, outputs, tasks to be completed in-between stages)
• List the outcomes of each process and when it is considered “done.“

8.2.6 Example: Opening a Bank Account

Let’s say you have implemented a banking system that allows customers to enter any branch and open a bank account for personal or business use. The process usually involves multiple people within that branch. Everyone is responsible for a particular step in the application’s approval process.

8.3 Download Template

Download a basic High-Level Design (HLD) template using the link below.

9. Systems Modeling Language

9.1 The Need for a Specialized Language

Describing and documenting IT system design has historically used natural language and visualizations. While both are powerful and rich, allowing the author much room for creativity and expression, they have considerable drawbacks.

These limitations are more pronounced when systems increase in complexity, with many layers of abstraction requiring multiple decomposition levels.

Below is a list of these problems:

• Natural language can be ambiguous, and the same sentence can be open to different interpretations.
• Errors are difficult to detect.
• Different system components can be described with varying depth and rigour, allowing inconsistencies to emerge.
• Documents written in natural language can overwhelm any individual’s cognitive abilities. The result is piles of barely used documentation.

9.2 Object-Process Methodology (OPM)

Thus, the need for a design modelling language arose. The anticipated language would address all the above issues at the price of confinement to a specific set of rules.

Any universal and practical language must adhere to specific rules governing the following areas:

• The language describes the objects and how they relate, group, categorize and subdivide.
• The meaning associated with these objects, semantics, is the corresponding field of study.
• The grammar would allow the proper construction of complex “sentences.” These rules are known as syntax.

Because objects can mean different things in different contexts, a special branch of philosophy called ontology engineering has emerged. Its job is to overcome the inter-domain problem of associating different meanings with different terms.

The Unified Modeling Language (UML) is one implementation of the above rules. However, it was not good enough and had to be replaced.

A universal, powerful, and domain-agnostic system modelling language was important enough to merit its ISO standard, the ISO/PAS 19450:2015.

This ISO standard is called Object-Process Methodology (OPM) and defines the following concepts:

• Entities:
• Objects: list of entities and building blocks of the system.
• Object States: an object can exist in one of several states during its lifetime and transition from one state to another.
• Processes are transformations that apply to objects. Processes can create, destroy, or transform objects between different object states.
• Links:
• Structural links define associations between objects (composition, inheritance, exhibition, instantiation)
• Procedural designating how objects, states, and processes affect each other.
• Event and condition: the Event-Condition-Action model describes an event that triggers an action if a specific condition is satisfied.

While the OPM model may seem cumbersome and nonintuitive at first, it is very powerful once you get used to it.

In general, whether using OPM or something else, it’s helpful to remember the following when writing technical documentation:

• Use consistent language, symbols, and expressions when describing the system (components, relationships, and functions).
• Consistency should apply to both text and diagrams, as well as content and style.
• Publish a documentation style guide that everybody can access and use.
• Use diagrams to convey complex ideas in a condensed yet very digestible format.

10. Final Words

The value of documentation and design has been downplayed significantly with Agile practices, although the advent of powerful content creation tools like Confluence has started to reverse the tide.

Technical documentation, architecture, and design, especially for large system integration projects, remain integral parts of the delivery of large software projects.

Doing it right can make an enormous difference in the quality of your deliveries, reflecting on the organisation’s image and value proposition.

Finding the right balance between quantity and productivity can take time, but it will be worth it. It also requires ingenuity to focus on critical and subtle information only and present it in a way that facilitates quick reading and assimilation.

A simple trick for managing documentation activities is maintaining templates that vary with project size; small projects use small templates, and large projects use large templates.

Collaboration, clarity, and stakeholder management keep everybody happy. The team’s morale remains high, and the customer remains satisfied and willing to do more business with you.

Those objectives and the best practices we described allow Operational Excellence in Software Development to become tangible: a safe and healthy environment for your people and a prosperous business at the service of the community.