Ingestion

The IRE is ServiceNow's core framework for processing all incoming CI data. Every CI that enters the CMDB — whether from Discovery, Import Sets, Service Graph Connectors, or direct API calls — passes through the IRE.

Primary Functions

• CI Identification — matching incoming data to existing CIs in the CMDB
• Reconciliation — resolving conflicts when multiple sources provide different values for the same attribute
• Relationship Processing — creating and maintaining CI-to-CI relationships
• Data Source Prioritization — determining which source is authoritative for each attribute

8-Step IRE Processing Flow

1. Payload Received — data arrives from Discovery, Import Set, SGC, or API
2. Normalization — Normalization Data Services (NDS) standardizes values (e.g., manufacturer names, OS names)
3. Identification Rules Evaluated — rules are checked in priority order; first match wins and IRE STOPS at the first match
4. Match Decision — single match means update existing CI; no match means create a new CI
5. Multiple Match = De-duplication Task created — NO update occurs and processing halts for that CI
6. Reconciliation — if updating, source precedence determines which values win per attribute
7. CMDB Update — CI is created or updated in the CMDB
8. Audit Logging — all changes are recorded in the CI's history

Identification Rule Components

• CI Class — the CMDB class the rule applies to
• Identification Attributes — the field combinations used for matching
• Order — determines evaluation sequence (first match wins)
• Conditions — optional filters to narrow when the rule applies

Example: Computer Identification

For computers, a common identification rule setup is: Rule Order 1 uses serial_number AND manufacturer with the condition that serial_number IS NOT EMPTY. This ensures strong, reliable matching before falling back to weaker identifiers.

Identification rules are defined per CI class and evaluated in priority order. They determine how incoming data is matched to existing CIs in the CMDB. Proper configuration of identification rules is essential for preventing duplicate CIs and ensuring data integrity.

Identifier Entries

Each identification rule contains identifier entries — attribute combinations with assigned priorities. A lower priority number means higher evaluation priority (checked first).

Independent vs Dependent CIs

• Independent CIs — can be identified on their own using their attributes alone (e.g., servers, network devices)
• Dependent CIs — require the parent CI to be matched first before identification can proceed (e.g., network interface cards, disk drives)

Inclusion Rules

Optional filters that narrow the set of candidate CIs considered during identification. They help reduce false matches by limiting the search scope.

Testing and Supported Sources

• Test identification rules using the CMDBTransformUtil API before deploying to production
• Supported sources: Discovery, Service Mapping, IntegrationHub ETL, IRE APIs

Reconciliation occurs when multiple data sources provide information about the same CI. The reconciliation engine determines which source's value wins for each individual attribute based on data source priority.

Priority Rules

A lower priority number indicates higher authority. For example, a source with priority 100 beats a source with priority 200. This is per official ServiceNow documentation.

Data Refresh Rules

When a higher-priority source stops updating, after a configurable period (e.g., 14 days), the next-priority source is allowed to update those attributes. This prevents stale data from blocking updates from active sources.

Non-Reconcileable Fields

Certain fields like Comments or Assigned To can be configured to never be overwritten by automated sources. This protects manually curated data from being overridden by Discovery or integrations.

CMDB 360

CMDB 360 stores ALL values from all sources in the cmdb_multisource_data table. Even when reconciliation picks a winner, the losing values are preserved. Administrators can compare values from different sources and manually override if needed.

sys_object_source Table

The sys_object_source table maps external native keys to ServiceNow sys_ids. This is critical for tracking which external system created or owns each CI, and it enables the IRE to maintain the link between external records and CMDB CIs.

Import Sets are temporary staging tables for external data before it is transformed and loaded into the CMDB. They serve as a buffer that allows data to be validated, mapped, and cleaned before it becomes part of the production CMDB.

Key Characteristics

• Support CSV, Excel, XML, and JSON formats
• Import Set Tables are created automatically based on the source data structure
• Data is temporary — automatically purged after successful transformation

Transform Maps

Transform Maps define the mapping rules between source (import set) columns and target (CMDB) fields. They control how raw data becomes structured CI records.

Transform Map Components

• Source Table — the import set table containing raw data
• Target Table — the CMDB table where data will be written
• Field Maps — individual column-to-field mappings
• Transform Scripts — custom logic executed during transformation

Coalesce Fields

The coalesce field determines which source field(s) are used to identify existing records for updates vs inserts. When a coalesce match is found, the existing record is updated. When no match is found, a new record is inserted.

Service Graph Connectors are pre-built, certified integration packages for importing third-party data into the CMDB following CSDM standards. They are the recommended approach for integrating external data sources whenever a connector is available.

Value Proposition

• Reduced implementation time — pre-built mappings and logic
• Improved data quality — certified identification rules
• CSDM compliance out-of-the-box
• ServiceNow-managed updates — connectors are maintained and upgraded by ServiceNow

Architecture

SGCs are built on IntegrationHub ETL as the processing engine. They use the Robust Transform Engine (RTE), NOT classic transform maps.

SGC Components

• Data Source Configuration — connection details and credentials
• Staging Tables — temporary storage for incoming data
• Transformation Logic — mapping and conversion rules
• Identification Rules — certified rules for CI matching
• Monitoring Dashboards — visibility into connector health and data flow

Setup Process

1. Installation from the ServiceNow Store
2. Configuration — set up credentials, connection parameters, and schedule
3. Mapping Review — verify and customize field mappings as needed
4. Test Execution — run the connector in a test environment
5. Production Deployment — enable the scheduled connector in production

Editing SGC Mappings

To edit SGC mappings: Access IntegrationHub ETL Studio, navigate to Step 3: Select CMDB Classes. You MUST mark Steps 1 and 2 as complete first before Step 3 mappings become editable. Then click Edit Mapping to customize.

Key Facts

• SGCs use Robust Transform Engine (RTE), NOT classic transform maps
• Connectors run in isolated application scopes for upgrade safety
• SGCs are ALWAYS preferred over custom Import Sets when a connector is available for the data source

Discovery is ServiceNow's automated tool for finding and mapping IT infrastructure. It probes networks, identifies devices and applications, and populates the CMDB with CI data. All Discovery data flows into the CMDB through the IRE.

Discovery Types

• Horizontal Discovery — broad network scanning to find devices across IP ranges; casts a wide net to identify everything on the network
• Top-Down Discovery — starts from known services and maps dependencies downward; focuses on service context and relationships

Discovery Patterns

• Behavior Pattern — defines how to probe and classify a device or application
• Quick Pattern — simplified pattern for rapid deployment
• Application Pattern — specific to application discovery and dependency mapping

Normalization standardizes CI attribute values from various raw sources into a consistent, canonical format. Without normalization, the same manufacturer or OS could appear as dozens of different strings, making reporting and management unreliable.

Normalization Data Services (NDS)

ServiceNow uses Normalization Data Services (NDS) to perform normalization. NDS is applied during the identification and reconciliation process, ensuring data is standardized before it is committed to the CMDB.

What Gets Normalized

• Manufacturer names (e.g., IBM vs International Business Machines → IBM)
• Model names (e.g., various model string formats → canonical model name)
• Operating system values (e.g., win10, Windows 10 Pro, W10 → Microsoft Windows 10)

Normalization Process Flow

Source data → Transform Map → Field Maps → Transform Scripts (normalization logic) → Standardized CMDB values. Normalization can occur at multiple points in the ingestion pipeline, but NDS is the primary mechanism.

Maintaining upgrade safety is critical when customizing CMDB ingestion processes. ServiceNow regularly releases platform upgrades, and any modifications to out-of-the-box (OOB) components may be overwritten or cause conflicts during upgrades.

Standard Upgrade-Safe Workflow

1. Create Custom Data Source
2. Create Custom Transform Map
3. Map Fields (source to target)
4. Write Transform Scripts for custom logic
5. Create Custom Identification Rules

Anti-Patterns (NOT Upgrade-Safe)

• Directly editing an OOB Discovery Pattern
• Modifying OOB Transform Map scripts
• Changing OOB CMDB field definitions
• Deleting OOB Identification Rules

Practice Scenarios