Publishing system

Pipes and the publishing system

You may require consuming, transforming, and exporting data from and to external sources. In the process, you may also need to implement data conversion between different formats and apply business logic. To do this, you use Sitefinity CMS publishing system module.

For example, the publishing system is used in Sitefinity CMS to implement functionality such as search indexes, RSS feeds, Twitter feeds, and so on. For more information, see List of inbound pipes.

In this article you will learn about the different parts of the publishing systems, their relation, and how you can use them to work with external data.

The core components of the Sitefinity CMS publishing system are:

• Publishing point - accumulates data from different sources.
• Inbound pipe - transfers data from source to publishing point.
• Outbound pipe - transfers data from publishing point to destination.

The data flows from the inbound pipes into the publishing point and is then processed by the outbound pipes.

Every time you save an item, Sitefinity CMS creates a scheduled task that invokes the publishing system in the current thread.

Sitefinity CMS allows you to create custom pipes or modify the built-in pipes.

Publishing point

The publishing point is a data structure that accumulates data from different sources and has associated inbound and/or outbound pipes. A publishing point can be Persistent or PassThrough.

• Persistent
  Stores the data coming from one or multiple inbound pipes into the database.
• PassThrough
  Passes the data directly to one or more outbound pipes without storing it in the database.

  For example, the search index, the RSS feed, and the Twitter feed are publishing points coming out-of-box with Sitefinity CMS.

  In addition, the publishing point can be configured to process Live items or Master items. For example, the latter is used internally in Sitefinity CMS to implement the backend search. For more information about item lifecycle, see Content Lifecycle.

  WrapperObject

  WrapperObject encapsulates data into a flexible structure suitable for data transformations. It enables you to dynamically transform source item to destination item based on predefined mapping settings. It is locale aware. WrapperObject is conceptually similar to DynamicObject class from the .NET BCL.

  Pipe

  A Sitefinity CMS interface for transforming data. For example, you can use pipes to add properties, change property value or type, or convert data based on predefined mappings.

  Pipes can be:

• Push pipe:
  It passes the data to the publishing point or destination when it receives it.
• Pull pipe:
  It pushes the data on demand when requested.

Pipes can also be classified as following:

Examples

Push Pipe - Persistent Point - Pull Pipe

You use this scenario when you import data that you need to store. You collect data that you cannot guarantee exists upon request from an external source.

For example: RSS feed

Push Pipe - Pass through Point - Push Pipe

You use this scenario when you need to flush data from Sitefinity CMS into an external source like Twitter, but you do not need to store the data.

Pull Pipe - Pass through Point - Pull Pipe

You use this scenario when you have the data and do not need to duplicate it in the publishing point.

For example: RSS feed in Sitefinity

You can also combine Pull and Push pipes on both sides of the publishing point.

Pipe components

Pipes have multiple settings (implemented as properties of the IPipe and related interfaces), but the two which you would use most often are the Mappings and Definitions. Pipe settings contain all information for creating a pipe. They are also used to restore a pipe saved in the database. The PipeSettings property is persistent. For more information, see Register pipe settings.

Mappings are exposed in the MappingSettings property and ultimately are a collection of objects of type Mapping. You use Mappings [TT1] [SR2] [TT3] to describe how a particular property of a WrapperObject is transformed when moving through the pipe – maybe in another type or in another property, or the value would change. The mappings are part of the PipeSettings and are persistent. For more information, see Register pipe mappings.

You use Definitions to store a collection of items describing the type members used by the mappings. They are used in the UI for changing the default mappings. The definitions are not persistent. For example, if you create a Mapping for a Price field, you can use a Definition to describe that the Price can only be a number bigger than 100. This is useful for Persistent publishing points to decide what are the appropriate database types to store the content. For more information, see Register pipe definitions.

As a convenience shortcut, Sitefinity CMS offers template pipes, which are .NET objects, representing a preset of pipes for creating a type of publishing point and are used in the UI. For example, Pipe templates describe which pipes will be displayed in the Create and Edit screens for search indexes and feeds.
To create a new search index, you can specify that you are using one of the search index template[TT4] [SR5] [TT6] s – SearchItemTemplate or BackendSearchItemTemplate, which will create a publishing point with inbound pipes for News, Blogs, Dynamic content types, and so on, and an outbound pipe for the search index.
In addition to the built-in pipe template, you can also create your custom templates[TT7] [SR8] . The templates are not persistent. For more information, see Site search API and Register pipe templates.

The following is a sample code for creating a custom search index pipe template[TT9] :

Using the publishing system, you can override and augment the built-in functionality of Sitefinity CMS, or you can implement interaction with completely different systems – for example, a search indexing service which is not supported out of box.

The Sitefinity CMS Publishing system is flexible and extensible. For example, here’s a diagram how the RSS feed is implemented:

 [TT1]Stoyan, what are the exact .NET types and Ipipe member names for Mappings and Definitions?

I don't think we need to specify that in the docs but here they are: [SR2]

Telerik.Sitefinity.Publishing.Model.MappingSettings

Telerik.Sitefinity.Publishing.Model.IDefinitionField

Telerik.Sitefinity.Publishing.IPipe

 [TT3]Okay, we won't document these

 [TT4]Stoyan: we need a code sample here

Here is a sample. https://gist.github.com/RuzmanovDev/8df605c89777d9cacc67a30f7302d132 [SR5]

 [TT6]thanks

 [TT7]Stoyan: do we want to document this? If we do, this will become a public API with support burden.

If we want to document them, please let me know what are the APIs. I do not think we need to have a sample code.

I see your point but currently those members are public and if anyone is extending the publishing system he is using them. I thing this diagram will be very helpful. One more thing - currently we have something like that in here https://www.progress.com/documentation/sitefinity-cms/for-developers-sitefinity-search---under-the-hood [SR8]

 [TT9]<script src="https://gist.github.com/sitefinitySDK/2d669dd7e3fe252fa41c052650e49dff.js"></script>