What is DEF?
From the official Data Exchange Framework documentation site:
Data Exchange Framework is designed to facilitate the transfer of data between two systems.
Basically, it allows Sitecore users/Content Editors/Administrators to integrate their Sitecore site with external systems. Integrate not by migrating (importing or exporting) only, but by preserving a seamless preconfigured flow of data that is triggered at scheduled intervals or when needed by a click of a button. It could be used to sync data into Sitecore, out of Sitecore, both or even neither (could be used for two external non-Sitecore systems).
Incorrect Use of DEF
If I am switching CMSs, from Umbraco for example into Sitecore, and I will be completely using Sitecore instead of Umbraco, I’m going to need a migration tool that would export all my templates, pages, users etc… from my old Umbraco CMS and then import it into Sitecore CMS. This tool/process should be run only once right before the new site goes live. I should not need to run this process ever again because the old Umbraco site is not running anymore and it should not have any new content after that migration.
In this case, DEF is not useful at all for me. Of course, it could be done with DEF but this is not the purpose of DEF. This is like scratching my back with my keyboard! In this case, there are probably tools that migrate data between CMSs that are more appropriate and would provide better options and utilities to make this transition easier.
Better Use of DEF
The more appropriate use of DEF would be (continuing on the above scenario) if, for some reason, I will not be giving up on my old Umbraco site. In other words, I will have two CMS sites running at the same time, an Umbraco site and a Sitecore one. I will be creating my users or blogs, for example, in my Umbraco CMS but I also want them to be available on Sitecore too.
In this case, DEF would do great, I would configure it and customize it to pull the required data from Umbraco, map them into Sitecore items and save them into Sitecore. I would configure DEF to run this process every day or week or whenever the site admin feels like it.
Do we need DEF?
Do we need another module to depend on?
One of the points to consider before installing DEF is that you’re going to depend on another module. This means:
- a new learning curve (not too steep)
- a new module to debug, face unsolved issues, report new issues
- a new module to worry about when deploying your site
- and most important of all a new module to prepare a set of painkiller pills for when upgrading Sitecore
Fortunately, this is an official Sitecore module which has its benefits. It will be, most likely, very widely used and this means less unsolved issues and issues to report, more articles and blog posts on how to configure it, customize it, hack it, etc… and best of all Sitecore would be responsible for providing its upgrade packages when upgrading Sitecore which SHOULD be quite reliable.
Its learning curve, luckily, is not too steep. Basic integrations could be achieved without any prior experiences with DEF in a day or two. Although it still lacks advanced documentations on how to further customize it, basic customizing is quite easy and could be achieved with following its official documentation and the Intro to DEF video on YouTube.
However, being an external module, it still poses extra efforts when deploying and extra pain when upgrading but I don’t think they would prevent me from moving forward with DEF.
Do we need DEF if we could do it manually?
DEF basically reads data from an external source, maps the data read into Sitecore items and saves them into Sitecore and vice versa. Thinking about it, many of our Sitecore sites may already have something similar to this. It is really not that difficult to implement. So why should I depend on a module to do something I can do myself.
This is a very valid point and there will be cases where implementing your own integration is the better choice especially:
- when dealing with very simple integrations that you’re sure will not have new requirement changes in the future. Working with DEF in such cases may be an overhead to your site. So analyzing the business needs and planning ahead is extremely important before you make your decision
- when dealing with integrations of extreme complexity that may require special processing requirements, chunking data or some other requirement that is not easily implementable with DEF. In such cases, you may face horrors if you attempt to do this with a framework that you’re not sure is capable of doing
Realistically speaking, most of our integrations won’t be neither of the above. But why should I choose DEF:
- Well tested and widely used framework. You can be sure that using a widely used framework is much safer than using something you just created
- DEF is very abstract and reusable. You can define pipelines, accessor sets, endpoints etc.. that could be used between different pipeline batches interchangeably. Each with its own specific logic. This point is really important and it may be the greatest feature of DEF
- I can update mappings, include new fields or remove existing fields in production by only configuring Sitecore items without having to edit any code
- Sitecore has provided an additional package that includes pipelines, accessor sets and endpoints that provide the ability to map and save synced data into Sitecore without a single line of code, other than the code the retrieves data from the other system of course.
- You can allow the end users to edit DEF settings to tailors their needs frequently at any time.
One example of DEF implementation that I really liked and was submitted in this year’s Hackathon and has lots of potentials, is team X-men’s (Sitecore Runner) submission which was an RSS blog importer.
Ideally, this module could provide endpoints for several blog sources such as WordPress, blogger, etc… with pipeline steps that accommodate all the available endpoints. End users would only have to specify the source system endpoint (either WordPress or blogger) and choose the preconfigured pipeline batch to run the data transfer process. The end users can then switch to any of the available blog endpoints at any time and immediately start transferring data. They could also configure several pipeline batches and transfer blogs from several blogs at the same time. This is really cool!
Seeing what DEF allows us to accomplish (which could really be an attraction factor for our clients) it does make sense to install it and start using it instead of manually writing integration code. You could argue that everything DEF does could be implemented manually including its reusability and its abstract nature. Of course, it could, but it wouldn’t be as well tested and reliable as this. Plus Sitecore would keep adding more features and updates which we, usually, don’t have time or money to do for our own implementations.
But how can we include new fields that weren’t in the initial requirements?
You can’t just include any field that wasn’t in the initial requirement unless you planned for it. Let me provide an example that my Hackathon team submitted which is a Trello Integration. Our module keeps pulling Trello cards and saving them as Sitecore items.
We have created a foundation layer project that was responsible for communicating with Trello APIs and returned a collection of Trello Cards. Trello cards could be returned to our feature layer Trello Provider project in two main ways:
- We used TrelloNet framework and so we could have returned a collection of the Card class that is used by TrelloNet. This way all the needed and unneeded fields would be supplied to our DEF implementation. However, this is not a good practice to depend on TrelloNet throughout our solution. We would never be able to switch to another Trello framework easily in the future.
- We chose to create an ICard interface and a Card class which we filled with most of the usable Trello Card fields. This included necessary fields such as ID, Name and Description in addition to currently unnecessary fields such DueDate, Closed? etc… Keep in mind that certain fields may need some pre-formatting in order to be usable directly. For example, the Members field provides a collection of Member class which contain several properties. To make things simple we only returned the member name. So our ICard interface had a Members property that potentially would have a comma separated Member names. This approach allowed us to provide several fields that could potentially be of use in the future without depending on an external framework
Regardless of the approach taken, our DEF pipeline context would now have a set of Trello Cards properties that are ready to be used to populate Sitecore items at any time. All we have to do then is configure (through Sitecore) the Accessor Sets, target item template and property-to-field mappings and our new fields would be available right away.
But isn’t our Endpoint and Pipeline Step dependent on Trello fields?
Isn’t this what Sitecore is all about? Looking at the Experience Editor, don’t we, developers, have to create a set of components that each is dependent on a specific template and HTML design? We then create placeholders and placeholder settings that allow the users to customize the design the way they need but within certain pre-planned requirements and dependencies. The more options we provide the more customization we allow the end user to apply.
This is also true with DEF. We create a set of options each with its own dependencies and let the user customize and choose the final structure of the integration according to our pre-planned implementations options.
- Highly reusable configuration items and abstract structure
- Well tested by Sitecore and the community
- Allows certain customizations by end user in production
- Official Sitecore module (support, official documentation, upgrades steps, new features by Sitecore)
- Sitecore provides additional Sitecore DEF items to make pulling data into Sitecore extremely easy
- Huge potential for additional community-created DEF pipelines, endpoints and implementations
- Still lacks advanced documentation that is much needed when customizing steps
- Still lacks articles, how-tos, questions and answers and advanced topics by the community
- It is still an additional module to worry about in a Sitecore site
Read about Sitecore’s Data Exchange Framework, watch the Intro to DEF video and try it out. It is worth it.