Sitecore Publishing Service — The Good, The Bad and The Ugly
Introduction
The concept of publishing content has been a massive part of Sitecore (and a lot of other Web Content Management Systems) for many versions. In this blog post, I will go through my learnings and experiences with Sitecore Publishing Service — The Good, The Bad and The Ugly.
Sitecore Publishing History
Publishing has been a big part of Sitecore since many versions ago. In a nutshell, publishing is the act of making appropriate or correct content available or unavailable to the public facing side of the web site, at the time of publishing. The process becomes complicated with all the publishing events, which provides additional processing or reporting. These complicated processes would eventually slow down the publishing process, consumes resources in Content Management instance, when lots of Sitecore items and the associated versions need to be processed, and determined whether it should be published and / or unpublished.
There were some great improvements that were introduced in 7.2. They were
- Dedicated Publishing Instance https://community.sitecore.net/technical_blogs/b/latest_technical_reinnovations/posts/dedicated-publishing-instance-configuration
- Parallel Publishing https://community.sitecore.net/technical_blogs/b/latest_technical_reinnovations/posts/enabling-parallel-publishing-with-sitecore-7-2
One issue with Dedicated Publishing Instance was that there is still a Sitecore.Kernel.dll attached to the instance. For clients who are still on Sitecore Perpetual licensing model, that means it would take up one production instance.
Parallel Publishing may work OK if the load on the Content Management instance is not heavy. However, our client has 50 content authors logged in concurrently during business hours. Parallel Publishing will further decrease the performance of the Content Management instance, as more CPU threads are required to process publishing tasks. I also experienced SQL Database deadlock problems when trying to max out the number of threads Parallel Publishing, which indicates that the SQL Server needs to increase its IOPs.
Sitecore Publishing Service was introduced in Sitecore 8.2. It is the first .Net Core application that was made generally availabe as a public release (Sitecore Commerce 9 was the first Sitecore application that was written in .Net Core, which was released later than Publishing Service). As it is a totally separate application with no Sitecore.Kernel.dll attached, it does not count as an instance in the Sitecore Perpetual licensing model.
One interesting thing to note, is that the latest documentation for Sitecore v9.2 has no more mention of the Dedicated Publishing Instance. Instead, Publishing Service is more heavily “promoted” to be the Publishing role.
The Good
Sitecore Publishing Service uses SQL Bulk Copy to copy data, which means the load is taken at the SQL Database instead of Sitecore application. This has a big improvement Content Management instance performance. As an example, I was testing on a 80 GB master database. Using good old publishing would just time out eventually. However using Publishing Service, republishing the entire site in Repair mode took 90 minutes to complete. Ehab Elgindy, product developer at Sitecore, has done a benchmark. http://www.ehabelgindy.com/the-awesome-performance-of-sitecore-publishing-service-v2/
Another feature of publishing service, Content Availability, operates just like the good old Live Mode in CM instances, in which the “website” uses the master database. Sitecore then works out the correct publishable versions and presents what it thinks is the correct version. This is now effectively done for the web database with Content Availability, except it does not have to filter as many Sitecore item versions.
As of SXA 1.8, publishing service is supported in version 3.1.3 and beyond. Sitecore has taken the direction of separating the Publishing Service web application, and the Publishing Service Module (which is installed as a Sitecore package in CM) release cycles as of v4.1
The Bad
As the work load is shifted to another separate web application, which is rewritten from the ground up, there are a few functions that are lost as a result.
The usage of History Engine in web database is gone. I have previously blogged about how useful the History Engine is. https://medium.com/@Vincent_Lui/sitecore-9-x-history-table-and-history-engine-101f39488391
Luckily I have a potential workaround for this scenario.
Custom processors to add additional items to be published as part of “Publish Related Items” is gone. Previously, the pipeline getItemReferences is used to add additional items / versions to the publishing job. This no longer works. There is a potential workaround to this, more on that later.
The following events are also no longer available:
publish:fail
publish:itemProcessing
publish:itemProcessed
publish:startPublishing
publish:statusUpdated
What changes are required to use Publishing Service ?
If a Sitecore solution has any type of publishing action implemented programmatically (eg. importing content from an external source), then the correct publishing implementation must be done to ensure that publishing will occur, whether using normal Sitecore publishing, or Publishing Service.
In Sitecore v8.0, PublishingManager was introduced as an abstract / provider layer for publishing. All implementation using Sitecore.Publishing.PublishOptions and Sitecore.Publishing.Publisher, must now switch to using Sitecore.Publishing.PublishManager instead. Examples of this is used in Hedgehog TDS PostDeploy Publish action. Unicorn 4.1.1 does not use PublishManager https://github.com/SitecoreUnicorn/Unicorn/blob/master/src/Unicorn/Publishing/ManualPublishQueueHandler.cs
Sitecore Powershell Extensions does not use PublishManager for its publish command. There is a workaround however https://sitecore.stackexchange.com/questions/15740/how-do-you-publish-with-the-publishing-service-using-spe.
Maybe someone should attempt to use PublishManager for both Unicron and SPE, and do a PR back into the repos.
I also noticed that there was a big spike on IOPs and DTUs on SQL database after using Publishing Service. It is very important that these metrics are monitored, and increase the resources if needed.
Sitecore has attempted to come up with a new pipeline to show the publishing operation results, which includes items / versions that have been published / unpublished. The changes are pushed to the pipeline publishEndResultBatch, with pipeline argument PublishEndResultBatchArgs. However, the item context is lost when it is deleted from the publishing target. I have since made some improvements which provides more context, and hopefully more align to what the History Engine used to provide.
I am attempting to look through the delete operations during publish, and trying to detect whether the delete action was due to Archive, Delete, or simply Publishing Restrictions or Workflow. Then exposing the results with more context to a class ChangedItem.
This can be extended to also replicate the getItemReferences pipeline that was available with the old Publishing.
Content Availability can be tricky to implement correctly. Publishing Events will not fire when a version expires. During publishing, multiple versions of an item that are valid currently and in the future will be published. This can be tricky if publishing events are required to perform other operations. A lot of thoughts, design and logic needs to be considered when designing page components that utilitses Content Search as well. Lots of testing is required to ensure Content Availability is suitable.
Here are a few more tips:
- Sitecore.Publishing.Service.Events.ItemSavedHandler should be enabled for multilingual sites. Default setting has been commented out.
- Sitecore.Publishing.Service.pipelines.BulkPublishingEnd.RaiseRemoteEvents rebuild indexing threshold needs some fine tuning on a case by case basis. The default value of 1000 may be too small for large sites, especially with language fallback enabled. The EventQueue clean up task also needs some tuning in conjunction, as the Clean Up process may have happened before the EventQueue is processed. Too big of a threshold value may also increase the number of rows to EventQueue table, which can lead to general Sitecore performance issues.
- For Sitecore 9.2 and above, investigate whether a dedicated Indexing role can help.
- In regards to Azure Web Apps sizing, I strongly recommend using S1 for non production, and P1V2 for Production. The Free and Basic tier has very poor network quality against SQL Azure.
The Ugly
The calculation of what should publish and unpublish was a total rewrite. Here are some of the findings that I have experienced, and the community has also experienced.
- Bugs can still happen, due to large legacy code base port over
- Publish calculations were different for Sitecore Publish vs Item Publish. This is now fixed for v4.1
- The .Net Core application introduces another complexity layer, something that developers are currently not used to. It does not contain good toolset compare to the Sitecore Commere Engine.
- There are, till this date, still ongoing issues when working with Cloned Items. Reach out to Sitecore if you are experiencing this immediately. Do not waste time trying to figure it out yourself.
- Language Fallback issues have only been recently resolved to a satisfactory level
Things will go wrong. It is best to turn the logging level to Diagnostic to expose more log information for Sitecore Support purposes. Be warned however, turning on that logging level will produce a lot of logging information. I will not recommend shipping this log to Application Insights, as this will blow out the Application Insights budget very quickly.
The amount of information required whenever logging a Sitecore Support ticket is insane. It takes an average of 45 minutes to raise a Sitecore Support ticket without even providing full databases backups. The required information of the Sitecore Support ticket includes
- Sitecore Support Generated Package
- Publishing Service Logs
- Database export of all Publishing_*, EventQueue, Properties tables in master and publishing target databases
- Sometimes Sitecore Support will request full databases backups
- Content Authors may get desperate, and transfer the item to the publishing target databases instead. When that occurs, the information retrieved for Sitecore Support are now polluted.
Some of my nightmare that I have experienced in v2 of Publishing Service were catastrophic, to the point where the entire web site was down. I will not go into detail as the latest v2 does not cause problem that drastically anymore.
Sitecore MVP Mikael Högberg has written a very detailed blog posts of the problems he encountered https://mikael.com/2019/07/learnings-from-a-year-of-implementing-sitecore-publishing-service/
We are currently toying with the idea of using Publishing Service 3.1.3 on Sitecore 8.2.7 with Publishing Service Module 2.2.1. However this is not officially supported. This will be one of the last resort if bugs that have been fixed for v3.1.3 are not being ported back to v2.
Based on the experiences above, I would only recommend using Publishing Service v3.1.3 above, with a lot of Sitecore patches included for Sitecore 9.0.2.
Should you use it ?
Here is the big question, should you still use it after finding out The Bad and The Ugly ? You will find a lot of performance benefits if:
- The content databases are massive in size, and often contains large publishing operations, and a multilingual environment. Large and frequent publishing jobs can build up into a big publishing queue, which takes time to get process. This can be a problem if scheduled content needs to go out. I have experienced content not publishing at the right time due to massive publishing queue before.
- If there are many concurrent content authors using Sitecore
- Publishing Service can improve performance on CM dramatically, especially in conjunction with a dedicated Indexing role in 9.2 and onwards
Sitecore MVP Jeremy Davis is using Publishing Service to solve a High Availability requirement across multiple Azure Regions. I was discussing this approach with him when he was researching how to tackle this. He has since documented his approach https://dev.to/jermdavis/an-edge-case-of-remote-events-with-publishing-service-5g0n
When setting multiple regions, different publishing targets should be setup. eg. web-southeastaust, web-eastasia etc. Each Solr Cluster should also contain index collections of all databases. Publishing Target drastically reduces the publishing latency due to wide geographical distributions.
Conclusion
I hope you find this blog post useful on Sitecore Publishing Service. Here is me presenting this topic at Sitecore Meetup in Melbourne Australia on Thursday 24th October.
You can find my presentation here on the day. Please excuse me for the cropped screen capture. The audio and my slide deck should be good enough for you to follow very closely. I made a mistake in the presentation stating that Horizontal Scaling is not supported. It is supported as per documentation in https://doc.sitecore.com/developers/92/platform-administration-and-architecture/en/content-publishing--publishing-.html
Youtube: https://www.youtube.com/watch?v=-3cKaZeWB1o
Slide Deck: https://docs.google.com/presentation/d/1ToDmBxDnl091LyoM2KX0pL9_arQQXayPacK-ZrysJv8/edit?usp=sharing
