Today I would like write about a quiet important topic for developers in Sitecore XC, how to persist entities in commerce engine. Once you start implementing your own plugins most probably you will also add new blocks, pipelines and command. And of course if you do that, you will most probably have to manipulate / change data of entities and save them back to Database.
Before I come to the point, where I show you a scenario I encountered in a customer project, I would like to start with writing about how in general the workflow how entities are retrieved from the system, and of course you can save them back after changing some data within your blocks.
Possibilities to retrieve an entity
Let us start with the basic topic of how to retrieve entities within a block. I would say there are basically 3 ways of doing that.
- From Database
- From Context
- From Args
But how do these 3 method differ from each others?
From Args: Let me start with the way of retrieving an entity via args. I guess it’s the most obvious and easiest way of getting an entity. Once you create a block you have to also give the block input and output parameters depending on the location, where you would like to hook in. For example if you would like to create a new block for the IAddCartLinesPipeline and want to hook n between AddCartLineBlock and AddContactBlock, you can see the output parameter of AddCartLineBlock is Sitecore.Commerce.Plugin.Carts.Cart and the input parameter of AddContactBlock is also Sitecore.Commerce.Plugin.Carts.Cart.
So to properly hook into that pipeline at that location, just create a block with input type matching the output type of the previous block and output type matching the input type of the next block.
If your new block now needs to modify the given cart, you now can easily access it via args object of the Run method, modify it and pass it on to the next block.
From Context: The next possibility to get entities within your block is to use the given CommercePipelineExecutionContext. Such an approach you have to do if you want to access entities, which are not OOTB passed via arguments through the pipeline to your block. Within your block you just have to access CommerceContext and extract your entity and you are able to change data, which are automatically updated in context.
In this example we are in the standard RMA pipeline of sitecore. Within that pipeline an object of type ReturnMerchandiseAuthorization is passed through the blocks. In addition the corresponding order object is saved to context, where you are able to get it from. Of course one prerequisite of doing that is, that some previous block saved the necessary entity to the context, from which you now want to grab it.
From Database: The last option to get an entity to be able to change data is the most direct one. To do that you just have to use the IFindEntityPipeline.
The result of that pipeline is a new instance of the entity based on the current state of the Database.
Possibility to save an entity
Now let’s shortly talk about, how you can save an entity to database again. This step is absolutely necessary when working with entities. If you would not save the changed entities in the end of processing back to Database, all the changes you made within the execution, within the pipelines and blocks, would be lost again.
Basically it is quiet easy to do that. To save an entity to database, you just have to call IPersistEntityPipeline with your entity.
Real world scenario
Now with these things in mind, let us have a look at an example I encountered, while working with Sitecore XC. The graphic below shows a sequence diagram of a pipeline created / used in Sitecore XC.
Within this execution you can see, which pipelines are called when and from whom. In addition it shows, where the entity comes from, which is necessary to process.
Let me shortly describe the given scenario. It all starts with the execution of a first pipeline. Based on the given arguments the first block in the pipeline just grabs the necessary entity from Database. Because this entity is not part of the arguments, which are passed through the pipeline, this block also saves that entity to context. After doing something the first block of the first pipeline finishes and the second block is executed. As you would expect it, the second block grabs the necessary entity from context and does various changes to the entity. In between this block itself executes another pipeline. The parameter of that new pipeline this time is exactly that entity, so it can be passed via arguments of pipelines and blocks. So the first block of that second pipeline uses that argument to also do something with the entity. This goes on till block two of the second pipeline. This block again executes a new pipeline to let something execute. But this time there is a break in architectural concept. The parameter, which is passed is just an id of the entity, which is needed later on. The only possibility of pipeline three to access the entity now is to grab it again from database. After that, this pipeline makes some changes to the entity and directly saves it back to database. Now the execution of pipeline three ends and after that also of pipeline two. In the end of pipeline two, there is a block, which takes the entity, which was at that time passed via argument from pipeline one, and saves the entity back to database. Then also pipeline two ends. Within the first, initial pipeline, now a block is executed, which is responsible for grabbing the entity from context and saves it permanently back to database to persist all the changes made in between.
What are the problems of that composition of pipelines?
The main question, you could also ask yourself at this point is:
How does the entity look like in the end of the overall processing?
The answer is quiet clear. It does not look like we would expect it to be. What I observed in the given scenario was an entity, which was in deed saved a few times, but always with a different state. Let me also describe, what most probably led to the inconsistency within the system.
As you know it from the beginning, we have multiple ways of getting an entity and that we always have to persist an entity to permanently save it. Within the given scenario, we have the problem, that all the ways of retrieving an entity for processing is mixed up together.In our scenario the main pain point lies in Pipeline 4 Block 1. This pipeline completely ignores given context and object within that context. The reason was quiet obvious. The pipeline was initially not designed to be placed, where it was finally placed, so it was not know, that there is an object in context, which can be used at this point, and more over, depending on the scenario, where it is used, there is not always a given entity in context, which could be used. In our scenario Pipeline 1 Block 1 prepares the context. If these pipelines are not there on other scenarios, also the context would not be prepared. But the workaround, which was approached there lead to the visible inconsistency.
Assume you have your entity and you are changing it within your pipelines and blocks. These changes are saved in the end to context. Now, when we enter the last pipeline, we grab a new instance of our entity from database. This results in a new entity, which of course does not include all the changes made earlier in the execution. Within that block we make some new changes like adding a necessary component with additional information. Once we save that entity it is permanently saved to database. The result is an updated entity in database, which only includes the changes of that specific block and nothing more.
Now, we could imagine, we have to scenarios. The first would be, that this new entity is given back to the previous blocks and pipelines and then be used in context. The other scenario would be to return nothing and just continue working on the saved entity in context. Let me say at this point, that both approaches are not optimal. It would only shift the problem and only the things, which are lost and overwritten would change.
In out scenario nothing was returned, so we continued working on the context entity object. In the end we know, that the last block now grabs the context object and again saves that entity to database to permanently save all the changes made while execution. But because, we never returned the new entity from the last pipeline, our context entity does not know these changes and on our last save, these changes are not included. In the following graphic you can see the operations, which directly have an influence on each others.
The reaction of the commerce system to such scenarios is now quiet different. In best case you directly get a commerce engine error, which tells you that there is a concurrency exception, that the version you try to save is outdated.
“Text”: “SQL:block:persistentity.Update.Exception: SQL.UpdateEntity.Fail: Id=’Entity-Catalog-Support Dummy’|Try=’1’|Environment=’Entity-CommerceEnvironment-HabitatAuthoring’|Message=’Concurrency error: The Entity version supplied (3) is no longer the current version.’|Number=’50000’|Procedure=’dbo.sp_CommerceEntitiesUpdateWithSharding’|Line=’26′”,
In worst case, the save is successfully executed AND the changes you made in between are completely overwritten.
Now imagine you would have returned the changed object back to the previous block and that you would have updated the context. Of course this would have saved the changes made in pipeline four. BUT because you would just overwrite the context object and you grabbed the entity from database, ALL the changes made earlier would be completely lost and overwritten at this point. So the issue would only be shifted.
What should be done?
Now the question is, how can you get rid of all these problems and issues?
The answer of that is also quiet simple. Do not make things more complicated, than they are already. Of course it is possible to call pipelines within pipelines within pipelines and split up the tasks. But then you have to keep an eye on every single pipeline; what goes on, what comes out, what is present in context. And all the time try to avoid grabbing exiting and adapted entities again from database.
In general, when you design a pipeline always keep in mind, that one of the first blocks checks and initializes everything and that, if possible, just the last block is responsible for saving the entity. If you work with pipelines in between, keep in mind to give the new pipeline the current entity and also to take it back after execution, so all the changes are just made on one single instance of the entity.
Within the today’s article you had an insight in a real world scenario I observed in a customer project. You learned about, how to get, handle and save entities in pipelines and what issues might happen, if you are not aware of what all the pipelines and blocks do while execution. In the end you also got some personal hints to also avoid such scenarios and get consistent data in the end.