This time I would like to show you, how to create a custom product importer within Sitecore Commerce Engine as a custom plugin. For demonstration reason I also created a small github repo with all the code I present within this article.
Note: This code should be used only for demonstration and learning purpose. It has not been tested extensively right now, and should give you only an idea, of how to create your own importer within Sitecore Commerce Engine. Based on that implementation, your development could be much faster, without stepping into every pitfall 😉
If you followed my blog posts, you will have noticed, that I have already written about the engine in general, revealed some misunderstandings in architecture and concepts and put everything together to create a new demo plugin based on some real customer business needs.
This time I though again about a topic, which comes from real world requirements and which might be complicated to implement without any previous idea or help. The outcome now is a plugin, which demonstrates, how to import all kind of commerce stuff related to a catalog.
The basic architecture used by commerce engine in general consists of Pipelines. Examples of such pipelines are ICalculateSellableItemSellPricePipeline or ICreateCategoryPipeline. Pipelines itselves consists of a various number of Blocks, which are executed one after another, while running through the pipline. If you want to have a look at all the pipelines and blocks used by commerce engine, you can read about it in the Chapter The Node Configuration File explained below.
The NodeConfiguration file explained
Beside the standard commerce engine log file, you can additionally find a NodeConfiguration_***.log file under /wwwroot/logs folder, like seen below.
For every startup of the application such configuration file is created. It holds all the information about used and configured pipelines of the current commerce engine.
An example of such a NodeConfiguration file you can see above. In the beginning it has some basic information about the instance. After that the structure is always the same. You see a pipeline with its same and parameter and with a small indent all the blocks also with parameters.
Above you can see one specific pipeline, the IGetSellableItemPipeline in detail. In red you can see the fully qualified name and in brackets the parameter used by that pipeline. In yellow I highlighted all the block, which belong to the IGetSellableItemPipeline. There are blocks like GetSellableItemBlock or GetSellableItemCatalogBlock. If this pipeline is executed all the blocks are respectively executed one after another from top to bottom. In green you can see two types highlighted. Every pipeline and every block consists of those information. The first parameter describes the input paramter used by the pipeline or the specific pipeline block. And the second parameter describes the output in the end.
Note: The output parameter of one block has to match to the input parameter of the next block 😉
Of course you are always able to create your own pipelines within your plugins from scratch or hook into existing pipelines by adding your own blocks. This is done by adding them correctly within the ConfigureSitecore.cs file. An example of such a configuration is seen below.
The Magic of Pipelines & Commands
The question now is, how do we build now our own import functionality with these information?
If you have a closer look to all the Pipelines, you will recognize, that there are a lot of pipelines we can directly use. For example we have all kind of pipelines to handle a catalog. The ones, which are interesting for us are for example CreateCatalogPipeline, GetCatalogPipeline, EditCatalogPipeline. The big advantage is, that these pipleines are parts of the commerce engine implementation, that they are already used by other parts of the commerce engine, for example within the UI interactions inside the Commerce Business tools and that they consist of all the blocks, which handle underlying functionality, like interacting directly with the database.
To use now such pipelines, commerce engine provides us with corresponding command classes. These commands are responsible for collecting needed parameters, creating the proper argument class, which the pipeline needs including verification and in the end to execute the corresponding pipeline properly. So we also get rid of all the logic to properly execute pipelines.
Whats import to know now for us is, that if we want to use commerce engine functionality within our custom plugin, we only have to find the proper command class, instantiate it and use its built in Process method to initiate the corresponding pipeline, which itself initiates the corresponding pipeline blocks.
Basic structure of the importer project
Based on the information we found out in the chapter before, we now can start implementing our own importer.
Below you can see the basic structure of the Plugin.Sample.Importer project.
This implementation consists of
- dedicated importer classes, each responsible for a specific commerce entity type (Catalog, Category, SellableItem, SellableItemVariation) within Services
- dedicated input parameter classes for the importer within Models
- an extension class for handling some logic within parameterDTOs within Extensions
- and a Controller within Controllers, which acts as external interface to test all the implemented functionality with Postman.
- Next Step: Next step would be to create an own Importer Minion within Minions, which interacts with an external PIM system, retrieves all the Catalog information, including Categories, SellableItems and SellableItemVariations and uses all the Services to import these information into the Commerce Engine.
Implementation of the importer
Now we will have a closer look at the individual implementations of the importers. Each one handled in a separated sub chapter, cause even if the basic approach of every importer is the same, importing a catalog, a category, a product, or variant, has its own pitfalls and procedures we have to obtain to get it work.
As we already know from the previous chapters, if we want to build our own importer, we can use commerce engine OOTB functionality to do so. So lets have a look at the implementation
Above you can directly see all the commands used by my catalog import. There are basic ones like GetCatalogCommand, EditCatalogCommand or CreateCatalogCommand. But there are also some ones, which are not so obvious, that they have to be used for proper result. These Commands are for example AssociateCatalogToBookCommand, DisassociateCatalogToBookCommand for Price- and Promotionbook handline or AssociateCatalogToInventorySetCommand and DisassociateCatalogFromInventorySetCommand.
Maybe some of you ask yourself, how can you find out, that these commands are needed? The answer is again, as always, I reflectored the existing code and existing pipelines, like DoUxActionPipeline. Within that Pipeline every UX interaction is handled from the Business Tools, including creating and editing catalogs and other commerce entities. While reflectoring this pipeline, I found out about the existence of these pipelines and commands, which are needed to fulfill such needs.
All these commands of course can be instantiated via Constructor Injection as seen above. Maybe to now get an idea of why it is useful to have such kind of implementation ready, cause even if you use existing pipelines and commands, you have to know about them and use them properly, which is in deed not so easy without any experience or trial and error. But now lets have a look at the code.
The ExecuteImporter function accepts as parameter the current context, the previously mentioned parameter object and a flag if existing catalog should be updated if already existing.
Now we can examine the single steps of importing / updating a catalog
- The first step of our function is, that we try to create a catalog based on the provided data. This command returns null if the catalog already exists, based on the provided Name. Otherwise it would return an instance of the created catalog.
- Next step if to check if the catalog was created previously or not to determine if it is allowed to be updated
- If the catalog was already existing, we now want to get this catalog from the commerce engine to work with that later on
- The next step is very catalog specific. It is about associating the current catalog to a Pricebook, Promotionbook and DefaultInventorySet. To achieve an edit within these properties, we have to be careful, when to call associate and have to be sure that, if it was already associated, it is disassociated before. The code below checks if the current catalog is already associated and if so, if the new association is different than before. If so it automatically disassociates the current book and associates the new one, if it is existing within Commerce Engine. For now these steps are done for both books and the inventory set in the same way only with the individual command.
- The last step now is to edit the current catalog entity and set the new information, including the information about Pricebook, Promotionbook and DefaultInventorySet. Note: Be careful, that these information match the associations done in the step before! It is possible to add here other names, than used in the assoication commands and therefore produce a missmatch. Note-2: Currently there seems to be an issue with programmatic associations and the Business Tools UX. It is possible that, even when you correctly use the commands for association and disassociation, the UX will display wrong, old information even if the engine itself has done the changes correctly. You can check, that the engine already did the changes and the UX display old information, if you then want to manually disassociate the catalog from the price book, that this is not possible and ends in an error message, that the catalog is not associated to the price book, even if it is displayed in the list below. So it is possible, that if you have changed the price book this way, the Business Tools will still display the catalog as part of the old price book, even if it is not anymore. Sitecore currently checks if the behavior is a feature wish or bug. Current Ticket ID is 515201. I will add a public reference number later on if one is existing
Next we will have a deeper look at the category import functionality. Compared to the catalog importer, this one is much less specific, as you will see below.
You can directly see, that we don’t need that extensive number of commands. We only need commands for creation, get, edit, association and “disassociation”.
The ExecuteImport has also nearly the same strcuture, than it was with the Catalog importer. Only the paramerter object has a different type for the specific category needs.
- The first step again is, that we try to create a new entity, this time of type category. Therefore we build up the entity ID of the category.
- Next we check if the category already existed and if we are permitted to update it.
- If the category exists, we try to get an instance of that.
- With that instance we try to update the values of that category, in case it was not created previously. If it would be created previously, an edit would be useless, so we try to skip that step then.
- In the end, we have to handle parent associations of the given category. Based on the list of given parents, we associate each as parent with the current category. One special thing at this point is, that we have to be careful, if the parent is a category or the catalog itself. Therefore we have to build up the parentId string diffrently.
Note: Maybe some of your already have seen it. Within step 5, when we associate parents to our category, there is missing one thing, which is essential. Currently we do NOT disassociate parents from the current category. Based on the current associations on the category and the given parentName list from the parameters and the deleteRelationShipCommand it would be possible to also do that. But right now this is not so easy to raelize. Below you see, a first approach of doing that
As mentioned earlier, with the deleteRelationshipCommand, everything would work well. The problem here is to get all the needed information. Parameters are SourceName, TargetName and RelationShipType. The first problem started with the relationshipType of type string.
What do we have to insert here?
You can see within the comment above, that I found out, what the possible values are, which can be inserted. How I did that? – Again with reflection.
First I had a look a the DeleteRelationshipPipeline and saw the two block belonging to that.
Then I examined the two blocks, till I found the place where the relationshipType is used and found out the possible values.
So this problem was solved. The next issue was now to fill sourceName and targetName properly. TargetName is no problem, cause its the current category entity. But the sourceName is a problem I did not solve till now. You can see, that I experimented with a hardcoded value, which worked perfectly fine. But how do I find out all the current associated categories / catalog from the current category, to find out which one is missing and therefore has to be disassociated. The category class holds information about parentCategoriesList and parentCatalogList as seen above in the code snippet. But there properties only hold Sitecore Item IDs and not entity IDs of commerce engine. At this point I stopped the investigation and opened a Sitecore ticket, of how to properly retrieve a list of parent categories / catalog. Once I get these information, I can continue implementing the disassociation.
The approach of the implementation of the product importer is nearly the same, than with the category importer. Therefore I will only show an overview of the whole process and will only point out some differences.
Above you can the the whole process from creation, update check, product retrieval, edit and association, like it is also within category importer. Again we have here the disassociation not implemented, cause of the same reasons.
One special issue I encountered with the editSellableItemCommand. This command accepts an existing instance of a sellableitem as parameter. Therefore I first use getSellableItemCommand to retieve an instance of the current sellableItem, edit it with the new values like, displayName, Description, tags etc. and use that changed instance in the edit command. Unfortunatly this currently throws an harsh SQL server PK already existing exception. It seems that the edit command tries to create a new entity of that already existing sellableItem, which would explain the given error. This issue was also already reported to Sitecore Support.
Another strange thing with this importer was the usage of the getSellableItemCommand. This command accepts an ItemId as parameter. At first I though I have to easily insert the entityId of the sellableitem, in the way I also added catalog or category entity IDs. The result was an error like this.
Expecting a CatalogId, ProductId and optionally a VariantId in the ItemId: Entity-SellableItem-0000004. Correct format is ‘catalogId|productId|[variantId]’.
So i changed my parameter to “HahniCatalogFINAL2|0000004”. But this produced exactly the same error. Therefore I started again to reflect the logic to see, what is expected as parameter.
What I learned from that piece of code is, that even if VariantId is not mandatory, I have to add the pipe separator in the end, so that we get an array of length 3 after splitting. Otherwise nothing works. So in the end my parameter looked loke this “HahniCatalogFINAL2|0000004|”, which finally worked well.
Variant importer again looks very similair to all the other importer, especially the product import. Cause most of the time, if we try to access a variant of a product, we first access the product and then access the variants on that product to edit them and then save again the whole product. Therefore most of the used commands are the same, then in the products importer.
Below you see the basic approach of creating and editing variants.
The only thing we have to know at this point is, that when we want to access product variants, we do that in first getting the product itself and then accessing the variants, edit them and saving again the whole sellable item. At this place we again have the same issue with editSellableItemCommand than we hat with product importer. In addition we do not need to set any association, because this is already done on creation time, with giving the entity ID of the product name.
How to test the Importer
To test all the created fucntionality, I added a dedicated CommandsController, which is responsible to expose all the ExecuteImports from my importer for external usage. Below you see the example of the CreateOrUpdateCatalog function which takes all the parameter, builds the parameter object and calls the ExecuteImport function of the CatalogImporter. In a real world scenario you would either way use such kind of external inferface or especially with PIM imports a scheduled task(Minion) to do all that.
This function is registered within ConfigureOpsServiceApiBlock with its name, parameter and return type as seen below.
The ConfigureOpsServiceApiBlock itself is registered within ConfigureSitecore.
To test the controller call you can simply use a postman, like seen below.
Within this article you learned about some architecture regarding commerce engine pipelines and commands. With these information, we were able to create some basic importer to demonstrate the usage of these commands and how they work together. In the end we created some basic plugin, with which we are able to build a new catalog with categories, sellable items and variants of sellable items. We learned how to avoid some tricky pitfalls, and which command has to be used with which parameter, to save our valuable time in the future.
But nevertheless we still have some issues and todos left open with editing sellable items and variants after creation and some mismatch in displaying changed associations within Business Tools. Hopefully in the near future we will get some feedback on these issues to finalize the basic functionalities.
Next steps then would be
- Create a minion as demonstration of external PIM system integration
- Add importer functionalities to also import custom view properties generated with the Composer
- Implement proper disassociation of categories and sellable items based on provided information by Sitecore Support.