Sitecore XC 9.0.2 – Walkthrough of creating your own product importer with various pitfalls

Sitecore XC 9.0.2 – Walkthrough of creating your own product importer with various pitfalls

This time I would like to show you, how to create a custom product importer within Sitecore Commerce Engine as a custom plugin. For demonstration reason I also created a small github repo with all the code I present within this article.

Note: This code should be used only for demonstration and learning purpose. It has not been tested extensively right now, and should give you only an idea, of how to create your own importer within Sitecore Commerce Engine. Based on that implementation, your development could be much faster, without stepping into every pitfall ūüėČ

Idea

If you followed my blog posts, you will have noticed, that I have already written about the engine in general, revealed some misunderstandings in architecture and concepts and put everything together to create a new demo plugin based on some real customer business needs.

This time I though again about a topic, which comes from real world requirements and which might be complicated to implement without any previous idea or help. The outcome now is a plugin, which demonstrates, how to import all kind of commerce stuff related to a catalog.

Basic Architecture

The basic architecture used by commerce engine in general consists of Pipelines. Examples of such pipelines are ICalculateSellableItemSellPricePipeline or ICreateCategoryPipeline. Pipelines itselves consists of a various number of Blocks, which are executed one after another, while running through the pipline. If you want to have a look at all the pipelines and blocks used by commerce engine, you can read about it in the Chapter The Node Configuration File explained below.

The NodeConfiguration file explained

Beside the standard commerce engine log file, you can additionally find a NodeConfiguration_***.log file under /wwwroot/logs folder, like seen below.

2018-09-09 13_42_32-logs.png

For every startup of the application such configuration file is created. It holds all the information about used and configured pipelines of the current commerce engine.

2018-09-09 13_42_01-_C__inetpub_wwwroot_CommerceAuthoring_Sc9_wwwroot_logs_NodeConfiguration_Deploym.png

An example of such a NodeConfiguration file you can see above. In the beginning it has some basic information about the instance. After that the structure is always the same. You see a pipeline with its same and parameter and with a small indent all the blocks also with parameters.

2018-09-09 13_39_57-_C__inetpub_wwwroot_CommerceAuthoring_Sc9_wwwroot_logs_NodeConfiguration_Deploym.png

Above you can see one specific pipeline, the IGetSellableItemPipeline in detail. In red you can see the fully qualified name and in brackets the parameter used by that pipeline. In yellow I highlighted all the block, which belong to the IGetSellableItemPipeline. There are blocks like GetSellableItemBlock or GetSellableItemCatalogBlock.  If this pipeline is executed all the blocks are respectively executed one after another from top to bottom. In green you can see two types highlighted. Every pipeline and every block consists of those information. The first parameter describes the input paramter used by the pipeline or the specific pipeline block. And the second parameter describes the output in the end.

Note: The output parameter of one block has to match to the input parameter of the next block ūüėČ

Of course you are always able to create your own pipelines within your plugins from scratch or hook into existing pipelines by adding your own blocks. This is done by adding them correctly within the ConfigureSitecore.cs file. An example of such a configuration is seen below.

2018-09-09 13_59_18-Customer.Sample.Solution - Microsoft Visual Studio.png

The Magic of Pipelines & Commands

The question now is, how do we build now our own import functionality with these information?

If you have a closer look to all the Pipelines, you will recognize, that there are a lot of pipelines we can directly use. For example we have all kind of pipelines to handle a catalog. The ones, which are interesting for us are for example CreateCatalogPipeline, GetCatalogPipeline, EditCatalogPipeline. The big advantage is, that these pipleines are parts of the commerce engine implementation, that they are already used by other parts of the commerce engine, for example within the UI interactions inside the Commerce Business tools and that they consist of all the blocks, which handle underlying functionality, like interacting directly with the database.

To use now such pipelines, commerce engine provides us with corresponding command classes. These commands are responsible for collecting needed parameters, creating the proper argument class, which the pipeline needs including verification and in the end to execute the corresponding pipeline properly. So we also get rid of all the logic to properly execute pipelines.

Whats import to know now for us is, that if we want to use commerce engine functionality within our custom plugin, we only have to find the proper command class, instantiate it and use its built in Process method to initiate the corresponding pipeline, which itself initiates the corresponding pipeline blocks.

Basic structure of the importer project

Based on the information we found out in the chapter before, we now can start implementing our own importer.

Below you can see the basic structure of the Plugin.Sample.Importer project.

2018-09-09 14_26_18-Customer.Sample.Solution - Microsoft Visual Studio

This implementation consists of

  • dedicated importer classes, each responsible for a specific commerce entity type (Catalog, Category, SellableItem, SellableItemVariation) within Services
  • dedicated input parameter classes for the importer within Models
  • an extension class for handling some logic within parameterDTOs within Extensions
  • and a Controller within Controllers, which acts as external interface to test all the implemented functionality with Postman.
  • Next Step:¬†Next step would be to create an own Importer Minion within Minions, which interacts with an external PIM system, retrieves all the Catalog information, including Categories, SellableItems and SellableItemVariations and uses all the Services to import these information into the Commerce Engine.

Implementation of the importer

Now we will have a closer look at the individual implementations of the importers. Each one handled in a separated sub chapter, cause even if the basic approach of every importer is the same, importing a catalog, a category, a product, or variant, has its own pitfalls and procedures we have to obtain to get it work.

Catalog Importer

As we already know from the previous chapters, if we want to build our own importer, we can use commerce engine OOTB functionality to do so. So lets have a look at the implementation

2018-09-09 14_49_56-Customer.Sample.Solution - Microsoft Visual Studio

Above you can directly see all the commands used by my catalog import. There are basic ones like GetCatalogCommand, EditCatalogCommand or CreateCatalogCommand. But there are also some ones, which are not so obvious, that they have to be used for proper result. These Commands are for example AssociateCatalogToBookCommand, DisassociateCatalogToBookCommand for Price- and Promotionbook handline or AssociateCatalogToInventorySetCommand and DisassociateCatalogFromInventorySetCommand.

Maybe some of you ask yourself, how can you find out, that these commands are needed? The answer is again, as always, I reflectored the existing code and existing pipelines, like DoUxActionPipeline. Within that Pipeline every UX interaction is handled from the Business Tools, including creating and editing catalogs and other commerce entities. While reflectoring this pipeline, I found out about the existence of these pipelines and commands, which are needed to fulfill such needs.2018-09-09 14_50_14-Customer.Sample.Solution - Microsoft Visual Studio

All these commands of course can be instantiated via Constructor Injection as seen above. Maybe to now get an idea of why it is useful to have such kind of implementation ready, cause even if you use existing pipelines and commands, you have to know about them and use them properly, which is in deed not so easy without any experience or trial and error. But now lets have a look at the code.

The ExecuteImporter function accepts as parameter the current context, the previously mentioned parameter object and a flag if existing catalog should be updated if already existing.

2018-09-09 15_05_10-Customer.Sample.Solution - Microsoft Visual Studio

Now we can examine the single steps of importing / updating a catalog

  1. The first step of our function is, that we try to create a catalog based on the provided data. This command returns null if the catalog already exists, based on the provided Name. Otherwise it would return an instance of the created catalog.2018-09-09 15_05_19-Customer.Sample.Solution - Microsoft Visual Studio
  2. Next step if to check if the catalog was created previously or not to determine if it is allowed to be updated2018-09-09 15_05_25-Customer.Sample.Solution - Microsoft Visual Studio
  3. If the catalog was already existing, we now want to get this catalog from the commerce engine to work with that later on2018-09-09 15_05_33-Customer.Sample.Solution - Microsoft Visual Studio
  4. The next step is very catalog specific. It is about associating the current catalog to a Pricebook, Promotionbook and DefaultInventorySet. To achieve an edit within these properties, we have to be careful, when to call associate and have to be sure that, if it was already associated, it is disassociated before. The code below checks if the current catalog is already associated and if so, if the new association is different than before. If so it automatically disassociates the current book and associates the new one, if it is existing within Commerce Engine. For now these steps are done for both books and the inventory set in the same way only with the individual command.2018-09-09 15_05_42-Customer.Sample.Solution - Microsoft Visual Studio
  5. The last step now is to edit the current catalog entity and set the new information, including the information about Pricebook, Promotionbook and DefaultInventorySet.  Note: Be careful, that these information match the associations done in the step before! It is possible to add here other names, than used in the assoication commands and therefore produce a missmatch. Note-2: Currently there seems to be an issue with programmatic associations and the Business Tools UX. It is possible that, even when you correctly use the commands for association and disassociation, the UX will display wrong, old information even if the engine itself has done the changes correctly. You can check, that the engine already did the changes and the UX display old information, if you then want to manually disassociate the catalog from the price book, that this is not possible and ends in an error message, that the catalog is not associated to the price book, even if it is displayed in the list below.2018-09-01 08_05_48-Microsoft Edge.png So it is possible, that if you have changed the price book this way, the Business Tools will still display the catalog as part of the old price book, even if it is not anymore. Sitecore currently checks if the behavior is a feature wish or bug. Current Ticket ID is 515201. I will add a public reference number later on if one is existing2018-09-09 15_05_51-Customer.Sample.Solution - Microsoft Visual Studio

Category Importer

Next we will have a deeper look at the category import functionality. Compared to the catalog importer, this one is much less specific, as you will see below.

2018-09-09 15_34_30-Customer.Sample.Solution - Microsoft Visual Studio

You can directly see, that we don’t need that extensive number of commands. We only need commands for creation, get, edit, association and “disassociation”.

The ExecuteImport has also nearly the same strcuture, than it was with the Catalog importer. Only the paramerter object has a different type for the specific category needs.2018-09-09 16_49_09-Customer.Sample.Solution - Microsoft Visual Studio

  1. The first step again is, that we try to create a new entity, this time of type category. Therefore we build up the entity ID of the category.2018-09-09 16_49_16-Customer.Sample.Solution - Microsoft Visual Studio
  2. Next we check if the category already existed and if we are permitted to update it.2018-09-09 16_49_22-Customer.Sample.Solution - Microsoft Visual Studio
  3. If the category exists, we try to get an instance of that.2018-09-09 16_49_27-Customer.Sample.Solution - Microsoft Visual Studio
  4. With that instance we try to update the values of that category, in case it was not created previously. If it would be created previously, an edit would be useless, so we try to skip that step then.2018-09-09 16_49_36-Customer.Sample.Solution - Microsoft Visual Studio
  5. In the end, we have to handle parent associations of the given category. Based on the list of given parents, we associate each as parent with the current category. One special thing at this point is, that we have to be careful, if the parent is a category or the catalog itself. Therefore we have to build up the parentId string diffrently.2018-09-09 16_49_46-Customer.Sample.Solution - Microsoft Visual Studio

Note: Maybe some of your already have seen it. Within step 5, when we associate parents to our category, there is missing one thing, which is essential. Currently we do NOT disassociate parents from the current category. Based on the current associations on the category and the given parentName list from the parameters and the deleteRelationShipCommand it would be possible to also do that. But right now this is not so easy to raelize. Below you see, a first approach of doing that

2018-09-09 17_08_01-Customer.Sample.Solution - Microsoft Visual Studio

As mentioned earlier, with the deleteRelationshipCommand, everything would work well. The problem here is to get all the needed information. Parameters are SourceName, TargetName and RelationShipType. The first problem started with the relationshipType of type string.

What do we have to insert here? 

You can see within the comment above, that I found out, what the possible values are, which can be inserted. How I did that? – Again with reflection.

First I had a look a the DeleteRelationshipPipeline and saw the two block belonging to that.

2018-09-09 17_14_02-_C__inetpub_wwwroot_CommerceAuthoring_Sc9_wwwroot_logs_NodeConfiguration_Deploym.png

Then I examined the two blocks, till I found the place where the relationshipType is used and found out the possible values.

2018-09-09 17_13_36-JetBrains dotPeek.png

So this problem was solved. The next issue was now to fill sourceName and targetName properly. TargetName is no problem, cause its the current category entity. But the sourceName is a problem I did not solve till now. You can see, that I experimented with a hardcoded value, which worked perfectly fine. But how do I find out all the current associated categories / catalog from the current category, to find out which one is missing and therefore has to be disassociated. The category class holds information about parentCategoriesList and parentCatalogList as seen above in the code snippet. But there properties only hold Sitecore Item IDs and not entity IDs of commerce engine. At this point I stopped the investigation and opened a Sitecore ticket, of how to properly retrieve a list of parent categories / catalog. Once I get these information, I can continue implementing the disassociation.

Product Importer

 The approach of the implementation of the product importer is nearly the same, than with the category importer. Therefore I will only show an overview of the whole process and will only point out some differences.

2018-09-09 17_47_30-Customer.Sample.Solution - Microsoft Visual Studio

Above you can the the whole process from creation, update check, product retrieval, edit and association, like it is also within category importer. Again we have here the disassociation not implemented, cause of the same reasons.

One special issue I encountered with the editSellableItemCommand. This command accepts an existing instance of a sellableitem as parameter. Therefore I first use getSellableItemCommand to retieve an instance of the current sellableItem, edit it with the new values like, displayName, Description, tags etc. and use that changed instance in the edit command. Unfortunatly this currently throws an harsh SQL server PK already existing exception. It seems that the edit command tries to create a new entity of that already existing sellableItem, which would explain the given error. This issue was also already reported to Sitecore Support.

Another strange thing with this importer was the usage of the getSellableItemCommand.  This command accepts an ItemId as parameter. At first I though I have to easily insert the entityId of the sellableitem, in the way I also added catalog or category entity IDs. The result was an error like this.

Expecting a CatalogId, ProductId and optionally a VariantId in the ItemId: Entity-SellableItem-0000004. Correct format is ‘catalogId|productId|[variantId]’.

So i changed my parameter to¬†“HahniCatalogFINAL2|0000004”.¬†But this produced exactly the same error. Therefore I started again to reflect the logic to see, what is expected as parameter.

2018-09-09 17_56_40-JetBrains dotPeek

What I learned from that piece of code is, that even if VariantId is not mandatory, I have to add the pipe separator in the end, so that we get an array of length 3 after splitting. Otherwise nothing works. So in the end my parameter looked loke this¬†“HahniCatalogFINAL2|0000004|”,¬†which finally worked well.

Variant Importer

Variant importer again looks very similair to all the other importer, especially the product import. Cause most of the time, if we try to access a variant of a product, we first access the product and then access the variants on that product to edit them and then save again the whole product. Therefore most of the used commands are the same, then in the products importer.

2018-09-09 18_07_27-Customer.Sample.Solution - Microsoft Visual Studio

Below you see the basic approach of creating and editing variants.

2018-09-09 18_05_21-Customer.Sample.Solution - Microsoft Visual Studio.png

The only thing we have to know at this point is, that when we want to access product variants, we do that in first getting the product itself and then accessing the variants, edit them and saving again the whole sellable item. At this place we again have the same issue with editSellableItemCommand than we hat with product importer. In addition we do not need to set any association, because this is already done on creation time, with giving the entity ID of the product name.

How to test the Importer

To test all the created fucntionality, I added a dedicated CommandsController, which is responsible to expose all the ExecuteImports from my importer for external usage. Below you see the example of the CreateOrUpdateCatalog function which takes all the parameter, builds the parameter object and calls the ExecuteImport function of the CatalogImporter. In a real world scenario you would either way use such kind of external inferface or especially with PIM imports a scheduled task(Minion) to do all that.

2018-09-09 14_46_58-Customer.Sample.Solution - Microsoft Visual Studio

This function is registered within ConfigureOpsServiceApiBlock with its name, parameter and return type as seen below. 2018-09-09 14_47_10-Customer.Sample.Solution - Microsoft Visual Studio

The ConfigureOpsServiceApiBlock itself is registered within ConfigureSitecore.

2018-09-09 17_28_06-Customer.Sample.Solution - Microsoft Visual Studio

To test the controller call you can simply use a postman, like seen below.

2018-09-09 14_47_38-Customer.Sample.Solution - Microsoft Visual Studio

Conclusion

Within this article you learned about some architecture regarding commerce engine pipelines and commands. With these information, we were able to create some basic importer to demonstrate the usage of these commands and how they work together. In the end we created some basic plugin, with which we are able to build a new catalog with categories, sellable items and variants of sellable items. We learned how to avoid some tricky pitfalls, and which command has to be used with which parameter, to save our valuable time in the future.

But nevertheless we still have some issues and todos left open with editing sellable items and variants after creation and some mismatch in displaying changed associations within Business Tools. Hopefully in the near future we will get some feedback on these issues to finalize the basic functionalities.

Next steps then would be

  • Create a minion as demonstration of external PIM system integration
  • Add importer functionalities to also import custom view properties generated with the Composer
  • Implement proper disassociation of categories and sellable items based on provided information by Sitecore Support.

Advertisements

10 thoughts on “Sitecore XC 9.0.2 – Walkthrough of creating your own product importer with various pitfalls

  1. Hi, Christian!
    It is a great article and I eager to try it but I can’t figure out what headers you use in your postman tests. Are they same as other Sitecore 9 postman tests (like authorization token, shop name, etc)? Could you add them to your project?

    Like

      1. Existing call for import catalog uses Shops Api, and you use Ops api, is that changed in XC 9.0.3 since 9.0.2? I’m currently using this version.

        Like

      2. I used OpsAPI cause in my oppinion it better fits to Dev Ops operations and mantainance, which uses Ops API.
        In the end it is up to you how you want to have it available, either way in shops or ops api. If you want to habve it available under shops API you just have to put the ConfigureServiceAPIBlock into the Shops Pipeline instead of Ops pipeline

        Like

  2. Thanks a lot for this guide, I’m creating catalogs now, they appear in DB and Business Tools Merchandising, but I still get errors like
    “MessageDate”: “2019-01-30T15:52:41.0753253Z”,
    “Code”: “Error”,
    “Text”: “SQL:block:persistentity.Update.Exception: SQL.UpdateEntity.Fail: Id=’Entity-Catalog-Support Test16’|Try=’1’|Environment=’Entity-CommerceEnvironment-HabitatAuthoring’|Message=’Concurrency error: The Entity version supplied (2) is no longer the current version.’|Number=’50000’|Procedure=’dbo.sp_CommerceEntitiesUpdateWithSharding’|Line=’26′”,
    “CommerceTermKey”: “SQL:block:persistentity”
    Google haven’t help me, or have answers like IIS reset, which obviously haven\t helped me. What do you suggest? Where should I look?

    Like

    1. Hello, check which entity version you are trying to grab / edit and save. There could be an inconsistent state of entity versions.
      Try to increment the entity version manually before you save the changed catalog.
      Normally commerce engine should take care of that on it’s own, when using persist entity, but maybe with editcatalog there is an issue

      I remember such kind of issue when working with a commerce entities and trying to save entities. Imagine the case of grabbing a commerce entity with e.g. version 2 and changing that. In parallel grabbing it again and again with version 2 and changing that. If you then try to save the first one, everything is fine. If you then try to save the second one, the system respons with such kind of error message to avoid merge conflicts.
      Hope that this already helps you.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s