1. General test conditions (baseline)
Test machine configuration
Intel Core i7-855U CPU @ 1.8 GHz
16,0 GB RAM
x64 Windows 10 Enterprise Operation System
Software Components
VirtoCommerce 3.0.0
Microsoft SQL Server 2016 Express Edition (64-bit)
Windows 10 Enterprise
Test data
Products - 10859
Variations - 7854
Categories - 894
Product properties - 344461
Prices - 37745
Pricelists - 20
Platform settings
Platform version - 3.1.0
Search engine - Lucene
Caching:CacheSlidingExpiration - 0:15:00
Catalog.Search.UseFullObjectIndexStoring - false
2. The tests were performed using the following scenarios:
#1 Indexed search products in the categories
endpoint - POST api/catalog/search/products
products page size - 20
response group - Assets|Properties|SeoInfo|Images|Outlines
unique data requests - 704
test method - sequential
#2 Get products by ids
endpoint - POST api/catalog/products/plenty
page size - 1-20 (random)
response group - Assets|Properties|SeoInfo|Images|Outlines
unique data requests - 1372
test method - sequential
#3 Save shopping cart
endpoint - POST api/carts
cart structure - 1 cart line item, no shipment and payment
uniqued data requests - 3K
test method - sequential
#4 Calculate prices for products
endpoint - POST api/pricing/evaluate
products count - 1-20
unqiue requests - 1372
#5 Create a new order
endpoint - POST api/order/customerOrders
line items - 7
shipments - 1
payments - 1
addresses - 1
#6 Mixed tests (consists from all these scenarios with the following % distribution)
Indexed search categories products - 30%
Get products by ids - 40%
Save exists shopping cart - 15%
Evaluate prices for products - 10%
Create a new order - 5%
3. Capacity test results
Max concurrent requests - 300
Initial concurrent requests count - 1
Step duration - 10 sec
Step concurrent requests count - 20
Load pattern - step
We conduct the same test that we performed on the 2nd version. This confirmed that there were no significant changes, except for some maintenance tests such as Full-indexation and restart time. The 3rd version was faster compared to the 2nd.
4. Conclusion
The performed capacity test has shown that the platform API preserves the desired quality of service only with a limited load. When the number of users simultaneously working with the site goes beyond 70 concurrent user sessions, average response time starts to grow proportionally. When the number of users is about 150, the average response time increases to 5 seconds. See the graph above.
The web server resource utilization especially high CPU utilization level, measured during the testing shows that the performance problems can related to application code and first suspect is Json serialization, because average response Json body size is about 50Kbytes.