1b.app
Link copied -

Evaluate app improvements

1. After importing a product with multiple suppliers (Import source.png), through the "CMS ExtraParts" application (box_auto_action_extraparts_import_products.php), duplicate / identical cards of this product are created in OneBox OS, each with a different supplier (Duplicates.png).

-It is necessary that when importing into OneBox OS, only one unique (Article-Brand link) product card is created, with all suppliers of this product (Suppliers grouping by product.png).

2. Application "CMS ExtraParts" (box_auto_action_extraparts_import_products.php) based on the levels (price_group.level) and the sign of the parent category (price_group.id_parent), and in total in CMS 4 levels (Categories.png), imports and builds a tree of categories in OneBox OS (Categories in Box OS.png).

- It is necessary to exclude the import of 2nd level categories so that the category tree in OneBox OS is built without 2nd level groups (they never carry products). That is, the parent (id_parent) of the 1st level groups (level = 1) must be the 3rd level groups (level = 3).

- Exclude categories with price_group.visible=0 from import into OneBox OS (Categories.png).

3. When importing products (box_auto_action_extraparts_import_products.php), the product's category affiliation is taken from the price.id_price_group (main category).

But since a product can belong to several categories at the same time (Additional product categories.png), you need to add the import of values: price.id_price_group1, price.id_price_group2, price.id_price_group3, price.id_price_group4 in Additional product categories (OneBox OS).
Original question is available on version: ru

Answers:

[code]
1. After importing a product with multiple suppliers (Import source.png), through the "CMS ExtraParts" application (box_auto_action_extraparts_import_products.php), duplicate / identical cards of this product are created in OneBox OS, each with a different supplier (Duplicates.png).
-It is necessary that when importing into OneBox OS, only one unique (Article-Brand link) product card is created, with all suppliers of this product (Suppliers grouping by product.png).
[/code]
it could have been inflated because in action the binding to the search by article using the "Uniqueness of articles" setting - in this case, the cards would be created uniquely by the link article + brand, but in reality - by article because the setting is turned off, it does not search
either you need to turn on the setting, or change the logic in action and search for goods by article - this is + 1h if finalized in the form of a setting
I also see that you are searching by id from the database, which is written to the external id - in fact, it is quite likely that you need to refine the setting to turn off the search by external id - to search exclusively by the article and then check the brand compliance - this is + 1h in form of setting
[code]
2. Application "CMS ExtraParts" (box_auto_action_extraparts_import_products.php), based on the levels (price_group.level) and the sign of the parent category (price_group.id_parent), and in total in CMS 4 levels (Categories.png), imports and builds a tree of categories in OneBox OS (Categories in Box OS.png).
- It is necessary to exclude the import of 2nd level categories so that the category tree in OneBox OS is built without 2nd level groups (they never carry products). That is, the parent (id_parent) of the 1st level groups (level = 1) must be the 3rd level groups (level = 3).
- Exclude categories with price_group.visible=0 from import into OneBox OS (Categories.png).
[/code]
in the form of settings - 2h
[code]
3. When importing products (box_auto_action_extraparts_import_products.php), the product's category affiliation is taken from the price.id_price_group (main category).
But since a product can belong to several categories at the same time (Additional product categories.png), you need to add the import of values: price.id_price_group1, price.id_price_group2, price.id_price_group3, price.id_price_group4 in Additional product categories (OneBox OS).
[/code]
in the form of settings - 2h
31.07.2023, 10:25
Original comment available on version: ru

Олександр Григорович
Support EP
Leave a message in this thread and the user's contacts will be shown to you
Good afternoon.
1) I checked the "Uniqueness of articles" setting - it controls only the Article field (not the combination of article + brand) and does not allow creating products with the same article but different brands, which does not suit me.
On this ok - we are finalizing.
2) ok
3) ok
I'm waiting for an invoice.
31.07.2023, 10:53
Original comment available on version: ru


Alexander wrote:
1. After importing a product with multiple suppliers (Import source.png), through the "CMS ExtraParts" application (box_auto_action_extraparts_import_products.php), duplicate / identical cards of this product are created in OneBox OS, each with a different supplier (Duplicates.png).
-It is necessary that when importing into OneBox OS, only one unique (Article-Brand link) product card is created, with all suppliers of this product (Suppliers grouping by product.png).

Improved settings:
"Ignore the uniqueness of the articles in the system and create products by linking the article and brand"
"Exclude product search by external ID"

2. Application "CMS ExtraParts" (box_auto_action_extraparts_import_products.php), based on the levels (price_group.level) and the sign of the parent category (price_group.id_parent), and in total in CMS 4 levels (Categories.png), imports and builds a tree of categories in OneBox OS (Categories in Box OS.png).
- It is necessary to exclude the import of 2nd level categories so that the category tree in OneBox OS is built without 2nd level groups (they never carry products). That is, the parent (id_parent) of the 1st level groups (level = 1) must be the 3rd level groups (level = 3).
- Exclude categories with price_group.visible=0 from import into OneBox OS (Categories.png).

Improved settings:
"Exclude import of 2nd level categories"
"Exclude import of categories with visible=0 attribute"

3. When importing products (box_auto_action_extraparts_import_products.php), the product's category affiliation is taken from the price.id_price_group (main category).
But since a product can belong to several categories at the same time (Additional product categories.png), you need to add the import of values: price.id_price_group1, price.id_price_group2, price.id_price_group3, price.id_price_group4 in Additional product categories (OneBox OS).

Improved settings:
"Import additional product categories"
"Delete non-imported additional categories"
07.08.2023, 15:05
Original comment available on version: ru

Олександр Григорович
Support EP
Leave a message in this thread and the user's contacts will be shown to you
I can't check the improvements. The "Automatization" application stopped working https://core.extraparts.com.ua/app/automatization/
Error: HTTP ERROR 500
07.08.2023, 15:45
Original comment available on version: ru


Олександр Григорович
Support EP
Leave a message in this thread and the user's contacts will be shown to you
1.1. Not all goods, suppliers are imported (import source missing.png)

1.2. Not all provider data is imported during import (suppliers.png)

Notice the price.post_date time missed.

-it can be seen that the failure occurs when a lot of updated data appears in the source (price).

The data update algorithm in the source (price table) is as follows:

Before inserting the data array with the id_provider key, the price and stock fields are set to zero in the price table using the same key.

With the addition/updating of data in the price, the post_date field is also updated, which starts box_auto_action_extraparts_import_products.php which has only 2 minutes. ($postDateFrom = DateTime_Object::Now()->addMinute(-2)->setFormat('Ymd H:i:s')->__toString();).

The selection (Selection.png) shows both updated (969981) rows in the range 05:47:11 - 05:58:44.

The source is updated by the id_provider key 1-2 times a day.

Let this be the first launch, but there is still a chance of skipping when restarting.

Maybe increase the number of minutes in $postDateFrom?
09.08.2023, 15:19
Original comment available on version: ru


Alexander wrote:
Let this be the first launch, but there is still a chance of skipping when restarting.
Maybe increase the number of minutes in $postDateFrom?

In theory, we can move this to a setting with a default value = 2
As I understand it, this will solve your entire described problem, or do you need to pay attention to something separately?
09.08.2023, 16:44
Original comment available on version: ru

Олександр Григорович
Support EP
Leave a message in this thread and the user's contacts will be shown to you
[/quote]
As I understand it, this will solve your entire described problem, or do you need to pay attention to something separately?
[/quote]
I described the problem above.
Regarding $postDateFrom - this is just my vision of the solution, but I'm not sure about it, I would like to hear an opinion, an option from the developer.
- If my conclusions are correct, then I can try to change the value of 2 to another, test it in practice, and based on the results we will decide what to do next.
09.08.2023, 17:04
Original comment available on version: ru

Олександр Григорович
Support EP
Leave a message in this thread and the user's contacts will be shown to you


Tyndyk Maxim Vadimovich

OneBox production wrote:



Alexander wrote:

Let this be the first launch, but there is still a chance of skipping when restarting.

Maybe increase the number of minutes in $postDateFrom?



In theory, we can move this to a setting with a default value = 2

As I understand it, this will solve your entire described problem, or do you need to pay attention to something separately?



Yes, the test with $postDateFrom showed a satisfactory result - you can move this to a setting with the default value = 2.

Regarding the third paragraph of the TOR:

1. Additional product categories are not imported (Additional categories does not display.png)

2. Categories no longer display linked products. (Additional categories does not display.png)
10.08.2023, 11:11
Original comment available on version: ru


Alexander wrote:
Yes, the test with $postDateFrom showed a satisfactory result - you can move this to a setting with the default value = 2.

put into setup

Alexander wrote:
Regarding the third paragraph of the TOR:
1. Additional product categories are not imported (Additional categories does not display.png)
2. Categories no longer display linked products. (Additional categories does not display.png)

on this point I do not see any problems with importing additional categories
can check on some specific example (id) from the price table - I will manually import only for the specified record
10.08.2023, 16:00
Original comment available on version: ru

Олександр Григорович
Support EP
Leave a message in this thread and the user's contacts will be shown to you

can check on some specific example (id) from the price table - I will manually import only for the specified record

The screenshot (Does not display additional categories.png) shows a specific example (id) from the price table - 2125275, its (id) in OneBOX -412732.
The screenshot also shows that id 2125275 (BOX id =412732) has the main category 597 (BOX id =1500) and additional: 887 (BOX id =1762), 1055 (BOX id = 1297), 1059 (BOX id =1928 ), ...
By opening any of the listed Additional categories in the box, we will not find a product with BOX id =412732.
10.08.2023, 17:00
Original comment available on version: ru

Олександр Григорович
Support EP
Leave a message in this thread and the user's contacts will be shown to you

In theory, we can move this to a setting with a default value = 2
As I understand it, this will solve your entire described problem, or do you need to pay attention to something separately?

This solved the problem, but not all...
- 806900 rows were updated in price in one second, but the import algorithm did not take this volume into work, that is, the import did not even start (when a smaller number is OK).
RAM / CPU on the server with a margin, the $postDateFrom setting does not play a role, maybe memory limit or time limit for script execution is not allowed?
10.08.2023, 20:27
Original comment available on version: ru

Олександр Григорович
Support EP
Leave a message in this thread and the user's contacts will be shown to you

Alexander wrote:

In theory, we can move this to a setting with a default value = 2
As I understand it, this will solve your entire described problem, or do you need to pay attention to something separately?

This solved the problem, but not all...
- 806900 rows were updated in price in one second, but the import algorithm did not take this volume into work, that is, the import did not even start (when a smaller number is OK).
RAM / CPU on the server with a margin, the $postDateFrom setting does not play a role, maybe memory limit or time limit for script execution is not allowed?

Decided!
- please allocate 4G of memory to a minute cron (/usr/bin/php -d memory_limit=4G -f ~/cron/cron-minute.php) and commit - this volume allows the script to work normally with 5M lines from price. (Verified)

Alexander wrote:

can check on some specific example (id) from the price table - I will manually import only for the specified record

The screenshot (Does not display additional categories.png) shows a specific example (id) from the price table - 2125275, its (id) in OneBOX -412732.
The screenshot also shows that id 2125275 (BOX id =412732) has the main category 597 (BOX id =1500) and additional: 887 (BOX id =1762), 1055 (BOX id = 1297), 1059 (BOX id =1928 ), ...
By opening any of the listed Additional categories in the box, we will not find a product with BOX id =412732.

This issue is still relevant.
16.08.2023, 16:27
Original comment available on version: ru

Олександр Григорович
Support EP
Leave a message in this thread and the user's contacts will be shown to you
Practical use has shown problems:

The work of the product import script can last from 2 minutes. up to 5 hours (for 5 million lines) consuming up to 4G of memory.

- running such a script along with other Minute Cron Automations, the latter will not start until the import script is finished, sometimes it is 5 hours.

- also the next import can start only after the end of the previous one, which makes the $postDateMinute setting ineffective and leads to missing imported data.

A solution to the problems of the import algorithm:

1. You need to run in parallel / independently of other tasks on the minute cron (for example, as /usr/bin/php -d memory_limit=512M -f ~/.../cron-supplierprice.php), allocating 4G of memory to it.

2. In order not to miss updating the data in price, after each run of the algorithm, it is necessary to fix the start time (for example, $LastDateStart), and take it into account taking into account this label (SELECT * FROM `price` WHERE `post_date` > '{$ LastDateStart}' ORDER BY `price`.`id` ASC).

In case the import algorithm has not been launched for a long time, and there is no need to import obsolete data, you can add a setting (Timestamp of the last import launch.png), checking which - the current server time will be assigned to the timestamp ($LastDateStart = DateTime_Object::Now()- >setFormat('Ymd H:i:s')->__toString();)

Please rate improvements.
17.08.2023, 15:17
Original comment available on version: ru


1. You need to run in parallel / independently of other tasks on the minute cron (for example, as /usr/bin/php -d memory_limit=512M -f ~/.../cron-supplierprice.php), allocating 4G of memory to it.

I can take out on your project the launch of the action in a separate cron + ignore it in the minute cron - it will take 1 hour

2. In order not to miss updating the data in price, after each run of the algorithm, it is necessary to fix the start time (for example, $LastDateStart), and take it into account taking into account this label (SELECT * FROM `price` WHERE `post_date` > '{$ LastDateStart}' ORDER BY `price`.`id` ASC).
In case the import algorithm has not been launched for a long time, and there is no need to import obsolete data, you can add a setting (Timestamp of the last import launch.png), checking which - the current server time will be assigned to the timestamp ($LastDateStart = DateTime_Object::Now()- >setFormat('Ymd H:i:s')->__toString();)

you can make such a setting in action - this is the same 1 hour

Alexander wrote:
The screenshot (Does not display additional categories.png) shows a specific example (id) from the price table - 2125275, its (id) in OneBOX -412732.
The screenshot also shows that id 2125275 (BOX id =412732) has the main category 597 (BOX id =1500) and additional: 887 (BOX id =1762), 1055 (BOX id = 1297), 1059 (BOX id =1928 ), ...
By opening any of the listed Additional categories in the box, we will not find a product with BOX id =412732.

I will check this issue soon.
25.08.2023, 14:26
Original comment available on version: ru

Олександр Григорович
Support EP
Leave a message in this thread and the user's contacts will be shown to you
I agree improvements 2h, ready to pay.
25.08.2023, 14:40
Original comment available on version: ru


Alexander wrote:
The screenshot (Does not display additional categories.png) shows a specific example (id) from the price table - 2125275, its (id) in OneBOX -412732.
The screenshot also shows that id 2125275 (BOX id =412732) has the main category 597 (BOX id =1500) and additional: 887 (BOX id =1762), 1055 (BOX id = 1297), 1059 (BOX id =1928 ), ...
By opening any of the listed Additional categories in the box, we will not find a product with BOX id =412732.

Corrected.
25.08.2023, 15:49
Original comment available on version: ru


Tyndyk Maxim Vadimovich
OneBox production wrote:

Alexander wrote:
The screenshot (Does not display additional categories.png) shows a specific example (id) from the price table - 2125275, its (id) in OneBOX -412732.
The screenshot also shows that id 2125275 (BOX id =412732) has the main category 597 (BOX id =1500) and additional: 887 (BOX id =1762), 1055 (BOX id = 1297), 1059 (BOX id =1928 ), ...
By opening any of the listed Additional categories in the box, we will not find a product with BOX id =412732.

Corrected.

+
25.08.2023, 16:19
Original comment available on version: ru

1. finalized the setting "Import data changed since the last launch of the action" (on startup it remembers the start time of the action and on the next launch it takes data from that time)
If this is the first launch with this setting, it will take N minutes specified in the action and remember the launch time for the next launch.
After that, remove N minutes in the setting - and the action works according to the date of the previous launch. If you need to parse from a specific time - uncheck the box and set how many minutes to parse one time, then return it.
2. moved the launch of the action to a separate minute cron, do not delete it from the automation.
28.08.2023, 09:47
Original comment available on version: ru


Tyndyk Maxim Vadimovich
OneBox production wrote:
1. finalized the setting "Import data changed since the last launch of the action" (on startup it remembers the start time of the action and on the next launch it takes data from that time)
If this is the first launch with this setting, it will take N minutes specified in the action and remember the launch time for the next launch.
After that, remove N minutes in the setting - and the action works according to the date of the previous launch. If you need to parse from a specific time - uncheck the box and set how many minutes to parse one time, then return it.

+

2. moved the launch of the action to a separate minute cron, do not delete it from the automation.

I look at the crons added memory, but for some reason the fatal-2023-08-28.log log complains every minute:
[2023-08-28 14:06:12]
point: /var/www/..../cron/cron-minute.php
engine_sessionid: 1693220763_1129
data: Array
(
[type] => 1
[message] => Allowed memory size of 536870912 bytes exhausted (tried to allocate 16384 bytes)
[file] => /var/www/..../web/packages/ConnectionManager/ConnectionManager_MySQLi.class.php
[line] => 125
And the import algorithm with 3G memory did not take 4909532 lines into operation, but with 4G it takes ok - please add this value.
28.08.2023, 14:18
Original comment available on version: ru


Alexander wrote:
I look at the crons added memory, but for some reason the fatal-2023-08-28.log log complains every minute:
[2023-08-28 14:06:12]
point: /var/www/..../cron/cron-minute.php
engine_sessionid: 1693220763_1129
data: Array
(
[type] => 1
[message] => Allowed memory size of 536870912 bytes exhausted (tried to allocate 16384 bytes)
[file] => /var/www/..../web/packages/ConnectionManager/ConnectionManager_MySQLi.class.php
[line] => 125

this minute crown did not have enough memory, you can raise it (I raised it to 3G)

Alexander wrote:
And the import algorithm with 3G memory did not take 4909532 lines into operation, but with 4G it takes ok - please add this value.

I see you already apparently did it yourself
28.08.2023, 17:47
Original comment available on version: ru


I see you already apparently did it yourself

I did this for a test, need to make a commit with 4G.
28.08.2023, 17:50
Original comment available on version: ru


Alexander wrote:
I did this for a test, need to make a commit with 4G.

put both 5 with a margin to
28.08.2023, 17:54
Original comment available on version: ru


Tyndyk Maxim Vadimovich
OneBox production wrote:

Alexander wrote:
I did this for a test, need to make a commit with 4G.

put both 5 with a margin to

+
28.08.2023, 17:55
Original comment available on version: ru



1. finalized the setting "Import data changed since the last launch of the action" (on startup it remembers the start time of the action and on the next launch it takes data from that time)

If this is the first launch with this setting, it will take N minutes specified in the action and remember the launch time for the next launch.

After that, remove N minutes in the setting - and the action works according to the date of the previous launch. If you need to parse from a specific time - uncheck the box and set how many minutes to parse one time, then return it.



Maxim, the setting skips the changed data since the last run of the action.

For clarity, I made a selection by the time the data was updated in the price table. (missing_array.png)

Red indicates the time of updated data that the import script did not take into work. (Import algorithm settings.png)

Having done several tests, I found out that the gaps do not depend on the amount of data or the data itself, that is, by updating the same data, the algorithm took them into work.

I think the problem lies in the launch trigger {$postDateFrom}, namely $postDateMinute = 2 confuses me, please check again ..
31.08.2023, 18:09
Original comment available on version: ru

Олександр Григорович
Support EP
Leave a message in this thread and the user's contacts will be shown to you
If it's not a bug, please rate the improvement.
After importing through the “CMS ExtraParts” application, data on quantity, purchase price, update date in Product Suppliers changes, but Automatic Recalculation of Prices and Product Availability does not occur, as through Downloading the Price List (xlsx) of the supplier.
I already described the problem, but there is no answer:
https://1b.app/ru/forum/supplier-and-price-list-management/17355-ne-proishodit-a...
https://1b.app/ru/forum/supplier-and-price-list-management/17381-ne-pratsyuie-za...
13.09.2023, 23:05
Original comment available on version: ru

Please join the conversation. If you have something to say - please write a comment. You will need a mobile phone and an SMS code for identification to enter. Log in and comment