Evaluate the refinement of the action
Now the mechanism for selecting updated data, in the action "Import products from CMS ExtraParts", is carried out by a request:
"SELECT * FROM `price` WHERE `post_date` >= '{$postDateFrom}' ORDER BY `price`.`id` ASC"
Where the variable '{$postDateFrom}' uses the time the action was last run.
The problem is that - when a large amount of updated data suddenly appears in `price' (for example, 4.6 Million), the action takes a large array of data (about 12G), the processing of which lasts for hours (about 5 hours), which leads to timeouts or errors in exported products or other data export failures ...
It is necessary to improve the mechanism for sampling updated data:
1. Take into account a limited (for example, 100,000 rows specified in the settings) amount of data for one action launch, process this array and stop the action.
At the Next Start of the Action, process the next amount of updated data and so on.
2. If two or more export actions work simultaneously (in parallel), the data between them must be distributed so that each of the actions does not process the same data.
Original question is available on version: ua
"SELECT * FROM `price` WHERE `post_date` >= '{$postDateFrom}' ORDER BY `price`.`id` ASC"
Where the variable '{$postDateFrom}' uses the time the action was last run.
The problem is that - when a large amount of updated data suddenly appears in `price' (for example, 4.6 Million), the action takes a large array of data (about 12G), the processing of which lasts for hours (about 5 hours), which leads to timeouts or errors in exported products or other data export failures ...
It is necessary to improve the mechanism for sampling updated data:
1. Take into account a limited (for example, 100,000 rows specified in the settings) amount of data for one action launch, process this array and stop the action.
At the Next Start of the Action, process the next amount of updated data and so on.
2. If two or more export actions work simultaneously (in parallel), the data between them must be distributed so that each of the actions does not process the same data.
Answers:
Please clarify, is the problem that it all takes a long time to work or is it putting a load on the server? If it’s something that’s been running for a long time, we can pour data into parallel streams like 30-40, if it’s something that’s loading the server, then that’s a different question.
Oleksandr Grigorovich wrote:
Brothers in the brow are surrounded (for example, it is indicated in the adjusted 100,000 rows)
Well, if you hit 100k, you will never catch up with the current state.
Oleksandr Grigorovich wrote:
Brothers in the brow are surrounded (for example, it is indicated in the adjusted 100,000 rows)
Well, if you hit 100k, you will never catch up with the current state.
22.04.2024, 13:28
Original comment available on version: ru
Perhaps it is also possible to speed up the loading of goods so that it is X times faster on such a volume
22.04.2024, 13:29
Original comment available on version: ru
Yes, the problem is that it takes a very long time, and the longer this process takes, the higher the likelihood of export failure.
There are enough server resources, that’s why I suggested exporting in parts and in parallel actions
There are enough server resources, that’s why I suggested exporting in parts and in parallel actions
22.04.2024, 13:56
Original comment available on version: ru
Oleksandr Hryhorovych wrote:
the higher the probability of export failure
so what failures do you observe now in such situations?
22.04.2024, 13:58
Original comment available on version: ua
Care Department
OneBox production wrote:
Alexander Grigorievich wrote:
the higher the probability of export failure
Yes, what kind of failures are you seeing now in such situations?
Most often, if the export lasts for a very long time, then brands begin to be erased for products or new ones are created without a brand (the $brandid variable returns an empty value...) - this creates big problems.
22.04.2024, 14:11
Original comment available on version: ru
If this is the only problem, then it will be fixed in 1 hour of revision.
22.04.2024, 14:26
Original comment available on version: ru
If there is a solution to speed up exports, which will allow you to successfully fill such a quantity and at the same time get rid of the loss of the brand, then issue an invoice.
22.04.2024, 14:34
Original comment available on version: ru
This solution will allow you to remember the entire list of brands and search for them from the cache, this will also speed up import a little, but I don’t think it will be global. Let's try, I'll issue an invoice
22.04.2024, 15:30
Original comment available on version: ru
Care Department
OneBox production wrote:
This solution will allow you to remember the entire list of brands and search for them from the cache, this will also speed up import a little, but I don’t think it will be global. Let's try, I'll issue an invoice
The decision to take a brand from Cash will not be enough, I checked.
I completely disabled the method for getting a brand in the class; updating it speeds up, but not enough.
Is it possible to implement pouring into multiple threads?
22.04.2024, 16:08
Original comment available on version: ru
I don’t care about the speed if in the end everything will flood normally. If we pour into several threads, we can heavily load the external database and our own.
22.04.2024, 16:32
Original comment available on version: ru
We can do it differently, I make it so that your download of 4 records works for example an hour, after which you pay according to the time spent.
22.04.2024, 16:33
Original comment available on version: ru
Care Department
OneBox production wrote:
We can do it differently, I make it so that your downloading of 4 records works for example an hour, after which you pay according to the time spent.
I agree, we’ll keep guessing longer, you’ll figure it out as we go.
Do it, I'll pay after the fact.
22.04.2024, 16:38
Original comment available on version: ru
Care Department
OneBox production wrote:
We can do it differently, I make it so that your download of 4 records works for example an hour, after which you pay according to the time spent.
Good afternoon.
Are there any progress on the task?
09.05.2024, 13:44
Original comment available on version: ru
Good afternoon. I rolled back all your changes and added them as settings + added my optimizations there and turned them on right away. During the first tests, a list of 1k products loaded in about 6 seconds, at the moment it takes about 2-3 (taking into account the import of categories, which always eats up about a second at the start of the action).
A list of 100k entries loads in about 130-150 seconds, i.e. the indicated 4 million records will arrive in about one and a half to two hours. Those. the action began to work several times faster.
At the moment, 3 hours have been spent, if you want, we can continue and upload 100k records in about a minute (i.e. 2 times faster, I think this is possible), but it will be more difficult since I have completed the basic optimizations, then you will have to analyze requests and every second will be a little more difficult.
A list of 100k entries loads in about 130-150 seconds, i.e. the indicated 4 million records will arrive in about one and a half to two hours. Those. the action began to work several times faster.
At the moment, 3 hours have been spent, if you want, we can continue and upload 100k records in about a minute (i.e. 2 times faster, I think this is possible), but it will be more difficult since I have completed the basic optimizations, then you will have to analyze requests and every second will be a little more difficult.
23.05.2024, 16:36
Original comment available on version: ru
Good afternoon. While I was making a report on the results of the modification, it turned off/on several times. import action and everything really worked, 3,112,663 records in 97 minutes. - an excellent indicator.
Thanks Dmitry! I'm waiting for the bill.
But there are also problems:
- during import, for some reason, some products overwrite the unsyncable value from 0 to 1 - does this affect further import into BOX or export to OpenCart?
- the current action importing manufacturers writes their cat.id (ExtraParts) in the shopbrand.code1c (BOX) field, this leads to an export conflict to opencart, described in https://1b.app/ru/forum/integrations-with-online -stores/18246-konflikt-u-roboti-dvoh-dodatkiv/ - is this necessary?
or how else can the conflict be resolved?
Thanks Dmitry! I'm waiting for the bill.
But there are also problems:
- during import, for some reason, some products overwrite the unsyncable value from 0 to 1 - does this affect further import into BOX or export to OpenCart?
- the current action importing manufacturers writes their cat.id (ExtraParts) in the shopbrand.code1c (BOX) field, this leads to an export conflict to opencart, described in https://1b.app/ru/forum/integrations-with-online -stores/18246-konflikt-u-roboti-dvoh-dodatkiv/ - is this necessary?
or how else can the conflict be resolved?
25.05.2024, 17:56
Original comment available on version: ru
Oleksandr Grigorovich wrote:
- when importing, some products will overwrite the unsyncable values from 0 to 1 - does this affect further import from BOX or export to OpenCart?
1. This can happen if some error occurred during the update process. Try to save the goods with your hands, is everything ok with it?
2. You are confusing something, it is probably the action that imports brands (separate). You can turn it off if you don't need to.
27.05.2024, 10:52
Original comment available on version: ru
Please join the conversation. If you have something to say - please write a comment. You will need a mobile phone and an SMS code for identification to enter.
Log in and comment
Copyright © OneBox 2024
