Evaluate the refinement of the action
Now the mechanism for selecting updated data, in the action "Import products from CMS ExtraParts", is carried out by a request:
"SELECT * FROM `price` WHERE `post_date` >= '{$postDateFrom}' ORDER BY `price`.`id` ASC"
Where the variable '{$postDateFrom}' uses the time the action was last run.
The problem is that - when a large amount of updated data suddenly appears in `price' (for example, 4.6 Million), the action takes a large array of data (about 12G), the processing of which lasts for hours (about 5 hours), which leads to timeouts or errors in exported products or other data export failures ...
It is necessary to improve the mechanism for sampling updated data:
1. Take into account a limited (for example, 100,000 rows specified in the settings) amount of data for one action launch, process this array and stop the action.
At the Next Start of the Action, process the next amount of updated data and so on.
2. If two or more export actions work simultaneously (in parallel), the data between them must be distributed so that each of the actions does not process the same data.
Original question is available on version: ua
"SELECT * FROM `price` WHERE `post_date` >= '{$postDateFrom}' ORDER BY `price`.`id` ASC"
Where the variable '{$postDateFrom}' uses the time the action was last run.
The problem is that - when a large amount of updated data suddenly appears in `price' (for example, 4.6 Million), the action takes a large array of data (about 12G), the processing of which lasts for hours (about 5 hours), which leads to timeouts or errors in exported products or other data export failures ...
It is necessary to improve the mechanism for sampling updated data:
1. Take into account a limited (for example, 100,000 rows specified in the settings) amount of data for one action launch, process this array and stop the action.
At the Next Start of the Action, process the next amount of updated data and so on.
2. If two or more export actions work simultaneously (in parallel), the data between them must be distributed so that each of the actions does not process the same data.
Answers:
Please clarify, is the problem that it all takes a long time to work or is it putting a load on the server? If it’s something that’s been running for a long time, we can pour data into parallel streams like 30-40, if it’s something that’s loading the server, then that’s a different question.
Oleksandr Grigorovich wrote:
Brothers in the brow are surrounded (for example, it is indicated in the adjusted 100,000 rows)
Well, if you hit 100k, you will never catch up with the current state.
Oleksandr Grigorovich wrote:
Brothers in the brow are surrounded (for example, it is indicated in the adjusted 100,000 rows)
Well, if you hit 100k, you will never catch up with the current state.
22.04.2024, 13:28
Original comment available on version: ru
Perhaps it is also possible to speed up the loading of goods so that it is X times faster on such a volume
22.04.2024, 13:29
Original comment available on version: ru
Yes, the problem is that it takes a very long time, and the longer this process takes, the higher the likelihood of export failure.
There are enough server resources, that’s why I suggested exporting in parts and in parallel actions
There are enough server resources, that’s why I suggested exporting in parts and in parallel actions
22.04.2024, 13:56
Original comment available on version: ru
Oleksandr Hryhorovych wrote:
the higher the probability of export failure
so what failures do you observe now in such situations?
22.04.2024, 13:58
Original comment available on version: ua
Care Department
OneBox production wrote:
Alexander Grigorievich wrote:
the higher the probability of export failure
Yes, what kind of failures are you seeing now in such situations?
Most often, if the export lasts for a very long time, then brands begin to be erased for products or new ones are created without a brand (the $brandid variable returns an empty value...) - this creates big problems.
22.04.2024, 14:11
Original comment available on version: ru
If this is the only problem, then it will be fixed in 1 hour of revision.
22.04.2024, 14:26
Original comment available on version: ru
If there is a solution to speed up exports, which will allow you to successfully fill such a quantity and at the same time get rid of the loss of the brand, then issue an invoice.
22.04.2024, 14:34
Original comment available on version: ru
This solution will allow you to remember the entire list of brands and search for them from the cache, this will also speed up import a little, but I don’t think it will be global. Let's try, I'll issue an invoice
22.04.2024, 15:30
Original comment available on version: ru
Care Department
OneBox production wrote:
This solution will allow you to remember the entire list of brands and search for them from the cache, this will also speed up import a little, but I don’t think it will be global. Let's try, I'll issue an invoice
The decision to take a brand from Cash will not be enough, I checked.
I completely disabled the method for getting a brand in the class; updating it speeds up, but not enough.
Is it possible to implement pouring into multiple threads?
22.04.2024, 16:08
Original comment available on version: ru
I don’t care about the speed if in the end everything will flood normally. If we pour into several threads, we can heavily load the external database and our own.
22.04.2024, 16:32
Original comment available on version: ru
We can do it differently, I make it so that your download of 4 records works for example an hour, after which you pay according to the time spent.
22.04.2024, 16:33
Original comment available on version: ru
Care Department
OneBox production wrote:
We can do it differently, I make it so that your downloading of 4 records works for example an hour, after which you pay according to the time spent.
I agree, we’ll keep guessing longer, you’ll figure it out as we go.
Do it, I'll pay after the fact.
22.04.2024, 16:38
Original comment available on version: ru
Please join the conversation. If you have something to say - please write a comment. You will need a mobile phone and an SMS code for identification to enter.
Log in and comment
Copyright © OneBox 2024
