Hello DHIS2 Community,
We’re encountering a challenge with the DHIS2 instance we manage for the ICRC, particularly with the analytics generation process. We are currently using DHIS2 version 2.38.5. Our datavalue table has grown quite large (around 115GB), and generating analytics for the last three years can take up to 6 hours. We need to refresh the data for the current year at least twice a day, while keeping the older data from previous years available in the analytics tables without having to regenerate it every time.
Based on the logs from our database, a significant portion of the time is spent on the creation of indexes. The system creates an index on every column, and for the 2024 data alone, we have around 119 columns in the analytics table, most of which represent categories, as most of our datasets have attributes.
At the moment, it seems necessary to include all years in the refresh process to keep the older data available. Continuous analytics doesn’t work for us either, as it refreshes everything once a day.
Has anyone faced similar challenges with a large datavalue table? How do you manage to refresh analytics for recent data efficiently while keeping older data accessible? Are there any strategies or configurations that could help us update new data without reprocessing everything from previous years?
I will also attach the cron job configuration we are using.
We would greatly appreciate any insights or shared experiences!
Best regards,
Tafsir
ICRC
6 posts - 3 participants