Data Validation

Data Processing Methodology:

To contribute to the broader field of study, the IBSO is committed to openness and academic rigour. Our comprehensive data processing methodology, integral to the academic research underpinning this analysis, is detailed in an academic journal paper here - Filtered dataset of Irish energy performance certificates: A data-driven approach for enhanced building stock modelling - ScienceDirect

Step 1: Loading Data

We utilise a range of Python libraries to import and manage our data efficiently. The data is initially loaded in its raw (.txt), unfiltered form for preliminary analysis.

Step 2: Data Consistency Checks

At this step, we add unique identifiers to each data entry and perform consistency checks. This process involves ensuring the accuracy of various informational data points, such as location and assessment dates, e.g. removing entries where the assessment date is in the future. We utilise a range of Python libraries to import and manage our data efficiently. The data is initially loaded in its raw (.txt), unfiltered form for preliminary analysis.

Step 3: Outlier Identification

Using advanced statistical methods, we identify and categorise outliers in the dataset. This step is crucial for maintaining the integrity of the data analysis. We utilise a range of Python libraries to import and manage our data efficiently. The data is initially loaded in its raw (.txt), unfiltered form for preliminary analysis.

Step 4: Data Segmentation

In cases of bimodal data distributions, we segment the data for more precise analysis. This step involves dividing data based on relevant categories to enhance the accuracy of our findings. We utilise a range of Python libraries to import and manage our data efficiently. The data is initially loaded in its raw (.txt), unfiltered form for preliminary analysis.

Step 5: Application of Filters

After identifying outliers and erroneous data, we apply specific filters to refine the dataset. This process ensures that only the most relevant and accurate data is used in our analysis. We utilise a range of Python libraries to import and manage our data efficiently. The data is initially loaded in its raw (.txt), unfiltered form for preliminary analysis.

Data Validation

Data Processing Methodology:

Card Collection for data-validation-cards

Step 1: Loading Data