dlookr: Data quality diagnosis

package dlookr

Introduction to data quality diagnostics in the dlookr package

Author

Affiliation

Choonghyun Ryu

Published

May 10, 2021

DOI

Preface

After you have acquired the data, you should do the following:

Diagnose data quality.
- If there is a problem with data quality,
- The data must be corrected or re-acquired.
Explore data to understand the data and find scenarios for performing the analysis.
Derive new variables or perform variable transformations.

The dlookr package makes these steps fast and easy:

Performs a data diagnosis or automatically generates a data diagnosis report.
Discover data in a variety of ways, and automatically generate EDA(exploratory data analysis) report.
Impute missing values and outliers, resolve skewed data, and binarize continuous variables into categorical variables. And generates an automated report to support it.

This document introduces Data Quality Diagnosis methods provided by the dlookr package. You will learn how to diagnose the quality of tbl_df data that inherits from data.frame and data.frame with functions provided by dlookr.

dlookr increases synergy when used with the dplyr package. Particularly in data exploration and data wrangle, it increases the efficiency of the tidyverse package group.

Overview

Overall Diagnose Data

Tasks	Descriptions	Functions	Support DBI
describe overview of data	Inquire basic information to understand the data in general	`overview()`
summary overview object	summary described overview of data	`summary.overview()`
plot overview object	plot described overview of data	`plot.overview()`
diagnose data quality of variables	The scope of data quality diagnosis is information on missing values and unique value information	`diagnose()`	x
diagnose data quality of categorical variables	frequency, ratio, rank by levels of each variables	`diagnose_category()`	x
diagnose data quality of numerical variables	descriptive statistics, number of zero, minus, outliers	`diagnose_numeric()`	x
diagnose data quality for outlier	number of outliers, ratio, mean of outliers, mean with outliers, mean without outliers	`diagnose_outlier()`	x
plot outliers information of numerical data	box plot and histogram whith outliers, without outliers	`plot_outlier.data.frame()`	x
plot outliers information of numerical data by target variable	box plot and density plot whith outliers, without outliers	`plot_outlier.target_df()`	x
diagnose combination of categorical variables	Check for sparse cases of level combinations of categorical variables	`diagnose_sparese()`

Visualize Missing Values

Tasks	Descriptions	Functions
pareto chart for missing value	visualize pareto chart for variables with missing value.	`plot_na_pareto()`
combination chart for missing value	visualize distribution of missing value by combination of variables.	`plot_na_hclust()`
plot the combination variables that is include missing value	visualize the combinations of missing value across cases..	`plot_na_intersect()`

Reporting

Types	Descriptions	Functions	Support DBI
reporting the information of data diagnosis into pdf file	report the information for diagnosing the quality of the data.	`diagnose_report()`	x
reporting the information of data diagnosis into html file	report the information for diagnosing the quality of the data.	`diagnose_report()`	x

Exercise data: nycflights13::flights

To illustrate basic use of the dlookr package, use the flights from the nycflights13 package. The flights data frame is data about departure and arrival on all flights departing from NYC in 2013.

library(dlookr)
library(dplyr)
library(ggplot2)
library(flextable)
library(nycflights13)

glimpse(flights)

Rows: 336,776
Columns: 19
$ year           <int> 2013, 2013, 2013, 2013, 2013, 2013, 2013, 201…
$ month          <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, …
$ day            <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, …
$ dep_time       <int> 517, 533, 542, 544, 554, 554, 555, 557, 557, …
$ sched_dep_time <int> 515, 529, 540, 545, 600, 558, 600, 600, 600, …
$ dep_delay      <dbl> 2, 4, 2, -1, -6, -4, -5, -3, -3, -2, -2, -2, …
$ arr_time       <int> 830, 850, 923, 1004, 812, 740, 913, 709, 838,…
$ sched_arr_time <int> 819, 830, 850, 1022, 837, 728, 854, 723, 846,…
$ arr_delay      <dbl> 11, 20, 33, -18, -25, 12, 19, -14, -8, 8, -2,…
$ carrier        <chr> "UA", "UA", "AA", "B6", "DL", "UA", "B6", "EV…
$ flight         <int> 1545, 1714, 1141, 725, 461, 1696, 507, 5708, …
$ tailnum        <chr> "N14228", "N24211", "N619AA", "N804JB", "N668…
$ origin         <chr> "EWR", "LGA", "JFK", "JFK", "LGA", "EWR", "EW…
$ dest           <chr> "IAH", "IAH", "MIA", "BQN", "ATL", "ORD", "FL…
$ air_time       <dbl> 227, 227, 160, 183, 116, 150, 158, 53, 140, 1…
$ distance       <dbl> 1400, 1416, 1089, 1576, 762, 719, 1065, 229, …
$ hour           <dbl> 5, 5, 5, 5, 6, 5, 6, 6, 6, 6, 6, 6, 6, 6, 6, …
$ minute         <dbl> 15, 29, 40, 45, 0, 58, 0, 0, 0, 0, 0, 0, 0, 0…
$ time_hour      <dttm> 2013-01-01 05:00:00, 2013-01-01 05:00:00, 20…

Overall Diagnose Data

Overview data

overview() inquire basic information to understand the data in general.

ov <- overview(flights)
ov %>% 
  flextable()

division	metrics	value
size	observations	336,776
size	variables	19
size	values	6,398,744
size	memory_size	40,650,104
missing	complete_obs	327,346
missing	missing_obs	9,430
missing	missing_vars	6
missing	missing_values	46,595
data_type	numerics	6
data_type	integers	8
data_type	factors	0
data_type	characters	4
data_type	others	1

summary.overview() tries to be smart about formatting 14 information of overview.

summary(ov)

── Data Scale ──────────────────────────────────────────────
● Number of observations           :     336,776
● Number of variables              :          19
● Number of values                 :   6,398,744
● Size of located memory(bytes)    :  40,650,104

── Missing Data ────────────────────────────────────────────
● Number of completed observations :     327,346
● Number of observations with N/A  :       9,430
● Number of variables with N/A     :           6
● Number of N/A                    :      46,595

── Data Type ───────────────────────────────────────────────
● Number of numeric variables      :           6
● Number of integer variables      :           8
● Number of factors variables      :           0
● Number of character variables    :           4
● Number of other variables        :           1

── Individual variables ────────────────────────────────────
        Variables Data Type
1            year   integer
2           month   integer
3             day   integer
4        dep_time   integer
5  sched_dep_time   integer
6       dep_delay   numeric
7        arr_time   integer
8  sched_arr_time   integer
9       arr_delay   numeric
10        carrier character
11         flight   integer
12        tailnum character
13         origin character
14           dest character
15       air_time   numeric
16       distance   numeric
17           hour   numeric
18         minute   numeric
19      time_hour   POSIXct

plot.overview() plot the data type, number of observations, and number of missing values for each variable.

plot(ov)

# sort by name of variables
plot(ov, order_type = "name")

# sort by data type of variables
plot(ov, order_type = "type")

General diagnosis

diagnose() allows you to diagnosis a variables in a data frame. Like function of dplyr, the first argument is the tibble (or data frame). The second and subsequent arguments refer to variables within that data frame.

The variables of the tbl_df object returned by diagnose() are as follows.

variables : variable name
types : the data type of the variable
missing_count : number of missing values
missing_percent : percentage of missing values
unique_count : number of unique values
unique_rate : rate of unique value. unique_count / number of observation

For example, we can diagnose all variables in flights:

diagnose(flights) %>% 
  flextable()

variables	types	missing_count	missing_percent	unique_count	unique_rate
year	integer	0	0.0000000	1	0.000002969333
month	integer	0	0.0000000	12	0.000035631993
day	integer	0	0.0000000	31	0.000092049315
dep_time	integer	8,255	2.4511842	1,319	0.003916549873
sched_dep_time	integer	0	0.0000000	1,021	0.003031688719
dep_delay	numeric	8,255	2.4511842	528	0.001567807682
arr_time	integer	8,713	2.5871796	1,412	0.004192697817
sched_arr_time	integer	0	0.0000000	1,163	0.003453333967
arr_delay	numeric	9,430	2.8000808	578	0.001716274319
carrier	character	0	0.0000000	16	0.000047509324
flight	integer	0	0.0000000	3,844	0.011414115020
tailnum	character	2,512	0.7458964	4,044	0.012007981566
origin	character	0	0.0000000	3	0.000008907998
dest	character	0	0.0000000	105	0.000311779937
air_time	numeric	9,430	2.8000808	510	0.001514359693
distance	numeric	0	0.0000000	214	0.000635437205
hour	numeric	0	0.0000000	20	0.000059386655
minute	numeric	0	0.0000000	60	0.000178159964
time_hour	POSIXct	0	0.0000000	6,936	0.020595291826

Missing Value(NA) : Variables with very large missing values, ie those with a missing_percent close to 100, should be excluded from the analysis.
Unique value : Variables with a unique value (unique_count = 1) are considered to be excluded from data analysis. And if the data type is not numeric (integer, numeric) and the number of unique values is equal to the number of observations (unique_rate = 1), then the variable is likely to be an identifier. Therefore, this variable is also not suitable for the analysis model.

year can be considered not to be used in the analysis model since unique_count is 1. However, you do not have to remove it if you configure date as a combination of year, month, and day.

For example, we can diagnose only a few selected variables:

# Select columns by name
diagnose(flights, year, month, day) %>% 
  flextable()

variables	types	missing_count	missing_percent	unique_count	unique_rate
year	integer	0	0	1	0.000002969333
month	integer	0	0	12	0.000035631993
day	integer	0	0	31	0.000092049315

# Select all columns between year and day (inclusive)
diagnose(flights, year:day) %>% 
  flextable()

variables	types	missing_count	missing_percent	unique_count	unique_rate
year	integer	0	0	1	0.000002969333
month	integer	0	0	12	0.000035631993
day	integer	0	0	31	0.000092049315

# Select all columns except those from year to day (inclusive)
diagnose(flights, -(year:day)) %>% 
  flextable()

variables	types	missing_count	missing_percent	unique_count	unique_rate
dep_time	integer	8,255	2.4511842	1,319	0.003916549873
sched_dep_time	integer	0	0.0000000	1,021	0.003031688719
dep_delay	numeric	8,255	2.4511842	528	0.001567807682
arr_time	integer	8,713	2.5871796	1,412	0.004192697817
sched_arr_time	integer	0	0.0000000	1,163	0.003453333967
arr_delay	numeric	9,430	2.8000808	578	0.001716274319
carrier	character	0	0.0000000	16	0.000047509324
flight	integer	0	0.0000000	3,844	0.011414115020
tailnum	character	2,512	0.7458964	4,044	0.012007981566
origin	character	0	0.0000000	3	0.000008907998
dest	character	0	0.0000000	105	0.000311779937
air_time	numeric	9,430	2.8000808	510	0.001514359693
distance	numeric	0	0.0000000	214	0.000635437205
hour	numeric	0	0.0000000	20	0.000059386655
minute	numeric	0	0.0000000	60	0.000178159964
time_hour	POSIXct	0	0.0000000	6,936	0.020595291826

By using dplyr, variables including missing values can be sorted by the weight of missing values.:

flights %>%
  diagnose() %>%
  select(-unique_count, -unique_rate) %>% 
  filter(missing_count > 0) %>% 
  arrange(desc(missing_count)) %>% 
  flextable()

variables	types	missing_count	missing_percent
arr_delay	numeric	9,430	2.8000808
air_time	numeric	9,430	2.8000808
arr_time	integer	8,713	2.5871796
dep_time	integer	8,255	2.4511842
dep_delay	numeric	8,255	2.4511842
tailnum	character	2,512	0.7458964

Diagnosis of numeric variables

diagnose_numeric() diagnoses numeric(continuous and discrete) variables in a data frame. Usage is the same as diagnose() but returns more diagnostic information. However, if you specify a non-numeric variable in the second and subsequent argument list, the variable is automatically ignored.

The variables of the tbl_df object returned by diagnose_numeric() are as follows.

min : minimum value
Q1 : 1st quartile, 25th percentile
mean : arithmetic mean
median : median, 50th percentile
Q3 : 3rd quartile, 75th percentile
max : maximum value
zero : number of observations with a value of 0
minus : number of observations with negative numbers
outlier : number of outliers

Applying the summary() function to a data frame can help you figure out the distribution of data by printing min, Q1, mean, median, Q3, and max give. However, the result is that analysts can only look at it with eyes. However, returning such information as a data frame structure like tbl_df widens the scope of utilization.

zero, minus, and outlier are useful for diagnosing the integrity of data. For example, numerical data in some cases may not have 0 or a negative number. Since the hypothetical numeric variable ‘employee salary’ can not have a negative or zero value, you should check for zero or negative numbers in the data diagnosis process.

diagnose_numeric() can diagnose all numeric variables of flights as follows.:

diagnose_numeric(flights) %>% 
  flextable()

variables	min	Q1	mean	median	Q3	max	zero	minus	outlier
year	2,013	2,013	2,013.000000	2,013	2,013	2,013	0	0	0
month	1	4	6.548510	7	10	12	0	0	0
day	1	8	15.710787	16	23	31	0	0	0
dep_time	1	907	1,349.109947	1,401	1,744	2,400	0	0	0
sched_dep_time	106	906	1,344.254840	1,359	1,729	2,359	0	0	0
dep_delay	-43	-5	12.639070	-2	11	1,301	16,514	183,575	43,216
arr_time	1	1,104	1,502.054999	1,535	1,940	2,400	0	0	0
sched_arr_time	1	1,124	1,536.380220	1,556	1,945	2,359	0	0	0
arr_delay	-86	-17	6.895377	-5	14	1,272	5,409	188,933	27,880
flight	1	553	1,971.923620	1,496	3,465	8,500	0	0	1
air_time	20	82	150.686460	129	192	695	0	0	5,448
distance	17	502	1,039.912604	872	1,389	4,983	0	0	715
hour	1	9	13.180247	13	17	23	0	0	0
minute	0	8	26.230100	29	44	59	60,696	0	0

If a numeric variable can not logically have a negative or zero value, it can be used with filter() to easily find a variable that does not logically match:

diagnose_numeric(flights) %>% 
  filter(minus > 0 | zero > 0) %>% 
  flextable()

variables	min	Q1	mean	median	Q3	max	zero	minus	outlier
dep_delay	-43	-5	12.639070	-2	11	1,301	16,514	183,575	43,216
arr_delay	-86	-17	6.895377	-5	14	1,272	5,409	188,933	27,880
minute	0	8	26.230100	29	44	59	60,696	0	0

Diagnosis of categorical variables

diagnose_category() diagnoses the categorical(factor, ordered, character) variables of a data frame. The usage is similar to diagnose () but returns more diagnostic information. If you specify a non-categorical variable in the second and subsequent argument list, the variable is automatically ignored. The top argument specifies the number of levels to return per variable. The default value is 10, which returns the top 10 level. Of course, if the number of levels is less than 10, all levels are returned.

The variables of the tbl_df object returned by diagnose_category() are as follows.

variables : variable names
levels: level names
N : Number of observation
freq : Number of observation at the levels
ratio : Percentage of observation at the levels
rank : Rank of occupancy ratio of levels

`diagnose_category() can diagnose all categorical variables of flights as follows.:

diagnose_category(flights) %>% 
  flextable()

variables	levels	N	freq	ratio	rank
carrier	UA	336,776	58,665	17.4195905	1
carrier	B6	336,776	54,635	16.2229494	2
carrier	EV	336,776	54,173	16.0857662	3
carrier	DL	336,776	48,110	14.2854598	4
carrier	AA	336,776	32,729	9.7183291	5
carrier	MQ	336,776	26,397	7.8381476	6
carrier	US	336,776	20,536	6.0978217	7
carrier	9E	336,776	18,460	5.4813882	8
carrier	WN	336,776	12,275	3.6448559	9
carrier	VX	336,776	5,162	1.5327696	10
tailnum		336,776	2,512	0.7458964	1
tailnum	N725MQ	336,776	575	0.1707366	2
tailnum	N722MQ	336,776	513	0.1523268	3
tailnum	N723MQ	336,776	507	0.1505452	4
tailnum	N711MQ	336,776	486	0.1443096	5
tailnum	N713MQ	336,776	483	0.1434188	6
tailnum	N258JB	336,776	427	0.1267905	7
tailnum	N298JB	336,776	407	0.1208518	8
tailnum	N353JB	336,776	404	0.1199610	9
tailnum	N351JB	336,776	402	0.1193672	10
origin	EWR	336,776	120,835	35.8799321	1
origin	JFK	336,776	111,279	33.0424377	2
origin	LGA	336,776	104,662	31.0776302	3
dest	ORD	336,776	17,283	5.1318978	1
dest	ATL	336,776	17,215	5.1117063	2
dest	LAX	336,776	16,174	4.8025988	3
dest	BOS	336,776	15,508	4.6048412	4
dest	MCO	336,776	14,082	4.1814144	5
dest	CLT	336,776	14,064	4.1760696	6
dest	SFO	336,776	13,331	3.9584175	7
dest	FLL	336,776	12,055	3.5795306	8
dest	MIA	336,776	11,728	3.4824334	9
dest	DCA	336,776	9,705	2.8817374	10

In collaboration with filter() in the dplyr package, we can see that the tailnum variable is ranked in top 1 with 2,512 missing values in the case where the missing value is included in the top 10:

diagnose_category(flights) %>% 
  filter(is.na(levels)) %>% 
  flextable()

variables	levels	N	freq	ratio	rank
tailnum		336,776	2,512	0.7458964	1

The following returns a list of levels less than or equal to 0.01%. It should be noted that the top argument has a generous specification of 500. If you use the default value of 10, values below 0.01% would not be included in the list:

flights %>%
  diagnose_category(top = 500)  %>%
  filter(ratio <= 0.01) %>% 
  flextable()

variables	levels	N	freq	ratio	rank
carrier	OO	336,776	32	0.0095018647	16
dest	JAC	336,776	25	0.0074233318	97
dest	PSP	336,776	19	0.0056417322	98
dest	EYW	336,776	17	0.0050478656	99
dest	HDN	336,776	15	0.0044539991	100
dest	MTJ	336,776	15	0.0044539991	100
dest	SBN	336,776	10	0.0029693327	102
dest	ANC	336,776	8	0.0023754662	103
dest	LEX	336,776	1	0.0002969333	104
dest	LGA	336,776	1	0.0002969333	104

In the analytic model, it is also possible to consider removing the small percentage of observations in the observations or joining them together.

Diagnosing outliers

diagnose_outlier() diagnoses the outliers of the numeric (continuous and discrete) variables of the data frame. The usage is the same as diagnose().

The variables of the tbl_df object returned by diagnose_outlier() are as follows.

outliers_cnt : Count of outliers
outliers_ratio : Percent of outliers
outliers_mean : Arithmetic Average of outliers
with_mean : Arithmetic Average of with outliers
without_mean : Arithmetic Average of without outliers

diagnose_outlier() can diagnose anomalies of all numeric variables of flights as follows:

diagnose_outlier(flights) %>% 
  flextable()

variables	outliers_cnt	outliers_ratio	outliers_mean	with_mean	without_mean
year	0	0.0000000000		2,013.000000	2,013.0000000
month	0	0.0000000000		6.548510	6.5485100
day	0	0.0000000000		15.710787	15.7107870
dep_time	0	0.0000000000		1,349.109947	1,349.1099473
sched_dep_time	0	0.0000000000		1,344.254840	1,344.2548400
dep_delay	43,216	12.8322683327	93.14666	12.639070	0.4443455
arr_time	0	0.0000000000		1,502.054999	1,502.0549986
sched_arr_time	0	0.0000000000		1,536.380220	1,536.3802201
arr_delay	27,880	8.2784996556	120.55624	6.895377	-3.6863417
flight	1	0.0002969333	8,500.00000	1,971.923620	1,971.9042358
air_time	5,448	1.6176924721	400.14189	150.686460	146.4645229
distance	715	0.2123072903	4,954.74266	1,039.912604	1,031.5834506
hour	0	0.0000000000		13.180247	13.1802474
minute	0	0.0000000000		26.230100	26.2300995

Numeric variables that contain anomalies are easily found with filter().:

diagnose_outlier(flights) %>% 
  filter(outliers_cnt > 0) %>% 
  flextable()

variables	outliers_cnt	outliers_ratio	outliers_mean	with_mean	without_mean
dep_delay	43,216	12.8322683327	93.14666	12.639070	0.4443455
arr_delay	27,880	8.2784996556	120.55624	6.895377	-3.6863417
flight	1	0.0002969333	8,500.00000	1,971.923620	1,971.9042358
air_time	5,448	1.6176924721	400.14189	150.686460	146.4645229
distance	715	0.2123072903	4,954.74266	1,039.912604	1,031.5834506

The following is a list of numeric variables with anomalies greater than 5%.:

diagnose_outlier(flights) %>% 
  filter(outliers_ratio > 5) %>% 
  mutate(rate = outliers_mean / with_mean) %>% 
  arrange(desc(rate)) %>% 
  select(-outliers_cnt) %>% 
  flextable()

variables	outliers_ratio	outliers_mean	with_mean	without_mean	rate
arr_delay	8.27850	120.55624	6.895377	-3.6863417	17.48363
dep_delay	12.83227	93.14666	12.639070	0.4443455	7.36974

If the outlier is larger than the average of all observations, it may be desirable to replace or remove the outlier in the data analysis process.

Visualization of outliers

plot_outlier() visualizes outliers of numerical variables(continuous and discrete) of data.frame. Usage is the same diagnose().

The plot derived from the numerical data diagnosis is as follows.

With outliers box plot
Without outliers box plot
With outliers histogram
Without outliers histogram

plot_outlier() can visualize an anomaly in the arr_delay variable of flights as follows:

flights %>%
  plot_outlier(arr_delay)

Use the function of the dplyr package and plot_outlier() and diagnose_outlier() to visualize anomaly values of all numeric variables with an outlier ratio of 0.5% or more.:

flights %>%
  plot_outlier(diagnose_outlier(flights) %>% 
                 filter(outliers_ratio >= 0.5) %>% 
                 select(variables) %>% 
                 unlist())

You should look at the visualization results and decide whether to remove or replace the outliers. In some cases, it is important to consider removing the variables that contain anomalies from the data analysis model.

In the visualization results, arr_delay has similar distributions to the normal distribution of the observed values. In the case of linear models, we can also consider removing or replacing anomalies. And air_time shows a roughly similar distribution before and after removing anomalies.

Check sparse cases

diagnose_sparese() checks for combinations of levels that do not appear as data among all combinations of levels of categorical variables.

glimpse(heartfailure)

Rows: 299
Columns: 13
$ age               <int> 75, 55, 65, 50, 65, 90, 75, 60, 65, 80, 75…
$ anaemia           <fct> No, No, No, Yes, Yes, Yes, Yes, Yes, No, Y…
$ cpk_enzyme        <dbl> 582, 7861, 146, 111, 160, 47, 246, 315, 15…
$ diabetes          <fct> No, No, No, No, Yes, No, No, Yes, No, No, …
$ ejection_fraction <dbl> 20, 38, 20, 20, 20, 40, 15, 60, 65, 35, 38…
$ hblood_pressure   <fct> Yes, No, No, No, No, Yes, No, No, No, Yes,…
$ platelets         <dbl> 265000, 263358, 162000, 210000, 327000, 20…
$ creatinine        <dbl> 1.9, 1.1, 1.3, 1.9, 2.7, 2.1, 1.2, 1.1, 1.…
$ sodium            <dbl> 130, 136, 129, 137, 116, 132, 137, 131, 13…
$ sex               <fct> Male, Male, Male, Male, Female, Male, Male…
$ smoking           <fct> No, No, Yes, No, No, Yes, No, Yes, No, Yes…
$ time              <int> 4, 6, 7, 7, 8, 8, 10, 10, 10, 10, 10, 10, …
$ death_event       <fct> Yes, Yes, Yes, Yes, Yes, Yes, Yes, Yes, Ye…

diagnose_sparese(heartfailure) %>% 
  flextable()

anaemia	diabetes	hblood_pressure	sex	smoking	death_event
No	No	No	Female	Yes	No
Yes	No	No	Female	Yes	No
No	Yes	No	Female	Yes	No
Yes	Yes	No	Female	Yes	No
Yes	No	Yes	Female	Yes	No
No	Yes	Yes	Female	Yes	No
Yes	Yes	Yes	Female	Yes	No
Yes	Yes	Yes	Male	No	Yes
No	No	No	Female	Yes	Yes
Yes	No	No	Female	Yes	Yes
No	Yes	No	Female	Yes	Yes
No	No	Yes	Female	Yes	Yes
Yes	Yes	Yes	Female	Yes	Yes
Yes	Yes	Yes	Male	Yes	Yes

# Adjust the threshold of limt to calculate
diagnose_sparese(heartfailure, limit = 50)

NULL

# List all combinations, including sparse cases
diagnose_sparese(heartfailure, type = "all") %>% 
  flextable()

anaemia	diabetes	hblood_pressure	sex	smoking	death_event	n_case
No	No	No	Female	No	No	10
Yes	No	No	Female	No	No	11
No	Yes	No	Female	No	No	14
Yes	Yes	No	Female	No	No	9
No	No	Yes	Female	No	No	8
Yes	No	Yes	Female	No	No	6
No	Yes	Yes	Female	No	No	6
Yes	Yes	Yes	Female	No	No	6
No	No	No	Male	No	No	12
Yes	No	No	Male	No	No	7
No	Yes	No	Male	No	No	13
Yes	Yes	No	Male	No	No	11
No	No	Yes	Male	No	No	8
Yes	No	Yes	Male	No	No	8
No	Yes	Yes	Male	No	No	5
Yes	Yes	Yes	Male	No	No	3
No	No	No	Female	Yes	No	0
Yes	No	No	Female	Yes	No	0
No	Yes	No	Female	Yes	No	0
Yes	Yes	No	Female	Yes	No	0
No	No	Yes	Female	Yes	No	1
Yes	No	Yes	Female	Yes	No	0
No	Yes	Yes	Female	Yes	No	0
Yes	Yes	Yes	Female	Yes	No	0
No	No	No	Male	Yes	No	26
Yes	No	No	Male	Yes	No	12
No	Yes	No	Male	Yes	No	8
Yes	Yes	No	Male	Yes	No	4
No	No	Yes	Male	Yes	No	5
Yes	No	Yes	Male	Yes	No	4
No	Yes	Yes	Male	Yes	No	4
Yes	Yes	Yes	Male	Yes	No	2
No	No	No	Female	No	Yes	5
Yes	No	No	Female	No	Yes	2
No	Yes	No	Female	No	Yes	3
Yes	Yes	No	Female	No	Yes	6
No	No	Yes	Female	No	Yes	2
Yes	No	Yes	Female	No	Yes	4
No	Yes	Yes	Female	No	Yes	3
Yes	Yes	Yes	Female	No	Yes	6
No	No	No	Male	No	Yes	8
Yes	No	No	Male	No	Yes	10
No	Yes	No	Male	No	Yes	4
Yes	Yes	No	Male	No	Yes	3
No	No	Yes	Male	No	Yes	4
Yes	No	Yes	Male	No	Yes	3
No	Yes	Yes	Male	No	Yes	3
Yes	Yes	Yes	Male	No	Yes	0
No	No	No	Female	Yes	Yes	0
Yes	No	No	Female	Yes	Yes	0
No	Yes	No	Female	Yes	Yes	0
Yes	Yes	No	Female	Yes	Yes	1
No	No	Yes	Female	Yes	Yes	0
Yes	No	Yes	Female	Yes	Yes	1
No	Yes	Yes	Female	Yes	Yes	1
Yes	Yes	Yes	Female	Yes	Yes	0
No	No	No	Male	Yes	Yes	6
Yes	No	No	Male	Yes	Yes	3
No	Yes	No	Male	Yes	Yes	4
Yes	Yes	No	Male	Yes	Yes	2
No	No	Yes	Male	Yes	Yes	3
Yes	No	Yes	Male	Yes	Yes	5
No	Yes	Yes	Male	Yes	Yes	4
Yes	Yes	Yes	Male	Yes	Yes	0

# collaboration with dplyr
heartfailure %>% 
  diagnose_sparese(type = "all") %>% 
  arrange(desc(n_case)) %>% 
  mutate(percent = round(n_case / sum(n_case) * 100, 1)) %>% 
  filter(percent > 3) %>% 
  flextable()

anaemia	diabetes	hblood_pressure	sex	smoking	death_event	n_case	percent
No	No	No	Male	Yes	No	26	8.7
No	Yes	No	Female	No	No	14	4.7
No	Yes	No	Male	No	No	13	4.3
No	No	No	Male	No	No	12	4.0
Yes	No	No	Male	Yes	No	12	4.0
Yes	No	No	Female	No	No	11	3.7
Yes	Yes	No	Male	No	No	11	3.7
No	No	No	Female	No	No	10	3.3
Yes	No	No	Male	No	Yes	10	3.3

Visualize missing values

Pareto chart

plot_na_pareto() visualize pareto chart for variables with missing value.

plot_na_pareto(flights)

# Visualize only variables containing missing values
plot_na_pareto(flights, only_na = TRUE)

# Change the grade
plot_na_pareto(flights, grade = list(High = 0.1, Middle = 0.6, Low = 1))

# Return the aggregate information about missing values.
plot_na_pareto(flights, only_na = TRUE, plot = FALSE)

# A tibble: 6 x 5
  variable  frequencies   ratio grade cumulative
  <fct>           <int>   <dbl> <fct>      <dbl>
1 air_time         9430 0.0280  Good        20.2
2 arr_delay        9430 0.0280  Good        40.5
3 arr_time         8713 0.0259  Good        59.2
4 dep_delay        8255 0.0245  Good        76.9
5 dep_time         8255 0.0245  Good        94.6
6 tailnum          2512 0.00746 Good       100

Combination chart using hclust

plot_na_hclust() visualize distribution of missing value by combination of variables using hclust().

# Generate data for the example
set.seed(123L)
flights2 <- flights[sample(nrow(flights), size = 1000), ]

# Visualize hcluster chart for variables with missing value.
plot_na_hclust(flights2)

Combination chart

plot_na_intersect() visualize the combinations of missing value across cases.

# Visualize the combination variables that is include missing value.
plot_na_intersect(flights2)

# Visualize variables containing missing values and complete case
plot_na_intersect(flights2, only_na = FALSE)

# Using n_vars argument
plot_na_intersect(flights2, n_vars = 5)

# Using n_intersects argument
plot_na_intersect(flights2, only_na = FALSE, n_intersacts = 3)

Create a diagnostic report

diagnose_report() performs data diagnosis of all variables of object inherited from data.frame(tbl_df, tbl, etc) or data.frame.

`diagnose_report() writes the report in two formats:

Latex based pdf file
html file

The contents of the report are as follows.:

Diagnose Data
- Overview of Diagnosis
  - List of all variables quality
  - Diagnosing Missing Data
  - Diagnosis of unique data(Text and Category)
  - Diagnosis of unique data(Numerical)
- Detailed data diagnosis
  - Diagnosis of categorical variables
  - Diagnosis of numerical variables
  - List of numerical diagnosis (zero)
  - List of numerical diagnosis (minus)
Diagnose Outliers
- Overview of Diagnosis
  - Diagnosis of numerical variable outliers
  - Detailed outliers diagnosis

The following creates a quality diagnostic report for flights, a tbl_df class object. The file format is pdf and file name is DataDiagnosis_Report.pdf.

flights %>%
  diagnose_report()

The following script creates an html report named DataDiagnosis_Report.html.

flights %>%
  diagnose_report(output_format = "html")

The following generates an HTML report named Diagn.html.

flights %>%
  diagnose_report(output_format = "html", output_file = "Diagn.html")

The Data Diagnostic Report is an automated report intended to aid in the data diagnosis process. It judged whether the data is supplemented or reacquired by referring to the report results.

Contents of pdf file

The cover of the report is shown in the following figure.:

The contents of the report are shown in the following figure.:

Most information is represented in the report as a table. An example of a table is shown in the following figure.:

In the data diagnosis report, the outlier diagnostic contents include visualization results. The result is shown in the following figure.:

Contents of html file

The title and contents of the report are shown in the following figure.:

Most of the information is represented in tables in reports. An example of a table in an html file is shown in the following figure.

In the data diagnosis report, the outlier diagnostic contents include visualization results. The result of the html file is shown in the following figure.

Reuse

Text and figures are licensed under Creative Commons Attribution CC BY 4.0. The figures that have been reused from other sources don't fall under this license and can be recognized by a note in their caption: "Figure from ...".

dlookr: Data quality diagnosis

Author

Affiliation

Published

DOI

Preface

Overview

Overall Diagnose Data

Visualize Missing Values

Reporting

Exercise data: nycflights13::flights

Overall Diagnose Data

Overview data

General diagnosis

Diagnosis of numeric variables

Diagnosis of categorical variables

Diagnosing outliers

Visualization of outliers

Check sparse cases

Visualize missing values

Pareto chart

Combination chart using hclust

Combination chart

Create a diagnostic report

Contents of pdf file

Contents of html file

Footnotes

Reuse