Home > Portfolio
The Brief
Arizona Engineering Services, an Arizona Group of Companies subsidiary, needs to validate data on its HR Analytics Report to ensure data quality.
As their BI Analyst, the HR manager would like you to;
- Validate the HR report to see if there are any data discrepancies or anomalies.
- Ensure optimal data quality.
Tools: SQL, MS SQL SERVER
Process and Implementation
Data Import
HR data was converted to CSV and imported into MS SQL Server. No primary key was set due to null values in the dataset. This can be removed from Excel, but we intend to use SQL for cleaning and transformations.
Create Database
Data Cleaning and Transformation
When conducting validations on a dashboard, it is important to go through the steps the team took to clean and transform the data before developing the dashboard. This is key in checking if there were any discrepancies in the cleaning process.
After importing the raw data, we would perform data cleaning which include;
- Removing nulls (from employee number, this would be the primary key)
- Removing duplicate values
- Changing data types, and
- Adding a primary key constraint
USE HR_Analytics_DB GO -- DATA CLEANUP & TRANSFORMATION -- Count the dataset to ensure all rows were imported SELECT COUNT(*) FROM employees; -- inspect the dataset SELECT * FROM employees; -- Dropped all columns where the employee number is null DELETE FROM employees WHERE Employee_Number IS NULL; -- Changed employee_number from varchar to integer with a not null constraint ALTER TABLE employees ALTER COLUMN Employee_Number INT NOT NULL; -- Added a primary key constraint ALTER TABLE employees ADD CONSTRAINT PK_Employees_Employee_Number PRIMARY KEY (Employee_Number); -- Primary key constraint returned error due to duplicate employee number -- Retrieve duplicate employee numbers SELECT * FROM ( SELECT Employee_Number, ROW_NUMBER() OVER(PARTITION BY Employee_Number ORDER BY Employee_Number) AS rn FROM employees )X WHERE X.rn > 1; -- To cross-check the duplicate rows SELECT * FROM employees WHERE Employee_Number = 1204033041; -- Duplicate employee numbers have different employee names, this was spotted in the HR Analysis & Report - Power BI version -- In a real scenario we would have to forward this back to HR for clarification, -- but for now, we would delete the duplicate record. -- USING CTE WITH cte AS ( SELECT *, ROW_NUMBER() OVER(PARTITION BY Employee_Number ORDER BY Employee_Number) AS rn FROM employees ) DELETE FROM cte WHERE rn > 1; -- We can now add a primary key constraint ALTER TABLE employees ADD CONSTRAINT PK_Employees_Employee_Number PRIMARY KEY (Employee_Number);
Data Analysis and Validation
Business Questions
1. Is there gender equality in the company’s employment?
2. Is there diversity in the workplace?
3. What demographics have the highest impact on low pay rates?
4. Identify the trend and pattern in the hiring process of the organization.
— The total number of employees is 300 compared to 301 in the Power BI report. We dropped the duplicate employee number because we needed to assign the primary key constraint.
1. Is there gender equality in the company’s employment?
The question relates to all employees recruited and not just current working employees.
SELECT
*,
(cast(no_of_employees as float)/cast(total_employees as float))* 100 as pct
FROM
(
SELECT
sex,
count(1) no_of_employees,
(select count(1) from employees) as total_employees
FROM employees
GROUP BY sex
)x;
-- We could achieve this with a window function to get better query performance
SELECT
sex,
count(1) as no_of_employees,
(cast(count(1) as float)/cast(sum(count(1)) over () as float))*100 as pct_of_total
FROM employees
GROUP BY sex;
sex | no_of_employees | total_employees | pct |
---|---|---|---|
Female | 174 | 300 | 58 |
Male | 126 | 300 | 42 |
sex | no_of_employees | pct_of_total |
---|---|---|
Female | 174 | 58 |
Male | 126 | 42 |
— # Females make-up 58% while males make-up 42%
2. Is there diversity in the workplace?
The question refers to current employees.
SELECT
racedesc as race,
count(1) as no_of_employees,
round(cast(count(1) as float)/cast(sum(count(1)) over () as float)*100, 2) as pct_of_total
FROM employees
WHERE Employment_Status not in ('Terminated for Cause','Voluntarily Terminated')
GROUP BY racedesc
ORDER BY no_of_employees DESC;
race | no_of_employees | pct_of_total |
---|---|---|
White | 124 | 62.31 |
Black or African American | 37 | 18.59 |
Asian | 20 | 10.05 |
Two or more races | 11 | 5.53 |
American Indian or Alaska Native | 4 | 2.01 |
Hispanic | 3 | 1.51 |
— # White race has the highest numbers with 124 and make up 62.31% of all races followed by blacks with 37 (18.59%)
3. What demographics are more impacted by low pay rates?
We would find out which age groups, race and gender have the lowest pay rate
SELECT
racedesc as race,
sex,
age_group,
COUNT(1) as no_of_employees,
ROUND(AVG(pay_rate),2) as avg_payrate
FROM
(
SELECT
racedesc,
sex,
age,
pay_rate,
CASE WHEN age >= 18 AND age <= 25 THEN '18-25' WHEN age >= 26 AND age <= 35 THEN '26-35' WHEN age >= 36 AND age <= 45 THEN '36-45' WHEN age >= 46 AND age <= 55 THEN '46-55' WHEN age >= 56 THEN '56+'
END as age_group
FROM employees
)x
GROUP BY racedesc,
sex,
age_group
ORDER BY avg_payrate ASC;
race | sex | age_group | no_of_employees | avg_payrate |
---|---|---|---|---|
Asian | Female | 46-55 | 2 | 14.5 |
American Indian or Alaska Native | Male | 46-55 | 1 | 16 |
Hispanic | Female | 36-45 | 1 | 17 |
Two or more races | Female | 36-45 | 4 | 17.13 |
Two or more races | Female | 46-55 | 2 | 19 |
White | Female | 18-25 | 2 | 22.5 |
Asian | Female | 56+ | 2 | 23.5 |
American Indian or Alaska Native | Female | 26-35 | 2 | 24.75 |
Black or African American | Male | 36-45 | 9 | 24.78 |
Asian | Female | 36-45 | 8 | 24.8 |
White | Male | 18-25 | 1 | 27 |
White | Female | 26-35 | 49 | 27.54 |
White | Female | 46-55 | 16 | 28.59 |
Asian | Male | 36-45 | 6 | 28.71 |
White | Male | 26-35 | 36 | 29.18 |
White | Female | 36-45 | 37 | 29.22 |
White | Male | 56+ | 4 | 29.38 |
White | Male | 36-45 | 27 | 31.07 |
Black or African American | Female | 56+ | 2 | 31.25 |
Asian | Female | 26-35 | 8 | 32.84 |
Black or African American | Female | 36-45 | 7 | 33.37 |
Black or African American | Female | 46-55 | 8 | 33.5 |
Two or more races | Female | 26-35 | 5 | 34.6 |
Two or more races | Male | 26-35 | 2 | 35.88 |
Black or African American | Female | 26-35 | 13 | 35.95 |
White | Female | 56+ | 6 | 36.43 |
White | Male | 46-55 | 11 | 37.44 |
Black or African American | Male | 26-35 | 11 | 37.91 |
Asian | Male | 26-35 | 4 | 38.2 |
Hispanic | Male | 36-45 | 2 | 39.5 |
Two or more races | Male | 36-45 | 2 | 40 |
Two or more races | Male | 46-55 | 3 | 43.83 |
Black or African American | Male | 46-55 | 4 | 49.55 |
Asian | Male | 46-55 | 1 | 50.5 |
American Indian or Alaska Native | Male | 26-35 | 1 | 56 |
Hispanic | Male | 26-35 | 1 | 63 |
— # Asian females within the age 46-55 are most impacted by a low pay rate
4. Identify the trend and pattern in the hiring process of the organization
WITH all_hiring
AS (
SELECT
FORMAT(date_of_hire,'yyyy-MM') as hire_date,
COUNT(1) as no_of_employees
FROM employees
GROUP BY FORMAT(date_of_hire,'yyyy-MM')
),
female_hiring AS
(
SELECT
FORMAT(date_of_hire,'yyyy-MM') as hire_date,
COUNT(1) as female_hires
FROM employees
WHERE lower(sex) = 'female'
GROUP BY FORMAT(date_of_hire,'yyyy-MM')
),
male_hiring AS
(
SELECT
FORMAT(date_of_hire,'yyyy-MM') as hire_date,
COUNT(1) as male_hires
FROM employees
WHERE lower(sex) = 'male'
GROUP BY FORMAT(date_of_hire,'yyyy-MM')
)
SELECT
ah.hire_date,
no_of_employees,
coalesce(female_hires,0) as female_hires,
coalesce(male_hires,0) as male_hires
FROM all_hiring ah
LEFT JOIN female_hiring fh ON ah.hire_date = fh.hire_date
LEFT JOIN male_hiring mh ON ah.hire_date = mh.hire_date
ORDER BY ah.hire_date;
hire_date | no_of_employees | female_hires | male_hires |
---|---|---|---|
2006-01 | 1 | 0 | 1 |
2007-06 | 1 | 0 | 1 |
2007-11 | 1 | 0 | 1 |
2008-01 | 1 | 1 | 0 |
2008-09 | 1 | 1 | 0 |
2008-10 | 1 | 1 | 0 |
2009-01 | 4 | 3 | 1 |
2009-04 | 1 | 1 | 0 |
2009-07 | 1 | 1 | 0 |
2009-10 | 1 | 1 | 0 |
2010-04 | 3 | 2 | 1 |
2010-05 | 1 | 1 | 0 |
2010-07 | 1 | 0 | 1 |
2010-08 | 2 | 0 | 2 |
2010-09 | 1 | 1 | 0 |
2010-10 | 1 | 1 | 0 |
2011-01 | 15 | 10 | 5 |
2011-02 | 6 | 3 | 3 |
2011-03 | 2 | 0 | 2 |
2011-04 | 8 | 5 | 3 |
2011-05 | 12 | 8 | 4 |
2011-06 | 2 | 2 | 0 |
2011-07 | 13 | 8 | 5 |
2011-08 | 5 | 2 | 3 |
2011-09 | 10 | 7 | 3 |
2011-10 | 1 | 1 | 0 |
2011-11 | 10 | 6 | 4 |
2012-01 | 8 | 6 | 2 |
2012-02 | 5 | 3 | 2 |
2012-03 | 2 | 1 | 1 |
2012-04 | 10 | 4 | 6 |
2012-05 | 4 | 2 | 2 |
2012-07 | 4 | 2 | 2 |
2012-08 | 4 | 1 | 3 |
2012-09 | 4 | 1 | 3 |
2012-10 | 1 | 1 | 0 |
2012-11 | 2 | 2 | 0 |
2013-01 | 5 | 3 | 2 |
2013-02 | 2 | 1 | 1 |
2013-04 | 2 | 2 | 0 |
2013-05 | 3 | 2 | 1 |
2013-07 | 9 | 3 | 6 |
2013-08 | 6 | 5 | 1 |
2013-09 | 9 | 5 | 4 |
2013-11 | 7 | 6 | 1 |
2014-01 | 6 | 3 | 3 |
2014-02 | 7 | 3 | 4 |
2014-03 | 3 | 1 | 2 |
2014-05 | 10 | 5 | 5 |
2014-07 | 9 | 6 | 3 |
2014-08 | 3 | 1 | 2 |
2014-09 | 13 | 8 | 5 |
2014-11 | 8 | 4 | 4 |
2014-12 | 1 | 0 | 1 |
2015-01 | 11 | 5 | 6 |
2015-02 | 8 | 7 | 1 |
2015-03 | 12 | 6 | 6 |
2015-05 | 2 | 1 | 1 |
2015-06 | 2 | 1 | 1 |
2015-07 | 1 | 1 | 0 |
2016-01 | 2 | 0 | 2 |
2016-05 | 1 | 1 | 0 |
2016-06 | 3 | 2 | 1 |
2016-07 | 5 | 3 | 2 |
— in Jan 2011, 15 employees were hired, 10 females and 5 males
— in May 2011, 12 employees were hired, 8 females and 4 males
— in Sep 2011, 10 employees were hired, 7 females, 3 males,
— however in Aug 2013, 9 employees were hired, 3 females and 6males
This shows that there is a higher preference for female employees than male employees. There are only very few occurrences where male candidates hired exceeded females.