diff --git "a/faiss_index/chunks.json" "b/faiss_index/chunks.json" new file mode 100644--- /dev/null +++ "b/faiss_index/chunks.json" @@ -0,0 +1,5947 @@ +[ + { + "text": "PRACTICAL SQL A Beginner’s Guide to Storytelling with Data by Anthony DeBarros San Francisco Estadísticos e-Books & Papers PRACTICAL SQL. Copyright © 2018 by Anthony DeBarros. All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher. ISBN-10: 1-59327-827-6 ISBN-13: 978-1-59327-827-4 Publisher: William Pollock Production Editor: Janelle Ludowise Cover Illustration: Josh Ellingson Interior Design: Octopod Studios Developmental Editors: Liz Chadwick and Annie Choi Technical Reviewer: Josh Berkus Copyeditor: Anne Marie Walker Compositor: Janelle Ludowise Proofreader: James Fraleigh For information on distribution, translations, or bulk sales, please contact No Starch Press, Inc. directly: No Starch Press, Inc. 245 8th Street, San Francisco, CA 94103 phone: 1.415.863.9900; info@nostarch.com www.nostarch.com Library of Congress Cataloging-in-Publication Data Names: DeBarros, Anthony, author. Title: Practical SQL : a beginner's guide to storytelling with data / Anthony DeBarros. Description: San Francisco : No Starch Press, 2018. | Includes index. Identifiers: LCCN 2018000030 (print) | LCCN 2017043947 (ebook) | ISBN 9781593278458 (epub) | ISBN 1593278454 (epub) | ISBN 9781593278274 (paperback) | ISBN 1593278276 (paperback) | ISBN 9781593278458 (ebook) Subjects: LCSH: SQL (Computer program language) | Database design. | BISAC: COMPUTERS / Programming Languages / SQL. | COMPUTERS / Database Management / General. | COMPUTERS / Database Management / Data Mining. Classification: LCC QA76.73.S67 (print) | LCC QA76.73.S67 D44 2018 (ebook) | DDC 005.75/6--dc23 LC record available at https://lccn.loc.gov/2018000030 No Starch Press and the No Starch Press logo are registered trademarks of No Starch Press, Inc. Other product and company names mentioned herein may be the trademarks of their respective owners. Rather than use a trademark symbol with every occurrence of a trademarked name, we are using the names only in an editorial fashion and to the benefit of the trademark owner, with no Estadísticos e-Books & Papers intention of infringement of the trademark. The information in this book is distributed on an “As Is” basis, without warranty. While every precaution has been taken in the preparation of this work, neither the author nor No Starch Press, Inc. shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in it. Estadísticos e-Books & Papers About the Author Anthony DeBarros is an award-winning journalist who has combined avid interests in data analysis, coding, and storytelling for much of his career. He spent more than 25 years with the Gannett company, including the Poughkeepsie Journal, USA TODAY, and Gannett Digital. He is currently senior vice president for content and product development for a publishing and events firm and lives and works in the Washington, D.C., area. Estadísticos e-Books & Papers About the Technical Reviewer Josh Berkus is a “hacker emeritus” for the PostgreSQL Project, where he served on the Core Team for 13 years. He was also a", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 0 + }, + { + "text": "and product development for a publishing and events firm and lives and works in the Washington, D.C., area. Estadísticos e-Books & Papers About the Technical Reviewer Josh Berkus is a “hacker emeritus” for the PostgreSQL Project, where he served on the Core Team for 13 years. He was also a database consultant for 15 years, working with PostgreSQL, MySQL, CitusDB, Redis, CouchDB, Hadoop, and Microsoft SQL Server. Josh currently works as a Kubernetes community manager at Red Hat, Inc. Estadísticos e-Books & Papers BRIEF CONTENTS Foreword by Sarah Frostenson Acknowledgments Introduction Chapter 1: Creating Your First Database and Table Chapter 2: Beginning Data Exploration with SELECT Chapter 3: Understanding Data Types Chapter 4: Importing and Exporting Data Chapter 5: Basic Math and Stats with SQL Chapter 6: Joining Tables in a Relational Database Chapter 7: Table Design That Works for You Chapter 8: Extracting Information by Grouping and Summarizing Chapter 9: Inspecting and Modifying Data Chapter 10: Statistical Functions in SQL Chapter 11: Working with Dates and Times Chapter 12: Advanced Query Techniques Chapter 13: Mining Text to Find Meaningful Data Chapter 14: Analyzing Spatial Data with PostGIS Chapter 15: Saving Time with Views, Functions, and Triggers Chapter 16: Using PostgreSQL from the Command Line Chapter 17: Maintaining Your Database Estadísticos e-Books & Papers Chapter 18: Identifying and Telling the Story Behind Your Data Appendix: Additional PostgreSQL Resources Index Estadísticos e-Books & Papers CONTENTS IN DETAIL FOREWORD by Sarah Frostenson ACKNOWLEDGMENTS INTRODUCTION What Is SQL? Why Use SQL? About This Book Using the Book’s Code Examples Using PostgreSQL Installing PostgreSQL Working with pgAdmin Alternatives to pgAdmin Wrapping Up 1 CREATING YOUR FIRST DATABASE AND TABLE Creating a Database Executing SQL in pgAdmin Connecting to the Analysis Database Creating a Table The CREATE TABLE Statement Making the teachers Table Inserting Rows into a Table The INSERT Statement Viewing the Data When Code Goes Bad Formatting SQL for Readability Wrapping Up Estadísticos e-Books & Papers Try It Yourself 2 BEGINNING DATA EXPLORATION WITH SELECT Basic SELECT Syntax Querying a Subset of Columns Using DISTINCT to Find Unique Values Sorting Data with ORDER BY Filtering Rows with WHERE Using LIKE and ILIKE with WHERE Combining Operators with AND and OR Putting It All Together Wrapping Up Try It Yourself 3 UNDERSTANDING DATA TYPES Characters Numbers Integers Auto-Incrementing Integers Decimal Numbers Choosing Your Number Data Type Dates and Times Using the interval Data Type in Calculations Miscellaneous Types Transforming Values from One Type to Another with CAST CAST Shortcut Notation Wrapping Up Try It Yourself 4 IMPORTING AND EXPORTING DATA Estadísticos e-Books & Papers Working with Delimited Text Files Quoting Columns that Contain Delimiters Handling Header Rows Using COPY to Import Data Importing Census Data Describing Counties Creating the us_counties_2010 Table Census Columns and Data Types Performing the Census Import with COPY Importing a Subset of Columns with COPY Adding a Default Value to a Column During Import Using COPY to Export Data Exporting All Data Exporting Particular Columns Exporting Query Results Importing and", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 1 + }, + { + "text": "Data Describing Counties Creating the us_counties_2010 Table Census Columns and Data Types Performing the Census Import with COPY Importing a Subset of Columns with COPY Adding a Default Value to a Column During Import Using COPY to Export Data Exporting All Data Exporting Particular Columns Exporting Query Results Importing and Exporting Through pgAdmin Wrapping Up Try It Yourself 5 BASIC MATH AND STATS WITH SQL Math Operators Math and Data Types Adding, Subtracting, and Multiplying Division and Modulo Exponents, Roots, and Factorials Minding the Order of Operations Doing Math Across Census Table Columns Adding and Subtracting Columns Finding Percentages of the Whole Tracking Percent Change Aggregate Functions for Averages and Sums Finding the Median Estadísticos e-Books & Papers Finding the Median with Percentile Functions Median and Percentiles with Census Data Finding Other Quantiles with Percentile Functions Creating a median() Function Finding the Mode Wrapping Up Try It Yourself 6 JOINING TABLES IN A RELATIONAL DATABASE Linking Tables Using JOIN Relating Tables with Key Columns Querying Multiple Tables Using JOIN JOIN Types JOIN LEFT JOIN and RIGHT JOIN FULL OUTER JOIN CROSS JOIN Using NULL to Find Rows with Missing Values Three Types of Table Relationships One-to-One Relationship One-to-Many Relationship Many-to-Many Relationship Selecting Specific Columns in a Join Simplifying JOIN Syntax with Table Aliases Joining Multiple Tables Performing Math on Joined Table Columns Wrapping Up Try It Yourself 7 TABLE DESIGN THAT WORKS FOR YOU Estadísticos e-Books & Papers Naming Tables, Columns, and Other Identifiers Using Quotes Around Identifiers to Enable Mixed Case Pitfalls with Quoting Identifiers Guidelines for Naming Identifiers Controlling Column Values with Constraints Primary Keys: Natural vs. Surrogate Foreign Keys Automatically Deleting Related Records with CASCADE The CHECK Constraint The UNIQUE Constraint The NOT NULL Constraint Removing Constraints or Adding Them Later Speeding Up Queries with Indexes B-Tree: PostgreSQL’s Default Index Considerations When Using Indexes Wrapping Up Try It Yourself 8 EXTRACTING INFORMATION BY GROUPING AND SUMMARIZING Creating the Library Survey Tables Creating the 2014 Library Data Table Creating the 2009 Library Data Table Exploring the Library Data Using Aggregate Functions Counting Rows and Values Using count() Finding Maximum and Minimum Values Using max() and min() Aggregating Data Using GROUP BY Wrapping Up Try It Yourself 9 Estadísticos e-Books & Papers INSPECTING AND MODIFYING DATA Importing Data on Meat, Poultry, and Egg Producers Interviewing the Data Set Checking for Missing Values Checking for Inconsistent Data Values Checking for Malformed Values Using length() Modifying Tables, Columns, and Data Modifying Tables with ALTER TABLE Modifying Values with UPDATE Creating Backup Tables Restoring Missing Column Values Updating Values for Consistency Repairing ZIP Codes Using Concatenation Updating Values Across Tables Deleting Unnecessary Data Deleting Rows from a Table Deleting a Column from a Table Deleting a Table from a Database Using Transaction Blocks to Save or Revert Changes Improving Performance When Updating Large Tables Wrapping Up Try It Yourself 10 STATISTICAL FUNCTIONS IN SQL Creating a Census Stats Table Measuring Correlation with corr(Y, X) Checking Additional Correlations Predicting Values with Regression Analysis", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 2 + }, + { + "text": "a Table Deleting a Table from a Database Using Transaction Blocks to Save or Revert Changes Improving Performance When Updating Large Tables Wrapping Up Try It Yourself 10 STATISTICAL FUNCTIONS IN SQL Creating a Census Stats Table Measuring Correlation with corr(Y, X) Checking Additional Correlations Predicting Values with Regression Analysis Finding the Effect of an Independent Variable with r-squared Creating Rankings with SQL Ranking with rank() and dense_rank() Estadísticos e-Books & Papers Ranking Within Subgroups with PARTITION BY Calculating Rates for Meaningful Comparisons Wrapping Up Try It Yourself 11 WORKING WITH DATES AND TIMES Data Types and Functions for Dates and Times Manipulating Dates and Times Extracting the Components of a timestamp Value Creating Datetime Values from timestamp Components Retrieving the Current Date and Time Working with Time Zones Finding Your Time Zone Setting Setting the Time Zone Calculations with Dates and Times Finding Patterns in New York City Taxi Data Finding Patterns in Amtrak Data Wrapping Up Try It Yourself 12 ADVANCED QUERY TECHNIQUES Using Subqueries Filtering with Subqueries in a WHERE Clause Creating Derived Tables with Subqueries Joining Derived Tables Generating Columns with Subqueries Subquery Expressions Common Table Expressions Cross Tabulations Installing the crosstab() Function Estadísticos e-Books & Papers Tabulating Survey Results Tabulating City Temperature Readings Reclassifying Values with CASE Using CASE in a Common Table Expression Wrapping Up Try It Yourself 13 MINING TEXT TO FIND MEANINGFUL DATA Formatting Text Using String Functions Case Formatting Character Information Removing Characters Extracting and Replacing Characters Matching Text Patterns with Regular Expressions Regular Expression Notation Turning Text to Data with Regular Expression Functions Using Regular Expressions with WHERE Additional Regular Expression Functions Full Text Search in PostgreSQL Text Search Data Types Creating a Table for Full Text Search Searching Speech Text Ranking Query Matches by Relevance Wrapping Up Try It Yourself 14 ANALYZING SPATIAL DATA WITH POSTGIS Installing PostGIS and Creating a Spatial Database The Building Blocks of Spatial Data Two-Dimensional Geometries Estadísticos e-Books & Papers Well-Known Text Formats A Note on Coordinate Systems Spatial Reference System Identifier PostGIS Data Types Creating Spatial Objects with PostGIS Functions Creating a Geometry Type from Well-Known Text Creating a Geography Type from Well-Known Text Point Functions LineString Functions Polygon Functions Analyzing Farmers’ Markets Data Creating and Filling a Geography Column Adding a GiST Index Finding Geographies Within a Given Distance Finding the Distance Between Geographies Working with Census Shapefiles Contents of a Shapefile Loading Shapefiles via the GUI Tool Exploring the Census 2010 Counties Shapefile Performing Spatial Joins Exploring Roads and Waterways Data Joining the Census Roads and Water Tables Finding the Location Where Objects Intersect Wrapping Up Try It Yourself 15 SAVING TIME WITH VIEWS, FUNCTIONS, AND TRIGGERS Using Views to Simplify Queries Creating and Querying Views Inserting, Updating, and Deleting Data Using a View Programming Your Own Functions Estadísticos e-Books & Papers Creating the percent_change() Function Using the percent_change() Function Updating Data with a Function Using the Python Language in a Function Automating Database Actions with Triggers Logging Grade Updates to", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 3 + }, + { + "text": "Queries Creating and Querying Views Inserting, Updating, and Deleting Data Using a View Programming Your Own Functions Estadísticos e-Books & Papers Creating the percent_change() Function Using the percent_change() Function Updating Data with a Function Using the Python Language in a Function Automating Database Actions with Triggers Logging Grade Updates to a Table Automatically Classifying Temperatures Wrapping Up Try It Yourself 16 USING POSTGRESQL FROM THE COMMAND LINE Setting Up the Command Line for psql Windows psql Setup macOS psql Setup Linux psql Setup Working with psql Launching psql and Connecting to a Database Getting Help Changing the User and Database Connection Running SQL Queries on psql Navigating and Formatting Results Meta-Commands for Database Information Importing, Exporting, and Using Files Additional Command Line Utilities to Expedite Tasks Adding a Database with createdb Loading Shapefiles with shp2pgsql Wrapping Up Try It Yourself 17 MAINTAINING YOUR DATABASE Estadísticos e-Books & Papers Recovering Unused Space with VACUUM Tracking Table Size Monitoring the autovacuum Process Running VACUUM Manually Reducing Table Size with VACUUM FULL Changing Server Settings Locating and Editing postgresql.conf Reloading Settings with pg_ctl Backing Up and Restoring Your Database Using pg_dump to Back Up a Database or Table Restoring a Database Backup with pg_restore Additional Backup and Restore Options Wrapping Up Try It Yourself 18 IDENTIFYING AND TELLING THE STORY BEHIND YOUR DATA Start with a Question Document Your Process Gather Your Data No Data? Build Your Own Database Assess the Data’s Origins Interview the Data with Queries Consult the Data’s Owner Identify Key Indicators and Trends over Time Ask Why Communicate Your Findings Wrapping Up Try It Yourself APPENDIX Estadísticos e-Books & Papers ADDITIONAL POSTGRESQL RESOURCES PostgreSQL Development Environments PostgreSQL Utilities, Tools, and Extensions PostgreSQL News Documentation INDEX Estadísticos e-Books & Papers FOREWORD When people ask which programming language I learned first, I often absent-mindedly reply, “Python,” forgetting that it was actually with SQL that I first learned to write code. This is probably because learning SQL felt so intuitive after spending years running formulas in Excel spreadsheets. I didn’t have a technical background, but I found SQL’s syntax, unlike that of many other programming languages, straightforward and easy to implement. For example, you run SELECT * on a SQL table to make every row and column appear. You simply use the JOIN keyword to return rows of data from different related tables, which you can then further group, sort, and analyze. I’m a graphics editor, and I’ve worked as a developer and journalist at a number of publications, including POLITICO, Vox, and USA TODAY. My daily responsibilities involve analyzing data and creating visualizations from what I find. I first used SQL when I worked at The Chronicle of Higher Education and its sister publication, The Chronicle of Philanthropy. Our team analyzed data ranging from nonprofit financials to faculty salaries at colleges and universities. Many of our projects included as much as 20 years’ worth of data, and one of my main tasks was to import all that data into a SQL", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 4 + }, + { + "text": "its sister publication, The Chronicle of Philanthropy. Our team analyzed data ranging from nonprofit financials to faculty salaries at colleges and universities. Many of our projects included as much as 20 years’ worth of data, and one of my main tasks was to import all that data into a SQL database and analyze it. I had to calculate the percent change in fund​raising dollars at a nonprofit or find the median endowment size at a university to measure an institution’s performance. I discovered SQL to be a powerful language, one that fundamentally shaped my understanding of what you can—and can’t—do with data. SQL excels at bringing order to messy, large data sets and helps you discover how different data sets are related. Plus, its queries and functions are easy to reuse within the same project or even in a different database. This leads me to Practical SQL. Looking back, I wish I’d read Chapter Estadísticos e-Books & Papers 4 on “Importing and Exporting Data” so I could have understood the power of bulk imports instead of writing long, cumbersome INSERT statements when filling a table. The statistical capabilities of PostgreSQL, covered in Chapters 5 and 10 in this book, are also something I wish I had grasped earlier, as my data analysis often involves calculating the percent change or finding the average or median values. I’m embarrassed to say that I didn’t know how percentile_cont(), covered in Chapter 5, could be used to easily calculate a median in PostgresSQL—with the added bonus that it also finds your data’s natural breaks or quantiles. But at that stage in my career, I was only scratching the surface of SQL’s capabilities. It wasn’t until 2014, when I became a data developer at Gannett Digital on a team led by Anthony DeBarros, that I learned to use PostgreSQL. I began to understand just how enormously powerful SQL was for creating a reproducible and sustainable workflow. When I met Anthony, he had been working at USA TODAY and other Gannett properties for more than 20 years, where he had led teams that built databases and published award-winning investigations. Anthony was able to show me the ins and outs of our team’s databases in addition to teaching me how to properly build and maintain my own. It was through working with Anthony that I truly learned how to code. One of the first projects Anthony and I collaborated on was the 2014 U.S. midterm elections. We helped build an election forecast data visualization to show USA TODAY readers the latest polling averages, campaign finance data, and biographical information for more than 1,300 candidates in more than 500 congressional and gubernatorial races. Building our data infrastructure was a complex, multistep process powered by a PostgreSQL database at its heart. Anthony taught me how to write code that funneled all the data from our sources into a half-dozen tables in PostgreSQL. From there, we could query the data into a format that would power the maps, charts, and", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 5 + }, + { + "text": "a complex, multistep process powered by a PostgreSQL database at its heart. Anthony taught me how to write code that funneled all the data from our sources into a half-dozen tables in PostgreSQL. From there, we could query the data into a format that would power the maps, charts, and front-end presentation of our election forecast. Around this time, I also learned one of my favorite things about PostgreSQL—its powerful suite of geographic functions (Chapter 14 in Estadísticos e-Books & Papers this book). By adding the PostGIS extension to the database, you can create spatial data that you can then export as GeoJSON or as a shapefile, a format that is easy to map. You can also perform complex spatial analysis, like calculating the distance between two points or finding the density of schools or, as Anthony shows in the chapter, all the farmers’ markets in a given radius. It’s a skill I’ve used repeatedly in my career. For example, I used it to build a data set of lead exposure risk at the census-tract level while at Vox, which I consider one of my crowning PostGIS achievements. Using this database, I was able to create a data set of every U.S. Census tract and its corresponding lead exposure risk in a spatial format that could be easily mapped at the national level. With so many different programming languages available—more than 200, if you can believe it—it’s truly overwhelming to know where to begin. One of the best pieces of advice I received when first starting to code was to find an inefficiency in my workflow that could be improved by coding. In my case, it was building a database to easily query a project’s data. Maybe you’re in a similar boat or maybe you just want to know how to analyze large data sets. Regardless, you’re probably looking for a no-nonsense guide that skips the programming jargon and delves into SQL in an easy-to-understand manner that is both practical and, more importantly, applicable. And that’s exactly what Practical SQL does. It gets away from programming theory and focuses on teaching SQL by example, using real data sets you’ll likely encounter. It also doesn’t shy away from showing you how to deal with annoying messy data pitfalls: misspelled names, missing values, and columns with unsuitable data types. This is important because, as you’ll quickly learn, there’s no such thing as clean data. Over the years, my role as a data journalist has evolved. I build fewer databases now and build more maps. I also report more. But the core requirement of my job, and what I learned when first learning SQL, remains the same: know thy data and to thine own data be true. In other words, the most important aspect of working with data is being able to Estadísticos e-Books & Papers understand what’s in it. You can’t expect to ask the right questions of your data or tell a compelling story if you don’t understand how to best", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 6 + }, + { + "text": "own data be true. In other words, the most important aspect of working with data is being able to Estadísticos e-Books & Papers understand what’s in it. You can’t expect to ask the right questions of your data or tell a compelling story if you don’t understand how to best analyze it. Fortunately, that’s where Practical SQL comes in. It’ll teach you the fundamentals of working with data so that you can discover your own stories and insights. Sarah Frostenson Graphics Editor at POLITICO Estadísticos e-Books & Papers ACKNOWLEDGMENTS Practical SQL is the work of many hands. My thanks, first, go to the team at No Starch Press. Thanks to Bill Pollock and Tyler Ortman for capturing the vision and sharpening the initial concept; to developmental editors Annie Choi and Liz Chadwick for refining each chapter; to copyeditor Anne Marie Walker for polishing the final drafts with an eagle eye; and to production editor Janelle Ludowise for laying out the book and keeping the process well organized. Josh Berkus, Kubernetes community manager for Red Hat, Inc., served as our technical reviewer. To work with Josh was to receive a master class in SQL and PostgreSQL. Thank you, Josh, for your patience and high standards. Thank you to Investigative Reporters and Editors (IRE) and its members and staff past and present for training journalists to find great stories in data. IRE is where I got my start with SQL and data journalism. During my years at USA TODAY, many colleagues either taught me SQL or imparted memorable lessons on data analysis. Special thanks to Paul Overberg for sharing his vast knowledge of demographics and the U.S. Census, to Lou Schilling for many technical lessons, to Christopher Schnaars for his SQL expertise, and to Sarah Frostenson for graciously agreeing to write the book’s foreword. My deepest appreciation goes to my dear wife, Elizabeth, and our sons. Thank you for making every day brighter and warmer, for your love, and for bearing with me as I completed this book. Estadísticos e-Books & Papers INTRODUCTION Shortly after joining the staff of USA TODAY I received a data set I would analyze almost every week for the next decade. It was the weekly Best-Selling Books list, which ranked the nation’s top-selling books based on confidential sales data. The list not only produced an endless stream of story ideas to pitch, but it also captured the zeitgeist of America in a singular way. For example, did you know that cookbooks sell a bit more during the week of Mother’s Day, or that Oprah Winfrey turned many obscure writers into number one best-selling authors just by having them on her show? Week after week, the book list editor and I pored over the sales figures and book genres, ranking the data in search of the next headline. Rarely did we come up empty: we chronicled everything from the rocket- rise of the blockbuster Harry Potter series to the fact that Oh, the Places You’ll Go! by Dr.", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 7 + }, + { + "text": "editor and I pored over the sales figures and book genres, ranking the data in search of the next headline. Rarely did we come up empty: we chronicled everything from the rocket- rise of the blockbuster Harry Potter series to the fact that Oh, the Places You’ll Go! by Dr. Seuss has become a perennial gift for new graduates. My technical companion during this time was the database programming language SQL (for Structured Query Language). Early on, I convinced USA TODAY’s IT department to grant me access to the SQL- based database system that powered our book list application. Using SQL, I was able to unlock the stories hidden in the database, which contained titles, authors, genres, and various codes that defined the publishing world. Analyzing data with SQL to discover interesting stories is exactly what you’ll learn to do using this book. Estadísticos e-Books & Papers What Is SQL? SQL is a widely used programming language that allows you to define and query databases. Whether you’re a marketing analyst, a journalist, or a researcher mapping neurons in the brain of a fruit fly, you’ll benefit from using SQL to manage database objects as well as create, modify, explore, and summarize data. Because SQL is a mature language that has been around for decades, it’s deeply ingrained in many modern systems. A pair of IBM researchers first outlined the syntax for SQL (then called SEQUEL) in a 1974 paper, building on the theoretical work of the British computer scientist Edgar F. Codd. In 1979, a precursor to the database company Oracle (then called Relational Software) became the first to use the language in a commercial product. Today, it continues to rank as one of the most-used computer languages in the world, and that’s unlikely to change soon. SQL comes in several variants, which are generally tied to specific database systems. The American National Standards Institute (ANSI) and International Organization for Standardization (ISO), which set standards for products and technologies, provide standards for the language and shepherd revisions to it. The good news is that the variants don’t stray far from the standard, so once you learn the SQL conventions for one database, you can transfer that knowledge to other systems. Why Use SQL? So why should you use SQL? After all, SQL is not usually the first tool people choose when they’re learning to analyze data. In fact, many people start with Microsoft Excel spreadsheets and their assortment of analytic functions. After working with Excel, they might graduate to Access, the database system built into Microsoft Office, which has a graphical query interface that makes it easy to get work done, making SQL skills optional. But as you might know, Excel and Access have their limits. Excel currently allows 1,048,576 rows maximum per worksheet, and Access limits database size to two gigabytes and limits columns to 255 per table. Estadísticos e-Books & Papers It’s not uncommon for data sets to surpass those limits, particularly when you’re working with data dumped", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 8 + }, + { + "text": "Excel and Access have their limits. Excel currently allows 1,048,576 rows maximum per worksheet, and Access limits database size to two gigabytes and limits columns to 255 per table. Estadísticos e-Books & Papers It’s not uncommon for data sets to surpass those limits, particularly when you’re working with data dumped from government systems. The last obstacle you want to discover while facing a deadline is that your database system doesn’t have the capacity to get the job done. Using a robust SQL database system allows you to work with terabytes of data, multiple related tables, and thousands of columns. It gives you improved programmatic control over the structure of your data, leading to efficiency, speed, and—most important—accuracy. SQL is also an excellent adjunct to programming languages used in the data sciences, such as R and Python. If you use either language, you can connect to SQL databases and, in some cases, even incorporate SQL syntax directly into the language. For people with no background in programming languages, SQL often serves as an easy-to-understand introduction into concepts related to data structures and programming logic. Additionally, knowing SQL can help you beyond data analysis. If you delve into building online applications, you’ll find that databases provide the backend power for many common web frameworks, interactive maps, and content management systems. When you need to dig beneath the surface of these applications, SQL’s capability to manipulate data and databases will come in very handy. About This Book Practical SQL is for people who encounter data in their everyday lives and want to learn how to analyze and transform it. To this end, I discuss real- world data and scenarios, such as U.S. Census demographics, crime statistics, and data about taxi rides in New York City. Along with information about databases and code, you’ll also learn tips on how to analyze and acquire data as well as other valuable insights I’ve accumulated throughout my career. I won’t focus on setting up servers or other tasks typically handled by a database administrator, but the SQL and PostgreSQL fundamentals you learn in this book will serve you well Estadísticos e-Books & Papers if you intend to go that route. I’ve designed the exercises for beginner SQL coders but will assume that you know your way around your computer, including how to install programs, navigate your hard drive, and download files from the internet. Although many chapters in this book can stand alone, you should work through the book sequentially to build on the fundamentals. Some data sets used in early chapters reappear later in the book, so following the book in order will help you stay on track. Practical SQL starts with the basics of databases, queries, tables, and data that are common to SQL across many database systems. Chapters 13 to 17 cover topics more specific to PostgreSQL, such as full text search and GIS. The following table of contents provides more detail about the topics discussed in each chapter: Chapter 1: Creating Your First", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 9 + }, + { + "text": "queries, tables, and data that are common to SQL across many database systems. Chapters 13 to 17 cover topics more specific to PostgreSQL, such as full text search and GIS. The following table of contents provides more detail about the topics discussed in each chapter: Chapter 1: Creating Your First Database and Table introduces PostgreSQL, the pgAdmin user interface, and the code for loading a simple data set about teachers into a new database. Chapter 2: Beginning Data Exploration with SELECT explores basic SQL query syntax, including how to sort and filter data. Chapter 3: Understanding Data Types explains the definitions for setting columns in a table to hold specific types of data, from text to dates to various forms of numbers. Chapter 4: Importing and Exporting Data explains how to use SQL commands to load data from external files and then export it. You’ll load a table of U.S. Census population data that you’ll use throughout the book. Chapter 5: Basic Math and Stats with SQL covers arithmetic operations and introduces aggregate functions for finding sums, averages, and medians. Chapter 6: Joining Tables in a Relational Database explains how to query multiple, related tables by joining them on key columns. You’ll learn how and when to use different types of joins. Estadísticos e-Books & Papers Chapter 7: Table Design that Works for You covers how to set up tables to improve the organization and integrity of your data as well as how to speed up queries using indexes. Chapter 8: Extracting Information by Grouping and Summarizing explains how to use aggregate functions to find trends in U.S. library use based on annual surveys. Chapter 9: Inspecting and Modifying Data explores how to find and fix incomplete or inaccurate data using a collection of records about meat, egg, and poultry producers as an example. Chapter 10: Statistical Functions in SQL introduces correlation, regression, and ranking functions in SQL to help you derive more meaning from data sets. Chapter 11: Working with Dates and Times explains how to create, manipulate, and query dates and times in your database, including working with time zones, using data on New York City taxi trips and Amtrak train schedules. Chapter 12: Advanced Query Techniques explains how to use more complex SQL operations, such as subqueries and cross tabulations, and the CASE statement to reclassify values in a data set on temperature readings. Chapter 13: Mining Text to Find Meaningful Data covers how to use PostgreSQL’s full text search engine and regular expressions to extract data from unstructured text, using a collection of speeches by U.S. presidents as an example. Chapter 14: Analyzing Spatial Data with PostGIS introduces data types and queries related to spatial objects, which will let you analyze geographical features like states, roads, and rivers. Chapter 15: Saving Time with Views, Functions, and Triggers explains how to automate database tasks so you can avoid repeating routine work. Estadísticos e-Books & Papers Chapter 16: Using PostgreSQL from the Command Line covers how to use", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 10 + }, + { + "text": "which will let you analyze geographical features like states, roads, and rivers. Chapter 15: Saving Time with Views, Functions, and Triggers explains how to automate database tasks so you can avoid repeating routine work. Estadísticos e-Books & Papers Chapter 16: Using PostgreSQL from the Command Line covers how to use text commands at your computer’s command prompt to connect to your database and run queries. Chapter 17: Maintaining Your Database provides tips and procedures for tracking the size of your database, customizing settings, and backing up data. Chapter 18: Identifying and Telling the Story Behind Your Data provides guidelines for generating ideas for analysis, vetting data, drawing sound conclusions, and presenting your findings clearly. Appendix: Additional PostgreSQL Resources lists software and documentation to help you grow your skills. Each chapter ends with a “Try It Yourself” section that contains exercises to help you reinforce the topics you learned. Using the Book’s Code Examples Each chapter includes code examples, and most use data sets I’ve already compiled. All the code and sample data in the book is available to download at https://www.nostarch.com/practicalSQL/. Click the Download the code from GitHub link to go to the GitHub repository that holds this material. At GitHub, you should see a “Clone or Download” button that gives you the option to download a ZIP file with all the materials. Save the file to your computer in a location where you can easily find it, such as your desktop. Inside the ZIP file is a folder for each chapter. Each folder contains a file named Chapter_XX (XX is the chapter number) that ends with a .sql extension. You can open those files with a text editor or with the PostgreSQL administrative tool you’ll install. You can copy and paste code when the book instructs you to run it. Note that in the book, several code examples are truncated to save space, but you’ll need the full listing from the .sql file to complete the exercise. You’ll know an example is truncated when you see --snip-- inside the listing. Estadísticos e-Books & Papers Also in the .sql files, you’ll see lines that begin with two hyphens (--) and a space. These are comments that provide the code’s listing number and additional context, but they’re not part of the code. These comments also note when the file has additional examples that aren’t in the book. NOTE After downloading data, Windows users might need to provide permission for the database to read files. To do so, right-click the folder containing the code and data, select Properties, and click the Security tab. Click Edit, then Add. Type the name Everyone into the object names box and click OK. Highlight Everyone in the user list, select all boxes under Allow, and then click Apply and OK. Using PostgreSQL In this book, I’ll teach you SQL using the open source PostgreSQL database system. PostgreSQL, or simply Postgres, is a robust database system that can handle very large amounts of data. Here are some", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 11 + }, + { + "text": "the user list, select all boxes under Allow, and then click Apply and OK. Using PostgreSQL In this book, I’ll teach you SQL using the open source PostgreSQL database system. PostgreSQL, or simply Postgres, is a robust database system that can handle very large amounts of data. Here are some reasons PostgreSQL is a great choice to use with this book: It’s free. It’s available for Windows, macOS, and Linux operating systems. Its SQL implementation closely follows ANSI standards. It’s widely used for analytics and data mining, so finding help online from peers is easy. Its geospatial extension, PostGIS, lets you analyze geometric data and perform mapping functions. It’s available in several variants, such as Amazon Redshift and Green​- plum, which focus on processing huge data sets. It’s a common choice for web applications, including those powered by the popular web frameworks Django and Ruby on Rails. Estadísticos e-Books & Papers Of course, you can also use another database system, such as Microsoft SQL Server or MySQL; many code examples in this book translate easily to either SQL implementation. However, some examples, especially later in the book, do not, and you’ll need to search online for equivalent solutions. Where appropriate, I’ll note whether an example code follows the ANSI SQL standard and may be portable to other systems or whether it’s specific to PostgreSQL. Installing PostgreSQL You’ll start by installing the PostgreSQL database and the graphical administrative tool pgAdmin, which is software that makes it easy to manage your database, import and export data, and write queries. One great benefit of working with PostgreSQL is that regardless of whether you work on Windows, macOS, or Linux, the open source community has made it easy to get PostgreSQL up and running. The following sections outline installation for all three operating systems as of this writing, but options might change as new versions are released. Check the documentation noted in each section as well as the GitHub repository with the book’s resources; I’ll maintain the files with updates and answers to frequently asked questions. NOTE Always install the latest available version of PostgreSQL for your operating system to ensure that it’s up to date on security patches and new features. For this book, I’ll assume you’re using version 10.0 or later. Windows Installation For Windows, I recommend using the installer provided by the company EnterpriseDB, which offers support and services for PostgreSQL users. EnterpriseDB’s package bundles PostgreSQL with pgAdmin and the company’s own Stack Builder, which also installs the spatial database Estadísticos e-Books & Papers extension PostGIS and programming language support, among other tools. To get the software, visit https://www.enterprisedb.com/ and create a free account. Then go to the downloads page at https://www.enterprisedb.com/software-downloads-postgres/. Select the latest available 64-bit Windows version of EDB Postgres Standard unless you’re using an older PC with 32-bit Windows. After you download the installer, follow these steps: 1. Right-click the installer and select Run as administrator. Answer Yes to the question about allowing the program to make changes", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 12 + }, + { + "text": "https://www.enterprisedb.com/software-downloads-postgres/. Select the latest available 64-bit Windows version of EDB Postgres Standard unless you’re using an older PC with 32-bit Windows. After you download the installer, follow these steps: 1. Right-click the installer and select Run as administrator. Answer Yes to the question about allowing the program to make changes to your computer. The program will perform a setup task and then present an initial welcome screen. Click through it. 2. Choose your installation directory, accepting the default. 3. On the Select Components screen, select the boxes to install PostgreSQL Server, the pgAdmin tool, Stack Builder, and Command Line Tools. 4. Choose the location to store data. You can choose the default, which is in a “data” subdirectory in the PostgreSQL directory. 5. Choose a password. PostgreSQL is robust with security and permissions. This password is for the initial database superuser account, which is called postgres. 6. Select a port number where the server will listen. Unless you have another database or application using it, the default of 5432 should be fine. If you have another version of PostgreSQL already installed or some other application is using that default, the value might be 5433 or another number, which is also okay. 7. Select your locale. Using the default is fine. Then click through the summary screen to begin the installation, which will take several minutes. 8. When the installation is done, you’ll be asked whether you want to launch EnterpriseDB’s Stack Builder to obtain additional packages. Select the box and click Finish. 9. When Stack Builder launches, choose the PostgreSQL installation Estadísticos e-Books & Papers on the drop-down menu and click Next. A list of additional applications should download. 10. Expand the Spatial Extensions menu and select either the 32-bit or 64-bit version of PostGIS Bundle for the version of Postgres you installed. Also, expand the Add-ons, tools and utilities menu and select EDB Language Pack, which installs support for programming languages including Python. Click through several times; you’ll need to wait while the installer downloads the additional components. 11. When installation files have been downloaded, click Next to install both components. For PostGIS, you’ll need to agree to the license terms; click through until you’re asked to Choose Components. Make sure PostGIS and Create spatial database are selected. Click Next, accept the default database location, and click Next again. 12. Enter your database password when prompted and continue through the prompts to finish installing PostGIS. 13. Answer Yes when asked to register GDAL. Also, answer Yes to the questions about setting POSTGIS_ENABLED_DRIVERS and enabling the POSTGIS_ENABLE_OUTDB_RASTERS environment variable. When finished, a PostgreSQL folder that contains shortcuts and links to documentation should be on your Windows Start menu. If you experience any hiccups installing PostgreSQL, refer to the “Troubleshooting” section of the EDB guide at https://www.enterprisedb.com/resources/product-documentation/. If you’re unable to install PostGIS via Stack Builder, try downloading a separate installer from the PostGIS site at http://postgis.net/windows_downloads/ and consult the guides at http://postgis.net/documentation/. macOS Installation For macOS users, I recommend obtaining", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 13 + }, + { + "text": "experience any hiccups installing PostgreSQL, refer to the “Troubleshooting” section of the EDB guide at https://www.enterprisedb.com/resources/product-documentation/. If you’re unable to install PostGIS via Stack Builder, try downloading a separate installer from the PostGIS site at http://postgis.net/windows_downloads/ and consult the guides at http://postgis.net/documentation/. macOS Installation For macOS users, I recommend obtaining Postgres.app, an open source macOS application that includes PostgreSQL as well as the PostGIS extension and a few other goodies: Estadísticos e-Books & Papers 1. Visit http://postgresapp.com/ and download the app’s Disk Image file that ends in .dmg. 2. Double-click the .dmg file to open it, and then drag and drop the app icon into your Applications folder. 3. Double-click the app icon. When Postgres.app opens, click Initialize to create and start a PostgreSQL database. A small elephant icon in your menu bar indicates that you now have a database running. To use included PostgreSQL command line tools, you’ll need to open your Terminal application and run the following code at the prompt (you can copy the code as a single line from the Postgres.app site at https://postgresapp.com/documentation/install.html): sudo mkdir -p /etc/paths.d && echo /Applications/Postgres.app/Contents/Versions/latest/bin | sudo tee /etc/paths.d/ postgresapp Next, because Postgres.app doesn’t include pgAdmin, you’ll need to follow these steps to download and run pgAdmin: 1. Visit the pgAdmin site’s page for macOS downloads at https://www.pgadmin.org/download/pgadmin-4-macos/. 2. Select the latest version and download the installer (look for a Disk Image file that ends in .dmg). 3. Double-click the .dmg file, click through the prompt to accept the terms, and then drag pgAdmin’s elephant app icon into your Applications folder. 4. Double-click the app icon to launch pgAdmin. NOTE On macOS, when you launch pgAdmin the first time, a dialog might appear that displays “pgAdmin4.app can’t be opened because it is from an unidentified developer.” Right-click the icon and select Open. The next Estadísticos e-Books & Papers dialog should give you the option to open the app; going forward, your Mac will remember you’ve granted this permission. Installation on macOS is relatively simple, but if you encounter any issues, review the documentation for Postgres.app at https://postgresapp.com/documentation/ and for pgAdmin at https://www.pgadmin.org/docs/. Linux Installation If you’re a Linux user, installing PostgreSQL becomes simultaneously easy and difficult, which in my experience is very much the way it is in the Linux universe. Most popular Linux distributions—including Ubuntu, Debian, and CentOS—bundle PostgreSQL in their standard package. However, some distributions stay on top of updates more than others. The best path is to consult your distribution’s documentation for the best way to install PostgreSQL if it’s not already included or if you want to upgrade to a more recent version. Alternatively, the PostgreSQL project maintains complete up-to-date package repositories for Red Hat variants, Debian, and Ubuntu. Visit https://yum.postgresql.org/ and https://wiki.postgresql.org/wiki/Apt for details. The packages you’ll want to install include the client and server for PostgreSQL, pgAdmin (if available), PostGIS, and PL/Python. The exact names of these packages will vary according to your Linux distribution. You might also need to manually start the PostgreSQL", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 14 + }, + { + "text": "Debian, and Ubuntu. Visit https://yum.postgresql.org/ and https://wiki.postgresql.org/wiki/Apt for details. The packages you’ll want to install include the client and server for PostgreSQL, pgAdmin (if available), PostGIS, and PL/Python. The exact names of these packages will vary according to your Linux distribution. You might also need to manually start the PostgreSQL database server. pgAdmin is rarely part of Linux distributions. To install it, refer to the pgAdmin site at https://www.pgadmin.org/download/ for the latest instructions and to see whether your platform is supported. If you’re feeling adventurous, you can find instructions on building the app from source code at https://www.pgadmin.org/download/pgadmin-4-source-code/. Working with pgAdmin Before you can start writing code, you’ll need to become familiar with Estadísticos e-Books & Papers pgAdmin, which is the administration and management tool for PostgreSQL. It’s free, but don’t underestimate its performance. In fact, pgAdmin is a full-featured tool similar to tools for purchase, such as Microsoft’s SQL Server Management Studio, in its capability to let you control multiple aspects of server operations. It includes a graphical interface for configuring and administrating your PostgreSQL server and databases, and—most appropriately for this book—offers a SQL query tool for writing, testing, and saving queries. If you’re using Windows, pgAdmin should come with the PostgreSQL package you downloaded from EnterpriseDB. On the Start menu, select PostgreSQL ▸ pgAdmin 4 (the version number of Postgres should also appear in the menu). If you’re using macOS and have installed pgAdmin separately, click the pgAdmin icon in your Applications folder, making sure you’ve also launched Postgres.app. When you open pgAdmin, it should look similar to Figure 1. Figure 1: The macOS version of the pgAdmin opening screen The left vertical pane displays an object browser where you can view available servers, databases, users, and other objects. Across the top of the Estadísticos e-Books & Papers screen is a collection of menu items, and below those are tabs to display various aspects of database objects and performance. Next, use the following steps to connect to the default database: 1. In the object browser, expand the plus sign (+) to the left of the Servers node to show the default server. Depending on your operating system, the default server name could be localhost or PostgreSQL x, where x is the Postgres version number. 2. Double-click the server name. Enter the password you chose during installation if prompted. A brief message appears while pgAdmin is establishing a connection. When you’re connected, several new object items should display under the server name. 3. Expand Databases and then expand the default database postgres. 4. Under postgres, expand the Schemas object, and then expand public. Your object browser pane should look similar to Figure 2. NOTE If pgAdmin doesn’t show a default under Servers, you’ll need to add it. Right-click Servers, and choose the Create Server option. In the dialog, type a name for your server in the General tab. On the Connection tab, in the Host name/address box, type localhost. Click Save, and you should see your server listed.", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 15 + }, + { + "text": "a default under Servers, you’ll need to add it. Right-click Servers, and choose the Create Server option. In the dialog, type a name for your server in the General tab. On the Connection tab, in the Host name/address box, type localhost. Click Save, and you should see your server listed. This collection of objects defines every feature of your database server. There’s a lot here, but for now we’ll focus on the location of tables. To view a table’s structure or perform actions on it with pgAdmin, this is where you can access the table. In Chapter 1, you’ll use this browser to create a new database and leave the default postgres as is. In addition, pgAdmin includes a Query Tool, which is where you write and execute code. To open the Query Tool, in pgAdmin’s object browser, click once on any database to highlight it. For example, click the Estadísticos e-Books & Papers postgres database and then select Tools ▸ Query Tool. The Query Tool has two panes: one for writing queries and one for output. It’s possible to open multiple tabs to connect to and write queries for different databases or just to organize your code the way you would like. To open another tab, click another database in the object browser and open the Query Tool again via the menu. Estadísticos e-Books & Papers Figure 2: The pgAdmin object browser Alternatives to pgAdmin Although pgAdmin is great for beginners, you’re not required to use it. If you prefer another administrative tool that works with PostgreSQL, feel free to use it. If you want to use your system’s command line for all the Estadísticos e-Books & Papers exercises in this book, Chapter 16 provides instructions on using the PostgreSQL command line tool psql. (The Appendix lists PostgreSQL resources you can explore to find additional administrative tools.) Wrapping Up Now that you’ve installed PostgreSQL and pgAdmin, you’re ready to start learning SQL and use it to discover valuable insights into your data! In Chapter 1, you’ll learn how to create a database and a table, and then you’ll load some data to explore its contents. Let’s get started! Estadísticos e-Books & Papers 1 CREATING YOUR FIRST DATABASE AND TABLE SQL is more than just a means for extracting knowledge from data. It’s also a language for defining the structures that hold data so we can organize relationships in the data. Chief among those structures is the table. A table is a grid of rows and columns that store data. Each row holds a collection of columns, and each column contains data of a specified type: most commonly, numbers, characters, and dates. We use SQL to define the structure of a table and how each table might relate to other tables in the database. We also use SQL to extract, or query, data from tables. Understanding tables is fundamental to understanding the data in your database. Whenever I start working with a fresh database, the first thing I do is", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 16 + }, + { + "text": "a table and how each table might relate to other tables in the database. We also use SQL to extract, or query, data from tables. Understanding tables is fundamental to understanding the data in your database. Whenever I start working with a fresh database, the first thing I do is look at the tables within. I look for clues in the table names and their column structure. Do the tables contain text, numbers, or both? How many rows are in each table? Next, I look at how many tables are in the database. The simplest database might have a single table. A full-bore application that handles customer data or tracks air travel might have dozens or hundreds. The number of tables tells me not only how much data I’ll need to analyze, but also hints that I should explore relationships among the data in each table. Estadísticos e-Books & Papers Before you dig into SQL, let’s look at an example of what the contents of tables might look like. We’ll use a hypothetical database for managing a school’s class enrollment; within that database are several tables that track students and their classes. The first table, called student_enrollment, shows the students that are signed up for each class section: student_id class_id class_section semester ---------- ---------- ------------- --------- CHRISPA004 COMPSCI101 3 Fall 2017 DAVISHE010 COMPSCI101 3 Fall 2017 ABRILDA002 ENG101 40 Fall 2017 DAVISHE010 ENG101 40 Fall 2017 RILEYPH002 ENG101 40 Fall 2017 This table shows that two students have signed up for COMPSCI101, and three have signed up for ENG101. But where are the details about each student and class? In this example, these details are stored in separate tables called students and classes, and each table relates to this one. This is where the power of a relational database begins to show itself. The first several rows of the students table include the following: student_id first_name last_name dob ---------- ---------- --------- ---------- ABRILDA002 Abril Davis 1999-01-10 CHRISPA004 Chris Park 1996-04-10 DAVISHE010 Davis Hernandez 1987-09-14 RILEYPH002 Riley Phelps 1996-06-15 The students table contains details on each student, using the value in the student_id column to identify each one. That value acts as a unique key that connects both tables, giving you the ability to create rows such as the following with the class_id column from student_enrollment and the first_name and last_name columns from students: class_id first_name last_name ---------- ---------- --------- COMPSCI101 Davis Hernandez COMPSCI101 Chris Park ENG101 Abril Davis ENG101 Davis Hernandez ENG101 Riley Phelps The classes table would work the same way, with a class_id column and Estadísticos e-Books & Papers several columns of detail about the class. Database builders prefer to organize data using separate tables for each main entity the database manages in order to reduce redundant data. In the example, we store each student’s name and date of birth just once. Even if the student signs up for multiple classes—as Davis Hernandez did—we don’t waste database space entering his name next to each class in the student_enrollment", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 17 + }, + { + "text": "entity the database manages in order to reduce redundant data. In the example, we store each student’s name and date of birth just once. Even if the student signs up for multiple classes—as Davis Hernandez did—we don’t waste database space entering his name next to each class in the student_enrollment table. We just include his student ID. Given that tables are a core building block of every database, in this chapter you’ll start your SQL coding adventure by creating a table inside a new database. Then you’ll load data into the table and view the completed table. Creating a Database The PostgreSQL program you downloaded in the Introduction is a database management system, a software package that allows you to define, manage, and query databases. When you installed PostgreSQL, it created a database server—an instance of the application running on your computer—that includes a default database called postgres. The database is a collection of objects that includes tables, functions, user roles, and much more. According to the PostgreSQL documentation, the default database is “meant for use by users, utilities and third party applications” (see https://www.postgresql.org/docs/current/static/app-initdb.html). In the exercises in this chapter, we’ll leave the default as is and instead create a new one. We’ll do this to keep objects related to a particular topic or application organized together. To create a database, you use just one line of SQL, shown in Listing 1-1. This code, along with all the examples in this book, is available for download via the resources at https://www.nostarch.com/practicalSQL/. CREATE DATABASE analysis; Listing 1-1: Creating a database named analysis This statement creates a database on your server named analysis using Estadísticos e-Books & Papers default PostgreSQL settings. Note that the code consists of two keywords —CREATE and DATABASE—followed by the name of the new database. The statement ends with a semicolon, which signals the end of the command. The semicolon ends all PostgreSQL statements and is part of the ANSI SQL standard. Sometimes you can omit the semicolon, but not always, and particularly not when running multiple statements in the admin. So, using the semicolon is a good habit to form. Executing SQL in pgAdmin As part of the Introduction to this book, you also installed the graphical administrative tool pgAdmin (if you didn’t, go ahead and do that now). For much of our work, you’ll use pgAdmin to run (or execute) the SQL statements we write. Later in the book in Chapter 16, I’ll show you how to run SQL statements in a terminal window using the PostgreSQL command line program psql, but getting started is a bit easier with a graphical interface. We’ll use pgAdmin to run the SQL statement in Listing 1-1 that creates the database. Then, we’ll connect to the new database and create a table. Follow these steps: 1. Run PostgreSQL. If you’re using Windows, the installer set PostgreSQL to launch every time you boot up. On macOS, you must double-click Postgres.app in your Applications folder. 2. Launch pgAdmin. As you did", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 18 + }, + { + "text": "the database. Then, we’ll connect to the new database and create a table. Follow these steps: 1. Run PostgreSQL. If you’re using Windows, the installer set PostgreSQL to launch every time you boot up. On macOS, you must double-click Postgres.app in your Applications folder. 2. Launch pgAdmin. As you did in the Introduction, in the left vertical pane (the object browser) expand the plus sign to the left of the Servers node to show the default server. Depending on how you installed PostgreSQL, the default server may be named localhost or PostgreSQL x, where x is the version of the application. 3. Double-click the server name. If you supplied a password during installation, enter it at the prompt. You’ll see a brief message that pgAdmin is establishing a connection. 4. In pgAdmin’s object browser, expand Databases and click once on the postgres database to highlight it, as shown in Figure 1-1. Estadísticos e-Books & Papers 5. Open the Query Tool by choosing Tools ▸ Query Tool. 6. In the SQL Editor pane (the top horizontal pane), type or copy the code from Listing 1-1. 7. Click the lightning bolt icon to execute the statement. PostgreSQL creates the database, and in the Output pane in the Query Tool under Messages you’ll see a notice indicating the query returned successfully, as shown in Figure 1-2. Figure 1-1: Connecting to the default postgres database Figure 1-2: Creating the analysis database Estadísticos e-Books & Papers 8. To see your new database, right-click Databases in the object browser. From the pop-up menu, select Refresh, and the analysis database will appear in the list, as shown in Figure 1-3. Good work! You now have a database called analysis, which you can use for the majority of the exercises in this book. In your own work, it’s generally a best practice to create a new database for each project to keep tables with related data together. Figure 1-3: The analysis database displayed in the object browser Connecting to the Analysis Database Before you create a table, you must ensure that pgAdmin is connected to the analysis database rather than to the default postgres database. To do that, follow these steps: 1. Close the Query Tool by clicking the X at the top right of the tool. You don’t need to save the file when prompted. 2. In the object browser, click once on the analysis database. 3. Reopen the Query Tool by choosing Tools ▸ Query Tool. 4. You should now see the label analysis on postgres@localhost at the top of the Query Tool window. (Again, instead of localhost, your version may show PostgreSQL.) Estadísticos e-Books & Papers Now, any code you execute will apply to the analysis database. Creating a Table As I mentioned earlier, tables are where data lives and its relationships are defined. When you create a table, you assign a name to each column (sometimes referred to as a field or attribute) and assign it a data type. These are the values the", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 19 + }, + { + "text": "analysis database. Creating a Table As I mentioned earlier, tables are where data lives and its relationships are defined. When you create a table, you assign a name to each column (sometimes referred to as a field or attribute) and assign it a data type. These are the values the column will accept—such as text, integers, decimals, and dates—and the definition of the data type is one way SQL enforces the integrity of data. For example, a column defined as date will take data in one of several standard formats, such as YYYY-MM-DD. If you try to enter characters not in a date format, for instance, the word peach, you’ll receive an error. Data stored in a table can be accessed and analyzed, or queried, with SQL statements. You can sort, edit, and view the data, and easily alter the table later if your needs change. Let’s make a table in the analysis database. The CREATE TABLE Statement For this exercise, we’ll use an often-discussed piece of data: teacher salaries. Listing 1-2 shows the SQL statement to create a table called teachers: ➊ CREATE TABLE teachers ( ➋ id bigserial, ➌ first_name varchar(25), last_name varchar(50), school varchar(50), ➍ hire_date date, ➎ salary numeric ➏ ); Listing 1-2: Creating a table named teachers with six columns This table definition is far from comprehensive. For example, it’s Estadísticos e-Books & Papers missing several constraints that would ensure that columns that must be filled do indeed have data or that we’re not inadvertently entering duplicate values. I cover constraints in detail in Chapter 7, but in these early chapters I’m omitting them to focus on getting you started on exploring data. The code begins with the two SQL keywords ➊ CREATE and TABLE that, together with the name teachers, signal PostgreSQL that the next bit of code describes a table to add to the database. Following an opening parenthesis, the statement includes a comma-separated list of column names along with their data types. For style purposes, each new line of code is on its own line and indented four spaces, which isn’t required, but it makes the code more readable. Each column name represents one discrete data element defined by a data type. The id column ➋ is of data type bigserial, a special integer type that auto-increments every time you add a row to the table. The first row receives the value of 1 in the id column, the second row 2, and so on. The bigserial data type and other serial types are PostgreSQL-specific implementations, but most database systems have a similar feature. Next, we create columns for the teacher’s first and last name, and the school where they teach ➌. Each is of the data type varchar, a text column with a maximum length specified by the number in parentheses. We’re assuming that no one in the database will have a last name of more than 50 characters. Although this is a safe assumption, you’ll discover over time that exceptions", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 20 + }, + { + "text": "Each is of the data type varchar, a text column with a maximum length specified by the number in parentheses. We’re assuming that no one in the database will have a last name of more than 50 characters. Although this is a safe assumption, you’ll discover over time that exceptions will always surprise you. The teacher’s hire_date ➍ is set to the data type date, and the salary column ➎ is a numeric. I’ll cover data types more thoroughly in Chapter 3, but this table shows some common examples of data types. The code block wraps up ➏ with a closing parenthesis and a semicolon. Now that you have a sense of how SQL looks, let’s run this code in pgAdmin. Making the teachers Table Estadísticos e-Books & Papers You have your code and you’re connected to the database, so you can make the table using the same steps we did when we created the database: 1. Open the pgAdmin Query Tool (if it’s not open, click once on the analysis database in pgAdmin’s object browser, and then choose Tools ▸ Query Tool). 2. Copy the CREATE TABLE script from Listing 1-2 into the SQL Editor. 3. Execute the script by clicking the lightning bolt icon. If all goes well, you’ll see a message in the pgAdmin Query Tool’s bottom output pane that reads, Query returned successfully with no result in 84 msec. Of course, the number of milliseconds will vary depending on your system. Now, find the table you created. Go back to the main pgAdmin window and, in the object browser, right-click the analysis database and choose Refresh. Choose Schemas ▸ public ▸ Tables to see your new table, as shown in Figure 1-4. Expand the teachers table node by clicking the plus sign to the left of its name. This reveals more details about the table, including the column names, as shown in Figure 1-5. Other information appears as well, such as indexes, triggers, and constraints, but I’ll cover those in later chapters. Clicking on the table name and then selecting the SQL menu in the pgAdmin workspace will display the SQL statement used to make the teachers table. Estadísticos e-Books & Papers Figure 1-4: The teachers table in the object browser Congratulations! So far, you’ve built a database and added a table to it. The next step is to add data to the table so you can write your first query. Estadísticos e-Books & Papers Figure 1-5: Table details for teachers Inserting Rows into a Table You can add data to a PostgreSQL table in several ways. Often, you’ll work with a large number of rows, so the easiest method is to import data from a text file or another database directly into a table. But just to get started, we’ll add a few rows using an INSERT INTO ... VALUES statement that specifies the target columns and the data values. Then we’ll view the data in its new home. The INSERT Statement To insert some", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 21 + }, + { + "text": "text file or another database directly into a table. But just to get started, we’ll add a few rows using an INSERT INTO ... VALUES statement that specifies the target columns and the data values. Then we’ll view the data in its new home. The INSERT Statement To insert some data into the table, you first need to erase the CREATE TABLE statement you just ran. Then, following the same steps as you did to create the database and table, copy the code in Listing 1-3 into your pgAdmin Query Tool: ➊ INSERT INTO teachers (first_name, last_name, school, hire_date, salary) ➋ VALUES ('Janet', 'Smith', 'F.D. Roosevelt HS', '2011-10-30', 36200), ('Lee', 'Reynolds', 'F.D. Roosevelt HS', '1993-05-22', 65000), ('Samuel', 'Cole', 'Myers Middle School', '2005-08-01', 43500), ('Samantha', 'Bush', 'Myers Middle School', '2011-10-30', 36200), ('Betty', 'Diaz', 'Myers Middle School', '2005-08-30', 43500), ('Kathleen', 'Roush', 'F.D. Roosevelt HS', '2010-10-22', 38500);➌ Listing 1-3: Inserting data into the teachers table This code block inserts names and data for six teachers. Here, the PostgreSQL syntax follows the ANSI SQL standard: after the INSERT INTO keywords is the name of the table, and in parentheses are the columns to be filled ➊. In the next row is the VALUES keyword and the data to insert into each column in each row ➋. You need to enclose the data for each row in a set of parentheses, and inside each set of parentheses, use a comma to separate each column value. The order of the values must also match the order of the columns specified after the table name. Each row of data ends with a comma, and the last row ends the entire statement Estadísticos e-Books & Papers with a semicolon ➌. Notice that certain values that we’re inserting are enclosed in single quotes, but some are not. This is a standard SQL requirement. Text and dates require quotes; numbers, including integers and decimals, don’t require quotes. I’ll highlight this requirement as it comes up in examples. Also, note the date format we’re using: a four-digit year is followed by the month and date, and each part is joined by a hyphen. This is the international standard for date formats; using it will help you avoid confusion. (Why is it best to use the format YYYY-MM-DD? Check out https://xkcd.com/1179/ to see a great comic about it.) PostgreSQL supports many additional date formats, and I’ll use several in examples. You might be wondering about the id column, which is the first column in the table. When you created the table, your script specified that column to be the bigserial data type. So as PostgreSQL inserts each row, it automatically fills the id column with an auto-incrementing integer. I’ll cover that in detail in Chapter 3 when I discuss data types. Now, run the code. This time the message in the Query Tool should include the words Query returned successfully: 6 rows affected. Viewing the Data You can take a quick look at the data you just loaded into the teachers table", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 22 + }, + { + "text": "detail in Chapter 3 when I discuss data types. Now, run the code. This time the message in the Query Tool should include the words Query returned successfully: 6 rows affected. Viewing the Data You can take a quick look at the data you just loaded into the teachers table using pgAdmin. In the object browser, locate the table and right- click. In the pop-up menu, choose View/Edit Data ▸ All Rows. As Figure 1-6 shows, you’ll see the six rows of data in the table with each column filled by the values in the SQL statement. Estadísticos e-Books & Papers Figure 1-6: Viewing table data directly in pgAdmin Notice that even though you didn’t insert a value for the id column, each teacher has an ID number assigned. You can view data using the pgAdmin interface in a few ways, but we’ll focus on writing SQL to handle those tasks. When Code Goes Bad There may be a universe where code always works, but unfortunately, we haven’t invented a machine capable of transporting us there. Errors happen. Whether you make a typo or mix up the order of operations, computer languages are unforgiving about syntax. For example, if you forget a comma in the code in Listing 1-3, PostgreSQL squawks back an error: ERROR: syntax error at or near \"(\" LINE 5: ('Samuel', 'Cole', 'Myers Middle School', '2005-08-01', 43... ^ ********** Error ********** Fortunately, the error message hints at what’s wrong and where: a syntax error is near an open parenthesis on line 5. But sometimes error messages can be more obscure. In that case, you do what the best coders do: a quick internet search for the error message. Most likely, someone else has experienced the same issue and might know the answer. Estadísticos e-Books & Papers Formatting SQL for Readability SQL requires no special formatting to run, so you’re free to use your own psychedelic style of uppercase, lowercase, and random indentations. But that won’t win you any friends when others need to work with your code (and sooner or later someone will). For the sake of readability and being a good coder, it’s best to follow these conventions: Uppercase SQL keywords, such as SELECT. Some SQL coders also uppercase the names of data types, such as TEXT and INTEGER. I use lowercase characters for data types in this book to separate them in your mind from keywords, but you can uppercase them if desired. Avoid camel case and instead use lowercase_and_underscores for object names, such as tables and column names (see more details about case in Chapter 7). Indent clauses and code blocks for readability using either two or four spaces. Some coders prefer tabs to spaces; use whichever works best for you or your organization. We’ll explore other SQL coding conventions as we go through the book, but these are the basics. Wrapping Up You accomplished quite a bit in this first chapter: you created a database and a table, and then loaded data into", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 23 + }, + { + "text": "spaces; use whichever works best for you or your organization. We’ll explore other SQL coding conventions as we go through the book, but these are the basics. Wrapping Up You accomplished quite a bit in this first chapter: you created a database and a table, and then loaded data into it. You’re on your way to adding SQL to your data analysis toolkit! In the next chapter, you’ll use this set of teacher data to learn the basics of querying a table using SELECT. TRY IT YOURSELF Here are two exercises to help you explore concepts related to databases, tables, and data relationships: Estadísticos e-Books & Papers 1. Imagine you’re building a database to catalog all the animals at your local zoo. You want one table to track the kinds of animals in the collection and another table to track the specifics on each animal. Write CREATE TABLE statements for each table that include some of the columns you need. Why did you include the columns you chose? 2. Now create INSERT statements to load sample data into the tables. How can you view the data via the pgAdmin tool? Create an additional INSERT statement for one of your tables. Purposely omit one of the required commas separating the entries in the VALUES clause of the query. What is the error message? Would it help you find the error in the code? Estadísticos e-Books & Papers 2 BEGINNING DATA EXPLORATION WITH SELECT For me, the best part of digging into data isn’t the prerequisites of gathering, loading, or cleaning the data, but when I actually get to interview the data. Those are the moments when I discover whether the data is clean or dirty, whether it’s complete, and most of all, what story the data can tell. Think of interviewing data as a process akin to interviewing a person applying for a job. You want to ask questions that reveal whether the reality of their expertise matches their resume. Interviewing is exciting because you discover truths. For example, you might find that half the respondents forgot to fill out the email field in the questionnaire, or the mayor hasn’t paid property taxes for the past five years. Or you might learn that your data is dirty: names are spelled inconsistently, dates are incorrect, or numbers don’t jibe with your expectations. Your findings become part of the data’s story. In SQL, interviewing data starts with the SELECT keyword, which retrieves rows and columns from one or more of the tables in a database. A SELECT statement can be simple, retrieving everything in a single table, or it can be complex enough to link dozens of tables while handling multiple calculations and filtering by exact criteria. We’ll start with simple SELECT statements. Estadísticos e-Books & Papers Basic SELECT Syntax Here’s a SELECT statement that fetches every row and column in a table called my_table: SELECT * FROM my_table; This single line of code shows the most basic form of a SQL query.", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 24 + }, + { + "text": "filtering by exact criteria. We’ll start with simple SELECT statements. Estadísticos e-Books & Papers Basic SELECT Syntax Here’s a SELECT statement that fetches every row and column in a table called my_table: SELECT * FROM my_table; This single line of code shows the most basic form of a SQL query. The asterisk following the SELECT keyword is a wildcard. A wildcard is like a stand-in for a value: it doesn’t represent anything in particular and instead represents everything that value could possibly be. Here, it’s shorthand for “select all columns.” If you had given a column name instead of the wildcard, this command would select the values in that column. The FROM keyword indicates you want the query to return data from a particular table. The semicolon after the table name tells PostgreSQL it’s the end of the query statement. Let’s use this SELECT statement with the asterisk wildcard on the teachers table you created in Chapter 1. Once again, open pgAdmin, select the analysis database, and open the Query Tool. Then execute the statement shown in Listing 2-1: SELECT * FROM teachers; Listing 2-1: Querying all rows and columns from the teachers table The result set in the Query Tool’s output pane contains all the rows and columns you inserted into the teachers table in Chapter 1. The rows may not always appear in this order, but that’s okay. Estadísticos e-Books & Papers Note that the id column (of type bigserial) automatically fills with sequential integers, even though you didn’t explicitly insert them. Very handy. This auto-incrementing integer acts as a unique identifier, or key, that not only ensures each row in the table is unique, but also will later give us a way to connect this table to other tables in the database. Let’s move on to refining this query. Querying a Subset of Columns Using the asterisk wildcard is helpful for discovering the entire contents of a table. But often it’s more practical to limit the columns the query retrieves, especially with large databases. You can do this by naming columns, separated by commas, right after the SELECT keyword. For example: SELECT some_column, another_column, amazing_column FROM table_name; With that syntax, the query will retrieve all rows from just those three columns. Let’s apply this to the teachers table. Perhaps in your analysis you want to focus on teachers’ names and salaries, not the school where they work or when they were hired. In that case, you might select only a few columns from the table instead of using the asterisk wildcard. Enter the statement shown in Listing 2-2. Notice that the order of the columns in the query is different than the order in the table: you’re able to retrieve columns in any order you’d like. SELECT last_name, first_name, salary FROM teachers; Listing 2-2: Querying a subset of columns Now, in the result set, you’ve limited the columns to three: last_name first_name salary --------- ---------- ------ Smith Janet 36200 Reynolds Lee 65000 Cole Samuel 43500 Bush Samantha", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 25 + }, + { + "text": "able to retrieve columns in any order you’d like. SELECT last_name, first_name, salary FROM teachers; Listing 2-2: Querying a subset of columns Now, in the result set, you’ve limited the columns to three: last_name first_name salary --------- ---------- ------ Smith Janet 36200 Reynolds Lee 65000 Cole Samuel 43500 Bush Samantha 36200 Estadísticos e-Books & Papers Diaz Betty 43500 Roush Kathleen 38500 Although these examples are basic, they illustrate a good strategy for beginning your interview of a data set. Generally, it’s wise to start your analysis by checking whether your data is present and in the format you expect. Are dates in a complete month-date-year format, or are they entered (as I once ruefully observed) as text with the month and year only? Does every row have a value? Are there mysteriously no last names starting with letters beyond “M”? All these issues indicate potential hazards ranging from missing data to shoddy recordkeeping somewhere in the workflow. We’re only working with a table of six rows, but when you’re facing a table of thousands or even millions of rows, it’s essential to get a quick read on your data quality and the range of values it contains. To do this, let’s dig deeper and add several SQL keywords. Using DISTINCT to Find Unique Values In a table, it’s not unusual for a column to contain rows with duplicate values. In the teachers table, for example, the school column lists the same school names multiple times because each school employs many teachers. To understand the range of values in a column, we can use the DISTINCT keyword as part of a query that eliminates duplicates and shows only unique values. Use the DISTINCT keyword immediately after SELECT, as shown in Listing 2-3: SELECT DISTINCT school FROM teachers; Listing 2-3: Querying distinct values in the school column The result is as follows: school ------------------- F.D. Roosevelt HS Myers Middle School Estadísticos e-Books & Papers Even though six rows are in the table, the output shows just the two unique school names in the school column. This is a helpful first step toward assessing data quality. For example, if a school name is spelled more than one way, those spelling variations will be easy to spot and correct. When you’re working with dates or numbers, DISTINCT will help highlight inconsistent or broken formatting. For example, you might inherit a data set in which dates were entered in a column formatted with a text data type. That practice (which you should avoid) allows malformed dates to exist: date --------- 5/30/2019 6//2019 6/1/2019 6/2/2019 The DISTINCT keyword also works on more than one column at a time. If we add a column, the query returns each unique pair of values. Run the code in Listing 2-4: SELECT DISTINCT school, salary FROM teachers; Listing 2-4: Querying distinct pairs of values in the school and salary columns Now the query returns each unique (or distinct) salary earned at each school. Because two teachers at Myers Middle School", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 26 + }, + { + "text": "each unique pair of values. Run the code in Listing 2-4: SELECT DISTINCT school, salary FROM teachers; Listing 2-4: Querying distinct pairs of values in the school and salary columns Now the query returns each unique (or distinct) salary earned at each school. Because two teachers at Myers Middle School earn $43,500, that pair is listed in just one row, and the query returns five rows rather than all six in the table: school salary ------------------- ------ Myers Middle School 43500 Myers Middle School 36200 F.D. Roosevelt HS 65000 F.D. Roosevelt HS 38500 F.D. Roosevelt HS 36200 This technique gives us the ability to ask, “For each x in the table, what are all the y values?” For each factory, what are all the chemicals it produces? For each election district, who are all the candidates running Estadísticos e-Books & Papers for office? For each concert hall, who are the artists playing this month? SQL offers more sophisticated techniques with aggregate functions that let us count, sum, and find minimum and maximum values. I’ll cover those in detail in Chapter 5 and Chapter 8. Sorting Data with ORDER BY Data can make more sense, and may reveal patterns more readily, when it’s arranged in order rather than jumbled randomly. In SQL, we order the results of a query using a clause containing the keywords ORDER BY followed by the name of the column or columns to sort. Applying this clause doesn’t change the original table, only the result of the query. Listing 2-5 shows an example using the teachers table: SELECT first_name, last_name, salary FROM teachers ORDER BY salary DESC; Listing 2-5: Sorting a column with ORDER BY By default, ORDER BY sorts values in ascending order, but here I sort in descending order by adding the DESC keyword. (The optional ASC keyword specifies sorting in ascending order.) Now, by ordering the salary column from highest to lowest, I can determine which teachers earn the most: first_name last_name salary ---------- --------- ------ Lee Reynolds 65000 Samuel Cole 43500 Betty Diaz 43500 Kathleen Roush 38500 Janet Smith 36200 Samantha Bush 36200 SORTING TEXT MAY SURPRISE YOU Sorting a column of numbers in PostgreSQL yields what you might expect: the data ranked from largest value to Estadísticos e-Books & Papers smallest or vice versa depending on whether or not you use the DESC keyword. But sorting a column with letters or other characters may return surprising results, especially if it has a mix of uppercase and lowercase characters, punctuation, or numbers that are treated as text. During PostgreSQL installation, the server is assigned a particular locale for collation, or ordering of text, as well as a character set. Both are based either on settings in the computer’s operating system or custom options supplied during installation. (You can read more about collation in the official PostgreSQL documentation at https://www.postgresql.org/docs/current/static/collation.html.) For example, on my Mac, my PostgreSQL install is set to the locale en_US, or U.S. English, and the character set UTF-8. You can", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 27 + }, + { + "text": "on settings in the computer’s operating system or custom options supplied during installation. (You can read more about collation in the official PostgreSQL documentation at https://www.postgresql.org/docs/current/static/collation.html.) For example, on my Mac, my PostgreSQL install is set to the locale en_US, or U.S. English, and the character set UTF-8. You can view your server’s collation setting by executing the statement SHOW ALL; and viewing the value of the parameter lc_collate. In a character set, each character gets a numerical value, and the sorting order depends on the order of those values. Based on UTF-8, PostgreSQL sorts characters in this order: 1. Punctuation marks, including quotes, parentheses, and math operators 2. Numbers 0 to 9 3. Additional punctuation, including the question mark 4. Capital letters from A to Z 5. More punctuation, including brackets and underscore 6. Lowercase letters a to z 7. Additional punctuation, special characters, and the extended alphabet Estadísticos e-Books & Papers Normally, the sorting order won’t be an issue because character columns usually just contain names, places, descriptions, and other straightforward text. But if you’re wondering why the word Ladybug appears before ladybug in your sort, you now have an explanation. The ability to sort in our queries gives us great flexibility in how we view and present data. For example, we’re not limited to sorting on just one column. Enter the statement in Listing 2-6: SELECT last_name, school, hire_date FROM teachers ➊ ORDER BY school ASC, hire_date DESC; Listing 2-6: Sorting multiple columns with ORDER BY In this case, we’re retrieving the last names of teachers, their school, and the date they were hired. By sorting the school column in ascending order and hire_date in descending order ➊, we create a listing of teachers grouped by school with the most recently hired teachers listed first. This shows us who the newest teachers are at each school. The result set should look like this: last_name school hire_date --------- ------------------- ---------- Smith F.D. Roosevelt HS 2011-10-30 Roush F.D. Roosevelt HS 2010-10-22 Reynolds F.D. Roosevelt HS 1993-05-22 Bush Myers Middle School 2011-10-30 Diaz Myers Middle School 2005-08-30 Cole Myers Middle School 2005-08-01 You can use ORDER BY on more than two columns, but you’ll soon reach a point of diminishing returns where the effect will be hardly noticeable. Imagine if you added columns about teachers’ highest college degree attained, the grade level taught, and birthdate to the ORDER BY clause. It would be difficult to understand the various sort directions in the output all at once, much less communicate that to others. Digesting data Estadísticos e-Books & Papers happens most easily when the result focuses on answering a specific question; therefore, a better strategy is to limit the number of columns in your query to only the most important, and then run several queries to answer each question you have. Filtering Rows with WHERE Sometimes, you’ll want to limit the rows a query returns to only those in which one or more columns meet certain criteria. Using teachers as an", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 28 + }, + { + "text": "columns in your query to only the most important, and then run several queries to answer each question you have. Filtering Rows with WHERE Sometimes, you’ll want to limit the rows a query returns to only those in which one or more columns meet certain criteria. Using teachers as an example, you might want to find all teachers hired before a particular year or all teachers making more than $75,000 at elementary schools. For these tasks, we use the WHERE clause. The WHERE keyword allows you to find rows that match a specific value, a range of values, or multiple values based on criteria supplied via an operator. You also can exclude rows based on criteria. Listing 2-7 shows a basic example. Note that in standard SQL syntax, the WHERE clause follows the FROM keyword and the name of the table or tables being queried: SELECT last_name, school, hire_date FROM teachers WHERE school = 'Myers Middle School'; Listing 2-7: Filtering rows using WHERE The result set shows just the teachers assigned to Myers Middle School: last_name school hire_date --------- ------------------- ---------- Cole Myers Middle School 2005-08-01 Bush Myers Middle School 2011-10-30 Diaz Myers Middle School 2005-08-30 Here, I’m using the equals comparison operator to find rows that exactly match a value, but of course you can use other operators with WHERE to customize your filter criteria. Table 2-1 provides a summary of the most commonly used comparison operators. Depending on your database Estadísticos e-Books & Papers system, many more might be available. Table 2-1: Comparison and Matching Operators in PostgreSQL OperatorFunction Example = Equal to WHERE school = 'Baker Middle' <> or != Not equal to* WHERE school <> 'Baker Middle' > Greater than WHERE salary > 20000 < Less than WHERE salary < 60500 >= Greater than or equal to WHERE salary >= 20000 <= Less than or equal to WHERE salary <= 60500 BETWEEN Within a range WHERE salary BETWEEN 20000 AND 40000 IN Match one of a set of values WHERE last_name IN ('Bush', 'Roush') LIKE Match a pattern (case sensitive) WHERE first_name LIKE 'Sam%' ILIKE Match a pattern (case insensitive) WHERE first_name ILIKE 'sam%' NOT Negates a condition WHERE first_name NOT ILIKE 'sam%' * The != operator is not part of standard ANSI SQL but is available in PostgreSQL and several other database systems. The following examples show comparison operators in action. First, we use the equals operator to find teachers whose first name is Janet: SELECT first_name, last_name, school FROM teachers WHERE first_name = 'Janet'; Next, we list all school names in the table but exclude F.D. Roosevelt HS using the not equal operator: SELECT school FROM teachers WHERE school != 'F.D. Roosevelt HS'; Estadísticos e-Books & Papers Here we use the less than operator to list teachers hired before January 1, 2000 (using the date format YYYY-MM-DD): SELECT first_name, last_name, hire_date FROM teachers WHERE hire_date < '2000-01-01'; Then we find teachers who earn $43,500 or more using the >= operator: SELECT first_name, last_name, salary", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 29 + }, + { + "text": "e-Books & Papers Here we use the less than operator to list teachers hired before January 1, 2000 (using the date format YYYY-MM-DD): SELECT first_name, last_name, hire_date FROM teachers WHERE hire_date < '2000-01-01'; Then we find teachers who earn $43,500 or more using the >= operator: SELECT first_name, last_name, salary FROM teachers WHERE salary >= 43500; The next query uses the BETWEEN operator to find teachers who earn between $40,000 and $65,000. Note that BETWEEN is inclusive, meaning the result will include values matching the start and end ranges specified. SELECT first_name, last_name, school, salary FROM teachers WHERE salary BETWEEN 40000 AND 65000; We’ll return to these operators throughout the book, because they’ll play a key role in helping us ferret out the data and answers we want to find. Using LIKE and ILIKE with WHERE Comparison operators are fairly straightforward, but LIKE and ILIKE deserve additional explanation. First, both let you search for patterns in strings by using two special characters: Percent sign (%) A wildcard matching one or more characters Underscore (_) A wildcard matching just one character For example, if you’re trying to find the word baker, the following LIKE patterns will match it: LIKE 'b%' LIKE '%ak%' Estadísticos e-Books & Papers LIKE '_aker' LIKE 'ba_er' The difference? The LIKE operator, which is part of the ANSI SQL standard, is case sensitive. The ILIKE operator, which is a PostgreSQL- only implementation, is case insensitive. Listing 2-8 shows how the two keywords give you different results. The first WHERE clause uses LIKE ➊ to find names that start with the characters sam, and because it’s case sensitive, it will return zero results. The second, using the case-insensitive ILIKE ➋, will return Samuel and Samantha from the table: SELECT first_name FROM teachers ➊ WHERE first_name LIKE 'sam%'; SELECT first_name FROM teachers ➋ WHERE first_name ILIKE 'sam%'; Listing 2-8: Filtering with LIKE and ILIKE Over the years, I’ve gravitated toward using ILIKE and wildcard operators in searches to make sure I’m not inadvertently excluding results from searches. I don’t assume that whoever typed the names of people, places, products, or other proper nouns always remembered to capitalize them. And if one of the goals of interviewing data is to understand its quality, using a case-insensitive search will help you find variations. Because LIKE and ILIKE search for patterns, performance on large databases can be slow. We can improve performance using indexes, which I’ll cover in “Speeding Up Queries with Indexes” on page 108. Combining Operators with AND and OR Comparison operators become even more useful when we combine them. To do this, we connect them using keywords AND and OR along with, if needed, parentheses. The statements in Listing 2-9 show three examples that combine operators this way: Estadísticos e-Books & Papers SELECT * FROM teachers ➊ WHERE school = 'Myers Middle School' AND salary < 40000; SELECT * FROM teachers ➋ WHERE last_name = 'Cole' OR last_name = 'Bush'; SELECT * FROM teachers ➌ WHERE school = 'F.D. Roosevelt", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 30 + }, + { + "text": "show three examples that combine operators this way: Estadísticos e-Books & Papers SELECT * FROM teachers ➊ WHERE school = 'Myers Middle School' AND salary < 40000; SELECT * FROM teachers ➋ WHERE last_name = 'Cole' OR last_name = 'Bush'; SELECT * FROM teachers ➌ WHERE school = 'F.D. Roosevelt HS' AND (salary < 38000 OR salary > 40000); Listing 2-9: Combining operators using AND and OR The first query uses AND in the WHERE clause ➊ to find teachers who work at Myers Middle School and have a salary less than $40,000. Because we connect the two conditions using AND, both must be true for a row to meet the criteria in the WHERE clause and be returned in the query results. The second example uses OR ➋ to search for any teacher whose last name matches Cole or Bush. When we connect conditions using OR, only one of the conditions must be true for a row to meet the criteria of the WHERE clause. The final example looks for teachers at Roosevelt whose salaries are either less than $38,000 or greater than $40,000 ➌. When we place statements inside parentheses, those are evaluated as a group before being combined with other criteria. In this case, the school name must be exactly F.D. Roosevelt HS and the salary must be either less or higher than specified for a row to meet the criteria of the WHERE clause. Putting It All Together You can begin to see how even the previous simple queries allow us to delve into our data with flexibility and precision to find what we’re looking for. You can combine comparison operator statements using the AND and OR keywords to provide multiple criteria for filtering, and you can include an ORDER BY clause to rank the results. Estadísticos e-Books & Papers With the preceding information in mind, let’s combine the concepts in this chapter into one statement to show how they fit together. SQL is particular about the order of keywords, so follow this convention: SELECT column_names FROM table_name WHERE criteria ORDER BY column_names; Listing 2-10 shows a query against the teachers table that includes all the aforementioned pieces: SELECT first_name, last_name, school, hire_date, salary FROM teachers WHERE school LIKE '%Roos%' ORDER BY hire_date DESC; Listing 2-10: A SELECT statement including WHERE and ORDER BY This listing returns teachers at Roosevelt High School, ordered from newest hire to earliest. We can see a clear correlation between a teacher’s hire date at the school and his or her current salary level: Wrapping Up Now that you’ve learned the basic structure of a few different SQL queries, you’ve acquired the foundation for many of the additional skills I’ll cover in later chapters. Sorting, filtering, and choosing only the most important columns from a table can yield a surprising amount of information from your data and help you find the story it tells. In the next chapter, you’ll learn about another foundational aspect of SQL: data types. Estadísticos e-Books &", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 31 + }, + { + "text": "in later chapters. Sorting, filtering, and choosing only the most important columns from a table can yield a surprising amount of information from your data and help you find the story it tells. In the next chapter, you’ll learn about another foundational aspect of SQL: data types. Estadísticos e-Books & Papers TRY IT YOURSELF Explore basic queries with these exercises: 1. The school district superintendent asks for a list of teachers in each school. Write a query that lists the schools in alphabetical order along with teachers ordered by last name A–Z. 2. Write a query that finds the one teacher whose first name starts with the letter S and who earns more than $40,000. 3. Rank teachers hired since January 1, 2010, ordered by highest paid to lowest. Estadísticos e-Books & Papers 3 UNDERSTANDING DATA TYPES Whenever I dig into a new database, I check the data type specified for each column in each table. If I’m lucky, I can get my hands on a data dictionary: a document that lists each column; specifies whether it’s a number, character, or other type; and explains the column values. Unfortunately, many organizations don’t create and maintain good documentation, so it’s not unusual to hear, “We don’t have a data dictionary.” In that case, I try to learn by inspecting the table structures in pgAdmin. It’s important to understand data types because storing data in the appropriate format is fundamental to building usable databases and performing accurate analysis. In addition, a data type is a programming concept applicable to more than just SQL. The concepts you’ll explore in this chapter will transfer well to additional languages you may want to learn. In a SQL database, each column in a table can hold one and only one data type, which is defined in the CREATE TABLE statement. You declare the data type after naming the column. Here’s a simple example that includes two columns, one a date and the other an integer: CREATE TABLE eagle_watch ( observed_date date, Estadísticos e-Books & Papers eagles_seen integer ); In this table named eagle_watch (for an annual inventory of bald eagles), the observed_date column is declared to hold date values by adding the date type declaration after its name. Similarly, eagles_seen is set to hold whole numbers with the integer type declaration. These data types are among the three categories you’ll encounter most: Characters Any character or symbol Numbers Includes whole numbers and fractions Dates and times Types holding temporal information Let’s look at each data type in depth; I’ll note whether they’re part of standard ANSI SQL or specific to PostgreSQL. Characters Character string types are general-purpose types suitable for any combination of text, numbers, and symbols. Character types include: char(n) A fixed-length column where the character length is specified by n. A column set at char(20) stores 20 characters per row regardless of how many characters you insert. If you insert fewer than 20 characters in any row, PostgreSQL pads the rest of that column", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 32 + }, + { + "text": "symbols. Character types include: char(n) A fixed-length column where the character length is specified by n. A column set at char(20) stores 20 characters per row regardless of how many characters you insert. If you insert fewer than 20 characters in any row, PostgreSQL pads the rest of that column with spaces. This type, which is part of standard SQL, also can be specified with the longer name character(n). Nowadays, char(n) is used infrequently and is mainly a remnant of legacy computer systems. varchar(n) A variable-length column where the maximum length is specified by n. If you insert fewer characters than the maximum, PostgreSQL will not store extra spaces. For example, the string blue will take four spaces, whereas the string 123 will take three. In large databases, this Estadísticos e-Books & Papers practice saves considerable space. This type, included in standard SQL, also can be specified using the longer name character varying(n). text A variable-length column of unlimited length. (According to the PostgreSQL documentation, the longest possible character string you can store is about 1 gigabyte.) The text type is not part of the SQL standard, but you’ll find similar implementations in other database systems, including Microsoft SQL Server and MySQL. According to PostgreSQL documentation at https://www.postgresql.org/docs/current/static/datatype-character.html, there is no substantial difference in performance among the three types. That may differ if you’re using another database manager, so it’s wise to check the docs. The flexibility and potential space savings of varchar and text seem to give them an advantage. But if you search discussions online, some users suggest that defining a column that will always have the same number of characters with char is a good way to signal what data it should contain. For instance, you might use char(2) for U.S. state postal abbreviations. To see these three character types in action, run the script in Listing 3-1. This script will build and load a simple table and then export the data to a text file on your computer. CREATE TABLE char_data_types ( ➊ varchar_column varchar(10), char_column char(10), text_column text ); ➋ INSERT INTO char_data_types VALUES ('abc', 'abc', 'abc'), ('defghi', 'defghi', 'defghi'); ➌ COPY char_data_types TO 'C:\\YourDirectory\\typetest.txt' ➍ WITH (FORMAT CSV, HEADER, DELIMITER '|'); Listing 3-1: Character data types in action Estadísticos e-Books & Papers The script defines three character columns ➊ of different types and inserts two rows of the same string into each ➋. Unlike the INSERT INTO statement you learned in Chapter 1, here we’re not specifying the names of the columns. If the VALUES statements match the number of columns in the table, the database will assume you’re inserting values in the order the column definitions were specified in the table. Next, the script uses the PostgreSQL COPY keyword ➌ to export the data to a text file named typetest.txt in a directory you specify. You’ll need to replace C:\\YourDirectory\\ with the full path to the directory on your computer where you want to save the file. The examples in this book use", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 33 + }, + { + "text": "script uses the PostgreSQL COPY keyword ➌ to export the data to a text file named typetest.txt in a directory you specify. You’ll need to replace C:\\YourDirectory\\ with the full path to the directory on your computer where you want to save the file. The examples in this book use Windows format and a path to a directory called YourDirectory on the C: drive. Linux and macOS file paths have a different format. On my Mac, the path to a file on the desktop is /Users/anthony/Desktop/. On Linux, my desktop is located at /home/anthony/Desktop/. The directory must exist already; PostgreSQL won’t create it for you. In PostgreSQL, COPY table_name FROM is the import function and COPY table_name TO is the export function. I’ll cover them in depth in Chapter 4; for now, all you need to know is that the WITH keyword options ➍ will format the data in the file with each column separated by a pipe character (|). That way, you can easily see where spaces fill out the unused portions of the char column. To see the output, open typetest.txt using a plain text editor (not Word or Excel, or another spreadsheet application). The contents should look like this: varchar_column|char_column|text_column abc|abc |abc defghi|defghi |defghi Even though you specified 10 characters for both the varchar and char columns, only the char column outputs 10 characters every time, padding unused characters with spaces. The varchar and text columns store only the characters you inserted. Again, there’s no real performance difference among the three types, although this example shows that char can potentially consume more Estadísticos e-Books & Papers storage space than needed. A few unused spaces in each column might seem negligible, but multiply that over millions of rows in dozens of tables and you’ll soon wish you had been more economical. Typically, using varchar with an n value sufficient to handle outliers is a solid strategy. Numbers Number columns hold various types of (you guessed it) numbers, but that’s not all: they also allow you to perform calculations on those numbers. That’s an important distinction from numbers you store as strings in a character column, which can’t be added, multiplied, divided, or perform any other math operation. Also, as I discussed in Chapter 2, numbers stored as characters sort differently than numbers stored as numbers, arranging in text rather than numerical order. So, if you’re doing math or the numeric order is important, use number types. The SQL number types include: Integers Whole numbers, both positive and negative Fixed-point and floating-point Two formats of fractions of whole numbers We’ll look at each type separately. Integers The integer data types are the most common number types you’ll find when exploring data in a SQL database. Think of all the places integers appear in life: your street or apartment number, the serial number on your refrigerator, the number on a raffle ticket. These are whole numbers, both positive and negative, including zero. The SQL standard provides three integer types: smallint,", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 34 + }, + { + "text": "exploring data in a SQL database. Think of all the places integers appear in life: your street or apartment number, the serial number on your refrigerator, the number on a raffle ticket. These are whole numbers, both positive and negative, including zero. The SQL standard provides three integer types: smallint, integer, and bigint. The difference between the three types is the maximum size of the numbers they can hold. Table 3-1 shows the upper and lower limits of Estadísticos e-Books & Papers each, as well as how much storage each requires in bytes. Table 3-1: Integer Data Types Data type Storage size Range smallint 2 bytes −32768 to +32767 integer 4 bytes −2147483648 to +2147483647 bigint 8 bytes −9223372036854775808 to +9223372036854775807 Even though it eats up the most storage, bigint will cover just about any requirement you’ll ever have with a number column. Its use is a must if you’re working with numbers larger than about 2.1 billion, but you can easily make it your go-to default and never worry. On the other hand, if you’re confident numbers will remain within the integer limit, that type is a good choice because it doesn’t consume as much space as bigint (a concern when dealing with millions of data rows). When the data values will remain constrained, smallint makes sense: days of the month or years are good examples. The smallint type will use half the storage as integer, so it’s a smart database design decision if the column values will always fit within its range. If you try to insert a number into any of these columns that is outside its range, the database will stop the operation and return an out of range error. Auto-Incrementing Integers In Chapter 1, when you made the teachers table, you created an id column with the declaration of bigserial: this and its siblings smallserial and serial are not so much true data types as a special implementation of the corresponding smallint, integer, and bigint types. When you add a column with a serial type, PostgreSQL will auto-increment the value in the column Estadísticos e-Books & Papers each time you insert a row, starting with 1, up to the maximum of each integer type. The serial types are implementations of the ANSI SQL standard for auto-numbered identity columns. Each database manager implements these in its own way. For example, Microsoft SQL Server uses an IDENTITY keyword to set a column to auto-increment. To use a serial type on a column, declare it in the CREATE TABLE statement as you would an integer type. For example, you could create a table called people that has an id column in each row: CREATE TABLE people ( id serial, person_name varchar(100) ); Every time a new person_name is added to the table, the id column will increment by 1. Table 3-2 shows the serial types and the ranges they cover. Table 3-2: Serial Data Types Data typeStorage sizeRange smallserial 2 bytes 1 to 32767 serial 4 bytes 1", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 35 + }, + { + "text": "serial, person_name varchar(100) ); Every time a new person_name is added to the table, the id column will increment by 1. Table 3-2 shows the serial types and the ranges they cover. Table 3-2: Serial Data Types Data typeStorage sizeRange smallserial 2 bytes 1 to 32767 serial 4 bytes 1 to 2147483647 bigserial 8 bytes 1 to 9223372036854775807 As with this example and in teachers in Chapter 1, makers of databases often employ a serial type to create a unique ID number, also known as a key, for each row in the table. Each row then has its own ID that other tables in the database can reference. I’ll cover this concept of relating tables in Chapter 6. Because the column is auto-incrementing, you don’t need to insert a number into that column when adding data; PostgreSQL handles that for you. Estadísticos e-Books & Papers NOTE Even though a column with a serial type auto-increments each time a row is added, some scenarios will create gaps in the sequence of numbers in the column. If a row is deleted, for example, the value in that row is never replaced. Or, if a row insert is aborted, the sequence for the column will still be incremented. Decimal Numbers As opposed to integers, decimals represent a whole number plus a fraction of a whole number; the fraction is represented by digits following a decimal point. In a SQL database, they’re handled by fixed-point and floating-point data types. For example, the distance from my house to the nearest grocery store is 6.7 miles; I could insert 6.7 into either a fixed- point or floating-point column with no complaint from PostgreSQL. The only difference is how the computer stores the data. In a moment, you’ll see that has important implications. Fixed-Point Numbers The fixed-point type, also called the arbitrary precision type, is numeric(precision,scale). You give the argument precision as the maximum number of digits to the left and right of the decimal point, and the argument scale as the number of digits allowable on the right of the decimal point. Alternately, you can specify this type using decimal(precision,scale). Both are part of the ANSI SQL standard. If you omit specifying a scale value, the scale will be set to zero; in effect, that creates an integer. If you omit specifying the precision and the scale, the database will store values of any precision and scale up to the maximum allowed. (That’s up to 131,072 digits before the decimal point and 16,383 digits after the decimal point, according to the PostgreSQL documentation at https://www.postgresql.org/docs/current/static/datatype- numeric.html.) Estadísticos e-Books & Papers For example, let’s say you’re collecting rainfall totals from several local airports—not an unlikely data analysis task. The U.S. National Weather Service provides this data with rainfall typically measured to two decimal places. (And, if you’re like me, you have a distant memory of your third- grade math teacher explaining that two digits after a decimal is the hundredths place.) To record rainfall in the database", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 36 + }, + { + "text": "task. The U.S. National Weather Service provides this data with rainfall typically measured to two decimal places. (And, if you’re like me, you have a distant memory of your third- grade math teacher explaining that two digits after a decimal is the hundredths place.) To record rainfall in the database using five digits total (the precision) and two digits maximum to the right of the decimal (the scale), you’d specify it as numeric(5,2). The database will always return two digits to the right of the decimal point, even if you don’t enter a number that contains two digits. For example, 1.47, 1.00, and 121.50. Floating-Point Types The two floating-point types are real and double precision. The difference between the two is how much data they store. The real type allows precision to six decimal digits, and double precision to 15 decimal points of precision, both of which include the number of digits on both sides of the point. These floating-point types are also called variable-precision types. The database stores the number in parts representing the digits and an exponent—the location where the decimal point belongs. So, unlike numeric, where we specify fixed precision and scale, the decimal point in a given column can “float” depending on the number. Using Fixed- and Floating-Point Types Each type has differing limits on the number of total digits, or precision, it can hold, as shown in Table 3-3. Table 3-3: Fixed-Point and Floating-Point Data Types Data type Storage size Storage type Range numeric, decimal variable Fixed- point Up to 131072 digits before the decimal point; up to 16383 digits after the decimal point Estadísticos e-Books & Papers real 4 bytes Floating- point 6 decimal digits precision double precision8 bytes Floating- point 15 decimal digits precision To see how each of the three data types handles the same numbers, create a small table and insert a variety of test cases, as shown in Listing 3-2: CREATE TABLE number_data_types ( ➊ numeric_column numeric(20,5), real_column real, double_column double precision ); ➋ INSERT INTO number_data_types VALUES (.7, .7, .7), (2.13579, 2.13579, 2.13579), (2.1357987654, 2.1357987654, 2.1357987654); SELECT * FROM number_data_types; Listing 3-2: Number data types in action We’ve created a table with one column for each of the fractional data types ➊ and loaded three rows into the table ➋. Each row repeats the same number across all three columns. When the last line of the script runs and we select everything from the table, we get the following: numeric_column real_column double_column -------------- ----------- ------------- 0.70000 0.7 0.7 2.13579 2.13579 2.13579 2.13580 2.1358 2.1357987654 Notice what happened. The numeric column, set with a scale of five, stores five digits after the decimal point whether or not you inserted that many. If fewer than five, it pads the rest with zeros. If more than five, it rounds them—as with the third-row number with 10 digits after the decimal. The real and double precision columns store only the number of digits Estadísticos e-Books & Papers present with no padding. Again on", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 37 + }, + { + "text": "many. If fewer than five, it pads the rest with zeros. If more than five, it rounds them—as with the third-row number with 10 digits after the decimal. The real and double precision columns store only the number of digits Estadísticos e-Books & Papers present with no padding. Again on the third row, the number is rounded when inserted into the real column because that type has a maximum of six digits of precision. The double precision column can hold up to 15 digits, so it stores the entire number. Trouble with Floating-Point Math If you’re thinking, “Well, numbers stored as a floating-point look just like numbers stored as fixed,” tread cautiously. The way computers store floating-point numbers can lead to unintended mathematical errors. Look at what happens when we do some calculations on these numbers. Run the script in Listing 3-3. SELECT ➊ numeric_column * 10000000 AS \"Fixed\", real_column * 10000000 AS \"Float\" FROM number_data_types ➋ WHERE numeric_column = .7; Listing 3-3: Rounding issues with float columns Here, we multiply the numeric_column and the real_column by 10 million ➊ and use a WHERE clause to filter out just the first row ➋. We should get the same result for both calculations, right? Here’s what the query returns: Fixed Float ------------- ---------------- 7000000.00000 6999999.88079071 Hello! No wonder floating-point types are referred to as “inexact.” It’s a good thing I’m not using this math to launch a mission to Mars or calculate the federal budget deficit. The reason floating-point math produces such errors is that the computer attempts to squeeze lots of information into a finite number of bits. The topic is the subject of a lot of writings and is beyond the scope of this book, but if you’re interested, you’ll find the link to a good synopsis at https://www.nostarch.com/practicalSQL/. The storage required by the numeric data type is variable, and Estadísticos e-Books & Papers depending on the precision and scale specified, numeric can consume considerably more space than the floating-point types. If you’re working with millions of rows, it’s worth considering whether you can live with relatively inexact floating-point math. Choosing Your Number Data Type For now, here are three guidelines to consider when you’re dealing with number data types: 1. Use integers when possible. Unless your data uses decimals, stick with integer types. 2. If you’re working with decimal data and need calculations to be exact (dealing with money, for example), choose numeric or its equivalent, decimal. Float types will save space, but the inexactness of floating- point math won’t pass muster in many applications. Use them only when exactness is not as important. 3. Choose a big enough number type. Unless you’re designing a database to hold millions of rows, err on the side of bigger. When using numeric or decimal, set the precision large enough to accommodate the number of digits on both sides of the decimal point. With whole numbers, use bigint unless you’re absolutely sure column values will be constrained to fit into the", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 38 + }, + { + "text": "millions of rows, err on the side of bigger. When using numeric or decimal, set the precision large enough to accommodate the number of digits on both sides of the decimal point. With whole numbers, use bigint unless you’re absolutely sure column values will be constrained to fit into the smaller integer or smallint types. Dates and Times Whenever you enter a date into a search form, you’re reaping the benefit of databases having an awareness of the current time (received from the server) plus the ability to handle formats for dates, times, and the nuances of the calendar, such as leap years and time zones. This is essential for storytelling with data, because the issue of when something occurred is usually as valuable a question as who, what, or how many were involved. PostgreSQL’s date and time support includes the four major data Estadísticos e-Books & Papers types shown in Table 3-4. Table 3-4: Date and Time Data Types Data typeStorage sizeDescription Range timestamp 8 bytes Date and time 4713 BC to 294276 AD date 4 bytes Date (no time) 4713 BC to 5874897 AD time 8 bytes Time (no date)00:00:00 to 24:00:00 interval 16 bytes Time interval +/− 178,000,000 years Here’s a rundown of data types for times and dates in PostgreSQL: timestamp Records date and time, which are useful for a range of situations you might track: departures and arrivals of passenger flights, a schedule of Major League Baseball games, or incidents along a timeline. Typically, you’ll want to add the keywords with time zone to ensure that the time recorded for an event includes the time zone where it occurred. Otherwise, times recorded in various places around the globe become impossible to compare. The format timestamp with time zone is part of the SQL standard; with PostgreSQL you can specify the same data type using timestamptz. date Records just the date. time Records just the time. Again, you’ll want to add the with time zone keywords. interval Holds a value representing a unit of time expressed in the format quantity unit. It doesn’t record the start or end of a time period, only its length. Examples include 12 days or 8 hours. (The PostgreSQL documentation at https://www.postgresql.org/docs/current/static/datatype- datetime.html lists unit values ranging from microsecond to millennium.) You’ll typically use this type for calculations or filtering on other date and time columns. Estadísticos e-Books & Papers Let’s focus on the timestamp with time zone and interval types. To see these in action, run the script in Listing 3-4. ➊ CREATE TABLE date_time_types ( timestamp_column timestamp with time zone, interval_column interval ); ➋ INSERT INTO date_time_types VALUES ('2018-12-31 01:00 EST','2 days'), ('2018-12-31 01:00 -8','1 month'), ('2018-12-31 01:00 Australia/Melbourne','1 century'), ➌ (now(),'1 week'); SELECT * FROM date_time_types; Listing 3-4: The timestamp and interval types in action Here, we create a table with a column for both types ➊ and insert four rows ➋. For the first three rows, our insert for the timestamp_column uses the same date and time", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 39 + }, + { + "text": "century'), ➌ (now(),'1 week'); SELECT * FROM date_time_types; Listing 3-4: The timestamp and interval types in action Here, we create a table with a column for both types ➊ and insert four rows ➋. For the first three rows, our insert for the timestamp_column uses the same date and time (December 31, 2018 at 1 AM) using the International Organization for Standardization (ISO) format for dates and times: YYYY- MM-DD HH:MM:SS. SQL supports additional date formats (such as MM/DD/YYYY), but ISO is recommended for portability worldwide. Following the time, we specify a time zone but use a different format in each of the first three rows: in the first row, we use the abbreviation EST, which is Eastern Standard Time in the United States. In the second row, we set the time zone with the value -8. That represents the number of hours difference, or offset, from Coordinated Universal Time (UTC). UTC refers to an overall world time standard as well as the value of UTC +/− 00:00, the time zone that covers the United Kingdom and Western Africa. (For a map of UTC time zones, see https://en.wikipedia.org/wiki/Coordinated_Universal_Time#/media/File:Stand ard_World_Time_Zones.png.) Using a value of -8 specifies a time zone eight hours behind UTC, which is the Pacific time zone in the United States and Canada. For the third row, we specify the time zone using the name of an area and location: Australia/Melbourne. That format uses values found in a standard time zone database often employed in computer programming. Estadísticos e-Books & Papers You can learn more about the time zone database at https://en.wikipedia.org/wiki/Tz_database. In the fourth row, instead of specifying dates, times, and time zones, the script uses PostgreSQL’s now() function ➌, which captures the current transaction time from your hardware. After the script runs, the output should look similar to (but not exactly like) this: timestamp_column interval_column ----------------------------- --------------- 2018-12-31 01:00:00-05 2 days 2018-12-31 04:00:00-05 1 mon 2018-12-30 09:00:00-05 100 years 2019-01-25 21:31:15.716063-05 7 days Even though we supplied the same date and time in the first three rows on the timestamp_column, each row’s output differs. The reason is that pgAdmin reports the date and time relative to my time zone, which in the results shown is indicated by the UTC offset of -05 at the end of each timestamp. A UTC offset of -05 means five hours behind UTC time, equivalent to the U.S. Eastern time zone, where I live. If you live in a different time zone, you’ll likely see a different offset; the times and dates also may differ from what’s shown here. We can change how PostgreSQL reports these timestamp values, and I’ll cover how to do that plus other tips for wrangling dates and times in Chapter 11. Finally, the interval_column shows the values you entered. PostgreSQL changed 1 century to 100 years and 1 week to 7 days because of its preferred default settings for interval display. Read the “Interval Input” section of the PostgreSQL documentation at https://www.postgresql.org/docs/current/static/datatype-datetime.html to learn more about options", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 40 + }, + { + "text": "times in Chapter 11. Finally, the interval_column shows the values you entered. PostgreSQL changed 1 century to 100 years and 1 week to 7 days because of its preferred default settings for interval display. Read the “Interval Input” section of the PostgreSQL documentation at https://www.postgresql.org/docs/current/static/datatype-datetime.html to learn more about options related to intervals. Using the interval Data Type in Calculations The interval data type is useful for easy-to-understand calculations on date and time data. For example, let’s say you have a column that holds Estadísticos e-Books & Papers the date a client signed a contract. Using interval data, you can add 90 days to each contract date to determine when to follow up with the client. To see how the interval data type works, we’ll use the date_time_types table we just created, as shown in Listing 3-5: SELECT timestamp_column interval_column, ➊ timestamp_column - interval_column AS new_date FROM date_time_types; Listing 3-5: Using the interval data type This is a typical SELECT statement except we’ll compute a column called new_date ➊ that contains the result of timestamp_column minus interval_column. (Computed columns are called expressions; we’ll use this technique often.) In each row, we subtract the unit of time indicated by the interval data type from the date. This produces the following result: Note that the new_date column by default is formatted as type timestamp with time zone, allowing for the display of time values as well as dates if the interval value uses them. Again, your output may be different based on your time zone. Miscellaneous Types The character, number, and date/time types you’ve learned so far will likely comprise the bulk of the work you do with SQL. But PostgreSQL supports many additional types, including but not limited to: A Boolean type that stores a value of true or false Estadísticos e-Books & Papers Geometric types that include points, lines, circles, and other two- dimensional objects Network address types, such as IP or MAC addresses A Universally Unique Identifier (UUID) type, sometimes used as a unique key value in tables XML and JSON data types that store information in those structured formats I’ll cover these types as required throughout the book. Transforming Values from One Type to Another with CAST Occasionally, you may need to transform a value from its stored data type to another type; for example, when you retrieve a number as a character so you can combine it with text or when you must treat a date stored as characters as an actual date type so you can sort it in date order or perform interval calculations. You can perform these conversions using the CAST() function. The CAST() function only succeeds when the target data type can accommodate the original value. Casting an integer as text is possible, because the character types can include numbers. Casting text with letters of the alphabet as a number is not. Listing 3-6 has three examples using the three data type tables we just created. The first two examples work, but the", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 41 + }, + { + "text": "the original value. Casting an integer as text is possible, because the character types can include numbers. Casting text with letters of the alphabet as a number is not. Listing 3-6 has three examples using the three data type tables we just created. The first two examples work, but the third will try to perform an invalid type conversion so you can see what a type casting error looks like. ➊ SELECT timestamp_column, CAST(timestamp_column AS varchar(10)) FROM date_time_types; ➋ SELECT numeric_column, CAST(numeric_column AS integer), CAST(numeric_column AS varchar(6)) FROM number_data_types; ➌ SELECT CAST(char_column AS integer) FROM char_data_types; Estadísticos e-Books & Papers Listing 3-6: Three CAST() examples The first SELECT statement ➊ returns the timestamp_column value as a varchar, which you’ll recall is a variable-length character column. In this case, I’ve set the character length to 10, which means when converted to a character string, only the first 10 characters are kept. That’s handy in this case, because that just gives us the date segment of the column and excludes the time. Of course, there are better ways to remove the time from a timestamp, and I’ll cover those in “Extracting the Components of a timestamp Value” on page 173. The second SELECT statement ➋ returns the numeric_column three times: in its original form and then as an integer and as a character. Upon conversion to an integer, PostgreSQL rounds the value to a whole number. But with the varchar conversion, no rounding occurs: the value is simply sliced at the sixth character. The final SELECT doesn’t work ➌: it returns an error of invalid input syntax for integer because letters can’t become integers! CAST Shortcut Notation It’s always best to write SQL that can be read by another person who might pick it up later, and the way CAST() is written makes what you intended when you used it fairly obvious. However, PostgreSQL also offers a less-obvious shortcut notation that takes less space: the double colon. Insert the double colon in between the name of the column and the data type you want to convert it to. For example, these two statements cast timestamp_column as a varchar: SELECT timestamp_column, CAST(timestamp_column AS varchar(10)) FROM date_time_types; SELECT timestamp_column::varchar(10) FROM date_time_types; Use whichever suits you, but be aware that the double colon is a Estadísticos e-Books & Papers PostgreSQL-only implementation not found in other SQL variants. Wrapping Up You’re now equipped to better understand the nuances of the data formats you encounter while digging into databases. If you come across monetary values stored as floating-point numbers, you’ll be sure to convert them to decimals before performing any math. And you’ll know how to use the right kind of text column to keep your database from growing too big. Next, I’ll continue with SQL foundations and show you how to import external data into your database. TRY IT YOURSELF Continue exploring data types with these exercises: 1. Your company delivers fruit and vegetables to local grocery stores, and you need to track the mileage driven", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 42 + }, + { + "text": "from growing too big. Next, I’ll continue with SQL foundations and show you how to import external data into your database. TRY IT YOURSELF Continue exploring data types with these exercises: 1. Your company delivers fruit and vegetables to local grocery stores, and you need to track the mileage driven by each driver each day to a tenth of a mile. Assuming no driver would ever travel more than 999 miles in a day, what would be an appropriate data type for the mileage column in your table? Why? 2. In the table listing each driver in your company, what are appropriate data types for the drivers’ first and last names? Why is it a good idea to separate first and last names into two columns rather than having one larger name column? 3. Assume you have a text column that includes strings formatted as dates. One of the strings is written as '4//2017'. What will happen when you try to convert that string to the timestamp data type? Estadísticos e-Books & Papers 4 IMPORTING AND EXPORTING DATA So far, you’ve learned how to add a handful of rows to a table using SQL INSERT statements. A row-by-row insert is useful for making quick test tables or adding a few rows to an existing table. But it’s more likely you’ll need to load hundreds, thousands, or even millions of rows, and no one wants to write separate INSERT statements in those situations. Fortunately, you don’t have to. If your data exists in a delimited text file (with one table row per line of text and each column value separated by a comma or other character) PostgreSQL can import the data in bulk via its COPY command. This command is a PostgreSQL-specific implementation with options for including or excluding columns and handling various delimited text types. In the opposite direction, COPY will also export data from PostgreSQL tables or from the result of a query to a delimited text file. This technique is handy when you want to share data with colleagues or move it into another format, such as an Excel file. I briefly touched on COPY for export in “Characters” on page 24, but in this chapter I’ll discuss import and export in more depth. For importing, I’ll start by introducing you to one of my favorite data sets: the Decennial U.S. Census population tally by county. Three steps form the outline of most of the imports you’ll do: Estadísticos e-Books & Papers 1. Prep the source data in the form of a delimited text file. 2. Create a table to store the data. 3. Write a COPY script to perform the import. After the import is done, we’ll check the data and look at additional options for importing and exporting. A delimited text file is the most common file format that’s portable across proprietary and open source systems, so we’ll focus on that file type. If you want to transfer data from another database program’s proprietary format directly", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 43 + }, + { + "text": "check the data and look at additional options for importing and exporting. A delimited text file is the most common file format that’s portable across proprietary and open source systems, so we’ll focus on that file type. If you want to transfer data from another database program’s proprietary format directly to PostgreSQL, such as Microsoft Access or MySQL, you’ll need to use a third-party tool. Check the PostgreSQL wiki at https://wiki.postgresql.org/wiki/ and search for “Converting from other Databases to PostgreSQL” for a list of tools. If you’re using SQL with another database manager, check the other database’s documentation for how it handles bulk imports. The MySQL database, for example, has a LOAD DATA INFILE statement, and Microsoft’s SQL Server has its own BULK INSERT command. Working with Delimited Text Files Many software applications store data in a unique format, and translating one data format to another is about as easy as a person trying to read the Cyrillic alphabet if they understand only English. Fortunately, most software can import from and export to a delimited text file, which is a common data format that serves as a middle ground. A delimited text file contains rows of data, and each row represents one row in a table. In each row, a character separates, or delimits, each data column. I’ve seen all kinds of characters used as delimiters, from ampersands to pipes, but the comma is most commonly used; hence the name of a file type you’ll see often: comma-separated values (CSV). The terms CSV and comma-delimited are interchangeable. Here’s a typical data row you might see in a comma-delimited file: Estadísticos e-Books & Papers John,Doe,123 Main St.,Hyde Park,NY,845-555-1212 Notice that a comma separates each piece of data—first name, last name, street, town, state, and phone—without any spaces. The commas tell the software to treat each item as a separate column, either upon import or export. Simple enough. Quoting Columns that Contain Delimiters Using commas as a column delimiter leads to a potential dilemma: what if the value in a column includes a comma? For example, sometimes people combine an apartment number with a street address, as in 123 Main St., Apartment 200. Unless the system for delimiting accounts for that extra comma, during import the line will appear to have an extra column and cause the import to fail. To handle such cases, delimited files wrap columns that contain a delimiter character with an arbitrary character called a text qualifier that tells SQL to ignore the delimiter character held within. Most of the time in comma-delimited files the text qualifier used is the double quote. Here’s the example data row again, but with the street name surrounded by double quotes: John,Doe,\"123 Main St., Apartment 200\",Hyde Park,NY,845-555-1212 On import, the database will recognize that double quotes signify one column regardless of whether it finds a delimiter within the quotes. When importing CSV files, PostgreSQL by default ignores delimiters inside double-quoted columns, but you can specify a different text qualifier if your import", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 44 + }, + { + "text": "Main St., Apartment 200\",Hyde Park,NY,845-555-1212 On import, the database will recognize that double quotes signify one column regardless of whether it finds a delimiter within the quotes. When importing CSV files, PostgreSQL by default ignores delimiters inside double-quoted columns, but you can specify a different text qualifier if your import requires it. (And, given the sometimes odd choices made by IT professionals, you may indeed need to employ a different character.) Handling Header Rows Another feature you’ll often find inside a delimited text file is the header Estadísticos e-Books & Papers row. As the name implies, it’s a single row at the top, or head, of the file that lists the name of each data field. Usually, a header is created during the export of data from a database. Here’s an example with the delimited row I’ve been using: FIRSTNAME,LASTNAME,STREET,CITY,STATE,PHONE John,Doe,\"123 Main St., Apartment 200\",Hyde Park,NY,845-555-1212 Header rows serve a few purposes. For one, the values in the header row identify the data in each column, which is particularly useful when you’re deciphering a file’s contents. Second, some database managers (although not PostgreSQL) use the header row to map columns in the delimited file to the correct columns in the import table. Because PostgreSQL doesn’t use the header row, we don’t want that row imported to a table, so we’ll use a HEADER option in the COPY command to exclude it. I’ll cover this with all COPY options in the next section. Using COPY to Import Data To import data from an external file into our database, first we need to check out a source CSV file and build the table in PostgreSQL to hold the data. Thereafter, the SQL statement for the import is relatively simple. All you need are the three lines of code in Listing 4-1: ➊ COPY table_name ➋ FROM 'C:\\YourDirectory\\your_file.csv' ➌ WITH (FORMAT CSV, HEADER); Listing 4-1: Using COPY for data import The block of code starts with the COPY keyword ➊ followed by the name of the target table, which must already exist in your database. Think of this syntax as meaning, “Copy data to my table called table_name.” The FROM keyword ➋ identifies the full path to the source file, including its name. The way you designate the path depends on your operating system. For Windows, begin with the drive letter, colon, backslash, and Estadísticos e-Books & Papers directory names. For example, to import a file located on my Windows desktop, the FROM line would read: FROM 'C:\\Users\\Anthony\\Desktop\\my_file.csv' On macOS or Linux, start at the system root directory with a forward slash and proceed from there. Here’s what the FROM line might look like when importing a file located on my Mac desktop: FROM '/Users/anthony/Desktop/my_file.csv' Note that in both cases the full path and filename are surrounded by single quotes. For the examples in the book, I use the Windows-style path C:\\YourDirectory\\ as a placeholder. Replace that with the path where you stored the file. The WITH keyword ➌ lets you specify options,", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 45 + }, + { + "text": "FROM '/Users/anthony/Desktop/my_file.csv' Note that in both cases the full path and filename are surrounded by single quotes. For the examples in the book, I use the Windows-style path C:\\YourDirectory\\ as a placeholder. Replace that with the path where you stored the file. The WITH keyword ➌ lets you specify options, surrounded by paren​- theses, that you can tailor to your input or output file. Here we specify that the external file should be comma-delimited, and that we should exclude the file’s header row in the import. It’s worth examining all the options in the official PostgreSQL documentation at https://www.postgresql.org/docs/current/static/sql-copy.html, but here is a list of the options you’ll commonly use: Input and output file format Use the FORMAT format_name option to specify the type of file you’re reading or writing. Format names are CSV, TEXT, or BINARY. Unless you’re deep into building technical systems, you’ll rarely encounter a need to work with BINARY, where data is stored as a sequence of bytes. More often, you’ll work with standard CSV files. In the TEXT format, a tab character is the delimiter by default (although you can specify another character) and backslash characters such as \\r are recognized as their ASCII equivalents—in this case, a carriage return. The TEXT format is used mainly by PostgreSQL’s built-in backup programs. Presence of a header row On import, use HEADER to specify that the source file has a header row. Estadísticos e-Books & Papers You can also specify it longhand as HEADER ON, which tells the database to start importing with the second line of the file, preventing the unwanted import of the header. You don’t want the column names in the header to become part of the data in the table. On export, using HEADER tells the database to include the column names as a header row in the output file, which is usually helpful to do. Delimiter The DELIMITER 'character' option lets you specify which character your import or export file uses as a delimiter. The delimiter must be a single character and cannot be a carriage return. If you use FORMAT CSV, the assumed delimiter is a comma. I include DELIMITER here to show that you have the option to specify a different delimiter if that’s how your data arrived. For example, if you received pipe-delimited data, you would treat the option this way: DELIMITER '|'. Quote character Earlier, you learned that in a CSV, commas inside a single column value will mess up your import unless the column value is surrounded by a character that serves as a text qualifier, telling the database to handle the value within as one column. By default, PostgreSQL uses the double quote, but if the CSV you’re importing uses a different character, you can specify it with the QUOTE 'quote_character' option. Now that you better understand delimited files, you’re ready to import one. Importing Census Data Describing Counties The data set you’ll work with in this import exercise is considerably larger than the teachers table", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 46 + }, + { + "text": "CSV you’re importing uses a different character, you can specify it with the QUOTE 'quote_character' option. Now that you better understand delimited files, you’re ready to import one. Importing Census Data Describing Counties The data set you’ll work with in this import exercise is considerably larger than the teachers table you made in Chapter 1. It contains census data about every county in the United States and is 3,143 rows deep and 91 columns wide. To understand the data, it helps to know a little about the U.S. Estadísticos e-Books & Papers Census. Every 10 years, the government conducts a full count of the population—one of several ongoing programs by the Census Bureau to collect demographic data. Each household in America receives a questionnaire about each person in it—their age, gender, race, and whether they are Hispanic or not. The U.S. Constitution mandates the count to determine how many members from each state make up the U.S. House of Representatives. Based on the 2010 Census, for example, Texas gained four seats in the House while New York and Ohio lost two seats each. Although apportioning House seats is the count’s main purpose, the data’s also a boon for trend trackers studying the population. A good synopsis of the 2010 count’s findings is available at https://www.census.gov/prod/cen2010/briefs/c2010br-01.pdf. The Census Bureau reports overall population totals and counts by race and ethnicity for various geographies including states, counties, cities, places, and school districts. For this exercise, I compiled a select collection of columns for the 2010 Census county-level counts into a file named us_counties_2010.csv. Download the us_counties_2010.csv file from https://www.nostarch.com/practicalSQL/ and save it to a folder on your computer. Open the file with a plain text editor. You should see a header row that begins with these columns: NAME,STUSAB,SUMLEV,REGION,DIVISION,STATE,COUNTY --snip-- Let’s explore some of the columns by examining the code for creating the import table. Creating the us_counties_2010 Table The code in Listing 4-2 shows only an abbreviated version of the CREATE TABLE script; many of the columns have been omitted. The full version is available (and annotated) along with all the code examples in the book’s resources. To import it properly, you’ll need to download the full table definition. Estadísticos e-Books & Papers CREATE TABLE us_counties_2010 ( ➊ geo_name varchar(90), ➋ state_us_abbreviation varchar(2), ➌ summary_level varchar(3), ➍ region smallint, division smallint, state_fips varchar(2), county_fips varchar(3), ➎ area_land bigint, area_water bigint, ➏ population_count_100_percent integer, housing_unit_count_100_percent integer, ➐ internal_point_lat numeric(10,7), internal_point_lon numeric(10,7), ➑ p0010001 integer, p0010002 integer, p0010003 integer, p0010004 integer, p0010005 integer, --snip-- p0040049 integer, p0040065 integer, p0040072 integer, h0010001 integer, h0010002 integer, h0010003 integer ); Listing 4-2: A CREATE TABLE statement for census county data To create the table, in pgAdmin click the analysis database that you created in Chapter 1. (It’s best to store the data in this book in analysis because we’ll reuse some of it in later chapters.) From the pgAdmin menu bar, select Tools ▸ Query Tool. Paste the script into the window and run it. Return to the", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 47 + }, + { + "text": "analysis database that you created in Chapter 1. (It’s best to store the data in this book in analysis because we’ll reuse some of it in later chapters.) From the pgAdmin menu bar, select Tools ▸ Query Tool. Paste the script into the window and run it. Return to the main pgAdmin window, and in the object browser, right-click and refresh the analysis database. Choose Schemas ▸ public ▸ Tables to see the new table. Although it’s empty, you can see the structure by running a basic SELECT query in pgAdmin’s Query Tool: SELECT * from us_counties_2010; When you run the SELECT query, you’ll see the columns in the table you created. No data rows exist yet. Estadísticos e-Books & Papers Census Columns and Data Types Before we import the CSV file into the table, let’s walk through several of the columns and the data types I chose in Listing 4-2. As my guide, I used the official census data dictionary for this data set found at http://www.census.gov/prod/cen2010/doc/pl94-171.pdf, although I give some columns more readable names in the table definition. Relying on a data dictionary when possible is good practice, because it helps you avoid misconfiguring columns or potentially losing data. Always ask if one is available, or do an online search if the data is public. In this set of census data, and thus the table you just made, each row describes the demographics of one county, starting with its geo_name ➊ and its two-character state abbreviation, the state_us_abbreviation ➋. Because both are text, we store them as varchar. The data dictionary indicates that the maximum length of the geo_name field is 90 characters, but because most names are shorter, using varchar will conserve space if we fill the field with a shorter name, such as Lee County, while allowing us to specify the maximum 90 characters. The geography, or summary level, represented by each row is described by summary_level ➌. We’re working only with county-level data, so the code is the same for each row: 050. Even though that code resembles a number, we’re treating it as text by again using varchar. If we used an integer type, that leading 0 would be stripped on import, leaving 50. We don’t want to do that because 050 is the complete summary level code, and we’d be altering the meaning of the data if the leading 0 were lost. Also, we won’t be doing any math with this value. Numbers from 0 to 9 in region and division ➍ represent the location of a county in the United States, such as the Northeast, Midwest, or South Atlantic. No number is higher than 9, so we define the columns with type smallint. We again use varchar for state_fips and county_fips, which are the standard federal codes for those entities, because those codes contain leading zeros that should not be stripped. It’s always important to distinguish codes from numbers; these state and county values are actually labels as opposed to numbers used", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 48 + }, + { + "text": "smallint. We again use varchar for state_fips and county_fips, which are the standard federal codes for those entities, because those codes contain leading zeros that should not be stripped. It’s always important to distinguish codes from numbers; these state and county values are actually labels as opposed to numbers used for math. Estadísticos e-Books & Papers The number of square meters for land and water in the county are recorded in area_land and area_water ➎, respectively. In certain places—such as Alaska, where there’s lots of land to go with all that snow—some values easily surpass the integer type’s maximum of 2,147,483,648. For that reason, we’re using bigint, which will handle the 376,855,656,455 square meters in the Yukon-Koyukuk Census Area with room to spare. Next, population_count_100_percent and housing_unit_count_100_percent ➏ are the total counts of population and housing units in the geography. In 2010, the United States had 308.7 million people and 131.7 million housing units. The population and housing units for any county fits well within the integer data type’s limits, so we use that for both. The latitude and longitude of a point near the center of the county, called an internal point, are specified in internal_point_lat and internal_point_lon ➐, respectively. The Census Bureau—along with many mapping systems—expresses latitude and longitude coordinates using a decimal degrees system. Latitude represents positions north and south on the globe, with the equator at 0 degrees, the North Pole at 90 degrees, and the South Pole at −90 degrees. Longitude represents locations east and west, with the Prime Meridian that passes through Greenwich in London at 0 degrees longitude. From there, longitude increases both east and west (positive numbers to the east and negative to the west) until they meet at 180 degrees on the opposite side of the globe. The location there, known as the antimeridian, is used as the basis for the International Date Line. When reporting interior points, the Census Bureau uses up to seven decimal places. With a value up to 180 to the left of the decimal, we need to account for a maximum of 10 digits total. So, we’re using numeric with a precision of 10 and a scale of 7. NOTE PostgreSQL, through the PostGIS extension, can store geometric data, which includes points that represent latitude and longitude in a single Estadísticos e-Books & Papers column. We’ll explore geometric data when we cover geographical queries in Chapter 14. Finally, we reach a series of columns ➑ that contain iterations of the population counts by race and ethnicity for the county as well as housing unit counts. The full set of 2010 Census data contains 291 of these columns. I’ve pared that down to 78 for this exercise, omitting many of the columns to make the data set more compact for these exercises. I won’t discuss all the columns now, but Table 4-1 shows a small sample. Table 4-1: Census Population-Count Columns Column name Description p0010001 Total population p0010002 Population of one race p0010003 Population of one race:", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 49 + }, + { + "text": "omitting many of the columns to make the data set more compact for these exercises. I won’t discuss all the columns now, but Table 4-1 shows a small sample. Table 4-1: Census Population-Count Columns Column name Description p0010001 Total population p0010002 Population of one race p0010003 Population of one race: White alone p0010004 Population of one race: Black or African American alone p0010005 Population of one race: American Indian and Alaska Native alone p0010006 Population of one race: Asian alone p0010007 Population of one race: Native Hawaiian and Other Pacific Islander alone p0010008 Population of one race: Some Other Race alone You’ll explore this data more in the next chapter when we look at math with SQL. For now, let’s run the import. Performing the Census Import with COPY Estadísticos e-Books & Papers Now you’re ready to bring the census data into the table. Run the code in Listing 4-3, remembering to change the path to the file to match the location of the data on your computer: COPY us_counties_2010 FROM 'C:\\YourDirectory\\us_counties_2010.csv' WITH (FORMAT CSV, HEADER); Listing 4-3: Importing census data using COPY When the code executes, you should see the following message in pgAdmin: Query returned successfully: 3143 rows affected That’s good news: the import CSV has the same number of rows. If you have an issue with the source CSV or your import statement, the database will throw an error. For example, if one of the rows in the CSV had more columns than in the target table, you’d see an error message that provides a hint as to how to fix it: ERROR: extra data after last expected column SQL state: 22P04 Context: COPY us_counties_2010, line 2: \"Autauga County,AL,050,3,6,01,001 ...\" Even if no errors are reported, it’s always a good idea to visually scan the data you just imported to ensure everything looks as expected. Start with a SELECT query of all columns and rows: SELECT * FROM us_counties_2010; There should be 3,143 rows displayed in pgAdmin, and as you scroll left and right through the result set, each field should have the expected values. Let’s review some columns that we took particular care to define with the appropriate data types. For example, run the following query to show the counties with the largest area_land values. We’ll use a LIMIT clause, which will cause the query to only return the number of rows we want; here, we’ll ask for three: SELECT geo_name, state_us_abbreviation, area_land Estadísticos e-Books & Papers FROM us_counties_2010 ORDER BY area_land DESC LIMIT 3; This query ranks county-level geographies from largest land area to smallest in square meters. We defined area_land as bigint because the largest values in the field are bigger than the upper range provided by regular integer. As you might expect, big Alaskan geographies are at the top: geo_name state_us_abbreviation area_land ------------------------- --------------------- ------------ Yukon-Koyukuk Census Area AK 376855656455 North Slope Borough AK 229720054439 Bethel Census Area AK 105075822708 Next, check the latitude and longitude columns of internal_point_lat and internal_point_lon, which we", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 50 + }, + { + "text": "range provided by regular integer. As you might expect, big Alaskan geographies are at the top: geo_name state_us_abbreviation area_land ------------------------- --------------------- ------------ Yukon-Koyukuk Census Area AK 376855656455 North Slope Borough AK 229720054439 Bethel Census Area AK 105075822708 Next, check the latitude and longitude columns of internal_point_lat and internal_point_lon, which we defined with numeric(10,7). This code sorts the counties by longitude from the greatest to smallest value. This time, we’ll use LIMIT to retrieve five rows: SELECT geo_name, state_us_abbreviation, internal_point_lon FROM us_counties_2010 ORDER BY internal_point_lon DESC LIMIT 5; Longitude measures locations from east to west, with locations west of the Prime Meridian in England represented as negative numbers starting with −1, −2, −3, and so on the farther west you go. We sorted in descending order, so we’d expect the easternmost counties of the United States to show at the top of the query result. Instead—surprise!—there’s a lone Alaska geography at the top: Estadísticos e-Books & Papers Here’s why: the Alaskan Aleutian Islands extend so far west (farther west than Hawaii) that they cross the antimeridian at 180 degrees longitude by less than 2 degrees. Once past the antimeridian, longitude turns positive, counting back down to 0. Fortunately, it’s not a mistake in the data; however, it’s a fact you can tuck away for your next trivia team competition. Congratulations! You have a legitimate set of government demographic data in your database. I’ll use it to demonstrate exporting data with COPY later in this chapter, and then you’ll use it to learn math functions in Chapter 5. Before we move on to exporting data, let’s examine a few additional importing techniques. Importing a Subset of Columns with COPY If a CSV file doesn’t have data for all the columns in your target database table, you can still import the data you have by specifying which columns are present in the data. Consider this scenario: you’re researching the salaries of all town supervisors in your state so you can analyze government spending trends by geography. To get started, you create a table called supervisor_salaries with the code in Listing 4-4: CREATE TABLE supervisor_salaries ( town varchar(30), county varchar(30), supervisor varchar(30), start_date date, salary money, benefits money ); Listing 4-4: Creating a table to track supervisor salaries You want columns for the town and county, the supervisor’s name, the date he or she started, and salary and benefits (assuming you just care about current levels). However, the first county clerk you contact says, “Sorry, we only have town, supervisor, and salary. You’ll need to get the rest from elsewhere.” You tell them to send a CSV anyway. You’ll import what you can. Estadísticos e-Books & Papers I’ve included such a sample CSV you can download in the book’s resources at https://www.nostarch.com/practicalSQL/, called supervisor_salaries.csv. You could try to import it using this basic COPY syntax: COPY supervisor_salaries FROM 'C:\\YourDirectory\\supervisor_salaries.csv' WITH (FORMAT CSV, HEADER); But if you do, PostgreSQL will return an error: ********** Error ********** ERROR: missing data for column \"start_date\" SQL state: 22P04", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 51 + }, + { + "text": "can download in the book’s resources at https://www.nostarch.com/practicalSQL/, called supervisor_salaries.csv. You could try to import it using this basic COPY syntax: COPY supervisor_salaries FROM 'C:\\YourDirectory\\supervisor_salaries.csv' WITH (FORMAT CSV, HEADER); But if you do, PostgreSQL will return an error: ********** Error ********** ERROR: missing data for column \"start_date\" SQL state: 22P04 Context: COPY supervisor_salaries, line 2: \"Anytown,Jones,27000\" The database complains that when it got to the fourth column of the table, start_date, it couldn’t find any data in the CSV. The workaround for this situation is to tell the database which columns in the table are present in the CSV, as shown in Listing 4-5: COPY supervisor_salaries ➊(town, supervisor, salary) FROM 'C:\\YourDirectory\\supervisor_salaries.csv' WITH (FORMAT CSV, HEADER); Listing 4-5: Importing salaries data from CSV to three table columns By noting in parentheses ➊ the three present columns after the table name, we tell PostgreSQL to only look for data to fill those columns when it reads the CSV. Now, if you select the first couple of rows from the table, you’ll see only those columns filled: Adding a Default Value to a Column During Import Estadísticos e-Books & Papers What if you want to populate the county column during the import, even though the value is missing from the CSV file? You can do so by using a temporary table. Temporary tables exist only until you end your database session. When you reopen the database (or lose your connection), those tables disappear. They’re handy for performing intermediary operations on data as part of your processing pipeline; we’ll use one to add a county name to the supervisor_salaries table as we import the CSV. Start by clearing the data you already imported into supervisor_salaries using a DELETE query: DELETE FROM supervisor_salaries; When that query finishes, run the code in Listing 4-6: ➊ CREATE TEMPORARY TABLE supervisor_salaries_temp (LIKE supervisor_salaries); ➋ COPY supervisor_salaries_temp (town, supervisor, salary) FROM 'C:\\YourDirectory\\supervisor_salaries.csv' WITH (FORMAT CSV, HEADER); ➌ INSERT INTO supervisor_salaries (town, county, supervisor, salary) SELECT town, 'Some County', supervisor, salary FROM supervisor_salaries_temp; ➍ DROP TABLE supervisor_salaries_temp; Listing 4-6: Using a temporary table to add a default value to a column during import This script performs four tasks. First, we create a temporary table called supervisor_salaries_temp ➊ based on the original supervisor_salaries table by passing as an argument the LIKE keyword (covered in “Using LIKE and ILIKE with WHERE” on page 19) followed by the parent table to copy. Then we import the supervisor_salaries.csv file ➋ into the temporary table using the now-familiar COPY syntax. Next, we use an INSERT statement to fill the salaries table ➌. Instead of specifying values, we employ a SELECT statement to query the temporary table. That query specifies the value for the second column, not as a column name, but as a string inside single quotes. Finally, we use DROP TABLE to erase the temporary table ➍. The Estadísticos e-Books & Papers temporary table will automatically disappear when you disconnect from the PostgreSQL session, but this removes it now in case we want to", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 52 + }, + { + "text": "not as a column name, but as a string inside single quotes. Finally, we use DROP TABLE to erase the temporary table ➍. The Estadísticos e-Books & Papers temporary table will automatically disappear when you disconnect from the PostgreSQL session, but this removes it now in case we want to run the query again against another CSV. After you run the query, run a SELECT statement on the first couple of rows to see the effect: Now you’ve filled the county field with a value. The path to this import might seem laborious, but it’s instructive to see how data processing can require multiple steps to get the desired results. The good news is that this temporary table demo is an apt indicator of the flexibility SQL offers to control data handling. Using COPY to Export Data The main difference between exporting and importing data with COPY is that rather than using FROM to identify the source data, you use TO for the path and name of the output file. You control how much data to export— an entire table, just a few columns, or to fine-tune it even more, the results of a query. Let’s look at three quick examples. Exporting All Data The simplest export sends everything in a table to a file. Earlier, you created the table us_counties_2010 with 91 columns and 3,143 rows of census data. The SQL statement in Listing 4-7 exports all the data to a text file named us_counties_export.txt. The WITH keyword option tells PostgreSQL to include a header row and use the pipe symbol instead of a comma for a delimiter. I’ve used the .txt file extension here for two Estadísticos e-Books & Papers reasons. First, it demonstrates that you can export to any text file format; second, we’re using a pipe for a delimiter, not a comma. I like to avoid calling files .csv unless they truly have commas as a separator. Remember to change the output directory to your preferred location. COPY us_counties_2010 TO 'C:\\YourDirectory\\us_counties_export.txt' WITH (FORMAT CSV, HEADER, DELIMITER '|'); Listing 4-7: Exporting an entire table with COPY Exporting Particular Columns You don’t always need (or want) to export all your data: you might have sensitive information, such as Social Security numbers or birthdates, that need to remain private. Or, in the case of the census county data, maybe you’re working with a mapping program and only need the county name and its geographic coordinates to plot the locations. We can export only these three columns by listing them in parentheses after the table name, as shown in Listing 4-8. Of course, you must enter these column names precisely as they’re listed in the data for PostgreSQL to recognize them. COPY us_counties_2010 (geo_name, internal_point_lat, internal_point_lon) TO 'C:\\YourDirectory\\us_counties_latlon_export.txt' WITH (FORMAT CSV, HEADER, DELIMITER '|'); Listing 4-8: Exporting selected columns from a table with COPY Exporting Query Results Additionally, you can add a query to COPY to fine-tune your output. In Listing 4-9 we export the name and state abbreviation of", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 53 + }, + { + "text": "them. COPY us_counties_2010 (geo_name, internal_point_lat, internal_point_lon) TO 'C:\\YourDirectory\\us_counties_latlon_export.txt' WITH (FORMAT CSV, HEADER, DELIMITER '|'); Listing 4-8: Exporting selected columns from a table with COPY Exporting Query Results Additionally, you can add a query to COPY to fine-tune your output. In Listing 4-9 we export the name and state abbreviation of only those counties whose name contains the letters mill in either uppercase or lowercase by using the case-insensitive ILIKE and the % wildcard character we covered in “Using LIKE and ILIKE with WHERE” on page 19. COPY ( SELECT geo_name, state_us_abbreviation FROM us_counties_2010 WHERE geo_name ILIKE '%mill%' Estadísticos e-Books & Papers ) TO 'C:\\YourDirectory\\us_counties_mill_export.txt' WITH (FORMAT CSV, HEADER, DELIMITER '|'); Listing 4-9: Exporting query results with COPY After running the code, your output file should have nine rows with county names including Miller, Roger Mills, and Vermillion. Importing and Exporting Through pgAdmin At times, the SQL COPY commands won’t be able to handle certain imports and exports, typically when you’re connected to a PostgreSQL instance running on a computer other than yours, perhaps elsewhere on a network. When that happens, you might not have access to that computer’s filesystem, which makes setting the path in the FROM or TO clause difficult. One workaround is to use pgAdmin’s built-in import/export wizard. In pgAdmin’s object browser (the left vertical pane), locate the list of tables in your analysis database by choosing Databases ▸ analysis ▸ Schemas ▸ public ▸ Tables. Next, right-click on the table you want to import to or export from, and select Import/Export. A dialog appears that lets you choose either to import or export from that table, as shown in Figure 4-1. Estadísticos e-Books & Papers Figure 4-1: The pgAdmin Import/Export dialog To import, move the Import/Export slider to Import. Then click the three dots to the right of the Filename box to locate your CSV file. From the Format drop-down list, choose csv. Then adjust the header, delimiter, quoting, and other options as needed. Click OK to import the data. To export, use the same dialog and follow similar steps. Wrapping Up Now that you’ve learned how to bring external data into your database, you can start digging into a myriad of data sets, whether you want to explore one of the thousands of publicly available data sets, or data related to your own career or studies. Plenty of data is available in CSV format or a format easily convertible to CSV. Look for data dictionaries to help you understand the data and choose the right data type for each field. The census data you imported as part of this chapter’s exercises will Estadísticos e-Books & Papers play a starring role in the next chapter in which we explore math functions with SQL. TRY IT YOURSELF Continue your exploration of data import and export with these exercises. Remember to consult the PostgreSQL documentation at https://www.postgresql.org/docs/current/static/sql-copy.html for hints: 1. Write a WITH statement to include with COPY to handle the import of an imaginary text file whose first", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 54 + }, + { + "text": "we explore math functions with SQL. TRY IT YOURSELF Continue your exploration of data import and export with these exercises. Remember to consult the PostgreSQL documentation at https://www.postgresql.org/docs/current/static/sql-copy.html for hints: 1. Write a WITH statement to include with COPY to handle the import of an imaginary text file whose first couple of rows look like this: id:movie:actor 50:#Mission: Impossible#:Tom Cruise 2. Using the table us_counties_2010 you created and filled in this chapter, export to a CSV file the 20 counties in the United States that have the most housing units. Make sure you export only each county’s name, state, and number of housing units. (Hint: Housing units are totaled for each county in the column housing_unit_count_100_percent.) 3. Imagine you’re importing a file that contains a column with these values: 17519.668 20084.461 18976.335 Will a column in your target table with data type numeric(3,8) work for these values? Why or why not? Estadísticos e-Books & Papers 5 BASIC MATH AND STATS WITH SQL If your data includes any of the number data types we explored in Chapter 3—integers, decimals, or floating points—sooner or later your analysis will include some calculations. For example, you might want to know the average of all the dollar values in a column, or add values in two columns to produce a total for each row. SQL handles calculations ranging from basic math through advanced statistics. In this chapter, I’ll start with the basics and progress to math functions and beginning statistics. I’ll also discuss calculations related to percentages and percent change. For several of the exercises, we’ll use the 2010 Decennial Census data you imported in Chapter 4. Math Operators Let’s start with the basic math you learned in grade school (and all’s forgiven if you’ve forgotten some of it). Table 5-1 shows nine math operators you’ll use most often in your calculations. The first four (addition, subtraction, multiplication, and division) are part of the ANSI SQL standard that are implemented in all database systems. The others are PostgreSQL-specific operators, although if you’re using another database, it likely has functions or operators to perform those operations. Estadísticos e-Books & Papers For example, the modulo operator (%) works in Microsoft SQL Server and MySQL as well as with PostgreSQL. If you’re using another database system, check its documentation. Table 5-1: Basic Math Operators OperatorDescription + Addition - Subtraction * Multiplication / Division (returns the quotient only, no remainder) % Modulo (returns just the remainder) ^ Exponentiation |/ Square root ||/ Cube root ! Factorial We’ll step through each of these operators by executing simple SQL queries on plain numbers rather than operating on a table or another database object. You can either enter the statements separately into the pgAdmin query tool and execute them one at a time, or if you copied the code for this chapter from the resources at https://www.nostarch.com/practicalSQL/, you can highlight each line before executing it. Math and Data Types As you work through the examples, note the data type of each result,", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 55 + }, + { + "text": "the pgAdmin query tool and execute them one at a time, or if you copied the code for this chapter from the resources at https://www.nostarch.com/practicalSQL/, you can highlight each line before executing it. Math and Data Types As you work through the examples, note the data type of each result, which is listed beneath each column name in the pgAdmin results grid. The type returned for a calculation will vary depending on the operation and the data type of the input numbers. Estadísticos e-Books & Papers In calculations with an operator between two numbers—addition, subtraction, multiplication, and division—the data type returned follows this pattern: Two integers return an integer. A numeric on either side of the operator returns a numeric. Anything with a floating-point number returns a floating-point number of type double precision. However, the exponentiation, root, and factorial functions are different. Each takes one number either before or after the operator and returns numeric and floating-point types, even when the input is an integer. Sometimes the result’s data type will suit your needs; other times, you may need to use CAST to change the data type, as mentioned in “Transforming Values from One Type to Another with CAST” on page 35, such as if you need to feed the result into a function that takes a certain type. I’ll note those times as we work through the book. Adding, Subtracting, and Multiplying Let’s start with simple integer addition, subtraction, and multiplication. Listing 5-1 shows three examples, each with the SELECT keyword followed by the math formula. Since Chapter 2, we’ve used SELECT for its main purpose: to retrieve data from a table. But with PostgreSQL, Microsoft’s SQL Server, MySQL, and some other database management systems, it’s possible to omit the table name for math and string operations while testing, as we do here. For readability’s sake, I recommend you use a single space before and after the math operator; although using spaces isn’t strictly necessary for your code to work, it is good practice. ➊ SELECT 2 + 2; ➋ SELECT 9 - 1; ➌ SELECT 3 * 4; Estadísticos e-Books & Papers Listing 5-1: Basic addition, subtraction, and multiplication with SQL None of these statements are rocket science, so you shouldn’t be surprised that running SELECT 2 + 2; ➊ in the query tool shows a result of 4. Similarly, the examples for subtraction ➋ and multiplication ➌ yield what you’d expect: 8 and 12. The output displays in a column, as with any query result. But because we’re not querying a table and specifying a column, the results appear beneath a ?column? name, signifying an unknown column: ?column? -------- 4 That’s okay. We’re not affecting any data in a table, just displaying a result. Division and Modulo Division with SQL gets a little trickier because of the difference between math with integers and math with decimals, which was mentioned earlier. Add in modulo, an operator that returns just the remainder in a division operation, and the results", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 56 + }, + { + "text": "in a table, just displaying a result. Division and Modulo Division with SQL gets a little trickier because of the difference between math with integers and math with decimals, which was mentioned earlier. Add in modulo, an operator that returns just the remainder in a division operation, and the results can be confusing. So, to make it clear, Listing 5-2 shows four examples: ➊ SELECT 11 / 6; ➋ SELECT 11 % 6; ➌ SELECT 11.0 / 6; ➍ SELECT CAST (11 AS numeric(3,1)) / 6; Listing 5-2: Integer and decimal division with SQL The first statement uses the / operator ➊ to divide the integer 11 by another integer, 6. If you do that math in your head, you know the answer is 1 with a remainder of 5. However, running this query yields 1, which is how SQL handles division of one integer by another—by reporting only the integer quotient. If you want to retrieve the remainder as an integer, you must perform the same calculation using the modulo operator %, as in Estadísticos e-Books & Papers ➋. That statement returns just the remainder, in this case 5. No single operation will provide you with both the quotient and the remainder as integers. Modulo is useful for more than just fetching a remainder: you can also use it as a test condition. For example, to check whether a number is even, you can test it using the % 2 operation. If the result is 0 with no remainder, the number is even. If you want to divide two numbers and have the result return as a numeric type, you can do so in two ways: first, if one or both of the numbers is a numeric, the result will by default be expressed as a numeric. That’s what happens when I divide 11.0 by 6 ➌. Execute that query, and the result is 1.83333. The number of decimal digits displayed may vary according to your PostgreSQL and system settings. Second, if you’re working with data stored only as integers and need to force decimal division, you can CAST one of the integers to a numeric type ➍. Executing this again returns 1.83333. Exponents, Roots, and Factorials Beyond the basics, PostgreSQL-flavored SQL also provides operators to square, cube, or otherwise raise a base number to an exponent, as well as find roots or the factorial of a number. Listing 5-3 shows these operations in action: ➊ SELECT 3 ^ 4; ➋ SELECT |/ 10; SELECT sqrt(10); ➌ SELECT ||/ 10; ➍ SELECT 4 !; Listing 5-3: Exponents, roots, and factorials with SQL The exponentiation operator (^) allows you to raise a given base number to an exponent, as in ➊, where 3 ^ 4 (colloquially, we’d call that three to the fourth power) returns 81. You can find the square root of a number in two ways: using the |/ Estadísticos e-Books & Papers operator ➋ or the sqrt(n) function. For a cube root, use the ||/ operator ➌.", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 57 + }, + { + "text": "➊, where 3 ^ 4 (colloquially, we’d call that three to the fourth power) returns 81. You can find the square root of a number in two ways: using the |/ Estadísticos e-Books & Papers operator ➋ or the sqrt(n) function. For a cube root, use the ||/ operator ➌. Both are prefix operators, named because they come before a single value. To find the factorial of a number, use the ! operator. It’s a suffix operator, coming after a single value. You’ll use factorials in many places in math, but perhaps the most common is to determine how many ways a number of items can be ordered. Say you have four photographs. How many ways could you order them next to each other on a wall? To find the answer, you’d calculate the factorial by starting with the number of items and multi​plying all the smaller positive integers. So, at ➍, the factorial statement of 4 ! is equivalent to 4 × 3 × 2 × 1. That’s 24 ways to order four photos. No wonder decorating takes so long sometimes! Again, these operators are specific to PostgreSQL; they’re not part of the SQL standard. If you’re using another database application, check its documentation for how it implements these operations. Minding the Order of Operations Can you recall from your earliest math lessons what the order of operations, or operator precedence, is on a mathematical expression? When you string together several numbers and operators, which calculations does SQL execute first? Not surprisingly, SQL follows the established math standard. For the PostgreSQL operators discussed so far, the order is: 1. Exponents and roots 2. Multiplication, division, modulo 3. Addition and subtraction Given these rules, you’ll need to encase an operation in parentheses if you want to calculate it in a different order. For example, the following two expressions yield different results: SELECT 7 + 8 * 9; SELECT (7 + 8) * 9; Estadísticos e-Books & Papers The first expression returns 79 because the multiplication operation receives precedence and is processed before the addition. The second returns 135 because the parentheses force the addition operation to occur first. Here’s a second example using exponents: SELECT 3 ^ 3 - 1; SELECT 3 ^ (3 - 1); Exponent operations take precedence over subtraction, so without parentheses the entire expression is evaluated left to right and the operation to find 3 to the power of 3 happens first. Then 1 is subtracted, returning 26. In the second example, the parentheses force the subtraction to happen first, so the operation results in 9, which is 3 to the power of 2. Keep operator precedence in mind to avoid having to correct your analysis later! Doing Math Across Census Table Columns Let’s try to use the most frequently used SQL math operators on real data by digging into the 2010 Decennial Census population table, us_counties_2010, that you imported in Chapter 4. Instead of using numbers in queries, we’ll use the names of the columns", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 58 + }, + { + "text": "later! Doing Math Across Census Table Columns Let’s try to use the most frequently used SQL math operators on real data by digging into the 2010 Decennial Census population table, us_counties_2010, that you imported in Chapter 4. Instead of using numbers in queries, we’ll use the names of the columns that contain the numbers. When we execute the query, the calculation will occur on each row of the table. To refresh your memory about the data, run the script in Listing 5-4. It should return 3,143 rows showing the name and state of each county in the United States, and the number of people who identified with one of six race categories or a combination of two or more races. The 2010 Census form received by each household—the so-called “short form”—allowed people to check either just one or multiple boxes under the question of race. (You can review the form at https://www.census.gov/2010census/pdf/2010_Questionnaire_Info.pdf.) People who checked one box were counted in categories such as “White Alone” Estadísticos e-Books & Papers or “Black or African American Alone.” Respondents who selected more than one box were tabulated in the overall category of “Two or More Races,” and the census data set breaks those down in detail. SELECT geo_name, state_us_abbreviation AS \"st\", p0010001 AS➊ \"Total Population\", p0010003 AS \"White Alone\", p0010004 AS \"Black or African American Alone\", p0010005 AS \"Am Indian/Alaska Native Alone\", p0010006 AS \"Asian Alone\", p0010007 AS \"Native Hawaiian and Other Pacific Islander Alone\", p0010008 AS \"Some Other Race Alone\", p0010009 AS \"Two or More Races\" FROM us_counties_2010; Listing 5-4: Selecting census population columns by race with aliases In us_counties_2010, each race and household data column contains a census code. For example, the “Asian Alone” column is reported as p0010006. Although those codes might be economical and compact, they make it difficult to understand which column is which when the query returns with just that code. In Listing 5-4, I employ a little trick to clarify the output by using the AS keyword ➊ to give each column a more readable alias in the result set. We could rename all the columns upon import, but with the census it’s best to use the code to refer to the same column names in the documentation if needed. Adding and Subtracting Columns Now, let’s try a simple calculation on two of the race columns in Listing 5-5, adding the number of people who identified as white alone or black alone in each county. SELECT geo_name, state_us_abbreviation AS \"st\", p0010003 AS \"White Alone\", p0010004 AS \"Black Alone\", ➊ p0010003 + p0010004 AS \"Total White and Black\" FROM us_counties_2010; Listing 5-5: Adding two columns in us_counties_2010 Estadísticos e-Books & Papers Providing p0010003 + p0010004 ➊ as one of the columns in the SELECT statement handles the calculation. Again, I use the AS keyword to provide a readable alias for the column. If you don’t provide an alias, PostgreSQL uses the label ?column?, which is far less than helpful. Run the query to see the results. The", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 59 + }, + { + "text": "one of the columns in the SELECT statement handles the calculation. Again, I use the AS keyword to provide a readable alias for the column. If you don’t provide an alias, PostgreSQL uses the label ?column?, which is far less than helpful. Run the query to see the results. The first few rows should resemble this output: A quick check with a calculator or pencil and paper confirms that the total column equals the sum of the columns you added. Excellent! Now, let’s build on this to test our data and validate that we imported columns correctly. The six race “Alone” columns plus the “Two or More Races” column should add up to the same number as the total population. The code in Listing 5-6 should show that it does: SELECT geo_name, state_us_abbreviation AS \"st\", ➊ p0010001 AS \"Total\", ➋ p0010003 + p0010004 + p0010005 + p0010006 + p0010007 + p0010008 + p0010009 AS \"All Races\", ➌ (p0010003 + p0010004 + p0010005 + p0010006 + p0010007 + p0010008 + p0010009) - p0010001 AS \"Difference\" FROM us_counties_2010 ➍ ORDER BY \"Difference\" DESC; Listing 5-6: Checking census data totals This query includes the population total ➊, followed by a calculation adding the seven race columns as All Races ➋. The population total and the races total should be identical, but rather than manually check, we also add a column that subtracts the population total column from the sum of the race columns ➌. That column, named Difference, should contain a zero in each row if all the data is in the right place. To avoid Estadísticos e-Books & Papers having to scan all 3,143 rows, we add an ORDER BY clause ➍ on the named column. Any rows showing a difference should appear at the top or bottom of the query result. Run the query; the first few rows should provide this result: geo_name st Total All Races Difference -------------- -- ------ --------- ---------- Autauga County AL 54571 54571 0 Baldwin County AL 182265 182265 0 Barbour County AL 27457 27457 0 With the Difference column showing zeros, we can be confident that our import was clean. Whenever I encounter or import a new data set, I like to perform little tests like this. They help me better understand the data and head off any potential issues before I dig into analysis. Finding Percentages of the Whole Let’s dig deeper into the census data to find meaningful differences in the population demographics of the counties. One way to do this (with any data set, in fact) is to calculate what percentage of the whole a particular variable represents. With the census data, we can learn a lot by comparing percentages from county to county and also by examining how percentages vary over time. To figure out the percentage of the whole, divide the number in question by the total. For example, if you had a basket of 12 apples and used 9 in a pie, that would be 9 / 12 or", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 60 + }, + { + "text": "to county and also by examining how percentages vary over time. To figure out the percentage of the whole, divide the number in question by the total. For example, if you had a basket of 12 apples and used 9 in a pie, that would be 9 / 12 or .75—commonly expressed as 75 percent. To try this on the census counties data, use the code in Listing 5-7, which calculates for each county the percentage of the population that reported their race as Asian: SELECT geo_name, state_us_abbreviation AS \"st\", (CAST ➊(p0010006 AS numeric(8,1)) / p0010001) * 100 AS \"pct_asian\" FROM us_counties_2010 ORDER BY \"pct_asian\" DESC; Estadísticos e-Books & Papers Listing 5-7: Calculating the percentage of the population that is Asian by county The key piece of this query divides p0010006, the column with the count of Asian alone, by p0010001, the column for total population ➊. If we use the data as their original integer types, we won’t get the fractional result we need: every row will display a result of 0, the quotient. Instead, we force decimal division by using CAST on one of the integers. The last part multiplies the result by 100 to present the result as a fraction of 100—the way most people understand percentages. By sorting from highest to lowest percentage, the top of the output is as follows: geo_name st pct_asian -------------------------- -- ----------------------- Honolulu County HI 43.89497769109962474000 Aleutians East Borough AK 35.97580388411333970100 San Francisco County CA 33.27165361664607226500 Santa Clara County CA 32.02237037519322063600 Kauai County HI 31.32461880132953749400 Aleutians West Census Area AK 28.87969789606185937800 Tracking Percent Change Another key indicator in data analysis is percent change: how much bigger, or smaller, is one number than another? Percent change calculations are often employed when analyzing change over time, and they’re particularly useful for comparing change among similar items. Some examples include: The year-over-year change in the number of vehicles sold by each automobile maker. The monthly change in subscriptions to each email list owned by a marketing firm. The annual increase or decrease in enrollment at schools across the nation. The formula to calculate percent change can be expressed like this: Estadísticos e-Books & Papers (new number – old number) / old number So, if you own a lemonade stand and sold 73 glasses of lemonade today and 59 glasses yesterday, you’d figure the day-to-day percent change like this: (73 – 59) / 59 = .237 = 23.7% Let’s try this with a small collection of test data related to spending in departments of a hypothetical local government. Listing 5-8 calculates which departments had the greatest percentage increase and loss: ➊ CREATE TABLE percent_change ( department varchar(20), spend_2014 numeric(10,2), spend_2017 numeric(10,2) ); ➋ INSERT INTO percent_change VALUES ('Building', 250000, 289000), ('Assessor', 178556, 179500), ('Library', 87777, 90001), ('Clerk', 451980, 650000), ('Police', 250000, 223000), ('Recreation', 199000, 195000); SELECT department, spend_2014, spend_2017, ➌ round( (spend_2017 - spend_2014) / spend_2014 * 100, 1) AS \"pct_change\" FROM percent_change; Listing 5-8: Calculating percent change Listing 5-8 creates a small table called", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 61 + }, + { + "text": "percent_change VALUES ('Building', 250000, 289000), ('Assessor', 178556, 179500), ('Library', 87777, 90001), ('Clerk', 451980, 650000), ('Police', 250000, 223000), ('Recreation', 199000, 195000); SELECT department, spend_2014, spend_2017, ➌ round( (spend_2017 - spend_2014) / spend_2014 * 100, 1) AS \"pct_change\" FROM percent_change; Listing 5-8: Calculating percent change Listing 5-8 creates a small table called percent_change ➊ and inserts six rows ➋ with data on department spending for the years 2014 and 2017. The percent change formula ➌ subtracts spend_2014 from spend_2017 and then divides by spend_2014. We multiply by 100 to express the result as a portion of 100. To simplify the output, this time I’ve added the round() function to remove all but one decimal place. The function takes two arguments: the Estadísticos e-Books & Papers column or expression to be rounded, and the number of decimal places to display. Because both numbers are type numeric, the result will also be a numeric. The script creates this result: department spend_2014 spend_2017 pct_change ---------- ---------- ---------- ---------- Building 250000.00 289000.00 15.6 Assessor 178556.00 179500.00 0.5 Library 87777.00 90001.00 2.5 Clerk 451980.00 650000.00 43.8 Police 250000.00 223000.00 -10.8 Recreation 199000.00 195000.00 -2.0 Now, it’s just a matter of finding out why the Clerk department’s spending has outpaced others in the town. Aggregate Functions for Averages and Sums So far, we’ve performed math operations across columns in each row of a table. SQL also lets you calculate a result from values within the same column using aggregate functions. You can see a full list of PostgreSQL aggregates, which calculate a single result from multiple inputs, at https://www.postgresql.org/docs/current/static/functions-aggregate.html. Two of the most-used aggregate functions in data analysis are avg() and sum(). Returning to the us_counties_2010 census table, it’s reasonable to want to calculate the total population of all counties plus the average population of all counties. Using avg() and sum() on column p0010001 (the total population) makes it easy, as shown in Listing 5-9. Again, we use the round() function to remove numbers after the decimal point in the average calculation. SELECT sum(p0010001) AS \"County Sum\", round(avg(p0010001), 0) AS \"County Average\" FROM us_counties_2010; Listing 5-9: Using the sum() and avg() aggregate functions Estadísticos e-Books & Papers This calculation produces the following result: County Sum County Average ---------- -------------- 308745538 98233 The population for all counties in the United States in 2010 added up to approximately 308.7 million, and the average county population was 98,233. Finding the Median The median value in a set of numbers is as important an indicator, if not more so, than the average. Here’s the difference between median and average, and why median matters: Average The sum of all the values divided by the number of values Median The “middle” value in an ordered set of values Why is median important for data analysis? Consider this example: let’s say six kids, ages 10, 11, 10, 9, 13, and 12, go on a field trip. It’s easy to add the ages and divide by six to get the group’s average age: (10 + 11", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 62 + }, + { + "text": "ordered set of values Why is median important for data analysis? Consider this example: let’s say six kids, ages 10, 11, 10, 9, 13, and 12, go on a field trip. It’s easy to add the ages and divide by six to get the group’s average age: (10 + 11 + 10 + 9 + 13 + 12) / 6 = 10.8 Because the ages are within a narrow range, the 10.8 average is a good representation of the group. But averages are less helpful when the values are bunched, or skewed, toward one end of the distribution, or if the group includes outliers. For example, what if an older chaperone joins the field trip? With ages of 10, 11, 10, 9, 13, 12, and 46, the average age increases considerably: (10 + 11 + 10 + 9 + 13 + 12 + 46) / 7 = 15.9 Now the average doesn’t represent the group well because the outlier skews it, making it an unreliable indicator. Estadísticos e-Books & Papers This is where medians shine. The median is the midpoint in an ordered list of values—the point at which half the values are more and half are less. Using the field trip, we order the attendees’ ages from lowest to highest: 9, 10, 10, 11, 12, 13, 46 The middle (median) value is 11. Half the values are higher, and half are lower. Given this group, the median of 11 is a better picture of the typical age than the average of 15.9. If the set of values is an even number, you average the two middle numbers to find the median. Let’s add another student (age 12) to the field trip: 9, 10, 10, 11, 12, 12, 13, 46 Now, the two middle values are 11 and 12. To find the median, we average them: 11.5. Medians are reported frequently in financial news. Reports on housing prices often use medians because a few sales of McMansions in a ZIP Code that is otherwise modest can make averages useless. The same goes for sports player salaries: one or two superstars can skew a team’s average. A good test is to calculate the average and the median for a group of values. If they’re close, the group is probably normally distributed (the familiar bell curve), and the average is useful. If they’re far apart, the values are not normally distributed and the median is the better representation. Finding the Median with Percentile Functions PostgreSQL (as with most relational databases) does not have a built-in median() function, similar to what you’d find in Excel or other spreadsheet programs. It’s also not included in the ANSI SQL standard. But we can use a SQL percentile function to find the median as well as other quantiles or cut points, which are the points that divide a group of numbers into Estadísticos e-Books & Papers equal sizes. Percentile functions are part of standard ANSI SQL. In statistics, percentiles indicate the point in an ordered set", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 63 + }, + { + "text": "SQL percentile function to find the median as well as other quantiles or cut points, which are the points that divide a group of numbers into Estadísticos e-Books & Papers equal sizes. Percentile functions are part of standard ANSI SQL. In statistics, percentiles indicate the point in an ordered set of data below which a certain percentage of the data is found. For example, a doctor might tell you that your height places you in the 60th percentile for an adult in your age group. That means 60 percent of people are your height or shorter. The median is equivalent to the 50th percentile—again, half the values are below and half above. SQL’s percentile functions allow us to calculate that easily, although we have to pay attention to a difference in how the two versions of the function—percentile_cont(n) and percentile_disc(n)— handle calculations. Both functions are part of the ANSI SQL standard and are present in PostgreSQL, Microsoft SQL Server, and other databases. The percentile_cont(n) function calculates percentiles as continuous values. That is, the result does not have to be one of the numbers in the data set but can be a decimal value in between two of the numbers. This follows the methodology for calculating medians on an even number of values, where the median is the average of the two middle numbers. On the other hand, percentile_disc(n) returns only discrete values. That is, the result returned will be rounded to one of the numbers in the set. To make this distinction clear, let’s use Listing 5-10 to make a test table and fill in six numbers. CREATE TABLE percentile_test ( numbers integer ); INSERT INTO percentile_test (numbers) VALUES (1), (2), (3), (4), (5), (6); SELECT ➊ percentile_cont(.5) WITHIN GROUP (ORDER BY numbers), ➋ percentile_disc(.5) WITHIN GROUP (ORDER BY numbers) FROM percentile_test; Listing 5-10: Testing SQL percentile functions Estadísticos e-Books & Papers In both the continuous ➊ and discrete ➋ percentile functions, we enter .5 to represent the 50th percentile, which is equivalent to the median. Running the code returns the following: percentile_cont percentile_disc --------------- --------------- 3.5 3 The percentile_cont() function returned what we’d expect the median to be: 3.5. But because percentile_disc() calculates discrete values, it reports 3, the last value in the first 50 percent of the numbers. Because the accepted method of calculating medians is to average the two middle values in an even-numbered set, use percentile_cont(.5) to find a median. Median and Percentiles with Census Data Our census data can show how a median tells a different story than an average. Listing 5-11 adds percentile_cont() alongside the sum() and avg() aggregates we’ve used so far: SELECT sum(p0010001) AS \"County Sum\", round(avg(p0010001), 0) AS \"County Average\", percentile_cont(.5) WITHIN GROUP (ORDER BY p0010001) AS \"County Median\" FROM us_counties_2010; Listing 5-11: Using sum(), avg(), and percentile_cont() aggregate functions Your result should equal the following: County Sum County Average County Median ---------- -------------- ------------- 308745538 98233 25857 The median and average are far apart, which shows that averages can mislead.", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 64 + }, + { + "text": "WITHIN GROUP (ORDER BY p0010001) AS \"County Median\" FROM us_counties_2010; Listing 5-11: Using sum(), avg(), and percentile_cont() aggregate functions Your result should equal the following: County Sum County Average County Median ---------- -------------- ------------- 308745538 98233 25857 The median and average are far apart, which shows that averages can mislead. As of 2010, half the counties in America had fewer than 25,857 people, whereas half had more. If you gave a presentation on U.S. demographics and told the audience that the “average county in America had 98,200 people,” they’d walk away with a skewed picture of reality. Nearly 40 counties had a million or more people as of the 2010 Estadísticos e-Books & Papers Decennial Census, and Los Angeles County had close to 10 million. That pushes the average higher. Finding Other Quantiles with Percentile Functions You can also slice data into smaller equal groups. Most common are quartiles (four equal groups), quintiles (five groups), and deciles (10 groups). To find any individual value, you can just plug it into a percentile function. For example, to find the value marking the first quartile, or the lowest 25 percent of data, you’d use a value of .25: percentile_cont(.25) However, entering values one at a time is laborious if you want to generate multiple cut points. Instead, you can pass values into percentile_cont() using an array, a SQL data type that contains a list of items. Listing 5-12 shows how to calculate all four quartiles at once: SELECT percentile_cont(➊array[.25,.5,.75]) WITHIN GROUP (ORDER BY p0010001) AS \"quartiles\" FROM us_counties_2010; Listing 5-12: Passing an array of values to percentile_cont() In this example, we create an array of cut points by enclosing values in a constructor ➊ called array[]. Inside the square brackets, we provide comma-separated values representing the three points at which to cut to create four quartiles. Run the query, and you should see this output: quartiles --------------------- {11104.5,25857,66699} Because we passed in an array, PostgreSQL returns an array, denoted by curly brackets. Each quartile is separated by commas. The first quartile is 11,104.5, which means 25 percent of counties have a population that is equal to or lower than this value. The second quartile is the same as the median: 25,857. The third quartile is 66,699, meaning the largest 25 Estadísticos e-Books & Papers percent of counties have at least this large of a population. Arrays come with a host of functions (noted for PostgreSQL at https://www.postgresql.org/docs/current/static/functions-array.html) that allow you to perform tasks such as adding or removing values or counting the elements. A handy function for working with the result returned in Listing 5-12 is unnest(), which makes the array easier to read by turning it into rows. Listing 5-13 shows the code: SELECT unnest( percentile_cont(array[.25,.5,.75]) WITHIN GROUP (ORDER BY p0010001) ) AS \"quartiles\" FROM us_counties_2010; Listing 5-13: Using unnest() to turn an array into rows Now the output should be in rows: quartiles --------- 11104.5 25857 66699 If we were computing deciles, pulling them from the resulting array and displaying", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 65 + }, + { + "text": "the code: SELECT unnest( percentile_cont(array[.25,.5,.75]) WITHIN GROUP (ORDER BY p0010001) ) AS \"quartiles\" FROM us_counties_2010; Listing 5-13: Using unnest() to turn an array into rows Now the output should be in rows: quartiles --------- 11104.5 25857 66699 If we were computing deciles, pulling them from the resulting array and displaying them in rows would be especially helpful. Creating a median() Function Although PostgreSQL does not have a built-in median() aggregate function, if you’re adventurous, the PostgreSQL wiki at http://wiki.postgresql.org/wiki/Aggregate_Median provides a script to create one. Listing 5-14 shows the script: ➊ CREATE OR REPLACE FUNCTION _final_median(anyarray) RETURNS float8 AS $$ WITH q AS ( SELECT val FROM unnest($1) val WHERE VAL IS NOT NULL ORDER BY 1 Estadísticos e-Books & Papers ), cnt AS ( SELECT COUNT(*) AS c FROM q ) SELECT AVG(val)::float8 FROM ( SELECT val FROM q LIMIT 2 - MOD((SELECT c FROM cnt), 2) OFFSET GREATEST(CEIL((SELECT c FROM cnt) / 2.0) - 1,0) ) q2; $$ LANGUAGE sql IMMUTABLE; ➋ CREATE AGGREGATE median(anyelement) ( SFUNC=array_append, STYPE=anyarray, FINALFUNC=_final_median, INITCOND='{}' ); Listing 5-14: Creating a median() aggregate function in PostgreSQL Given what you’ve learned so far, the code for making a median() aggregate function may look inscrutable. I’ll cover functions in more depth later in the book, but for now note that the code contains two main blocks: one to make a function called _final_median ➊ that sorts the values in the column and finds the midpoint, and a second that serves as the callable aggregate function median() ➋ and passes values to _final_median. For now, you can skip reviewing the script line by line and simply execute the code. Let’s add the median() function to the census query and try it next to percentile_cont(), as shown in Listing 5-15: SELECT sum(p0010001) AS \"County Sum\", round(AVG(p0010001), 0) AS \"County Average\", median(p0010001) AS \"County Median\", percentile_cont(.5) WITHIN GROUP (ORDER BY p0010001) AS \"50th Percentile\" FROM us_counties_2010; Listing 5-15: Using a median() aggregate function The query results show that the median function and the percentile function return the same value: Estadísticos e-Books & Papers County Sum County Average County Median 50th Percentile ---------- -------------- ------------- --------------- 308745538 98233 25857 25857 So when should you use median() instead of a percentile function? There is no simple answer. The median() syntax is easier to remember, albeit a chore to set up for each database, and it’s specific to PostgreSQL. Also, in practice, median() executes more slowly and may perform poorly on large data sets or slow machines. On the other hand, percentile_cont() is portable across several SQL database managers, including Microsoft SQL Server, and allows you to find any percentile from 0 to 100. Ultimately, you can try both and decide. Finding the Mode Additionally, we can find the mode, the value that appears most often, using the PostgreSQL mode() function. The function is not part of standard SQL and has a syntax similar to the percentile functions. Listing 5-16 shows a mode() calculation on p0010001, the total population column: SELECT mode()", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 66 + }, + { + "text": "the Mode Additionally, we can find the mode, the value that appears most often, using the PostgreSQL mode() function. The function is not part of standard SQL and has a syntax similar to the percentile functions. Listing 5-16 shows a mode() calculation on p0010001, the total population column: SELECT mode() WITHIN GROUP (ORDER BY p0010001) FROM us_counties_2010; Listing 5-16: Finding the most frequent value with mode() The result is 21720, a population count shared by counties in Mississippi, Oregon, and West Virginia. Wrapping Up Working with numbers is a key step in acquiring meaning from your data, and with the math skills covered in this chapter, you’re ready to handle the foundations of numerical analysis with SQL. Later in the book, you’ll learn about deeper statistical concepts including regression and correlation. At this point, you have the basics of sums, averages, and percentiles. You’ve also learned how a median can be a fairer assessment Estadísticos e-Books & Papers of a group of values than an average. That alone can help you avoid inaccurate conclusions. In the next chapter, I’ll introduce you to the power of joining data in two or more tables to increase your options for data analysis. We’ll use the 2010 Census data you’ve already loaded into the analysis database and explore additional data sets. TRY IT YOURSELF Here are three exercises to test your SQL math skills: 1. Write a SQL statement for calculating the area of a circle whose radius is 5 inches. (If you don’t remember the formula, it’s an easy web search.) Do you need parentheses in your calculation? Why or why not? 2. Using the 2010 Census county data, find out which New York state county has the highest percentage of the population that identified as “American Indian/Alaska Native Alone.” What can you learn about that county from online research that explains the relatively large proportion of American Indian population compared with other New York counties? 3. Was the 2010 median county population higher in California or New York? Estadísticos e-Books & Papers 6 JOINING TABLES IN A RELATIONAL DATABASE In Chapter 1, I introduced the concept of a relational database, an application that supports data stored across multiple, related tables. In a relational model, each table typically holds data on one entity—such as students, cars, purchases, houses—and each row in the table describes one of those entities. A process known as a table join allows us to link rows in one table to rows in other tables. The concept of relational databases came from the British computer scientist Edgar F. Codd. While working for IBM in 1970, he published a paper called “A Relational Model of Data for Large Shared Data Banks.” His ideas revolutionized database design and led to the development of SQL. Using the relational model, you can build tables that eliminate duplicate data, are easier to maintain, and provide for increased flexibility in writing queries to get just the data you want. Linking Tables Using JOIN To connect tables", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 67 + }, + { + "text": "His ideas revolutionized database design and led to the development of SQL. Using the relational model, you can build tables that eliminate duplicate data, are easier to maintain, and provide for increased flexibility in writing queries to get just the data you want. Linking Tables Using JOIN To connect tables in a query, we use a JOIN ... ON statement (or one of the other JOIN variants I’ll cover in this chapter). The JOIN statement links one table to another in the database during a query, using matching values in columns we specify in both tables. The syntax takes this form: Estadísticos e-Books & Papers SELECT * FROM table_a JOIN table_b ON table_a.key_column = table_b.foreign_key_column This is similar to the basic SELECT syntax you’ve already learned, but instead of naming one table in the FROM clause, we name a table, give the JOIN keyword, and then name a second table. The ON keyword follows, where we specify the columns we want to use to match values. When the query runs, it examines both tables and then returns columns from both tables where the values match in the columns specified in the ON clause. Matching based on equality between values is the most common use of the ON clause, but you can use any expression that evaluates to the Boolean results true or false. For example, you could match where values from one column are greater than or equal to values in the other: ON table_a.key_column >= table_b.foreign_key_column That’s rare, but it’s an option if your analysis requires it. Relating Tables with Key Columns Consider this example of relating tables with key columns: imagine you’re a data analyst with the task of checking on a public agency’s payroll spending by department. You file a Freedom of Information Act request for that agency’s salary data, expecting to receive a simple spreadsheet listing each employee and their salary, arranged like this: dept location first_name last_name salary ---- -------- ---------- --------- ------ Tax Atlanta Nancy Jones 62500 Tax Atlanta Lee Smith 59300 IT Boston Soo Nguyen 83000 IT Boston Janet King 95000 But that’s not what arrives. Instead, the agency sends you a data dump from its payroll system: a dozen CSV files, each representing one table in its database. You read the document explaining the data layout (be sure to always ask for it!) and start to make sense of the columns in each table. Estadísticos e-Books & Papers Two of the tables stand out: one named employees and another named departments. Using the code in Listing 6-1, let’s create versions of these tables, insert rows, and examine how to join the data in both tables. Using the analysis database you’ve created for these exercises, run all the code, and then look at the data either by using a basic SELECT statement or clicking the table name in pgAdmin and selecting View/Edit Data ▸ All Rows. CREATE TABLE departments ( dept_id bigserial, dept varchar(100), city varchar(100), ➊ CONSTRAINT dept_key PRIMARY KEY (dept_id), ➋", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 68 + }, + { + "text": "these exercises, run all the code, and then look at the data either by using a basic SELECT statement or clicking the table name in pgAdmin and selecting View/Edit Data ▸ All Rows. CREATE TABLE departments ( dept_id bigserial, dept varchar(100), city varchar(100), ➊ CONSTRAINT dept_key PRIMARY KEY (dept_id), ➋ CONSTRAINT dept_city_unique UNIQUE (dept, city) ); CREATE TABLE employees ( emp_id bigserial, first_name varchar(100), last_name varchar(100), salary integer, ➌ dept_id integer REFERENCES departments (dept_id), ➍ CONSTRAINT emp_key PRIMARY KEY (emp_id), ➎ CONSTRAINT emp_dept_unique UNIQUE (emp_id, dept_id) ); INSERT INTO departments (dept, city) VALUES ('Tax', 'Atlanta'), ('IT', 'Boston'); INSERT INTO employees (first_name, last_name, salary, dept_id) VALUES ('Nancy', 'Jones', 62500, 1), ('Lee', 'Smith', 59300, 1), ('Soo', 'Nguyen', 83000, 2), ('Janet', 'King', 95000, 2); Listing 6-1: Creating the departments and employees tables The two tables follow Codd’s relational model in that each describes attributes about a single entity, in this case the agency’s departments and employees. In the departments table, you should see the following contents: dept_id dept city ------- ---- ------- 1 Tax Atlanta Estadísticos e-Books & Papers 2 IT Boston The dept_id column is the table’s primary key. A primary key is a column or collection of columns whose values uniquely identify each row in a table. A valid primary key column enforces certain constraints: The column or collection of columns must have a unique value for each row. The column or collection of columns can’t have missing values. You define the primary key for departments ➊ and employees ➍ using a CONSTRAINT keyword, which I’ll cover in depth with additional constraint types in Chapter 7. The dept_id column uniquely identifies the department, and although this example contains only a department name and city, such a table would likely include additional information, such as an address or contact information. The employees table should have the following contents: emp_id first_name last_name salary dept_id ------ ---------- --------- ------ ------- 1 Nancy Jones 62500 1 2 Lee Smith 59300 1 3 Soo Nguyen 83000 2 4 Janet King 95000 2 The emp_id column uniquely identifies each row in the employees table. For you to know which department each employee works in, the table includes a dept_id column. The values in this column refer to values in the departments table’s primary key. We call this a foreign key, which you add as a constraint ➌ when creating the table. A foreign key constraint requires a value entered in a column to already exist in the primary key of the table it references. So, values in dept_id in the employees table must exist in dept_id in the departments table; otherwise, you can’t add them. Unlike a primary key, a foreign key column can be empty, and it can contain duplicate values. In this example, the dept_id associated with the employee Nancy Jones is 1; this refers to the value of 1 in the departments table’s primary key, dept_id. Estadísticos e-Books & Papers That tells us that Nancy Jones is part of the Tax department located in", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 69 + }, + { + "text": "it can contain duplicate values. In this example, the dept_id associated with the employee Nancy Jones is 1; this refers to the value of 1 in the departments table’s primary key, dept_id. Estadísticos e-Books & Papers That tells us that Nancy Jones is part of the Tax department located in Atlanta. NOTE Primary key values only need to be unique within a table. That’s why it’s okay for both the employees table and the departments table to have primary key values using the same numbers. Both tables also include a UNIQUE constraint, which I’ll also discuss in more depth in “The UNIQUE Constraint” on page 105. Briefly, it guarantees that values in a column, or a combination of values in more than one column, are unique. In departments, it requires that each row have a unique pair of values for dept and city ➋. In employees, each row must have a unique pair of emp_id and dept_id ➎. You add these constraints to avoid duplicate data. For example, you can’t have two tax departments in Atlanta. You might ask: what is the advantage of breaking apart data into components like this? Well, consider what this sample of data would look like if you had received it the way you initially thought you would, all in one table: dept location first_name last_name salary ---- -------- ---------- --------- ------ Tax Atlanta Nancy Jones 62500 Tax Atlanta Lee Smith 59300 IT Boston Soo Nguyen 83000 IT Boston Janet King 95000 First, when you combine data from various entities in one table, inevitably you have to repeat information. This happens here: the department name and location is spelled out for each employee. This is fine when the table consists of four rows like this, or even 4,000. But when a table holds millions of rows, repeating lengthy strings is redundant and wastes precious space. Second, cramming unrelated data into one table makes managing the data difficult. What if the Marketing department changes its name to Brand Marketing? Each row in the table would require an update. It’s Estadísticos e-Books & Papers simpler to store department names and locations in just one table and update it only once. Now that you know the basics of how tables can relate, let’s look at how to join them in a query. Querying Multiple Tables Using JOIN When you join tables in a query, the database connects rows in both tables where the columns you specified for the join have matching values. The query results then include columns from both tables if you requested them as part of the query. You also can use columns from the joined tables to filter results using a WHERE clause. Queries that join tables are similar in syntax to basic SELECT statements. The difference is that the query also specifies the following: The tables and columns to join, using a SQL JOIN ... ON statement The type of join to perform using variations of the JOIN keyword Let’s look at the overall", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 70 + }, + { + "text": "join tables are similar in syntax to basic SELECT statements. The difference is that the query also specifies the following: The tables and columns to join, using a SQL JOIN ... ON statement The type of join to perform using variations of the JOIN keyword Let’s look at the overall JOIN ... ON syntax first and then explore various types of joins. To join the example employees and departments tables and see all related data from both, start by writing a query like the one in Listing 6-2: ➊ SELECT * ➋ FROM employees JOIN departments ➌ ON employees.dept_id = departments.dept_id; Listing 6-2: Joining the employees and departments tables In the example, you include an asterisk wildcard with the SELECT statement to choose all columns from both tables ➊. Next, the JOIN keyword ➋ goes between the two tables you want data from. Finally, you specify the columns to join the tables using the ON keyword ➌. For each table, you provide the table name, a period, and the column that contains the key values. An equal sign goes between the two table and column Estadísticos e-Books & Papers names. When you run the query, the results include all values from both tables where values in the dept_id columns match. In fact, even the dept_id field appears twice because you selected all columns of both tables: So, even though the data lives in two tables, each with a focused set of columns, you can query those tables to pull the relevant data back together. In “Selecting Specific Columns in a Join” on page 85, I’ll show you how to retrieve only the columns you want from both tables. JOIN Types There’s more than one way to join tables in SQL, and the type of join you’ll use depends on how you want to retrieve data. The following list describes the different types of joins. While reviewing each, it’s helpful to think of two tables side by side, one on the left of the JOIN keyword and the other on the right. A data-driven example of each join follows the list: JOIN Returns rows from both tables where matching values are found in the joined columns of both tables. Alternate syntax is INNER JOIN. LEFT JOIN Returns every row from the left table plus rows that match values in the joined column from the right table. When a left table row doesn’t have a match in the right table, the result shows no values from the right table. RIGHT JOIN Returns every row from the right table plus rows that match the key values in the key column from the left table. When a right table row doesn’t have a match in the left table, the result shows no values from the left table. Estadísticos e-Books & Papers FULL OUTER JOIN Returns every row from both tables and matches rows; then joins the rows where values in the joined columns match. If there’s no match for a value in either the", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 71 + }, + { + "text": "in the left table, the result shows no values from the left table. Estadísticos e-Books & Papers FULL OUTER JOIN Returns every row from both tables and matches rows; then joins the rows where values in the joined columns match. If there’s no match for a value in either the left or right table, the query result contains an empty row for the other table. CROSS JOIN Returns every possible combination of rows from both tables. These join types are best illustrated with data. Say you have two simple tables that hold names of schools. To better visualize join types, let’s call the tables schools_left and schools_right. There are four rows in schools_left: id left_school -- ------------------------ 1 Oak Street School 2 Roosevelt High School 5 Washington Middle School 6 Jefferson High School There are five rows in schools_right: id right_school -- --------------------- 1 Oak Street School 2 Roosevelt High School 3 Morrison Elementary 4 Chase Magnet Academy 6 Jefferson High School Notice that only schools with the id of 1, 2, and 6 match in both tables. Working with two tables of similar data is a common scenario for a data analyst, and a common task would be to identify which schools exist in both tables. Using different joins can help you find those schools, plus other details. Again using your analysis database, run the code in Listing 6-3 to build and populate these two tables: CREATE TABLE schools_left ( ➊ id integer CONSTRAINT left_id_key PRIMARY KEY, left_school varchar(30) ); Estadísticos e-Books & Papers CREATE TABLE schools_right ( ➋ id integer CONSTRAINT right_id_key PRIMARY KEY, right_school varchar(30) ); ➌ INSERT INTO schools_left (id, left_school) VALUES (1, 'Oak Street School'), (2, 'Roosevelt High School'), (5, 'Washington Middle School'), (6, 'Jefferson High School'); INSERT INTO schools_right (id, right_school) VALUES (1, 'Oak Street School'), (2, 'Roosevelt High School'), (3, 'Morrison Elementary'), (4, 'Chase Magnet Academy'), (6, 'Jefferson High School'); Listing 6-3: Creating two tables to explore JOIN types We create and fill two tables: the declarations for these should by now look familiar, but there’s one new element: we add a primary key to each table. After the declaration for the schools_left id column ➊ and schools_right id column, ➋ the keywords CONSTRAINT key_name PRIMARY KEY indicate that those columns will serve as the primary key for their table. That means for each row in both tables, the id column must be filled and contain a value that is unique for each row in that table. Finally, we use the familiar INSERT statements ➌ to add the data to the tables. JOIN We use JOIN, or INNER JOIN, when we want to return rows that have a match in the columns we used for the join. To see an example of this, run the code in Listing 6-4, which joins the two tables you just made: SELECT * FROM schools_left JOIN schools_right ON schools_left.id = schools_right.id; Listing 6-4: Using JOIN Similar to the method we used in Listing 6-2, we specify the", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 72 + }, + { + "text": "used for the join. To see an example of this, run the code in Listing 6-4, which joins the two tables you just made: SELECT * FROM schools_left JOIN schools_right ON schools_left.id = schools_right.id; Listing 6-4: Using JOIN Similar to the method we used in Listing 6-2, we specify the two tables to join around the JOIN keyword. Then we specify which columns we’re Estadísticos e-Books & Papers joining on, in this case the id columns of both tables. Three school IDs match in both tables, so JOIN returns only the three rows of those IDs that match. Schools that exist only in one of the two tables don’t appear in the result. Notice also that the columns from the left table display on the left of the result table: id left_school id right_school -- --------------------- -- --------------------- 1 Oak Street School 1 Oak Street School 2 Roosevelt High School 2 Roosevelt High School 6 Jefferson High School 6 Jefferson High School When should you use JOIN? Typically, when you’re working with well- structured, well-maintained data sets and only need to find rows that exist in all the tables you’re joining. Because JOIN doesn’t provide rows that exist in only one of the tables, if you want to see all the data in one or more of the tables, use one of the other join types. LEFT JOIN and RIGHT JOIN In contrast to JOIN, the LEFT JOIN and RIGHT JOIN keywords each return all rows from one table and display blank rows from the other table if no matching values are found in the joined columns. Let’s look at LEFT JOIN in action first. Execute the code in Listing 6-5: SELECT * FROM schools_left LEFT JOIN schools_right ON schools_left.id = schools_right.id; Listing 6-5: Using LEFT JOIN The result of the query shows all four rows from schools_left as well as the three rows in schools_right where the id fields matched. Because schools_right doesn’t contain a value of 5 in its right_id column, there’s no match, so LEFT JOIN shows an empty row on the right rather than omitting the entire row from the left table as with JOIN. The rows from schools_right that don’t match any values in schools_left are omitted from the results: id left_school id right_school -- ------------------------ -- --------------------- Estadísticos e-Books & Papers 1 Oak Street School 1 Oak Street School 2 Roosevelt High School 2 Roosevelt High School 5 Washington Middle School 6 Jefferson High School 6 Jefferson High School We see similar but opposite behavior by running RIGHT JOIN, as in Listing 6-6: SELECT * FROM schools_left RIGHT JOIN schools_right ON schools_left.id = schools_right.id; Listing 6-6: Using RIGHT JOIN This time, the query returns all rows from schools_right plus rows from schools_left where the id columns have matching values, but the query doesn’t return the rows of schools_left that don’t have a match with schools_right: id left_school id right_school -- --------------------- -- --------------------- 1 Oak Street School 1 Oak Street School 2 Roosevelt High", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 73 + }, + { + "text": "all rows from schools_right plus rows from schools_left where the id columns have matching values, but the query doesn’t return the rows of schools_left that don’t have a match with schools_right: id left_school id right_school -- --------------------- -- --------------------- 1 Oak Street School 1 Oak Street School 2 Roosevelt High School 2 Roosevelt High School 3 Morrison Elementary 4 Chase Magnet Academy 6 Jefferson High School 6 Jefferson High School You’d use either of these join types in a few circumstances: You want your query results to contain all the rows from one of the tables. You want to look for missing values in one of the tables; for example, when you’re comparing data about an entity representing two different time periods. When you know some rows in a joined table won’t have matching values. FULL OUTER JOIN When you want to see all rows from both tables in a join, regardless of whether any match, use the FULL OUTER JOIN option. To see it in action, run Estadísticos e-Books & Papers Listing 6-7: SELECT * FROM schools_left FULL OUTER JOIN schools_right ON schools_left.id = schools_right.id; Listing 6-7: Using FULL OUTER JOIN The result gives every row from the left table, including matching rows and blanks for missing rows from the right table, followed by any leftover missing rows from the right table: id left_school id right_school -- ------------------------ -- --------------------- 1 Oak Street School 1 Oak Street School 2 Roosevelt High School 2 Roosevelt High School 5 Washington Middle School 6 Jefferson High School 6 Jefferson High School 4 Chase Magnet Academy 3 Morrison Elementary A full outer join is admittedly less useful and used less often than inner and left or right joins. Still, you can use it for a couple of tasks: to merge two data sources that partially overlap or to visualize the degree to which the tables share matching values. CROSS JOIN In a CROSS JOIN query, the result (also known as a Cartesian product) lines up each row in the left table with each row in the right table to present all possible combinations of rows. Listing 6-8 shows the CROSS JOIN syntax; because the join doesn’t need to find matches between key fields, there’s no need to provide the clause using the ON keyword. SELECT * FROM schools_left CROSS JOIN schools_right; Listing 6-8: Using CROSS JOIN The result has 20 rows—the product of four rows in the left table times five rows in the right: Estadísticos e-Books & Papers id left_school id right_school -- ------------------------ -- --------------------- 1 Oak Street School 1 Oak Street School 1 Oak Street School 2 Roosevelt High School 1 Oak Street School 3 Morrison Elementary 1 Oak Street School 4 Chase Magnet Academy 1 Oak Street School 6 Jefferson High School 2 Roosevelt High School 1 Oak Street School 2 Roosevelt High School 2 Roosevelt High School 2 Roosevelt High School 3 Morrison Elementary 2 Roosevelt High School 4 Chase Magnet Academy 2 Roosevelt High School 6 Jefferson", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 74 + }, + { + "text": "School 4 Chase Magnet Academy 1 Oak Street School 6 Jefferson High School 2 Roosevelt High School 1 Oak Street School 2 Roosevelt High School 2 Roosevelt High School 2 Roosevelt High School 3 Morrison Elementary 2 Roosevelt High School 4 Chase Magnet Academy 2 Roosevelt High School 6 Jefferson High School 5 Washington Middle School 1 Oak Street School 5 Washington Middle School 2 Roosevelt High School 5 Washington Middle School 3 Morrison Elementary 5 Washington Middle School 4 Chase Magnet Academy 5 Washington Middle School 6 Jefferson High School 6 Jefferson High School 1 Oak Street School 6 Jefferson High School 2 Roosevelt High School 6 Jefferson High School 3 Morrison Elementary 6 Jefferson High School 4 Chase Magnet Academy 6 Jefferson High School 6 Jefferson High School Unless you want to take an extra-long coffee break, I’d suggest avoiding a CROSS JOIN query on large tables. Two tables with 250,000 records each would produce a result set of 62.5 billion rows and tax even the hardiest server. A more practical use would be generating data to create a checklist, such as all colors you’d want to offer for each shirt style in a warehouse. Using NULL to Find Rows with Missing Values Being able to reveal missing data from one of the tables is valuable when you’re digging through data. Any time you join tables, it’s wise to vet the quality of the data and understand it better by discovering whether all key values in one table appear in another. There are many reasons why a discrepancy might exist, such as a clerical error, incomplete output from the database, or some change in the data over time. All this information is important context for making correct inferences about the data. When you have only a handful of rows, eyeballing the data is an easy way to look for rows with missing data. For large tables, you need a better strategy: filtering to show all rows without a match. To do this, we Estadísticos e-Books & Papers employ the keyword NULL. In SQL, NULL is a special value that represents a condition in which there’s no data present or where the data is unknown because it wasn’t included. For example, if a person filling out an address form skips the “Middle Initial” field, rather than storing an empty string in the database, we’d use NULL to represent the unknown value. It’s important to keep in mind that NULL is different from 0 or an empty string that you’d place in a character field using two quotes (\"\"). Both those values could have some unintended meaning that’s open to misinterpretation, so you use NULL to show that the value is unknown. And unlike 0 or an empty string, you can use NULL across data types. When a SQL join returns empty rows in one of the tables, those columns don’t come back empty but instead come back with the value NULL. In Listing 6-9, we’ll find those rows by", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 75 + }, + { + "text": "is unknown. And unlike 0 or an empty string, you can use NULL across data types. When a SQL join returns empty rows in one of the tables, those columns don’t come back empty but instead come back with the value NULL. In Listing 6-9, we’ll find those rows by adding a WHERE clause to filter for NULL by using the phrase IS NULL on the right_id column. If we wanted to look for columns with data, we’d use IS NOT NULL. SELECT * FROM schools_left LEFT JOIN schools_right ON schools_left.id = schools_right.id WHERE schools_right.id IS NULL; Listing 6-9: Filtering to show missing values with IS NULL Now the result of the join shows only the one row from the left table that didn’t have a match on the right side. id left_school id right_school -- ------------------------ -- ------------ 5 Washington Middle School Three Types of Table Relationships Part of the science (or art, some may say) of joining tables involves understanding how the database designer intends for the tables to relate, also known as the database’s relational model. The three types of table relationships are one to one, one to many, and many to many. Estadísticos e-Books & Papers One-to-One Relationship In our JOIN example in Listing 6-4, there is only one match for an id in each of the two tables. In addition, there are no duplicate id values in either table: only one row in the left table exists with an id of 1, and only one row in the right table has an id of 1. In database parlance, this is called a one-to-one relationship. Consider another example: joining two tables with state-by-state census data. One table might contain household income data and the other data on educational attainment. Both tables would have 51 rows (one for each state plus Washington, D.C.), and if we wanted to join them on a key such as state name, state abbreviation, or a standard geography code, we’d have only one match for each key value in each table. One-to-Many Relationship In a one-to-many relationship, a key value in the first table will have multiple matching values in the second table’s joined column. Consider a database that tracks automobiles. One table would hold data on automobile manufacturers, with one row each for Ford, Honda, Kia, and so on. A second table with model names, such as Focus, Civic, Sedona, and Accord, would have several rows matching each row in the manufacturers’ table. Many-to-Many Relationship In a many-to-many relationship, multiple rows in the first table will have multiple matching rows in the second table. As an example, a table of baseball players could be joined to a table of field positions. Each player can be assigned to multiple positions, and each position can be played by multiple people. Understanding these relationships is essential because it helps us discern whether the results of queries accurately reflect the structure of the database. Estadísticos e-Books & Papers Selecting Specific Columns in a Join So", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 76 + }, + { + "text": "Each player can be assigned to multiple positions, and each position can be played by multiple people. Understanding these relationships is essential because it helps us discern whether the results of queries accurately reflect the structure of the database. Estadísticos e-Books & Papers Selecting Specific Columns in a Join So far, we’ve used the asterisk wildcard to select all columns from both tables. That’s okay for quick data checks, but more often you’ll want to specify a subset of columns. You can focus on just the data you want and avoid inadvertently changing the query results if someone adds a new column to a table. As you learned in single-table queries, to select particular columns you use the SELECT keyword followed by the desired column names. When joining tables, the syntax changes slightly: you must include the column as well as its table name. The reason is that more than one table can contain columns with the same name, which is certainly true of our joined tables so far. Consider the following query, which tries to fetch an id column without naming the table: SELECT id FROM schools_left LEFT JOIN schools_right ON schools_left.id = schools_right.id; Because id exists in both schools_left and schools_right, the server throws an error that appears in pgAdmin’s results pane: column reference \"id\" is ambiguous. It’s not clear which table id belongs to. To fix the error, we need to add the table name in front of each column we’re querying, as we do in the ON clause. Listing 6-10 shows the syntax, specifying that we want the id column from schools_left. We’re also fetching the school names from both tables. SELECT schools_left.id, schools_left.left_school, schools_right.right_school FROM schools_left LEFT JOIN schools_right ON schools_left.id = schools_right.id; Listing 6-10: Querying specific columns in a join We simply prefix each column name with the table it comes from, and the rest of the query syntax is the same. The result returns the requested Estadísticos e-Books & Papers columns from each table: id left_school right_school -- ------------------------ --------------------- 1 Oak Street School Oak Street School 2 Roosevelt High School Roosevelt High School 5 Washington Middle School 6 Jefferson High School Jefferson High School We can also add the AS keyword we used previously with census data to make it clear in the results that the id column is from schools_left. The syntax would look like this: SELECT schools_left.id AS left_id, ... This would display the name of the schools_left id column as left_id. We could do this for all the other columns we select using the same syntax, but the next section describes another, better method we can use to rename multiple columns. Simplifying JOIN Syntax with Table Aliases Naming the table for a column is easy enough, but doing so for multiple columns clutters your code. One of the best ways to serve your colleagues is to write code that’s readable, which should generally not involve making them wade through table names repeated for 25 columns! The way to write", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 77 + }, + { + "text": "table for a column is easy enough, but doing so for multiple columns clutters your code. One of the best ways to serve your colleagues is to write code that’s readable, which should generally not involve making them wade through table names repeated for 25 columns! The way to write more concise code is to use a shorthand approach called table aliases. To create a table alias, we place a character or two after the table name when we declare it in the FROM clause. (You can use more than a couple of characters for an alias, but if the goal is to simplify code, don’t go overboard.) Those characters then serve as an alias we can use instead of the full table name anywhere we reference the table in the code. Listing 6-11 demonstrates how this works: SELECT lt.id, lt.left_school, rt.right_school ➊ FROM schools_left AS lt LEFT JOIN schools_right AS rt Estadísticos e-Books & Papers ON lt.id = rt.id; Listing 6-11: Simplifying code with table aliases In the FROM clause, we declare the alias lt to represent schools_left and the alias rt to represent schools_right ➊ using the AS keyword. Once that’s in place, we can use the aliases instead of the full table names everywhere else in the code. Immediately, our SQL looks more compact, and that’s ideal. Joining Multiple Tables Of course, SQL joins aren’t limited to two tables. We can continue adding tables to the query as long as we have columns with matching values to join on. Let’s say we obtain two more school-related tables and want to join them to schools_left in a three-table join. Here are the tables: schools_enrollment has the number of students per school: id enrollment -- ---------- 1 360 2 1001 5 450 6 927 The schools_grades table contains the grade levels housed in each building: id grades -- ------ 1 K-3 2 9-12 5 6-8 6 9-12 To write the query, we’ll use Listing 6-12 to create the tables and load the data: CREATE TABLE schools_enrollment ( id integer, enrollment integer ); Estadísticos e-Books & Papers CREATE TABLE schools_grades ( id integer, grades varchar(10) ); INSERT INTO schools_enrollment (id, enrollment) VALUES (1, 360), (2, 1001), (5, 450), (6, 927); INSERT INTO schools_grades (id, grades) VALUES (1, 'K-3'), (2, '9-12'), (5, '6-8'), (6, '9-12'); SELECT lt.id, lt.left_school, en.enrollment, gr.grades ➊ FROM schools_left AS lt LEFT JOIN schools_enrollment AS en ON lt.id = en.id ➋ LEFT JOIN schools_grades AS gr ON lt.id = gr.id; Listing 6-12: Joining multiple tables After we run the CREATE TABLE and INSERT portions of the script, the results consist of schools_enrollment and schools_grades tables, each with records that relate to schools_left from earlier in the chapter. We then connect all three tables. In the SELECT query, we join schools_left to schools_enrollment ➊ using the tables’ id fields. We also declare table aliases to keep the code compact. Next, the query joins schools_left to school_grades again on the id fields ➋. Our result now includes columns from", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 78 + }, + { + "text": "We then connect all three tables. In the SELECT query, we join schools_left to schools_enrollment ➊ using the tables’ id fields. We also declare table aliases to keep the code compact. Next, the query joins schools_left to school_grades again on the id fields ➋. Our result now includes columns from all three tables: id left_school enrollment grades -- ------------------------ ---------- ------ 1 Oak Street School 360 K-3 2 Roosevelt High School 1001 9-12 5 Washington Middle School 450 6-8 6 Jefferson High School 927 9-12 If you need to, you can add even more tables to the query using additional joins. You can also join on different columns, depending on the tables’ relationships. Although there is no hard limit in SQL to the number of tables you can join in a single query, some database systems Estadísticos e-Books & Papers might impose one. Check the documentation. Performing Math on Joined Table Columns The math functions we explored in Chapter 5 are just as usable when working with joined tables. We just need to include the table name when referencing a column in an operation, as we did when selecting table columns. If you work with any data that has a new release at regular intervals, you’ll find this concept useful for joining a newly released table to an older one and exploring how values have changed. That’s certainly what I and many journalists do each time a new set of census data is released. We’ll load the new data and try to find patterns in the growth or decline of the population, income, education, and other indicators. Let’s look at how to do this by revisiting the us_counties_2010 table we created in Chapter 4 and loading similar county data from the previous Decennial Census, in 2000, to a new table. Run the code in Listing 6-13, making sure you’ve saved the CSV file somewhere first: ➊ CREATE TABLE us_counties_2000 ( geo_name varchar(90), state_us_abbreviation varchar(2), state_fips varchar(2), county_fips varchar(3), p0010001 integer, p0010002 integer, p0010003 integer, p0010004 integer, p0010005 integer, p0010006 integer, p0010007 integer, p0010008 integer, p0010009 integer, p0010010 integer, p0020002 integer, p0020003 integer ); ➋ COPY us_counties_2000 FROM 'C:\\YourDirectory\\us_counties_2000.csv' WITH (FORMAT CSV, HEADER); ➌ SELECT c2010.geo_name, c2010.state_us_abbreviation AS state, c2010.p0010001 AS pop_2010, Estadísticos e-Books & Papers c2000.p0010001 AS pop_2000 c2010.p0010001 - c2000.p0010001 AS raw_change, ➍ round( (CAST(c2010.p0010001 AS numeric(8,1)) - c2000.p0010001) / c2000.p0010001 * 100, 1 ) AS pct_change FROM us_counties_2010 c2010 INNER JOIN us_counties_2000 c2000 ➎ ON c2010.state_fips = c2000.state_fips AND c2010.county_fips = c2000.county_fips ➏ AND c2010.p0010001 <> c2000.p0010001 ➐ ORDER BY pct_change DESC; Listing 6-13: Performing math on joined census tables In this code, we’re building on earlier foundations. We have the familiar CREATE TABLE statement ➊, which for this exercise includes state and county codes, a geo_name column with the full name of the state and county, and nine columns with population counts including total population and counts by race. The COPY statement ➋ imports a CSV file with the census data; you can find us_counties_2000.csv along with", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 79 + }, + { + "text": "this exercise includes state and county codes, a geo_name column with the full name of the state and county, and nine columns with population counts including total population and counts by race. The COPY statement ➋ imports a CSV file with the census data; you can find us_counties_2000.csv along with all of the book’s resources at https://www.nostarch.com/practicalSQL/. After you’ve downloaded the file, you’ll need to change the file path to the location where you saved it. When you’ve finished the import, you should have a table named us_counties_2000 with 3,141 rows. As with the 2010 data, this table has a column named p0010001 that contains the total population for each county in the United States. Because both tables have the same column, it makes sense to calculate the percent change in population for each county between 2000 and 2010. Which counties have led the nation in growth? Which ones have a decline in population? We’ll use the percent change calculation we used in Chapter 5 to get the answer. The SELECT statement ➌ includes the county’s name and state abbreviation from the 2010 table, which is aliased with c2010. Next are the p0010001 total population columns from the 2010 and 2000 tables, both renamed with unique names using AS to distinguish them in the results. To get the raw change in population, we subtract the 2000 population from the 2010 count, and to find the percent change, we employ a formula ➍ and round the results to one decimal point. We join by matching values in two columns in both tables: state_fips Estadísticos e-Books & Papers and county_fips ➎. The reason to join on two columns instead of one is that in both tables, we need the combination of a state code and a county code to find a unique county. I’ve added a third condition ➏ to illustrate using an inequality. This limits the join to counties where the p0010001 population column has a different value. We combine all three conditions using the AND keyword. Using that syntax, a join happens when all three conditions are satisfied. Finally, the results are sorted in descending order by percent change ➐ so we can see the fastest growers at the top. That’s a lot of work, but it’s worth it. Here’s what the first five rows of the results indicate: Two counties, Kendall in Illinois and Pinal in Arizona, more than doubled their population in 10 years, with counties in Florida, South Dakota, and Virginia not far behind. That’s a valuable story we’ve extracted from this analysis and a starting point for understanding national population trends. If you were to dig into the data further, you might find that many of the counties with the largest growth from 2000 to 2010 were suburban bedroom communities that benefited from the decade’s housing boom, and that a more recent trend sees Americans leaving rural areas to move to cities. That could make for an interesting analysis following the 2020 Decennial Census. Wrapping Up", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 80 + }, + { + "text": "the counties with the largest growth from 2000 to 2010 were suburban bedroom communities that benefited from the decade’s housing boom, and that a more recent trend sees Americans leaving rural areas to move to cities. That could make for an interesting analysis following the 2020 Decennial Census. Wrapping Up Given that table relationships are foundational to database architecture, learning to join tables in queries allows you to handle many of the more complex data sets you’ll encounter. Experimenting with the different Estadísticos e-Books & Papers types of joins on tables can tell you a great deal about how data have been gathered and reveal when there’s a quality issue. Make trying various joins a routine part of your exploration of a new data set. Moving forward, we’ll continue building on these bigger concepts as we drill deeper into finding information in data sets and working with the finer nuances of handling data types and making sure we have quality data. But first, we’ll look at one more foundational element: employing best practices to build reliable, speedy databases with SQL. TRY IT YOURSELF Continue your exploration of joins with these exercises: 1. The table us_counties_2010 contains 3,143 rows, and us_counties_2000 has 3,141. That reflects the ongoing adjustments to county-level geographies that typically result from government decision making. Using appropriate joins and the NULL value, identify which counties don’t exist in both tables. For fun, search online to find out why they’re missing. 2. Using either the median() or percentile_cont() functions in Chapter 5, determine the median of the percent change in county population. 3. Which county had the greatest percentage loss of population between 2000 and 2010? Do you have any idea why? (Hint: A major weather event happened in 2005.) Estadísticos e-Books & Papers 7 TABLE DESIGN THAT WORKS FOR YOU Obsession with detail can be a good thing. When you’re running out the door, it’s reassuring to know your keys will be hanging on the hook where you always leave them. The same holds true for database design. When you need to excavate a nugget of information from dozens of tables and millions of rows, you’ll appreciate a dose of that same detail obsession. When you organize data into a finely tuned, smartly named set of tables, the analysis experience becomes more manageable. In this chapter, I’ll build on Chapter 6 by introducing best practices for organizing and tuning SQL databases, whether they’re yours or ones you inherit for analysis. You already know how to create basic tables and add columns with the appropriate data type and a primary key. Now, we’ll dig deeper into table design by exploring naming rules and conventions, ways to maintain the integrity of your data, and how to add indexes to tables to speed up queries. Naming Tables, Columns, and Other Identifiers Developers tend to follow different SQL style patterns when naming tables, columns, and other objects (called identifiers). Some prefer to use camel case, as in berrySmoothie, where words are strung", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 81 + }, + { + "text": "of your data, and how to add indexes to tables to speed up queries. Naming Tables, Columns, and Other Identifiers Developers tend to follow different SQL style patterns when naming tables, columns, and other objects (called identifiers). Some prefer to use camel case, as in berrySmoothie, where words are strung together and the first Estadísticos e-Books & Papers letter of each word is capitalized except for the first word. Pascal case, as in BerrySmoothie, follows a similar pattern but capitalizes the first letter of the first word too. With snake case, as in berry_smoothie, all the words are lowercase and separated by underscores. So far, I’ve been using snake case in most of the examples, such as in the table us_counties_2010. You’ll find passionate supporters of each naming convention, and some preferences are tied to individual database applications or programming languages. For example, Microsoft recommends Pascal case for its SQL Server users. Whichever convention you prefer, it’s most important to choose a style and apply it consistently. Be sure to check whether your organization has a style guide or offer to collaborate on one, and then follow it religiously. Mixing styles or following none generally leads to a mess. It will be difficult to know which table is the most current, which is the backup, or the difference between two similarly named tables. For example, imagine connecting to a database and finding the following collection of tables: Customers customers custBackup customer_analysis customer_test2 customer_testMarch2012 customeranalysis In addition, working without a consistent naming scheme makes it problematic for others to dive into your data and makes it challenging for you to pick up where you left off. Let’s explore considerations related to naming identifiers and suggestions for best practices. Using Quotes Around Identifiers to Enable Mixed Case Standard ANSI SQL and many database-specific variants of SQL treat identifiers as case-insensitive unless you provide a delimiter around them —typically double quotes. Consider these two hypothetical CREATE TABLE statements for PostgreSQL: Estadísticos e-Books & Papers CREATE TABLE customers ( customer_id serial, --snip-- ); CREATE TABLE Customers ( customer_id serial, --snip-- ); When you execute these statements in order, the first CREATE TABLE command creates a table called customers. But rather than creating a second table called Customers, the second statement will throw an error: relation \"customers\" already exists. Because you didn’t quote the identifier, PostgreSQL treats customers and Customers as the same identifier, disregarding the case. If you want to preserve the uppercase letter and create a separate table named Customers, you must surround the identifier with quotes, like this: CREATE TABLE \"Customers\" ( customer_id serial, --snip-- ); Now, PostgreSQL retains the uppercase C and creates Customers as well as customers. Later, to query Customers rather than customers, you’ll have to quote its name in the SELECT statement: SELECT * FROM \"Customers\"; Of course, you wouldn’t want two tables with such similar names because of the high risk of a mix-up. This example simply illustrates the behavior of SQL in PostgreSQL. Pitfalls with Quoting", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 82 + }, + { + "text": "Customers rather than customers, you’ll have to quote its name in the SELECT statement: SELECT * FROM \"Customers\"; Of course, you wouldn’t want two tables with such similar names because of the high risk of a mix-up. This example simply illustrates the behavior of SQL in PostgreSQL. Pitfalls with Quoting Identifiers Using quotation marks also permits characters not otherwise allowed in an identifier, including spaces. But be aware of the negatives of using this method: for example, you might want to throw quotes around \"trees planted\" and use that as a column name in a reforestation database, but Estadísticos e-Books & Papers then all users will have to provide quotes on every subsequent reference to that column. Omit the quotes and the database will respond with an error, identifying trees and planted as separate columns missing a comma between them. A more readable and reliable option is to use snake case, as in trees_planted. Another downside to quoting is that it lets you use SQL reserved keywords, such as TABLE, WHERE, or SELECT, as an identifier. Reserved keywords are words SQL designates as having special meaning in the language. Most database developers frown on using reserved keywords as identifiers. At a minimum it’s confusing, and at worst neglecting or forgetting to quote that keyword later will result in an error because the database will interpret the word as a command instead of an identifier. NOTE For PostgreSQL, you can find a list of keywords documented at https://www.postgresql.org/docs/current/static/sql-keywords- appendix.html. In addition, many code editors and database tools, including pgAdmin, will automatically highlight keywords in a particular color. Guidelines for Naming Identifiers Given the extra burden of quoting and its potential problems, it’s best to keep your identifier names simple, unquoted, and consistent. Here are my recommendations: Use snake case. Snake case is readable and reliable, as shown in the earlier trees_planted example. It’s used throughout the official PostgreSQL documentation and helps make multiword names easy to understand: video_on_demand makes more sense at a glance than videoondemand. Make names easy to understand and avoid cryptic abbreviations. If you’re building a database related to travel, Estadísticos e-Books & Papers arrival_time is a better reminder of the content as a column name than arv_tm. For table names, use plurals. Tables hold rows, and each row represents one instance of an entity. So, use plural names for tables, such as teachers, vehicles, or departments. Mind the length. The maximum number of characters allowed for an identifier name varies by database application: the SQL standard is 128 characters, but PostgreSQL limits you to 63, and the Oracle system maximum is 30. If you’re writing code that may get reused in another database system, lean toward shorter identifier names. When making copies of tables, use names that will help you manage them later. One method is to append a YYYY_MM_DD date to the table name when you create it, such as tire_sizes_2017_10_20. An additional benefit is that the table names will sort in date order. Controlling Column", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 83 + }, + { + "text": "identifier names. When making copies of tables, use names that will help you manage them later. One method is to append a YYYY_MM_DD date to the table name when you create it, such as tire_sizes_2017_10_20. An additional benefit is that the table names will sort in date order. Controlling Column Values with Constraints A column’s data type already broadly defines the kind of data it will accept: integers versus characters, for example. But SQL provides several additional constraints that let us further specify acceptable values for a column based on rules and logical tests. With constraints, we can avoid the “garbage in, garbage out” phenomenon, which is what happens when poor-quality data result in inaccurate or incomplete analysis. Constraints help maintain the quality of the data and ensure the integrity of the relationships among tables. In Chapter 6, you learned about primary and foreign keys, which are two of the most commonly used constraints. Let’s review them as well as the following additional constraint types: CHECK Evaluates whether the data falls within values we specify UNIQUE Ensures that values in a column or group of columns are unique in each row in the table NOT NULL Prevents NULL values in a column Estadísticos e-Books & Papers We can add constraints in two ways: as a column constraint or as a table constraint. A column constraint only applies to that column. It’s declared with the column name and data type in the CREATE TABLE statement, and it gets checked whenever a change is made to the column. With a table constraint, we can supply criteria that apply to one or more columns. We declare it in the CREATE TABLE statement immediately after defining all the table columns, and it gets checked whenever a change is made to a row in the table. Let’s explore these constraints, their syntax, and their usefulness in table design. Primary Keys: Natural vs. Surrogate In Chapter 6, you learned about giving a table a primary key: a column or collection of columns whose values uniquely identify each row in a table. A primary key is a constraint, and it imposes two rules on the column or columns that make up the key: 1. Each column in the key must have a unique value for each row. 2. No column in the key can have missing values. Primary keys also provide a means of relating tables to each other and maintaining referential integrity, which is ensuring that rows in related tables have matching values when we expect them to. The simple primary key example in “Relating Tables with Key Columns” on page 74 had a single ID field that used an integer inserted by us, the user. However, as with most areas of SQL, you can implement primary keys in several ways. Often, the data will suggest the best path. But first we must assess whether to use a natural key or a surrogate key as the primary key. Using Existing Columns for Natural Keys You implement", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 84 + }, + { + "text": "However, as with most areas of SQL, you can implement primary keys in several ways. Often, the data will suggest the best path. But first we must assess whether to use a natural key or a surrogate key as the primary key. Using Existing Columns for Natural Keys You implement a natural key by using one or more of the table’s existing columns rather than creating a column and filling it with artificial values to act as keys. If a column’s values obey the primary key constraint— Estadísticos e-Books & Papers unique for every row and never empty—it can be used as a natural key. A value in the column can change as long as the new value doesn’t cause a violation of the constraint. An example of a natural key is a driver’s license identification number issued by a local Department of Motor Vehicles. Within a governmental jurisdiction, such as a state in the United States, we’d reasonably expect that all drivers would receive a unique ID on their licenses. But if we were compiling a national driver’s license database, we might not be able to make that assumption; several states could independently issue the same ID code. In that case, the driver_id column may not have unique values and cannot be used as the natural key unless it’s combined with one or more additional columns. Regardless, as you build tables, you’ll encounter many values suitable for natural keys: a part number, a serial number, or a book’s ISBN are all good examples. Introducing Columns for Surrogate Keys Instead of relying on existing data, a surrogate key typically consists of a single column that you fill with artificial values. This might be a sequential number auto-generated by the database; for example, using a serial data type (covered in “Auto-Incrementing Integers” on page 27). Some developers like to use a Universally Unique Identifier (UUID), which is a code comprised of 32 hexadecimal digits that identifies computer hardware or software. Here’s an example: 2911d8a8-6dea-4a46-af23-d64175a08237 Pros and Cons of Key Types As with most SQL debates, there are arguments for using either type of primary key. Reasons cited for using natural keys often include the following: The data already exists in the table, and you don’t need to add a column to create a key. Estadísticos e-Books & Papers Because the natural key data has meaning, it can reduce the need to join tables when searching. Alternatively, advocates of surrogate keys highlight these points in favor: Because a surrogate key doesn’t have any meaning in itself and its values are independent of the data in the table, if your data changes later, you’re not limited by the key structure. Natural keys tend to consume more storage than the integers typically used for surrogate keys. A well-designed table should have one or more columns that can serve as a natural key. An example is a product table with a unique product code. But in a table of employees, it might be difficult to", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 85 + }, + { + "text": "to consume more storage than the integers typically used for surrogate keys. A well-designed table should have one or more columns that can serve as a natural key. An example is a product table with a unique product code. But in a table of employees, it might be difficult to find any single column, or even multiple columns, that would be unique on a row-by- row basis to serve as a primary key. In that case, you can create a surrogate key, but you probably should reconsider the table structure. Primary Key Syntax In “JOIN Types” on page 78, you created primary keys on the schools_left and schools_right tables to try out JOIN types. In fact, these were surrogate keys: in both tables, you created columns called id to use as the key and used the keywords CONSTRAINT key_name PRIMARY KEY to declare them as primary keys. Let’s work through several more primary key examples. In Listing 7-1, we declare a primary key using the column constraint and table constraint methods on a table similar to the driver’s license example mentioned earlier. Because we expect the driver’s license IDs to always be unique, we’ll use that column as a natural key. CREATE TABLE natural_key_example ( ➊ license_id varchar(10) CONSTRAINT license_key PRIMARY KEY, first_name varchar(50), last_name varchar(50) ); ➋ DROP TABLE natural_key_example; Estadísticos e-Books & Papers CREATE TABLE natural_key_example ( license_id varchar(10), first_name varchar(50), last_name varchar(50), ➌ CONSTRAINT license_key PRIMARY KEY (license_id) ); Listing 7-1: Declaring a single-column natural key as a primary key We first use the column constraint syntax to declare license_id as the primary key by adding the CONSTRAINT keyword ➊ followed by a name for the key and then the keywords PRIMARY KEY. An advantage of using this syntax is that it’s easy to understand at a glance which column is designated as the primary key. Note that in the column constraint syntax you can omit the CONSTRAINT keyword and name for the key, and simply use PRIMARY KEY. Next, we delete the table from the database by using the DROP TABLE command ➋ to prepare for the table constraint example. To add the same primary key using the table constraint syntax, we declare the CONSTRAINT after listing the final column ➌ with the column we want to use as the key in parentheses. In this example, we end up with the same column for the primary key as we did with the column constraint syntax. However, you must use the table constraint syntax when you want to create a primary key using more than one column. In that case, you would list the columns in parentheses, separated by commas. We’ll explore that in a moment. First, let’s look at how having a primary key protects you from ruining the integrity of your data. Listing 7-2 contains two INSERT statements: INSERT INTO natural_key_example (license_id, first_name, last_name) VALUES ('T229901', 'Lynn', 'Malero'); INSERT INTO natural_key_example (license_id, first_name, last_name) VALUES ('T229901', 'Sam', 'Tracy'); Listing 7-2: An example of a primary", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 86 + }, + { + "text": "let’s look at how having a primary key protects you from ruining the integrity of your data. Listing 7-2 contains two INSERT statements: INSERT INTO natural_key_example (license_id, first_name, last_name) VALUES ('T229901', 'Lynn', 'Malero'); INSERT INTO natural_key_example (license_id, first_name, last_name) VALUES ('T229901', 'Sam', 'Tracy'); Listing 7-2: An example of a primary key violation When you execute the first INSERT statement on its own, the server loads a row into the natural_key_example table without any issue. When you Estadísticos e-Books & Papers attempt to execute the second, the server replies with an error: ERROR: duplicate key value violates unique constraint \"license_key\" DETAIL: Key (license_id)=(T229901) already exists. Before adding the row, the server checked whether a license_id of T229901 was already present in the table. Because it was, and because a primary key by definition must be unique for each row, the server rejected the operation. The rules of the fictional DMV state that no two drivers can have the same license ID, so checking for and rejecting duplicate data is one way for the database to enforce that rule. Creating a Composite Primary Key If we want to create a natural key but a single column in the table isn’t sufficient for meeting the primary key requirements for uniqueness, we may be able to create a suitable key from a combination of columns, which is called a composite primary key. As a hypothetical example, let’s use a table that tracks student school attendance. The combination of a student ID column and a date column would give us unique data for each row, tracking whether or not the student was in school each day during a school year. To create a composite primary key from two or more columns, you must declare it using the table constraint syntax mentioned earlier. Listing 7-3 creates an example table for the student attendance scenario. The school database would record each student_id only once per school_day, creating a unique value for the row. A present column of data type boolean indicates whether the student was there on that day. CREATE TABLE natural_key_composite_example ( student_id varchar(10), school_day date, present boolean, CONSTRAINT student_key PRIMARY KEY (student_id, school_day) ); Listing 7-3: Declaring a composite primary key as a natural key The syntax in Listing 7-3 follows the same table constraint format for Estadísticos e-Books & Papers adding a primary key for one column, but we pass two (or more) columns as arguments rather than one. Again, we can simulate a key violation by attempting to insert a row where the combination of values in the two key columns—student_id and school_day—is not unique to the table. Run the code in Listing 7-4: INSERT INTO natural_key_composite_example (student_id, school_day, present) VALUES(775, '1/22/2017', 'Y'); INSERT INTO natural_key_composite_example (student_id, school_day, present) VALUES(775, '1/23/2017', 'Y'); INSERT INTO natural_key_composite_example (student_id, school_day, present) VALUES(775, '1/23/2017', 'N'); Listing 7-4: Example of a composite primary key violation The first two INSERT statements execute fine because there’s no duplication of values in the combination of key columns. But the third statement", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 87 + }, + { + "text": "INSERT INTO natural_key_composite_example (student_id, school_day, present) VALUES(775, '1/23/2017', 'Y'); INSERT INTO natural_key_composite_example (student_id, school_day, present) VALUES(775, '1/23/2017', 'N'); Listing 7-4: Example of a composite primary key violation The first two INSERT statements execute fine because there’s no duplication of values in the combination of key columns. But the third statement causes an error because the student_id and school_day values it contains match a combination that already exists in the table: ERROR: duplicate key value violates unique constraint \"student_key\" DETAIL: Key (student_id, school_day)=(775, 2017-01-23) already exists. You can create composite keys with more than two columns. The specific database you’re using imposes the limit to the number of columns you can use. Creating an Auto-Incrementing Surrogate Key If a table you’re creating has no columns suitable for a natural primary key, you may have a data integrity problem; in that case, it’s best to reconsider how you’re structuring the database. If you’re inheriting data for analysis or feel strongly about using surrogate keys, you can create a column and fill it with unique values. Earlier, I mentioned that some developers use UUIDs for this; others rely on software to generate a unique code. For our purposes, an easy way to create a surrogate primary key is with an auto-incrementing integer using one of the serial data types discussed in “Auto-Incrementing Integers” on page 27. Estadísticos e-Books & Papers Recall the three serial types: smallserial, serial, and bigserial. They correspond to the integer types smallint, integer, and bigint in terms of the range of values they handle and the amount of disk storage they consume. For a primary key, it may be tempting to try to save disk space by using serial, which handles numbers as large as 2,147,483,647. But many a database developer has received a late-night call from a user frantic to know why their application is broken, only to discover that the database is trying to generate a number one greater than the data type’s maximum. For this reason, with PostgreSQL, it’s generally wise to use bigserial, which accepts numbers as high as 9.2 quintillion. You can set it and forget it, as shown in the first column defined in Listing 7-5: CREATE TABLE surrogate_key_example ( ➊ order_number bigserial, product_name varchar(50), order_date date, ➋ CONSTRAINT order_key PRIMARY KEY (order_number) ); ➌ INSERT INTO surrogate_key_example (product_name, order_date) VALUES ('Beachball Polish', '2015-03-17'), ('Wrinkle De-Atomizer', '2017-05-22'), ('Flux Capacitor', '1985-10-26'); SELECT * FROM surrogate_key_example; Listing 7-5: Declaring a bigserial column as a surrogate key Listing 7-5 shows how to declare the bigserial ➊ data type for an order_number column and set the column as the primary key ➋. When you insert data into the table ➌, you can omit the order_number column. With order_number set to bigserial, the database will create a new value for that column on each insert. The new value will be one greater than the largest already created for the column. Run SELECT * FROM surrogate_key_example; to see how the column fills in automatically: order_number product_name order_date ------------ -------------------", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 88 + }, + { + "text": "order_number set to bigserial, the database will create a new value for that column on each insert. The new value will be one greater than the largest already created for the column. Run SELECT * FROM surrogate_key_example; to see how the column fills in automatically: order_number product_name order_date ------------ ------------------- ---------- 1 Beachball Polish 2015-03-17 2 Wrinkle De-Atomizer 2017-05-22 3 Flux Capacitor 1985-10-26 Estadísticos e-Books & Papers The database will add one to order_number each time a new row is inserted. But it won’t fill any gaps in the sequence created after rows are deleted. Foreign Keys With the foreign key constraint, SQL very helpfully provides a way to ensure data in related tables doesn’t end up unrelated, or orphaned. A foreign key is one or more columns in a table that match the primary key of another table. But a foreign key also imposes a constraint: values entered must already exist in the primary key or other unique key of the table it references. If not, the value is rejected. This constraint ensures that we don’t end up with rows in one table that have no relation to rows in the other tables we can join them to. To illustrate, Listing 7-6 shows two tables from a hypothetical database tracking motor vehicle activity: CREATE TABLE licenses ( license_id varchar(10), first_name varchar(50), last_name varchar(50), ➊ CONSTRAINT licenses_key PRIMARY KEY (license_id) ); CREATE TABLE registrations ( registration_id varchar(10), registration_date date, ➋ license_id varchar(10) REFERENCES licenses (license_id), CONSTRAINT registration_key PRIMARY KEY (registration_id, license_id) ); ➌ INSERT INTO licenses (license_id, first_name, last_name) VALUES ('T229901', 'Lynn', 'Malero'); ➍ INSERT INTO registrations (registration_id, registration_date, license_id) VALUES ('A203391', '3/17/2017', 'T229901'); ➎ INSERT INTO registrations (registration_id, registration_date, license_id) VALUES ('A75772', '3/17/2017', 'T000001'); Listing 7-6: A foreign key example The first table, licenses, is similar to the natural_key_example table we Estadísticos e-Books & Papers made earlier and uses a driver’s unique license_id ➊ as a natural primary key. The second table, registrations, is for tracking vehicle registrations. A single license ID might be connected to multiple vehicle registrations, because each licensed driver can register multiple vehicles over a number of years. Also, a single vehicle could be registered to multiple license holders, establishing, as you learned in Chapter 6, a many-to-many relationship. Here’s how that relationship is expressed via SQL: in the registrations table, we designate the column license_id as a foreign key by adding the REFERENCES keyword, followed by the table name and column for it to reference ➋. Now, when we insert a row into registrations, the database will test whether the value inserted into license_id already exists in the license_id primary key column of the licenses table. If it doesn’t, the database returns an error, which is important. If any rows in registrations didn’t correspond to a row in licenses, we’d have no way to write a query to find the person who registered the vehicle. To see this constraint in action, create the two tables and execute the INSERT statements one at a time.", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 89 + }, + { + "text": "which is important. If any rows in registrations didn’t correspond to a row in licenses, we’d have no way to write a query to find the person who registered the vehicle. To see this constraint in action, create the two tables and execute the INSERT statements one at a time. The first adds a row to licenses ➌ that includes the value T229901 for the license_id. The second adds a row to registrations ➍ where the foreign key contains the same value. So far, so good, because the value exists in both tables. But we encounter an error with the third insert, which tries to add a row to registrations ➎ with a value for license_id that’s not in licenses: ERROR: insert or update on table \"registrations\" violates foreign key constraint \"registrations_license_id_fkey\" DETAIL: Key (license_id)=(T000001) is not present in table \"licenses\". The resulting error is good because it shows the database is keeping the data clean. But it also indicates a few practical implications: first, it affects the order we insert data. We cannot add data to a table that contains a foreign key before the other table referenced by the key has the related records, or we’ll get an error. In this example, we’d have to create a driver’s license record before inserting a related registration Estadísticos e-Books & Papers record (if you think about it, that’s what your local department of motor vehicles probably does). Second, the reverse applies when we delete data. To maintain referential integrity, the foreign key constraint prevents us from deleting a row from licenses before removing any related rows in registrations, because doing so would leave an orphaned record. We would have to delete the related row in registrations first, and then delete the row in licenses. However, ANSI SQL provides a way to handle this order of operations automatically using the ON DELETE CASCADE keywords, which I’ll discuss next. Automatically Deleting Related Records with CASCADE To delete a row in licenses and have that action automatically delete any related rows in registrations, we can specify that behavior by adding ON DELETE CASCADE when defining the foreign key constraint. When we create the registrations table, the keywords would go at the end of the definition of the license_id column, like this: CREATE TABLE registrations ( registration_id varchar(10), registration_date date, license_id varchar(10) REFERENCES licenses (license_id) ON DELETE CASCADE, CONSTRAINT registration_key PRIMARY KEY (registration_id, license_id) ); Now, deleting a row in licenses should also delete all related rows in registrations. This allows us to delete a driver’s license without first having to manually remove any registrations to it. It also maintains data integrity by ensuring deleting a license doesn’t leave orphaned rows in registrations. The CHECK Constraint A CHECK constraint evaluates whether data added to a column meets the expected criteria, which we specify with a logical test. If the criteria aren’t met, the database returns an error. The CHECK constraint is extremely Estadísticos e-Books & Papers valuable because it can prevent columns from getting", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 90 + }, + { + "text": "CHECK Constraint A CHECK constraint evaluates whether data added to a column meets the expected criteria, which we specify with a logical test. If the criteria aren’t met, the database returns an error. The CHECK constraint is extremely Estadísticos e-Books & Papers valuable because it can prevent columns from getting loaded with nonsensical data. For example, a new employee’s birthdate probably shouldn’t be more than 120 years in the past, so you can set a cap on birthdates. Or, in most schools I know, Z isn’t a valid letter grade for a course (although my barely passing algebra grade felt like it), so we might insert constraints that only accept the values A–F. As with primary keys, we can implement a CHECK constraint as a column constraint or a table constraint. For a column constraint, declare it in the CREATE TABLE statement after the column name and data type: CHECK (logical expression). As a table constraint, use the syntax CONSTRAINT constraint_name CHECK (logical expression) after all columns are defined. Listing 7-7 shows a CHECK constraint applied to two columns in a table we might use to track the user role and salary of employees within an organization. It uses the table constraint syntax for the primary key and the CHECK constraint. CREATE TABLE check_constraint_example ( user_id bigserial, user_role varchar(50), salary integer, CONSTRAINT user_id_key PRIMARY KEY (user_id), ➊ CONSTRAINT check_role_in_list CHECK (user_role IN('Admin', 'Staff')), ➋ CONSTRAINT check_salary_not_zero CHECK (salary > 0) ); Listing 7-7: Examples of CHECK constraints We create the table and set the user_id column as an auto- incrementing surrogate primary key. The first CHECK ➊ tests whether values entered into the user_role column match one of two predefined strings, Admin or Staff, by using the SQL IN operator. The second CHECK tests whether values entered in the salary column are greater than 0, because no one should be earning a negative amount ➋. Both tests are another example of a Boolean expression, a statement that evaluates as either true or false. If a value tested by the constraint evaluates as true, the check passes. NOTE Estadísticos e-Books & Papers Developers may debate whether check logic belongs in the database, in the application in front of the database, such as a human resources system, or both. One advantage of checks in the database is that the database will maintain data integrity in the case of changes to the application, even if a new system gets built or users are given alternate ways to add data. When values are inserted or updated, the database checks them against the constraint. If the values in either column violate the constraint—or, for that matter, if the primary key constraint is violated—the database will reject the change. If we use the table constraint syntax, we also can combine more than one test in a single CHECK statement. Say we have a table related to student achievement. We could add the following: CONSTRAINT grad_check CHECK (credits >= 120 AND tuition = 'Paid') Notice that we", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 91 + }, + { + "text": "reject the change. If we use the table constraint syntax, we also can combine more than one test in a single CHECK statement. Say we have a table related to student achievement. We could add the following: CONSTRAINT grad_check CHECK (credits >= 120 AND tuition = 'Paid') Notice that we combine two logical tests by enclosing them in parentheses and connecting them with AND. Here, both Boolean expressions must evaluate as true for the entire check to pass. You can also test values across columns, as in the following example where we want to make sure an item’s sale price is a discount on the original, assuming we have columns for both values: CONSTRAINT sale_check CHECK (sale_price < retail_price) Inside the parentheses, the logical expression checks that the sale price is less than the retail price. The UNIQUE Constraint We can also ensure that a column has a unique value in each row by using the UNIQUE constraint. If ensuring unique values sounds similar to the purpose of a primary key, it is. But UNIQUE has one important difference. In a primary key, no values can be NULL, but a UNIQUE constraint permits multiple NULL values in a column. Estadísticos e-Books & Papers To show the usefulness of UNIQUE, look at the code in Listing 7-8, which is a table for tracking contact info: CREATE TABLE unique_constraint_example ( contact_id bigserial CONSTRAINT contact_id_key PRIMARY KEY, first_name varchar(50), last_name varchar(50), email varchar(200), ➊ CONSTRAINT email_unique UNIQUE (email) ); INSERT INTO unique_constraint_example (first_name, last_name, email) VALUES ('Samantha', 'Lee', 'slee@example.org'); INSERT INTO unique_constraint_example (first_name, last_name, email) VALUES ('Betty', 'Diaz', 'bdiaz@example.org'); INSERT INTO unique_constraint_example (first_name, last_name, email) ➋ VALUES ('Sasha', 'Lee', 'slee@example.org'); Listing 7-8: A UNIQUE constraint example In this table, contact_id serves as a surrogate primary key, uniquely identifying each row. But we also have an email column, the main point of contact with each person. We’d expect this column to contain only unique email addresses, but those addresses might change over time. So, we use UNIQUE ➊ to ensure that any time we add or update a contact’s email we’re not providing one that already exists. If we do try to insert an email that already exists ➋, the database will return an error: ERROR: duplicate key value violates unique constraint \"email_unique\" DETAIL: Key (email)=(slee@example.org) already exists. Again, the error shows the database is working for us. The NOT NULL Constraint In Chapter 6, you learned about NULL, a special value in SQL that represents a condition where no data is present in a row in a column or the value is unknown. You’ve also learned that NULL values are not allowed in a primary key, because primary keys need to uniquely identify each row in a table. But there will be other columns besides primary keys where Estadísticos e-Books & Papers you don’t want to allow empty values. For example, in a table listing each student in a school, it would be necessary for columns containing first and last names to be", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 92 + }, + { + "text": "each row in a table. But there will be other columns besides primary keys where Estadísticos e-Books & Papers you don’t want to allow empty values. For example, in a table listing each student in a school, it would be necessary for columns containing first and last names to be filled for each row. To require a value in a column, SQL provides the NOT NULL constraint, which simply prevents a column from accepting empty values. Listing 7-9 demonstrates the NOT NULL syntax: CREATE TABLE not_null_example ( student_id bigserial, first_name varchar(50) NOT NULL, last_name varchar(50) NOT NULL, CONSTRAINT student_id_key PRIMARY KEY (student_id) ); Listing 7-9: A NOT NULL constraint example Here, we declare NOT NULL for the first_name and last_name columns because it’s likely we’d require those pieces of information in a table tracking student information. If we attempt an INSERT on the table and don’t include values for those columns, the database will notify us of the violation. Removing Constraints or Adding Them Later So far, we’ve been placing constraints on tables at the time of creation. You can also remove a constraint or later add one to an existing table using ALTER TABLE, the SQL command that makes changes to tables and columns. We’ll work with ALTER TABLE more in Chapter 9, but for now we’ll review the syntax for adding and removing constraints. To remove a primary key, foreign key, or a UNIQUE constraint, you would write an ALTER TABLE statement in this format: ALTER TABLE table_name DROP CONSTRAINT constraint_name; To drop a NOT NULL constraint, the statement operates on the column, so you must use the additional ALTER COLUMN keywords, like so: ALTER TABLE table_name ALTER COLUMN column_name DROP NOT NULL; Estadísticos e-Books & Papers Let’s use these statements to modify the not_null_example table you just made, as shown in Listing 7-10: ALTER TABLE not_null_example DROP CONSTRAINT student_id_key; ALTER TABLE not_null_example ADD CONSTRAINT student_id_key PRIMARY KEY (student_id); ALTER TABLE not_null_example ALTER COLUMN first_name DROP NOT NULL; ALTER TABLE not_null_example ALTER COLUMN first_name SET NOT NULL; Listing 7-10: Dropping and adding a primary key and a NOT NULL constraint Execute the statements one at a time to make changes to the table. Each time, you can view the changes to the table definition in pgAdmin by clicking the table name once, and then clicking the SQL tab above the query window. With the first ALTER TABLE statement, we use DROP CONSTRAINT to remove the primary key named student_id_key. We then add the primary key back using ADD CONSTRAINT. We’d use that same syntax to add a constraint to any existing table. NOTE You can only add a constraint to an existing table if the data in the target column obeys the limits of the constraint. For example, you can’t place a primary key constraint on a column that has duplicate or empty values. In the third statement, ALTER COLUMN and DROP NOT NULL remove the NOT NULL constraint from the first_name column. Finally, SET NOT NULL adds", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 93 + }, + { + "text": "target column obeys the limits of the constraint. For example, you can’t place a primary key constraint on a column that has duplicate or empty values. In the third statement, ALTER COLUMN and DROP NOT NULL remove the NOT NULL constraint from the first_name column. Finally, SET NOT NULL adds the constraint. Speeding Up Queries with Indexes In the same way that a book’s index helps you find information more quickly, you can speed up queries by adding an index to one or more columns. The database uses the index as a shortcut rather than scanning each row to find data. That’s admittedly a simplistic picture of what, in SQL databases, is a nontrivial topic. I could write several chapters on Estadísticos e-Books & Papers SQL indexes and tuning databases for performance, but instead I’ll offer general guidance on using indexes and a PostgreSQL-specific example that demonstrates their benefits. B-Tree: PostgreSQL’s Default Index While following along in this book, you’ve already created several indexes, perhaps without knowing. Each time you add a primary key or UNIQUE constraint to a table, PostgreSQL (as well as most database systems) places an index on the column. Indexes are stored separately from the table data, but they’re accessed automatically when you run a query and are updated every time a row is added or removed from the table. In PostgreSQL, the default index type is the B-Tree index. It’s created automatically on the columns designated for the primary key or a UNIQUE constraint, and it’s also the type created by default when you execute a CREATE INDEX statement. B-Tree, short for balanced tree, is so named because the structure organizes the data in a way that when you search for a value, it looks from the top of the tree down through branches until it locates the data you want. (Of course, the process is a lot more complicated than that. A good start on understanding more about the B-Tree is the B-Tree Wikipedia entry.) A B-Tree index is useful for data that can be ordered and searched using equality and range operators, such as <, <=, =, >=, >, and BETWEEN. PostgreSQL incorporates additional index types, including the Generalized Inverted Index (GIN) and the Generalized Search Tree (GiST). Each has distinct uses, and I’ll incorporate them in later chapters on full text search and queries using geometry types. For now, let’s see a B-Tree index speed a simple search query. For this exercise, we’ll use a large data set comprising more than 900,000 New York City street addresses, compiled by the OpenAddresses project at https://openaddresses.io/. The file with the data, city_of_new_york.csv, is available for you to download along with all the resources for this book from https://www.nostarch.com/practicalSQL/. Estadísticos e-Books & Papers After you’ve downloaded the file, use the code in Listing 7-11 to create a new_york_addresses table and import the address data. You’re a pro at this by now, although the import will take longer than the tiny data sets you’ve loaded so", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 94 + }, + { + "text": "this book from https://www.nostarch.com/practicalSQL/. Estadísticos e-Books & Papers After you’ve downloaded the file, use the code in Listing 7-11 to create a new_york_addresses table and import the address data. You’re a pro at this by now, although the import will take longer than the tiny data sets you’ve loaded so far. The final, loaded table is 126MB, and on one of my systems, it took nearly a minute for the COPY command to complete. CREATE TABLE new_york_addresses ( longitude numeric(9,6), latitude numeric(9,6), street_number varchar(10), street varchar(32), unit varchar(7), postcode varchar(5), id integer CONSTRAINT new_york_key PRIMARY KEY ); COPY new_york_addresses FROM 'C:\\YourDirectory\\city_of_new_york.csv' WITH (FORMAT CSV, HEADER); Listing 7-11: Importing New York City address data When the data loads, run a quick SELECT query to visually check that you have 940,374 rows and seven columns. A common use for this data might be to search for matches in the street column, so we’ll use that example for exploring index performance. Benchmarking Query Performance with EXPLAIN We’ll measure how well an index can improve query speed by checking the performance before and after adding one. To do this, we’ll use PostgreSQL’s EXPLAIN command, which is specific to PostgreSQL and not part of standard SQL. The EXPLAIN command provides output that lists the query plan for a specific database query. This might include how the database plans to scan the table, whether or not it will use indexes, and so on. If we add the ANALYZE keyword, EXPLAIN will carry out the query and show the actual execution time, which is what we want for the current exercise. Recording Some Control Execution Times Estadísticos e-Books & Papers Run each of the three queries in Listing 7-12 one at a time. We’re using typical SELECT queries with a WHERE clause but with the keywords EXPLAIN ANALYZE included at the beginning. Instead of showing the query results, these keywords tell the database to execute the query and display statistics about the query process and how long it took to execute. EXPLAIN ANALYZE SELECT * FROM new_york_addresses WHERE street = 'BROADWAY'; EXPLAIN ANALYZE SELECT * FROM new_york_addresses WHERE street = '52 STREET'; EXPLAIN ANALYZE SELECT * FROM new_york_addresses WHERE street = 'ZWICKY AVENUE'; Listing 7-12: Benchmark queries for index performance On my system, the first query returns these stats: ➊ Seq Scan on new_york_addresses (cost=0.00..20730.68 rows=3730 width=46) (actual time=0.055..289.426 rows=3336 loops=1) Filter: ((street)::text = 'BROADWAY'::text) Rows Removed by Filter: 937038 Planning time: 0.617 ms ➋ Execution time: 289.838 ms Not all the output is relevant here, so I won’t decode it all, but two lines are pertinent. The first indicates that to find any rows where street = 'BROADWAY', the database will conduct a sequential scan ➊ of the table. That’s a synonym for a full table scan: each row will be examined, and the database will remove any row that doesn’t match BROADWAY. The execution time (on my computer about 290 milliseconds) ➋ is how long this will take. Your time will depend on factors including", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 95 + }, + { + "text": "of the table. That’s a synonym for a full table scan: each row will be examined, and the database will remove any row that doesn’t match BROADWAY. The execution time (on my computer about 290 milliseconds) ➋ is how long this will take. Your time will depend on factors including your computer hardware. Run each query in Listing 7-12 and record the execution time for each. Adding the Index Now, let’s see how adding an index changes the query’s search method and how fast it works. Listing 7-13 shows the SQL statement for creating Estadísticos e-Books & Papers the index with PostgreSQL: CREATE INDEX street_idx ON new_york_addresses (street); Listing 7-13: Creating a B-Tree index on the new_york_addresses table Notice that it’s similar to the commands for creating constraints we’ve covered in the chapter already. (Other database systems have their own variants and options for creating indexes, and there is no ANSI standard.) We give the CREATE INDEX keywords followed by a name we choose for the index, in this case street_idx. Then ON is added, followed by the target table and column. Execute the CREATE INDEX statement, and PostgreSQL will scan the values in the street column and build the index from them. We only need to create the index once. When the task finishes, rerun each of the three queries in Listing 7-12 and record the execution times reported by EXPLAIN ANALYZE. For example: Bitmap Heap Scan on new_york_addresses (cost=65.80..5962.17 rows=2758 width=46) (actual time=1.792..9.816 rows=3336 loops=1) Recheck Cond: ((street)::text = 'BROADWAY'::text) Heap Blocks: exact=2157 ➊ -> Bitmap Index Scan on street_idx (cost=0.00..65.11 rows=2758 width=0) (actual time=1.253..1.253 rows=3336 loops=1) Index Cond: ((street)::text = 'BROADWAY'::text) Planning time: 0.163 ms ➋ Execution time: 5.887 ms Do you notice a change? First, instead of a sequential scan, the EXPLAIN ANALYZE statistics for each query show that the database is now using an index scan on street_idx ➊ instead of visiting each row. Also, the query speed is now markedly faster ➋. Table 7-1 shows the execution times (rounded) from my computer before and after adding the index. Table 7-1: Measuring Index Performance Query Filter Before IndexAfter Index WHERE street = 'BROADWAY' 290 ms 6 ms WHERE street = '52 STREET' 271 ms 6 ms Estadísticos e-Books & Papers WHERE street = 'ZWICKY AVENUE'306 ms 1 ms The execution times are much, much better, effectively a quarter second faster or more per query. Is a quarter second that impressive? Well, whether you’re seeking answers in data using repeated querying or creating a database system for thousands of users, the time savings adds up. If you ever need to remove an index from a table—perhaps if you’re testing the performance of several index types—use the DROP INDEX command followed by the name of the index to remove. Considerations When Using Indexes You’ve seen that indexes have significant performance benefits, so does that mean you should add an index to every column in a table? Not so fast! Indexes are valuable, but they’re not always needed.", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 96 + }, + { + "text": "DROP INDEX command followed by the name of the index to remove. Considerations When Using Indexes You’ve seen that indexes have significant performance benefits, so does that mean you should add an index to every column in a table? Not so fast! Indexes are valuable, but they’re not always needed. In addition, they do enlarge the database and impose a maintenance cost on writing data. Here are a few tips for judging when to uses indexes: Consult the documentation for the database manager you’re using to learn about the kinds of indexes available and which to use on particular data types. PostgreSQL, for example, has five more index types in addition to B-Tree. One, called GiST, is particularly suited to the geometry data types I’ll discuss later in the book. Full text search, which you’ll learn in Chapter 13, also benefits from indexing. Consider adding indexes to any columns you’ll use in table joins. Primary keys are indexed by default in PostgreSQL, but foreign key columns in related tables are not and are a good target for indexes. Add indexes to columns that will frequently end up in a query WHERE clause. As you’ve seen, search performance is significantly improved via indexes. Use EXPLAIN ANALYZE to test performance under a variety of configurations if you’re unsure. Optimization is a process! Estadísticos e-Books & Papers Wrapping Up With the tools you’ve added to your toolbox in this chapter, you’re ready to ensure that the databases you build or inherit are best suited for your collection and exploration of data. Your queries will run faster, you can exclude unwanted values, and your database objects will have consistent organization. That’s a boon for you and for others who share your data. This chapter concludes the first part of the book, which focused on giving you the essentials to dig into SQL databases. I’ll continue building on these foundations as we explore more complex queries and strategies for data analysis. In the next chapter, we’ll use SQL aggregate functions to assess the quality of a data set and get usable information from it. TRY IT YOURSELF Are you ready to test yourself on the concepts covered in this chapter? Consider the following two tables from a database you’re making to keep track of your vinyl LP collection. Start by reviewing these CREATE TABLE statements: CREATE TABLE albums ( album_id bigserial, album_catalog_code varchar(100), album_title text, album_artist text, album_release_date date, album_genre varchar(40), album_description text ); CREATE TABLE songs ( song_id bigserial, song_title text, song_artist text, album_id bigint ); The albums table includes information specific to the overall collection of songs on the disc. The songs table catalogs each track on the album. Each song has a title and Estadísticos e-Books & Papers its own artist column, because each song might feature its own collection of artists. Use the tables to answer these questions: 1. Modify these CREATE TABLE statements to include primary and foreign keys plus additional constraints on both tables. Explain why you made your", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 97 + }, + { + "text": "title and Estadísticos e-Books & Papers its own artist column, because each song might feature its own collection of artists. Use the tables to answer these questions: 1. Modify these CREATE TABLE statements to include primary and foreign keys plus additional constraints on both tables. Explain why you made your choices. 2. Instead of using album_id as a surrogate key for your primary key, are there any columns in albums that could be useful as a natural key? What would you have to know to decide? 3. To speed up queries, which columns are good candidates for indexes? Estadísticos e-Books & Papers 8 EXTRACTING INFORMATION BY GROUPING AND SUMMARIZING Every data set tells a story, and it’s the data analyst’s job to find out what that story is. In Chapter 2, you learned about interviewing data using SELECT statements, which included sorting columns, finding distinct values, and filtering results. You’ve also learned the fundamentals of SQL math, data types, table design, and joining tables. With all these tools under your belt, you’re ready to summarize data using grouping and SQL functions. Summarizing data allows us to identify useful information we wouldn’t be able to see otherwise. In this chapter, we’ll use the well-known institution of your local library as our example. Despite changes in the way people consume information, libraries remain a vital part of communities worldwide. But the internet and advancements in library technology have changed how we use libraries. For example, ebooks and online access to digital materials now have a permanent place in libraries along with books and periodicals. In the United States, the Institute of Museum and Library Services (IMLS) measures library activity as part of its annual Public Libraries Survey. The survey collects data from more than 9,000 library Estadísticos e-Books & Papers administrative entities, defined by the survey as agencies that provide library services to a particular locality. Some agencies are county library systems, and others are part of school districts. Data on each agency includes the number of branches, staff, books, hours open per year, and so on. The IMLS has been collecting data each year since 1988 and includes all public library agencies in the 50 states plus the District of Columbia and several territories, such as American Samoa. (Read more about the program at https://www.imls.gov/research-evaluation/data- collection/public-libraries-survey/.) For this exercise, we’ll assume the role of an analyst who just received a fresh copy of the library data set to produce a report describing trends from the data. We’ll need to create two tables, one with data from the 2014 survey and the second from the 2009 survey. Then we’ll summarize the more interesting data in each table and join the tables to see the five- year trends. During the analysis, you’ll learn SQL techniques for summarizing data using aggregate functions and grouping. Creating the Library Survey Tables Let’s create the 2014 and 2009 library survey tables and import the data. We’ll use appropriate data types for each column and add constraints and", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 98 + }, + { + "text": "see the five- year trends. During the analysis, you’ll learn SQL techniques for summarizing data using aggregate functions and grouping. Creating the Library Survey Tables Let’s create the 2014 and 2009 library survey tables and import the data. We’ll use appropriate data types for each column and add constraints and an index to each table to preserve data integrity and speed up queries. Creating the 2014 Library Data Table We’ll start by creating the table for the 2014 library data. Using the CREATE TABLE statement, Listing 8-1 builds pls_fy2014_pupld14a, a table for the fiscal year 2014 Public Library Data File from the Public Libraries Survey. The Public Library Data File summarizes data at the agency level, counting activity at all agency outlets, which include central libraries, branch libraries, and bookmobiles. The annual survey generates two additional files we won’t use: one summarizes data at the state level, and the other has data on individual outlets. For this exercise, those files are redundant, but you can read about the data they contain in the 2014 data dictionary, Estadísticos e-Books & Papers available from the IMLS at https://www.imls.gov/sites/default/files/fy2014_pls_data_file_documentation.pdf For convenience, I’ve created a naming scheme for the tables: pls refers to the survey title, fy2014 is the fiscal year the data covers, and pupld14a is the name of the particular file from the survey. For simplicity, I’ve selected just 72 of the more relevant columns from the 159 in the original survey file to fill the pls_fy2014_pupld14a table, excluding data like the codes that explain the source of individual responses. When a library didn’t provide data, the agency derived the data using other means, but we don’t need that information for this exercise. Note that Listing 8-1 is abbreviated for convenience. The full data set and code for creating and loading this table is available for download with all the book’s resources at https://www.nostarch.com/practicalSQL/. CREATE TABLE pls_fy2014_pupld14a ( stabr varchar(2) NOT NULL, ➊ fscskey varchar(6) CONSTRAINT fscskey2014_key PRIMARY KEY, libid varchar(20) NOT NULL, libname varchar(100) NOT NULL, obereg varchar(2) NOT NULL, rstatus integer NOT NULL, statstru varchar(2) NOT NULL, statname varchar(2) NOT NULL, stataddr varchar(2) NOT NULL, --snip-- wifisess integer NOT NULL, yr_sub integer NOT NULL ); ➋ CREATE INDEX libname2014_idx ON pls_fy2014_pupld14a (libname); CREATE INDEX stabr2014_idx ON pls_fy2014_pupld14a (stabr); CREATE INDEX city2014_idx ON pls_fy2014_pupld14a (city); CREATE INDEX visits2014_idx ON pls_fy2014_pupld14a (visits); ➌ COPY pls_fy2014_pupld14a FROM 'C:\\YourDirectory\\pls_fy2014_pupld14a.csv' WITH (FORMAT CSV, HEADER); Listing 8-1: Creating and filling the 2014 Public Libraries Survey table After finding the code and data file for Listing 8-1, connect to your analysis database in pgAdmin and run it. Remember to change C:\\YourDirectory\\ to the path where you saved the CSV file. Here’s what it does: first, the code makes the table via CREATE TABLE. We Estadísticos e-Books & Papers assign a primary key constraint to the column named fscskey ➊, a unique code the data dictionary says is assigned to each library. Because it’s unique, present in each row, and unlikely to change, it can serve as a", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 99 + }, + { + "text": "makes the table via CREATE TABLE. We Estadísticos e-Books & Papers assign a primary key constraint to the column named fscskey ➊, a unique code the data dictionary says is assigned to each library. Because it’s unique, present in each row, and unlikely to change, it can serve as a natural primary key. The definition for each column includes the appropriate data type and NOT NULL constraints where the columns have no missing values. If you look carefully in the data dictionary, you’ll notice that I changed the column named database in the CSV file to databases in the table. The reason is that database is a SQL reserved keyword, and it’s unwise to use keywords as identifiers because it can lead to unintended consequences in queries or other functions. The startdat and enddat columns contain dates, but we’ve set their data type to varchar(10) in the code because in the CSV file those columns include non-date values, and our import will fail if we try to use a date data type. In Chapter 9, you’ll learn how to clean up cases like these. For now, those columns are fine as is. After creating the table, we add indexes ➋ to columns we’ll use for queries. This provides faster results when we search the column for a particular library. The COPY statement ➌ imports the data from a CSV file named pls_fy2014_pupld14a.csv using the file path you provide. Creating the 2009 Library Data Table Creating the table for the 2009 library data follows similar steps, as shown in Listing 8-2. Most ongoing surveys will have a handful of year- to-year changes because the makers of the survey either think of new questions or modify existing ones, so the included columns will be slightly different in this table. That’s one reason the data providers create new tables instead of adding rows to a cumulative table. For example, the 2014 file has a wifisess column, which lists the annual number of Wi-Fi sessions the library provided, but this column doesn’t exist in the 2009 data. The data dictionary for this survey year is at https://www.imls.gov/sites/default/files/fy2009_pls_data_file_documentation.pdf Estadísticos e-Books & Papers After you build this table, import the CSV file pls_fy2009_pupld09a. This file is also available to download along with all the book’s resources at https://www.nostarch.com/practicalSQL/. When you’ve saved the file and added the correct file path to the COPY statement, execute the code in Listing 8-2: CREATE TABLE pls_fy2009_pupld09a ( stabr varchar(2) NOT NULL, ➊ fscskey varchar(6) CONSTRAINT fscskey2009_key PRIMARY KEY, libid varchar(20) NOT NULL, libname varchar(100) NOT NULL, address varchar(35) NOT NULL, city varchar(20) NOT NULL, zip varchar(5) NOT NULL, zip4 varchar(4) NOT NULL, cnty varchar(20) NOT NULL, --snip-- fipsst varchar(2) NOT NULL, fipsco varchar(3) NOT NULL ); ➋ CREATE INDEX libname2009_idx ON pls_fy2009_pupld09a (libname); CREATE INDEX stabr2009_idx ON pls_fy2009_pupld09a (stabr); CREATE INDEX city2009_idx ON pls_fy2009_pupld09a (city); CREATE INDEX visits2009_idx ON pls_fy2009_pupld09a (visits); COPY pls_fy2009_pupld09a FROM 'C:\\YourDirectory\\pls_fy2009_pupld09a.csv' WITH (FORMAT CSV, HEADER); Listing 8-2: Creating and filling the 2009 Public Libraries Survey table", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 100 + }, + { + "text": "NULL, fipsco varchar(3) NOT NULL ); ➋ CREATE INDEX libname2009_idx ON pls_fy2009_pupld09a (libname); CREATE INDEX stabr2009_idx ON pls_fy2009_pupld09a (stabr); CREATE INDEX city2009_idx ON pls_fy2009_pupld09a (city); CREATE INDEX visits2009_idx ON pls_fy2009_pupld09a (visits); COPY pls_fy2009_pupld09a FROM 'C:\\YourDirectory\\pls_fy2009_pupld09a.csv' WITH (FORMAT CSV, HEADER); Listing 8-2: Creating and filling the 2009 Public Libraries Survey table We use fscskey as the primary key again ➊, and we create an index on libname and other columns ➋. Now, let’s mine the two tables of library data from 2014 and 2009 to discover their stories. Exploring the Library Data Using Aggregate Functions Aggregate functions combine values from multiple rows and return a single result based on an operation on those values. For example, you might return the average of values with the avg() function, as you learned in Chapter 5. That’s just one of many aggregate functions in SQL. Some are part of the SQL standard, and others are specific to PostgreSQL and Estadísticos e-Books & Papers other database managers. Most of the aggregate functions used in this chapter are part of standard SQL (a full list of PostgreSQL aggregates is at https://www.postgresql.org/docs/current/static/functions-aggregate.html). In this section, we’ll work through the library data using aggregates on single and multiple columns, and then explore how you can expand their use by grouping the results they return with values from additional columns. Counting Rows and Values Using count() After importing a data set, a sensible first step is to make sure the table has the expected number of rows. For example, the IMLS documentation for the 2014 data says the file we imported has 9,305 rows, and the 2009 file has 9,299 rows. When we count the number of rows in those tables, the results should match those counts. The count() aggregate function, which is part of the ANSI SQL standard, makes it easy to check the number of rows and perform other counting tasks. If we supply an asterisk as an input, such as count(*), the asterisk acts as a wildcard, so the function returns the number of table rows regardless of whether they include NULL values. We do this in both statements in Listing 8-3: SELECT count(*) FROM pls_fy2014_pupld14a; SELECT count(*) FROM pls_fy2009_pupld09a; Listing 8-3: Using count() for table row counts Run each of the commands in Listing 8-3 one at a time to see the table row counts. For pls_fy2014_pupld14a, the result should be: count ----- 9305 And for pls_fy2009_pupld09a, the result should be: Estadísticos e-Books & Papers count ----- 9299 Both results match the number of rows we expected. NOTE You can also check the row count using the pgAdmin interface, but it’s clunky. Right-clicking the table name in pgAdmin’s object browser and selecting View/Edit Data ▸ All Rows executes a SQL query for all rows. Then, a pop-up message in the results pane shows the row count, but it disappears after a few seconds. Comparing the number of table rows to what the documentation says is important because it will alert us to issues such as", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 101 + }, + { + "text": "All Rows executes a SQL query for all rows. Then, a pop-up message in the results pane shows the row count, but it disappears after a few seconds. Comparing the number of table rows to what the documentation says is important because it will alert us to issues such as missing rows or cases where we might have imported the wrong file. Counting Values Present in a Column To return the number of rows in a specific column that contain values, we supply the name of a column as input to the count() function rather than an asterisk. For example, if you scan the CREATE TABLE statements for both library tables closely, you’ll notice that we omitted the NOT NULL constraint for the salaries column plus several others. The reason is that not every library agency reported salaries, and some rows have NULL values. To count the number of rows in the salaries column from 2014 that have values, run the count() function in Listing 8-4: SELECT count(salaries) FROM pls_fy2014_pupld14a; Listing 8-4: Using count() for the number of values in a column The result shows 5,983 rows have a value in salaries: count ----- Estadísticos e-Books & Papers 5983 This number is far lower than the number of rows that exist in the table. In the 2014 data, slightly less than two-thirds of the agencies reported salaries, and you’d want to note that fact when reporting any results of calculations performed on those columns. This check is important because the extent to which values are present in a column might influence your decision on whether to proceed with analysis at all. Checking with experts on the topic and digging deeper into the data is usually a good idea, and I recommend seeking expert advice as part of a broader analysis methodology (for more on this topic, see Chapter 18). Counting Distinct Values in a Column In Chapter 2, I covered the DISTINCT keyword, which is part of the SQL standard. When added after SELECT in a query, DISTINCT returns a list of unique values. We can use it to see unique values in one column, or we can see unique combinations of values from multiple columns. Another use of DISTINCT is to add it to the count() function, which causes the function to return a count of distinct values from a column. Listing 8-5 shows two queries. The first counts all values in the 2014 table’s libname column. The second does the same but includes DISTINCT in front of the column name. Run them both, one at a time. SELECT count(libname) FROM pls_fy2014_pupld14a; SELECT count(DISTINCT libname) FROM pls_fy2014_pupld14a; Listing 8-5: Using count() for the number of distinct values in a column The first query returns a row count that matches the number of rows in the table that we found using Listing 8-3: count ----- 9305 That’s good. We expect to have the library agency name listed in Estadísticos e-Books & Papers every row. But the second query returns a", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 102 + }, + { + "text": "column The first query returns a row count that matches the number of rows in the table that we found using Listing 8-3: count ----- 9305 That’s good. We expect to have the library agency name listed in Estadísticos e-Books & Papers every row. But the second query returns a smaller number: count ----- 8515 Using DISTINCT to remove duplicates reduces the number of library names to the 8,515 that are unique. My closer inspection of the data shows that 530 library agencies share their name with one or more other agencies. As one example, nine library agencies are named OXFORD PUBLIC LIBRARY in the table, each one in a city or town named Oxford in different states, including Alabama, Connecticut, Kansas, and Pennsylvania, among others. We’ll write a query to see combinations of distinct values in “Aggregating Data Using GROUP BY” on page 120. Finding Maximum and Minimum Values Using max() and min() Knowing the largest and smallest numbers in a column is useful for a couple of reasons. First, it helps us get a sense of the scope of the values reported for a particular variable. Second, the functions used, max() and min(), can reveal unexpected issues with the data, as you’ll see now with the libraries data. Both max() and min() work the same way: you use a SELECT statement followed by the function with the name of a column supplied. Listing 8-6 uses max() and min() on the 2014 table with the visits column as input. The visits column records the number of annual visits to the library agency and all of its branches. Run the code, and then we’ll review the output. SELECT max(visits), min(visits) FROM pls_fy2014_pupld14a; Listing 8-6: Finding the most and fewest visits using max() and min() The query returns the following results: max min -------- --- 17729020 -3 Estadísticos e-Books & Papers Well, that’s interesting. The maximum value of more than 17.7 million is reasonable for a large city library system, but -3 as the minimum? On the surface, that result seems like a mistake, but it turns out that the creators of the library survey are employing a problematic yet common convention in data collection: using a negative number or some artificially high value as an indicator. In this case, the survey creators used negative numbers to indicate the following conditions: 1. A value of -1 indicates a “nonresponse” to that question. 2. A value of -3 indicates “not applicable” and is used when a library agency has closed either temporarily or permanently. We’ll need to account for and exclude negative values as we explore the data, because summing a column and including the negative values will result in an incorrect total. We can do this using a WHERE clause to filter them. It’s a good thing we discovered this issue now rather than later after spending a lot of time on deeper analysis! NOTE A better alternative for this negative value scenario is to use NULL in rows in the", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 103 + }, + { + "text": "total. We can do this using a WHERE clause to filter them. It’s a good thing we discovered this issue now rather than later after spending a lot of time on deeper analysis! NOTE A better alternative for this negative value scenario is to use NULL in rows in the visits column where response data is absent, and then create a separate visits_flag column to hold codes explaining why. This technique separates number values from information about them. Aggregating Data Using GROUP BY When you use the GROUP BY clause with aggregate functions, you can group results according to the values in one or more columns. This allows us to perform operations like sum() or count() for every state in our table or for every type of library agency. Let’s explore how using GROUP BY with aggregates works. On its own, Estadísticos e-Books & Papers GROUP BY, which is also part of standard ANSI SQL, eliminates duplicate values from the results, similar to DISTINCT. Listing 8-7 shows the GROUP BY clause in action: SELECT stabr FROM pls_fy2014_pupld14a ➊ GROUP BY stabr ORDER BY stabr; Listing 8-7: Using GROUP BY on the stabr column The GROUP BY clause ➊ follows the FROM clause and includes the column name to group. In this case, we’re selecting stabr, which contains the state abbreviation, and grouping by that same column. We then use ORDER BY stabr as well so that the grouped results are in alphabetical order. This will yield a result with unique state abbreviations from the 2014 table. Here’s a portion of the results: stabr ----- AK AL AR AS AZ CA --snip-- WV WY Notice that there are no duplicates in the 56 rows returned. These standard two-letter postal abbreviations include the 50 states plus Washington, D.C., and several U.S. territories, such as American Samoa and the U.S. Virgin Islands. You’re not limited to grouping just one column. In Listing 8-8, we use the GROUP BY clause on the 2014 data to specify the city and stabr columns for grouping: SELECT city, stabr FROM pls_fy2014_pupld14a GROUP BY city, stabr ORDER BY city, stabr; Estadísticos e-Books & Papers Listing 8-8: Using GROUP BY on the city and stabr columns The results get sorted by city and then by state, and the output shows unique combinations in that order: city stabr ---------- ----- ABBEVILLE AL ABBEVILLE LA ABBEVILLE SC ABBOTSFORD WI ABERDEEN ID ABERDEEN SD ABERNATHY TX --snip-- This grouping returns 9,088 rows, 217 fewer than the total table rows. The result indicates there are multiple occasions where the file includes more than one library agency for a particular city and state combination. Combining GROUP BY with count() If we combine GROUP BY with an aggregate function, such as count(), we can pull more descriptive information from our data. For example, we know 9,305 library agencies are in the 2014 table. We can get a count of agencies by state and sort them to see which states have the most. Listing 8-9 shows", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 104 + }, + { + "text": "with an aggregate function, such as count(), we can pull more descriptive information from our data. For example, we know 9,305 library agencies are in the 2014 table. We can get a count of agencies by state and sort them to see which states have the most. Listing 8-9 shows how: ➊ SELECT stabr, count(*) FROM pls_fy2014_pupld14a ➋ GROUP BY stabr ➌ ORDER BY count(*) DESC; Listing 8-9: Using GROUP BY with count() on the stabr column Unlike in earlier examples, we’re now asking for the values in the stabr column and a count of those values. In the list of columns to query ➊, we specify stabr and the count() function with an asterisk as its input. As before, the asterisk causes count() to include NULL values. Also, when we select individual columns along with an aggregate function, we must include the columns in a GROUP BY clause ➋. If we don’t, the database will Estadísticos e-Books & Papers return an error telling us to do so. The reason is that you can’t group values by aggregating and have ungrouped column values in the same query. To sort the results and have the state with the largest number of agencies at the top, we can ORDER BY the count() function ➌ in descending order using DESC. Run the code in Listing 8-9. The results show New York, Illinois, and Texas as the states with the greatest number of library agencies in 2014: stabr count ----- ----- NY 756 IL 625 TX 556 IA 543 PA 455 MI 389 WI 381 MA 370 --snip-- Remember that our table represents library agencies that serve a locality. Just because New York, Illinois, and Texas have the greatest number of library agencies doesn’t mean they have the greatest number of outlets where you can walk in and peruse the shelves. An agency might have one central library only, or it might have no central libraries but 23 branches spread around a county. To count outlets, each row in the table also has values in the columns centlib and branlib, which record the number of central and branch libraries, respectively. To find totals, we would use the sum() aggregate function on both columns. Using GROUP BY on Multiple Columns with count() We can glean yet more information from our data by combining GROUP BY with the count() function and multiple columns. For example, the stataddr column in both tables contains a code indicating whether the agency’s address changed in the last year. The values in stataddr are: 00 No change from last year Estadísticos e-Books & Papers 07 Moved to a new location 15 Minor address change Listing 8-10 shows the code for counting the number of agencies in each state that moved, had a minor address change, or had no change using GROUP BY with stabr and stataddr and adding count(): ➊ SELECT stabr, stataddr, count(*) FROM pls_fy2014_pupld14a ➋ GROUP BY stabr, stataddr ➌ ORDER BY stabr ASC, count(*) DESC; Listing 8-10: Using", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 105 + }, + { + "text": "the number of agencies in each state that moved, had a minor address change, or had no change using GROUP BY with stabr and stataddr and adding count(): ➊ SELECT stabr, stataddr, count(*) FROM pls_fy2014_pupld14a ➋ GROUP BY stabr, stataddr ➌ ORDER BY stabr ASC, count(*) DESC; Listing 8-10: Using GROUP BY with count() of the stabr and stataddr columns The key sections of the query are the column names and the count() function after SELECT ➊, and making sure both columns are reflected in the GROUP BY clause ➋. The effect of grouping by two columns is that count() will show the number of unique combinations of stabr and stataddr. To make the output easier to read, let’s sort first by the state code in ascending order and then by the count in descending order ➌. Here are the results: stabr stataddr count ----- -------- ----- AK 00 70 AK 15 10 AK 07 5 AL 00 221 AL 07 3 AR 00 58 AS 00 1 AZ 00 91 --snip-- The first few rows of the results show that code 00 (no change in address) is the most common value for each state. We’d expect that because it’s likely there are more library agencies that haven’t changed address than those that have. The result helps assure us that we’re analyzing the data in a sound way. If code 07 (moved to a new location) was the most frequent in each state, that would raise a question about Estadísticos e-Books & Papers whether we’ve written the query correctly or whether there’s an issue with the data. Revisiting sum() to Examine Library Visits So far, we’ve combined grouping with aggregate functions, like count(), on columns within a single table to provide results grouped by a column’s values. Now let’s expand the technique to include grouping and aggregating across joined tables using the 2014 and 2009 libraries data. Our goal is to identify trends in library visits spanning that five-year period. To do this, we need to calculate totals using the sum() aggregate function. Before we dig into these queries, let’s address the issue of using the values -3 and -1 to indicate “not applicable” and “nonresponse.” To prevent these negative numbers with no meaning as quantities from affecting the analysis, we’ll filter them out using a WHERE clause to limit the queries to rows where values in visits are zero or greater. Let’s start by calculating the sum of annual visits to libraries from the individual 2014 and 2009 tables. Run each SELECT statement in Listing 8- 11 separately: SELECT sum(visits) AS visits_2014 FROM pls_fy2014_pupld14a WHERE visits >= 0; SELECT sum(visits) AS visits_2009 FROM pls_fy2009_pupld09a WHERE visits >= 0; Listing 8-11: Using the sum() aggregate function to total visits to libraries in 2014 and 2009 For 2014, visits totaled approximately 1.4 billion. visits_2014 ----------- 1425930900 For 2009, visits totaled approximately 1.6 billion. We’re onto something here, but it may not be good news. The trend seems to point downward with", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 106 + }, + { + "text": "8-11: Using the sum() aggregate function to total visits to libraries in 2014 and 2009 For 2014, visits totaled approximately 1.4 billion. visits_2014 ----------- 1425930900 For 2009, visits totaled approximately 1.6 billion. We’re onto something here, but it may not be good news. The trend seems to point downward with visits dropping about 10 percent from 2009 to 2014. Estadísticos e-Books & Papers visits_2009 ----------- 1591799201 These queries sum overall visits. But from the row counts we ran earlier in the chapter, we know that each table contains a different number of library agencies: 9,305 in 2014 and 9,299 in 2009 due to agencies opening, closing, or merging. So, let’s determine how the sum of visits will differ if we limit the analysis to library agencies that exist in both tables. We can do that by joining the tables, as shown in Listing 8- 12: ➊ SELECT sum(pls14.visits) AS visits_2014, sum(pls09.visits) AS visits_2009 ➋ FROM pls_fy2014_pupld14a pls14 JOIN pls_fy2009_pupld09a pls09 ON pls14.fscskey = pls09.fscskey ➌ WHERE pls14.visits >= 0 AND pls09.visits >= 0; Listing 8-12: Using the sum() aggregate function to total visits on joined 2014 and 2009 library tables This query pulls together a few concepts we covered in earlier chapters, including table joins. At the top, we use the sum() aggregate function ➊ to total the visits columns from the 2014 and 2009 tables. When we join the tables on the tables’ primary keys, we’re declaring table aliases ➋ as we explored in Chapter 6. Here, we declare pls14 as the alias for the 2014 table and pls09 as the alias for the 2009 table to avoid having to write the lengthier full table names throughout the query. Note that we use a standard JOIN, also known as an INNER JOIN. That means the query results will only include rows where the primary key values of both tables (the column fscskey) match. Using the WHERE clause ➌, we return rows where both tables have a value of zero or greater in the visits column. As we did in Listing 8-11, we specify that the result should include only those rows where visits are greater than or equal to 0 in both tables. This will prevent the artificial negative values from impacting the sums. Run the query. The results should look like this: Estadísticos e-Books & Papers visits_2014 visits_2009 ----------- ----------- 1417299241 1585455205 The results are similar to what we found by querying the tables separately, although these totals are six to eight million smaller. The reason is that the query referenced only agencies with an fscskey in both tables. Still, the downward trend holds. We’ll need to dig a little deeper to get the full story. NOTE Although we joined the tables on fscskey, it’s entirely possible that some library agencies that appear in both tables merged or split between 2009 and 2014. A call to the IMLS asking about caveats for working with this data is a good idea. Grouping Visit Sums by State Now that we", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 107 + }, + { + "text": "we joined the tables on fscskey, it’s entirely possible that some library agencies that appear in both tables merged or split between 2009 and 2014. A call to the IMLS asking about caveats for working with this data is a good idea. Grouping Visit Sums by State Now that we know library visits dropped for the United States as a whole between 2009 and 2014, you might ask yourself, “Did every part of the country see a decrease, or did the degree of the trend vary by region?” We can answer this question by modifying our preceding query to group by the state code. Let’s also use a percent-change calculation to compare the trend by state. Listing 8-13 contains the full code: ➊ SELECT pls14.stabr, sum(pls14.visits) AS visits_2014, sum(pls09.visits) AS visits_2009, round( (CAST(sum(pls14.visits) AS decimal(10,1)) - sum(pls09.visits)) / sum(pls09.visits) * 100, 2 ) AS pct_change➋ FROM pls_fy2014_pupld14a pls14 JOIN pls_fy2009_pupld09a pls09 ON pls14.fscskey = pls09.fscskey WHERE pls14.visits >= 0 AND pls09.visits >= 0 ➌ GROUP BY pls14.stabr ➍ ORDER BY pct_change DESC; Listing 8-13: Using GROUP BY to track percent change in library visits by state Estadísticos e-Books & Papers We follow the SELECT keyword with the stabr column ➊ from the 2014 table; that same column appears in the GROUP BY clause ➌. It doesn’t matter which table’s stabr column we use because we’re only querying agencies that appear in both tables. After SELECT, we also include the now-familiar percent-change calculation you learned in Chapter 5, which gets the alias pct_change ➋ for readability. We end the query with an ORDER BY clause ➍, using the pct_change column alias. When you run the query, the top of the results shows 10 states or territories with an increase in visits from 2009 to 2014. The rest of the results show a decline. Oklahoma, at the bottom of the ranking, had a 35 percent drop! stabr visits_2014 visits_2009 pct_change ----- ----------- ----------- ---------- GU 103593 60763 70.49 DC 4230790 2944774 43.67 LA 17242110 15591805 10.58 MT 4582604 4386504 4.47 AL 17113602 16933967 1.06 AR 10762521 10660058 0.96 KY 19256394 19113478 0.75 CO 32978245 32782247 0.60 SC 18178677 18105931 0.40 SD 3899554 3890392 0.24 MA 42011647 42237888 -0.54 AK 3486955 3525093 -1.08 ID 8730670 8847034 -1.32 NH 7508751 7675823 -2.18 WY 3666825 3756294 -2.38 --snip-- RI 5259143 6612167 -20.46 NC 33952977 43111094 -21.24 PR 193279 257032 -24.80 GA 28891017 40922598 -29.40 OK 13678542 21171452 -35.39 This useful data should lead a data analyst to investigate what’s driving the changes, particularly the largest ones. Data analysis can sometimes raise as many questions as it answers, but that’s part of the process. It’s always worth a phone call to a person with knowledge about the data to provide context for the results. Sometimes, they may have a very good explanation. Other times, an expert will say, “That doesn’t sound right.” That answer might send you back to the keeper of the data or the Estadísticos e-Books & Papers documentation to find out if you", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 108 + }, + { + "text": "the data to provide context for the results. Sometimes, they may have a very good explanation. Other times, an expert will say, “That doesn’t sound right.” That answer might send you back to the keeper of the data or the Estadísticos e-Books & Papers documentation to find out if you overlooked a code or a nuance with the data. Filtering an Aggregate Query Using HAVING We can refine our analysis by examining a subset of states and territories that share similar characteristics. With percent change in visits, it makes sense to separate large states from small states. In a small state like Rhode Island, one library closing could have a significant effect. A single closure in California might be scarcely noticed in a statewide count. To look at states with a similar volume in visits, we could sort the results by either of the visits columns, but it would be cleaner to get a smaller result set in our query. To filter the results of aggregate functions, we need to use the HAVING clause that’s part of standard ANSI SQL. You’re already familiar with using WHERE for filtering, but aggregate functions, such as sum(), can’t be used within a WHERE clause because they operate at the row level, and aggregate functions work across rows. The HAVING clause places conditions on groups created by aggregating. The code in Listing 8-14 modifies the query in Listing 8-13 by inserting the HAVING clause after GROUP BY: SELECT pls14.stabr, sum(pls14.visits) AS visits_2014, sum(pls09.visits) AS visits_2009, round( (CAST(sum(pls14.visits) AS decimal(10,1)) - sum(pls09.visits)) / sum(pls09.visits) * 100, 2 ) AS pct_change FROM pls_fy2014_pupld14a pls14 JOIN pls_fy2009_pupld09a pls09 ON pls14.fscskey = pls09.fscskey WHERE pls14.visits >= 0 AND pls09.visits >= 0 GROUP BY pls14.stabr ➊ HAVING sum(pls14.visits) > 50000000 ORDER BY pct_change DESC; Listing 8-14: Using a HAVING clause to filter the results of an aggregate query In this case, we’ve set our query results to include only rows with a sum of visits in 2014 greater than 50 million. That’s an arbitrary value I chose to show only the very largest states. Adding the HAVING clause ➊ reduces the number of rows in the output to just six. In practice, you might experiment with various values. Here are the results: Estadísticos e-Books & Papers stabr visits_2014 visits_2009 pct_change ----- ----------- ----------- ---------- TX 72876601 78838400 -7.56 CA 162787836 182181408 -10.65 OH 82495138 92402369 -10.72 NY 106453546 119810969 -11.15 IL 72598213 82438755 -11.94 FL 73165352 87730886 -16.60 Each of the six states has experienced a decline in visits, but notice that the percent-change variation isn’t as wide as in the full set of states and territories. Depending on what we learn from library experts, looking at the states with the most activity as a group might be helpful in describing trends, as would looking at other groupings. Think of a sentence or bullet point you might write that would say, “In the nation’s largest states, visits decreased between 8 percent and 17 percent between 2009 and 2014.” You", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 109 + }, + { + "text": "with the most activity as a group might be helpful in describing trends, as would looking at other groupings. Think of a sentence or bullet point you might write that would say, “In the nation’s largest states, visits decreased between 8 percent and 17 percent between 2009 and 2014.” You could write similar sentences about medium-sized states and small states. Wrapping Up If this chapter has inspired you to visit your local library and check out a couple of books, ask a librarian whether their branch has seen a rise or drop in visits over the last few years. Chances are, you can guess the answer now. In this chapter, you learned how to use standard SQL techniques to summarize data in a table by grouping values and using a handful of aggregate functions. By joining data sets, you were able to identify some interesting five-year trends. You also learned that data doesn’t always come perfectly packaged. The use of negative values in columns as an indicator rather than as an actual numeric value forced us to filter out those rows. Unfortunately, data sets offer those kinds of challenges more often than not. In the next chapter, you’ll learn techniques to clean up a data set that has a number of issues. In subsequent chapters, you’ll also discover more aggregate functions to help you find the stories in your data. Estadísticos e-Books & Papers TRY IT YOURSELF Put your grouping and aggregating skills to the test with these challenges: 1. We saw that library visits have declined recently in most places. But what is the pattern in the use of technology in libraries? Both the 2014 and 2009 library survey tables contain the columns gpterms (the number of internet-connected computers used by the public) and pitusr (uses of public internet computers per year). Modify the code in Listing 8-13 to calculate the percent change in the sum of each column over time. Watch out for negative values! 2. Both library survey tables contain a column called obereg, a two- digit Bureau of Economic Analysis Code that classifies each library agency according to a region of the United States, such as New England, Rocky Mountains, and so on. Just as we calculated the percent change in visits grouped by state, do the same to group percent changes in visits by U.S. region using obereg. Consult the survey documentation to find the meaning of each region code. For a bonus challenge, create a table with the obereg code as the primary key and the region name as text, and join it to the summary query to group by the region name rather than the code. 3. Thinking back to the types of joins you learned in Chapter 6, which join type will show you all the rows in both tables, including those without a match? Write such a query and add an IS NULL filter in a WHERE clause to show agencies not included in one or the other table. Estadísticos e-Books", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 110 + }, + { + "text": "joins you learned in Chapter 6, which join type will show you all the rows in both tables, including those without a match? Write such a query and add an IS NULL filter in a WHERE clause to show agencies not included in one or the other table. Estadísticos e-Books & Papers 9 INSPECTING AND MODIFYING DATA If you asked me to propose a toast to a newly minted class of data analysts, I’d probably raise my glass and say, “May your data always be free of errors and may it always arrive perfectly structured!” Life would be ideal if these sentiments were feasible. In reality, you’ll sometimes receive data in such a sorry state that it’s hard to analyze without modifying it in some way. This is called dirty data, which is a general label for data with errors, missing values, or poor organization that makes standard queries ineffective. When data is converted from one file type to another or when a column receives the wrong data type, information can be lost. Typos and spelling inconsistencies can also result in dirty data. Whatever the cause may be, dirty data is the bane of the data analyst. In this chapter, you’ll use SQL to clean up dirty data as well as perform other useful maintenance tasks. You’ll learn how to examine data to assess its quality and how to modify data and tables to make analysis easier. But the techniques you’ll learn will be useful for more than just cleaning data. The ability to make changes to data and tables gives you options for updating or adding new information to your database as it becomes available, elevating your database from a static collection to a living record. Let’s begin by importing our data. Estadísticos e-Books & Papers Importing Data on Meat, Poultry, and Egg Producers For this example, we’ll use a directory of U.S. meat, poultry, and egg producers. The Food Safety and Inspection Service (FSIS), an agency within the U.S. Department of Agriculture, compiles and updates this database every month. The FSIS is responsible for inspecting animals and food at more than 6,000 meat processing plants, slaughterhouses, farms, and the like. If inspectors find a problem, such as bacterial contamination or mislabeled food, the agency can issue a recall. Anyone interested in agriculture business, food supply chain, or outbreaks of foodborne illnesses will find the directory useful. Read more about the agency on its site at https://www.fsis.usda.gov/. The file we’ll use comes from the directory’s page on https://www.data.gov/, a website run by the U.S. federal government that catalogs thousands of data sets from various federal agencies (https://catalog.data.gov/dataset/meat-poultry-and-egg-inspection-directory-by- establishment-name/). We’ll examine the original data as it was available for download, with the exception of the ZIP Codes column (I’ll explain why later). You’ll find the data in the file MPI_Directory_by_Establishment_Name.csv along with other resources for this book at https://www.nostarch.com/practicalSQL/. To import the file into PostgreSQL, use the code in Listing 9-1 to create a table called meat_poultry_egg_inspect and use COPY", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 111 + }, + { + "text": "with the exception of the ZIP Codes column (I’ll explain why later). You’ll find the data in the file MPI_Directory_by_Establishment_Name.csv along with other resources for this book at https://www.nostarch.com/practicalSQL/. To import the file into PostgreSQL, use the code in Listing 9-1 to create a table called meat_poultry_egg_inspect and use COPY to add the CSV file to the table. As in previous examples, use pgAdmin to connect to your analysis database, and then open the Query Tool to run the code. Remember to change the path in the COPY statement to reflect the location of your CSV file. CREATE TABLE meat_poultry_egg_inspect ( ➊ est_number varchar(50) CONSTRAINT est_number_key PRIMARY KEY, company varchar(100), street varchar(100), city varchar(30), st varchar(2), zip varchar(5), phone varchar(14), grant_date date, ➋ activities text, Estadísticos e-Books & Papers dbas text ); ➌ COPY meat_poultry_egg_inspect FROM 'C:\\YourDirectory\\MPI_Directory_by_Establishment_Name.csv' WITH (FORMAT CSV, HEADER, DELIMITER ','); ➍ CREATE INDEX company_idx ON meat_poultry_egg_inspect (company); Listing 9-1: Importing the FSIS Meat, Poultry, and Egg Inspection Directory The meat_poultry_egg_inspect table has 10 columns. We add a natural primary key constraint to the est_number column ➊, which contains a unique value for each row that identifies the establishment. Most of the remaining columns relate to the company’s name and location. You’ll use the activities column ➋, which describes activities at the company, in the “Try It Yourself” exercise at the end of this chapter. We set the activities and dbas columns to text, a data type that in PostgreSQL affords us up to 1GB of characters, because some of the strings in the columns are thousands of characters long. We import the CSV file ➌ and then create an index on the company column ➍ to speed up searches for particular companies. For practice, let’s use the count() aggregate function introduced in Chapter 8 to check how many rows are in the meat_poultry_egg_inspect table: SELECT count(*) FROM meat_poultry_egg_inspect; The result should show 6,287 rows. Now let’s find out what the data contains and determine whether we can glean useful information from it as is, or if we need to modify it in some way. Interviewing the Data Set Interviewing data is my favorite part of analysis. We interview a data set to discover its details: what it holds, what questions it can answer, and how suitable it is for our purposes, the same way a job interview reveals whether a candidate has the skills required for the position. Estadísticos e-Books & Papers The aggregate queries you learned in Chapter 8 are a useful interviewing tool because they often expose the limitations of a data set or raise questions you may want to ask before drawing conclusions in your analysis and assuming the validity of your findings. For example, the meat_poultry_egg_inspect table’s rows describe food producers. At first glance, we might assume that each company in each row operates at a distinct address. But it’s never safe to assume in data analysis, so let’s check using the code in Listing 9-2: SELECT company, street, city, st, count(*) AS", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 112 + }, + { + "text": "example, the meat_poultry_egg_inspect table’s rows describe food producers. At first glance, we might assume that each company in each row operates at a distinct address. But it’s never safe to assume in data analysis, so let’s check using the code in Listing 9-2: SELECT company, street, city, st, count(*) AS address_count FROM meat_poultry_egg_inspect GROUP BY company, street, city, st HAVING count(*) > 1 ORDER BY company, street, city, st; Listing 9-2: Finding multiple companies at the same address Here, we group companies by unique combinations of the company, street, city, and st columns. Then we use count(*), which returns the number of rows for each combination of those columns and gives it the alias address_count. Using the HAVING clause introduced in Chapter 8, we filter the results to show only cases where more than one row has the same combination of values. This should return all duplicate addresses for a company. The query returns 23 rows, which means there are close to two dozen cases where the same company is listed multiple times at the same address: This is not necessarily a problem. There may be valid reasons for a company to appear multiple times at the same address. For example, two Estadísticos e-Books & Papers types of processing plants could exist with the same name. On the other hand, we may have found data entry errors. Either way, it’s sound practice to eliminate concerns about the validity of a data set before relying on it, and the result should prompt us to investigate individual cases before we draw conclusions. However, this data set has other issues that we need to look at before we can get meaningful information from it. Let’s work through a few examples. Checking for Missing Values Let’s start checking for missing values by asking a basic question: how many of the meat, poultry, and egg processing companies are in each state? Finding out whether we have values from all states and whether any rows are missing a state code will serve as another useful check on the data. We’ll use the aggregate function count() along with GROUP BY to determine this, as shown in Listing 9-3: SELECT st, count(*) AS st_count FROM meat_poultry_egg_inspect GROUP BY st ORDER BY st; Listing 9-3: Grouping and counting states The query is a simple count similar to the examples in Chapter 8. When you run the query, it tallies the number of times each state postal code (st) appears in the table. Your result should include 57 rows, grouped by the state postal code in the column st. Why more than the 50 U.S. states? Because the data includes Puerto Rico and other unincorporated U.S. territories, such as Guam and American Samoa. Alaska (AK) is at the top of the results with a count of 17 establishments: st st_count -- -------- AK 17 AL 93 AR 87 AS 1 --snip-- WA 139 Estadísticos e-Books & Papers WI 184 WV 23 WY 1 3 However, the row at the bottom", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 113 + }, + { + "text": "and American Samoa. Alaska (AK) is at the top of the results with a count of 17 establishments: st st_count -- -------- AK 17 AL 93 AR 87 AS 1 --snip-- WA 139 Estadísticos e-Books & Papers WI 184 WV 23 WY 1 3 However, the row at the bottom of the list has a count of 3 and a NULL value in the st_count column. To find out what this means, let’s query the rows where the st column has NULL values. NOTE Depending on the database implementation, NULL values will either appear first or last in a sorted column. In PostgreSQL, they appear last by default. The ANSI SQL standard doesn’t specify one or the other, but it lets you add NULLS FIRST or NULLS LAST to an ORDER BY clause to specify a preference. For example, to make NULL values appear first in the preceding query, the clause would read ORDER BY st NULLS FIRST. In Listing 9-4, we use the technique covered in “Using NULL to Find Rows with Missing Values” on page 83, adding a WHERE clause with the st column and the IS NULL keywords to find which rows are missing a state code: SELECT est_number, company, city, st, zip FROM meat_poultry_egg_inspect WHERE st IS NULL; Listing 9-4: Using IS NULL to find missing values in the st column This query returns three rows that don’t have a value in the st column: Estadísticos e-Books & Papers If we want an accurate count of establishments per state, these missing values would lead to an incorrect result. To find the source of this dirty data, it’s worth making a quick visual check of the original file downloaded from https://www.data.gov/. Unless you’re working with files in the gigabyte range, you can usually open a CSV file in a text editor and search for the row. If you’re working with larger files, you might be able to examine the source data using utilities such as grep (on Linux and macOS) or findstr (on Windows). In this case, a visual check confirms that, indeed, there was no state listed in those rows in the CSV file, so the error is organic to the data, not one introduced during import. In our interview of the data so far, we’ve discovered that we’ll need to add missing values to the st column to clean up this table. Let’s look at what other issues exist in our data set and make a list of cleanup tasks. Checking for Inconsistent Data Values Inconsistent data is another factor that can hamper our analysis. We can check for inconsistently entered data within a column by using GROUP BY with count(). When you scan the unduplicated values in the results, you might be able to spot variations in the spelling of names or other attributes. For example, many of the 6,200 companies in our table are multiple locations owned by a few multinational food corporations, such as Cargill or Tyson Foods. To find out how", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 114 + }, + { + "text": "unduplicated values in the results, you might be able to spot variations in the spelling of names or other attributes. For example, many of the 6,200 companies in our table are multiple locations owned by a few multinational food corporations, such as Cargill or Tyson Foods. To find out how many locations each company owns, we would try to count the values in the company column. Let’s see what happens when we do, using the query in Listing 9-5: SELECT company, count(*) AS company_count FROM meat_poultry_egg_inspect GROUP BY company ORDER BY company ASC; Listing 9-5: Using GROUP BY and count() to find inconsistent company names Scrolling through the results reveals a number of cases in which a company’s name is spelled several different ways. For example, notice the Estadísticos e-Books & Papers entries for the Armour-Eckrich brand: company company_count --------------------------- ------------- --snip-- Armour - Eckrich Meats, LLC 1 Armour-Eckrich Meats LLC 3 Armour-Eckrich Meats, Inc. 1 Armour-Eckrich Meats, LLC 2 --snip-- At least four different spellings are shown for seven establishments that are likely owned by the same company. If we later perform any aggregation by company, it would help to standardize the names so all of the items counted or summed are grouped properly. Let’s add that to our list of items to fix. Checking for Malformed Values Using length() It’s a good idea to check for unexpected values in a column that should be consistently formatted. For example, each entry in the zip column in the meat_poultry_egg_inspect table should be formatted in the style of U.S. ZIP Codes with five digits. However, that’s not what is in our data set. Solely for the purpose of this example, I replicated an error I’ve committed before. When I converted the original Excel file to a CSV file, I stored the ZIP Code in the “General” number format in the spreadsheet instead of as a text value. By doing so, any ZIP Code that begins with a zero, such as 07502 for Paterson, NJ, lost the leading zero because an integer can’t start with a zero. As a result, 07502 appears in the table as 7502. You can make this error in a variety of ways, including by copying and pasting data into Excel columns set to “General.” After being burned a few times, I learned to take extra caution with numbers that should be formatted as text. My deliberate error appears when we run the code in Listing 9-6. The example introduces length(), a string function that counts the number of characters in a string. We combine length() with count() and GROUP BY to determine how many rows have five characters in the zip field and how Estadísticos e-Books & Papers many have a value other than five. To make it easy to scan the results, we use length() in the ORDER BY clause. SELECT length(zip), count(*) AS length_count FROM meat_poultry_egg_inspect GROUP BY length(zip) ORDER BY length(zip) ASC; Listing 9-6: Using length() and count() to test the zip column The", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 115 + }, + { + "text": "Papers many have a value other than five. To make it easy to scan the results, we use length() in the ORDER BY clause. SELECT length(zip), count(*) AS length_count FROM meat_poultry_egg_inspect GROUP BY length(zip) ORDER BY length(zip) ASC; Listing 9-6: Using length() and count() to test the zip column The results confirm the formatting error. As you can see, 496 of the ZIP Codes are four characters long, and 86 are three characters long, which means these numbers originally had two leading zeros that my conversion erroneously eliminated: length length_count ------ ------------ 3 86 4 496 5 5705 Using the WHERE clause, we can check the details of the results to see which states these shortened ZIP Codes correspond to, as shown in Listing 9-7: SELECT st, count(*) AS st_count FROM meat_poultry_egg_inspect ➊ WHERE length(zip) < 5 GROUP BY st ORDER BY st ASC; Listing 9-7: Filtering with length() to find short zip values The length() function inside the WHERE clause ➊ returns a count of rows where the ZIP Code is less than five characters for each state code. The result is what we would expect. The states are largely in the Northeast region of the United States where ZIP Codes often start with a zero: st st_count -- -------- CT 55 MA 101 ME 24 Estadísticos e-Books & Papers NH 18 NJ 244 PR 84 RI 27 VI 2 VT 27 Obviously, we don’t want this error to persist, so we’ll add it to our list of items to correct. So far, we need to correct the following issues in our data set: Missing values for three rows in the st column Inconsistent spelling of at least one company’s name Inaccurate ZIP Codes due to file conversion Next, we’ll look at how to use SQL to fix these issues by modifying your data. Modifying Tables, Columns, and Data Almost nothing in a database, from tables to columns and the data types and values they contain, is set in concrete after it’s created. As your needs change, you can add columns to a table, change data types on existing columns, and edit values. Fortunately, you can use SQL to modify, delete, or add to existing data and structures. Given the issues we discovered in the meat_poultry_egg_inspect table, being able to modify our database will come in handy. To make changes to our database, we’ll use two SQL commands: the first command, ALTER TABLE, is part of the ANSI SQL standard and provides options to ADD COLUMN, ALTER COLUMN, and DROP COLUMN, among others. Typically, PostgreSQL and other databases include implementation-specific extensions to ALTER TABLE that provide an array of options for managing database objects (see https://www.postgresql.org/docs/current/static/sql- altertable.html). For our exercises, we’ll stick with the core options. The second command, UPDATE, also included in the SQL standard, allows you to change values in a table’s columns. You can supply criteria Estadísticos e-Books & Papers using the WHERE clause to choose which rows to update. Let’s explore the basic syntax and", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 116 + }, + { + "text": "exercises, we’ll stick with the core options. The second command, UPDATE, also included in the SQL standard, allows you to change values in a table’s columns. You can supply criteria Estadísticos e-Books & Papers using the WHERE clause to choose which rows to update. Let’s explore the basic syntax and options for both commands, and then use them to fix the issues in our data set. WHEN TO TOSS YOUR DATA If your interview of the data reveals too many missing values or values that defy common sense—such as numbers ranging in the billions when you expected thousands—it’s time to reevaluate its use. The data may not be reliable enough to serve as the foundation of your analysis. If you suspect as much, the first step is to revisit the original data file. Make sure you imported it correctly and that values in all the source columns are located in the same columns in the table. You might need to open the original spreadsheet or CSV file and do a visual comparison. The second step is to call the agency or company that produced the data to confirm what you see and seek an explanation. You might also ask for advice from others who have used the same data. More than once I’ve had to toss a data set after determining that it was poorly assembled or simply incomplete. Sometimes, the amount of work required to make a data set usable undermines its usefulness. These situations require you to make a tough judgment call. But it’s better to start over or find an alternative than to use bad data that can lead to faulty conclusions. Modifying Tables with ALTER TABLE Estadísticos e-Books & Papers We can use the ALTER TABLE statement to modify the structure of tables. The following examples show the syntax for common operations that are part of standard ANSI SQL. The code for adding a column to a table looks like this: ALTER TABLE table ADD COLUMN column data_type; Similarly, we can remove a column with the following syntax: ALTER TABLE table DROP COLUMN column; To change the data type of a column, we would use this code: ALTER TABLE table ALTER COLUMN column SET DATA TYPE data_type; Adding a NOT NULL constraint to a column will look like the following: ALTER TABLE table ALTER COLUMN column SET NOT NULL; Note that in PostgreSQL and some other systems, adding a constraint to the table causes all rows to be checked to see whether they comply with the constraint. If the table has millions of rows, this could take a while. Removing the NOT NULL constraint looks like this: ALTER TABLE table ALTER COLUMN column DROP NOT NULL; When you execute an ALTER TABLE statement with the placeholders filled in, you should see a message that reads ALTER TABLE in the pgAdmin output screen. If an operation violates a constraint or if you attempt to change a column’s data type and the existing values in the column won’t", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 117 + }, + { + "text": "When you execute an ALTER TABLE statement with the placeholders filled in, you should see a message that reads ALTER TABLE in the pgAdmin output screen. If an operation violates a constraint or if you attempt to change a column’s data type and the existing values in the column won’t conform to the new data type, PostgreSQL returns an error. But PostgreSQL won’t give you any warning about deleting data when you drop a column, so use extra caution before dropping a column. Modifying Values with UPDATE The UPDATE statement modifies the data in a column in all rows or in a Estadísticos e-Books & Papers subset of rows that meet a condition. Its basic syntax, which would update the data in every row in a column, follows this form: UPDATE table SET column = value; We first pass UPDATE the name of the table to update, and then pass the SET clause the column that contains the values to change. The new value to place in the column can be a string, number, the name of another column, or even a query or expression that generates a value. We can update values in multiple columns at a time by adding additional columns and source values, and separating each column and value statement with a comma: UPDATE table SET column_a = value, column_b = value; To restrict the update to particular rows, we add a WHERE clause with some criteria that must be met before the update can happen: UPDATE table SET column = value WHERE criteria; We can also update one table with values from another table. Standard ANSI SQL requires that we use a subquery, a query inside a query, to specify which values and rows to update: UPDATE table SET column = (SELECT column FROM table_b WHERE table.column = table_b.column) WHERE EXISTS (SELECT column FROM table_b WHERE table.column = table_b.column); The value portion of the SET clause is a subquery, which is a SELECT statement inside parentheses that generates the values for the update. Similarly, the WHERE EXISTS clause uses a SELECT statement to generate values that serve as the filter for the update. If we didn’t use this clause, we Estadísticos e-Books & Papers might inadvertently set some values to NULL without planning to. (If this syntax looks somewhat complicated, that’s okay. I’ll cover subqueries in detail in Chapter 12.) Some database managers offer additional syntax for updating across tables. PostgreSQL supports the ANSI standard but also a simpler syntax using a FROM clause for updating values across tables: UPDATE table SET column = table_b.column FROM table_b WHERE table.column = table_b.column; When you execute an UPDATE statement, PostgreSQL returns a message stating UPDATE along with the number of rows affected. Creating Backup Tables Before modifying a table, it’s a good idea to make a copy for reference and backup in case you accidentally destroy some data. Listing 9-8 shows how to use a variation of the familiar CREATE TABLE statement to make a new table based", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 118 + }, + { + "text": "number of rows affected. Creating Backup Tables Before modifying a table, it’s a good idea to make a copy for reference and backup in case you accidentally destroy some data. Listing 9-8 shows how to use a variation of the familiar CREATE TABLE statement to make a new table based on the existing data and structure of the table we want to duplicate: CREATE TABLE meat_poultry_egg_inspect_backup AS SELECT * FROM meat_poultry_egg_inspect; Listing 9-8: Backing up a table After running the CREATE TABLE statement, the result should be a pristine copy of your table with the new specified name. You can confirm this by counting the number of records in both tables with one query: SELECT (SELECT count(*) FROM meat_poultry_egg_inspect) AS original, (SELECT count(*) FROM meat_poultry_egg_inspect_backup) AS backup; The results should return a count of 6,287 from both tables, like this: original backup -------- ------ 6287 6287 Estadísticos e-Books & Papers If the counts match, you can be sure your backup table is an exact copy of the structure and contents of the original table. As an added measure and for easy reference, we’ll use ALTER TABLE to make copies of column data within the table we’re updating. NOTE Indexes are not copied when creating a table backup using the CREATE TABLE statement. If you decide to run queries on the backup, be sure to create a separate index on that table. Restoring Missing Column Values Earlier in this chapter, the query in Listing 9-4 revealed that three rows in the meat_poultry_egg_inspect table don’t have a value in the st column: To get a complete count of establishments in each state, we need to fill those missing values using an UPDATE statement. Creating a Column Copy Even though we’ve backed up this table, let’s take extra caution and make a copy of the st column within the table so we still have the original data if we make some dire error somewhere! Let’s create the copy and fill it with the existing st column values using the SQL statements in Listing 9- 9: ➊ ALTER TABLE meat_poultry_egg_inspect ADD COLUMN st_copy varchar(2); UPDATE meat_poultry_egg_inspect ➋ SET st_copy = st; Estadísticos e-Books & Papers Listing 9-9: Creating and filling the st_copy column with ALTER TABLE and UPDATE The ALTER TABLE statement ➊ adds a column called st_copy using the same varchar data type as the original st column. Next, the UPDATE statement’s SET clause ➋ fills our newly created st_copy column with the values in column st. Because we don’t specify any criteria using a WHERE clause, values in every row are updated, and PostgreSQL returns the message UPDATE 6287. Again, it’s worth noting that on a very large table, this operation could take some time and also substantially increase the table’s size. Making a column copy in addition to a table backup isn’t entirely necessary, but if you’re the patient, cautious type, it can be worthwhile. We can confirm the values were copied properly with a simple SELECT query on both columns,", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 119 + }, + { + "text": "take some time and also substantially increase the table’s size. Making a column copy in addition to a table backup isn’t entirely necessary, but if you’re the patient, cautious type, it can be worthwhile. We can confirm the values were copied properly with a simple SELECT query on both columns, as in Listing 9-10: SELECT st, st_copy FROM meat_poultry_egg_inspect ORDER BY st; Listing 9-10: Checking values in the st and st_copy columns The SELECT query returns 6,287 rows showing both columns holding values except the three rows with missing values: st st_copy -- ------- AK AK AK AK AK AK AK AK --snip-- Now, with our original data safely stored in the st_copy column, we can update the three rows with missing state codes. This is now our in-table backup, so if something goes drastically wrong while we’re updating the missing data in the original column, we can easily copy the original data back in. I’ll show you how after we apply the first updates. Updating Rows Where Values Are Missing Estadísticos e-Books & Papers To update those rows missing values, we first find the values we need with a quick online search: Atlas Inspection is located in Minnesota; Hall- Namie Packing is in Alabama; and Jones Dairy is in Wisconsin. Add those states to the appropriate rows using the code in Listing 9-11: UPDATE meat_poultry_egg_inspect SET st = 'MN' ➊ WHERE est_number = 'V18677A'; UPDATE meat_poultry_egg_inspect SET st = 'AL' WHERE est_number = 'M45319+P45319'; UPDATE meat_poultry_egg_inspect SET st = 'WI' WHERE est_number = 'M263A+P263A+V263A'; Listing 9-11: Updating the st column for three establishments Because we want each UPDATE statement to affect a single row, we include a WHERE clause ➊ for each that identifies the company’s unique est_number, which is the table’s primary key. When we run each query, PostgreSQL responds with the message UPDATE 1, showing that only one row was updated for each query. If we rerun the code in Listing 9-4 to find rows where st is NULL, the query should return nothing. Success! Our count of establishments by state is now complete. Restoring Original Values What happens if we botch an update by providing the wrong values or updating the wrong rows? Because we’ve backed up the entire table and the st column within the table, we can easily copy the data back from either location. Listing 9-12 shows the two options. ➊ UPDATE meat_poultry_egg_inspect SET st = st_copy; ➋ UPDATE meat_poultry_egg_inspect original SET st = backup.st FROM meat_poultry_egg_inspect_backup backup WHERE original.est_number = backup.est_number; Estadísticos e-Books & Papers Listing 9-12: Restoring original st column values To restore the values from the backup column in meat_poultry_egg_inspect you created in Listing 9-9, run an UPDATE query ➊ that sets st to the values in st_copy. Both columns should again have the identical original values. Alternatively, you can create an UPDATE ➋ that sets st to values in the st column from the meat_poultry_egg_inspect_backup table you made in Listing 9-8. Updating Values for Consistency In Listing 9-5 we", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 120 + }, + { + "text": "that sets st to the values in st_copy. Both columns should again have the identical original values. Alternatively, you can create an UPDATE ➋ that sets st to values in the st column from the meat_poultry_egg_inspect_backup table you made in Listing 9-8. Updating Values for Consistency In Listing 9-5 we discovered several cases where a single company’s name was entered inconsistently. If we want to aggregate data by company name, such inconsistencies will hinder us from doing so. Here are the spelling variations of Armour-Eckrich Meats in Listing 9-5: --snip-- Armour - Eckrich Meats, LLC Armour-Eckrich Meats LLC Armour-Eckrich Meats, Inc. Armour-Eckrich Meats, LLC --snip-- We can standardize the spelling of this company’s name by using an UPDATE statement. To protect our data, we’ll create a new column for the standardized spellings, copy the names in company into the new column, and work in the new column to avoid tampering with the original data. Listing 9-13 has the code for both actions: ALTER TABLE meat_poultry_egg_inspect ADD COLUMN company_standard varchar(100); UPDATE meat_poultry_egg_inspect SET company_standard = company; Listing 9-13: Creating and filling the company_standard column Now, let’s say we want any name in company that contains the string Armour to appear in company_standard as Armour-Eckrich Meats. (This assumes we’ve checked all entries containing Armour and want to standardize Estadísticos e-Books & Papers them.) We can update all the rows matching the string Armour by using a WHERE clause. Run the two statements in Listing 9-14: UPDATE meat_poultry_egg_inspect SET company_standard = 'Armour-Eckrich Meats' ➊ WHERE company LIKE 'Armour%'; SELECT company, company_standard FROM meat_poultry_egg_inspect WHERE company LIKE 'Armour%'; Listing 9-14: Using an UPDATE statement to modify field values that match a string The important piece of this query is the WHERE clause that uses the LIKE keyword ➊ that was introduced with filtering in Chapter 2. Including the wildcard syntax % at the end of the string Armour updates all rows that start with those characters regardless of what comes after them. The clause lets us target all the varied spellings used for the company’s name. The SELECT statement in Listing 9-14 returns the results of the updated company_standard column next to the original company column: company company_standard --------------------------- -------------------- Armour-Eckrich Meats LLC Armour-Eckrich Meats Armour - Eckrich Meats, LLC Armour-Eckrich Meats Armour-Eckrich Meats LLC Armour-Eckrich Meats Armour-Eckrich Meats LLC Armour-Eckrich Meats Armour-Eckrich Meats, Inc. Armour-Eckrich Meats Armour-Eckrich Meats, LLC Armour-Eckrich Meats Armour-Eckrich Meats, LLC Armour-Eckrich Meats The values for Armour-Eckrich in company_standard are now standardized with consistent spelling. If we want to standardize other company names in the table, we would create an UPDATE statement for each case. We would also keep the original company column for reference. Repairing ZIP Codes Using Concatenation Our final fix repairs values in the zip column that lost leading zeros as the result of my deliberate data faux pas. For companies in Puerto Rico and the U.S. Virgin Islands, we need to restore two leading zeros to the values in zip because (aside from an IRS processing", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 121 + }, + { + "text": "Our final fix repairs values in the zip column that lost leading zeros as the result of my deliberate data faux pas. For companies in Puerto Rico and the U.S. Virgin Islands, we need to restore two leading zeros to the values in zip because (aside from an IRS processing facility in Holtsville, NY) Estadísticos e-Books & Papers they’re the only locations in the United States where ZIP Codes start with two zeros. Then, for the other states, located mostly in New England, we’ll restore a single leading zero. We’ll use UPDATE again but this time in conjunction with the double- pipe string operator (||), which performs concatenation. Concatenation combines two or more string or non-string values into one. For example, inserting || between the strings abc and 123 results in abc123. The double- pipe operator is a SQL standard for concatenation supported by PostgreSQL. You can use it in many contexts, such as UPDATE queries and SELECT, to provide custom output from existing as well as new data. First, Listing 9-15 makes a backup copy of the zip column in the same way we made a backup of the st column earlier: ALTER TABLE meat_poultry_egg_inspect ADD COLUMN zip_copy varchar(5); UPDATE meat_poultry_egg_inspect SET zip_copy = zip; Listing 9-15: Creating and filling the zip_copy column Next, we use the code in Listing 9-16 to perform the first update: UPDATE meat_poultry_egg_inspect ➊ SET zip = '00' || zip ➋ WHERE st IN('PR','VI') AND length(zip) = 3; Listing 9-16: Modifying codes in the zip column missing two leading zeros We use SET to set the zip column ➊ to a value that is the result of the concatenation of the string 00 and the existing content of the zip column. We limit the UPDATE to only those rows where the st column has the state codes PR and VI ➋ using the IN comparison operator from Chapter 2 and add a test for rows where the length of zip is 3. This entire statement will then only update the zip values for Puerto Rico and the Virgin Islands. Run the query; PostgreSQL should return the message UPDATE 86, which is the number of rows we expect to change based on our earlier count in Listing 9-6. Let’s repair the remaining ZIP Codes using a similar query in Listing Estadísticos e-Books & Papers 9-17: UPDATE meat_poultry_egg_inspect SET zip = '0' || zip WHERE st IN('CT','MA','ME','NH','NJ','RI','VT') AND length(zip) = 4; Listing 9-17: Modifying codes in the zip column missing one leading zero PostgreSQL should return the message UPDATE 496. Now, let’s check our progress. Earlier in the chapter, when we aggregated rows in the zip column by length, we found 86 rows with three characters and 496 with four: length count ------ ----- 3 86 4 496 5 5705 Using the same query in Listing 9-6 now returns a more desirable result: all the rows have a five-digit ZIP Code. length count ------ ----- 5 6287 In this example we used concatenation, but you can", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 122 + }, + { + "text": "and 496 with four: length count ------ ----- 3 86 4 496 5 5705 Using the same query in Listing 9-6 now returns a more desirable result: all the rows have a five-digit ZIP Code. length count ------ ----- 5 6287 In this example we used concatenation, but you can employ additional SQL string functions to modify data with UPDATE by changing words from uppercase to lowercase, trimming unwanted spaces, replacing characters in a string, and more. I’ll discuss additional string functions in Chapter 13 when we consider advanced techniques for working with text. Updating Values Across Tables In “Modifying Values with UPDATE” on page 138, I showed the standard ANSI SQL and PostgreSQL-specific syntax for updating values in one table based on values in another. This syntax is particularly valuable in a relational database where primary keys and foreign keys establish table relationships. It’s also useful when data in one table may be necessary context for updating values in another. Estadísticos e-Books & Papers For example, let’s say we’re setting an inspection date for each of the companies in our table. We want to do this by U.S. regions, such as Northeast, Pacific, and so on, but those regional designations don’t exist in our table. However, they do exist in a data set we can add to our database that also contains matching st state codes. This means we can use that other data as part of our UPDATE statement to provide the necessary information. Let’s begin with the New England region to see how this works. Enter the code in Listing 9-18, which contains the SQL statements to create a state_regions table and fill the table with data: CREATE TABLE state_regions ( st varchar(2) CONSTRAINT st_key PRIMARY KEY, region varchar(20) NOT NULL ); COPY state_regions FROM 'C:\\YourDirectory\\state_regions.csv' WITH (FORMAT CSV, HEADER, DELIMITER ','); Listing 9-18: Creating and filling a state_regions table We’ll create two columns in a state_regions table: one containing the two-character state code st and the other containing the region name. We set the primary key constraint to the st column, which holds a unique st_key value to identify each state. In the data you’re importing, each state is present and assigned to a U.S. Census region, and territories outside the United States are labeled as outlying areas. We’ll update the table one region at a time. Next, let’s return to the meat_poultry_egg_inspect table, add a column for inspection dates, and then fill in that column with the New England states. Listing 9-19 shows the code: ALTER TABLE meat_poultry_egg_inspect ADD COLUMN inspection_date date; ➊ UPDATE meat_poultry_egg_inspect inspect ➋ SET inspection_date = '2019-12-01' ➌ WHERE EXISTS (SELECT state_regions.region FROM state_regions WHERE inspect.st = state_regions.st AND state_regions.region = 'New England'); Estadísticos e-Books & Papers Listing 9-19: Adding and updating an inspection_date column The ALTER TABLE statement creates the inspection_date column in the meat_poultry_egg_inspect table. In the UPDATE statement, we start by naming the table using an alias of inspect to make the code easier to read ➊. Next,", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 123 + }, + { + "text": "'New England'); Estadísticos e-Books & Papers Listing 9-19: Adding and updating an inspection_date column The ALTER TABLE statement creates the inspection_date column in the meat_poultry_egg_inspect table. In the UPDATE statement, we start by naming the table using an alias of inspect to make the code easier to read ➊. Next, the SET clause assigns a date value of 2019-12-01 to the new inspection_date column ➋. Finally, the WHERE EXISTS clause includes a subquery that connects the meat_poultry_egg_inspect table to the state_regions table we created in Listing 9-18 and specifies which rows to update ➌. The subquery (in parentheses, beginning with SELECT) looks for rows in the state_regions table where the region column matches the string New England. At the same time, it joins the meat_poultry_egg_inspect table with the state_regions table using the st column from both tables. In effect, the query is telling the database to find all the st codes that correspond to the New England region and use those codes to filter the update. When you run the code, you should receive a message of UPDATE 252, which is the number of companies in New England. You can use the code in Listing 9-20 to see the effect of the change: SELECT st, inspection_date FROM meat_poultry_egg_inspect GROUP BY st, inspection_date ORDER BY st; Listing 9-20: Viewing updated inspection_date values The results should show the updated inspection dates for all New England companies. The top of the output shows Connecticut has received a date, for example, but states outside New England remain NULL because we haven’t updated them yet: st inspection_date -- --------------- --snip-- CA CO CT 2019-12-01 DC --snip-- Estadísticos e-Books & Papers To fill in dates for additional regions, substitute a different region for New England in Listing 9-19 and rerun the query. Deleting Unnecessary Data The most irrevocable way to modify data is to remove it entirely. SQL includes options to remove rows and columns from a table along with options to delete an entire table or database. We want to perform these operations with caution, removing only data or tables we don’t need. Without a backup, the data is gone for good. NOTE It’s easy to exclude unwanted data in queries using a WHERE clause, so decide whether you truly need to delete the data or can just filter it out. Cases where deleting may be the best solution include data with errors or data imported incorrectly. In this section, we’ll use a variety of SQL statements to delete unnecessary data. For removing rows from a table, we’ll use the DELETE FROM statement. To remove a column from a table, we’ll use ALTER TABLE. And to remove a whole table from the database, we’ll use the DROP TABLE statement. Writing and executing these statements is fairly simple, but doing so comes with a caveat. If deleting rows, a column, or a table would cause a violation of a constraint, such as the foreign key constraint covered in Chapter 7, you need to deal with that", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 124 + }, + { + "text": "the DROP TABLE statement. Writing and executing these statements is fairly simple, but doing so comes with a caveat. If deleting rows, a column, or a table would cause a violation of a constraint, such as the foreign key constraint covered in Chapter 7, you need to deal with that constraint first. That might involve removing the constraint, deleting data in another table, or deleting another table. Each case is unique and will require a different way to work around the constraint. Deleting Rows from a Table Using a DELETE FROM statement, we can remove all rows from a table, or we can use a WHERE clause to delete only the portion that matches an Estadísticos e-Books & Papers expression we supply. To delete all rows from a table, use the following syntax: DELETE FROM table_name; If your table has a large number of rows, it might be faster to erase the table and create a fresh version using the original CREATE TABLE statement. To erase the table, use the DROP TABLE command discussed in “Deleting a Table from a Database” on page 148. To remove only selected rows, add a WHERE clause along with the matching value or pattern to specify which ones you want to delete: DELETE FROM table_name WHERE expression; For example, if we want our table of meat, poultry, and egg processors to include only establishments in the 50 U.S. states, we can remove the companies in Puerto Rico and the Virgin Islands from the table using the code in Listing 9-21: DELETE FROM meat_poultry_egg_inspect WHERE st IN('PR','VI'); Listing 9-21: Deleting rows matching an expression Run the code; PostgreSQL should return the message DELETE 86. This means the 86 rows where the st column held either PR or VI have been removed from the table. Deleting a Column from a Table While working on the zip column in the meat_poultry_egg_inspect table earlier in this chapter, we created a backup column called zip_copy. Now that we’ve finished working on fixing the issues in zip, we no longer need zip_copy. We can remove the backup column, including all the data within the column, from the table by using the DROP keyword in the ALTER TABLE statement. The syntax for removing a column is similar to other ALTER TABLE Estadísticos e-Books & Papers statements: ALTER TABLE table_name DROP COLUMN column_name; The code in Listing 9-22 removes the zip_copy column: ALTER TABLE meat_poultry_egg_inspect DROP COLUMN zip_copy; Listing 9-22: Removing a column from a table using DROP PostgreSQL returns the message ALTER TABLE, and the zip_copy column should be deleted. Deleting a Table from a Database The DROP TABLE statement is a standard ANSI SQL feature that deletes a table from the database. This statement might come in handy if, for example, you have a collection of backups, or working tables, that have outlived their usefulness. It’s also useful in other situations, such as when you need to change the structure of a table significantly; in that case, rather than using", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 125 + }, + { + "text": "the database. This statement might come in handy if, for example, you have a collection of backups, or working tables, that have outlived their usefulness. It’s also useful in other situations, such as when you need to change the structure of a table significantly; in that case, rather than using too many ALTER TABLE statements, you can just remove the table and create another one by running a new CREATE TABLE statement. The syntax for the DROP TABLE command is simple: DROP TABLE table_name; For example, Listing 9-23 deletes the backup version of the meat_poultry_egg_inspect table: DROP TABLE meat_poultry_egg_inspect_backup; Listing 9-23: Removing a table from a database using DROP Run the query; PostgreSQL should respond with the message DROP TABLE to indicate the table has been removed. Using Transaction Blocks to Save or Revert Changes Estadísticos e-Books & Papers The alterations you made on data using the techniques in this chapter so far are final. That is, after you run a DELETE or UPDATE query (or any other query that alters your data or database structure), the only way to undo the change is to restore from a backup. However, you can check your changes before finalizing them and cancel the change if it’s not what you intended. You do this by wrapping the SQL statement within a transaction block, which is a group of statements you define using the following keywords at the beginning and end of the query: START TRANSACTION signals the start of the transaction block. In PostgreSQL, you can also use the non-ANSI SQL BEGIN keyword. COMMIT signals the end of the block and saves all changes. ROLLBACK signals the end of the block and reverts all changes. Usually, database programmers employ a transaction block to define the start and end of a sequence of operations that perform one unit of work in a database. An example is when you purchase tickets to a Broadway show. A successful transaction might involve two steps: charging your credit card and reserving your seats so someone else can’t buy them. A database programmer would either want both steps in the transaction to happen (say, when your card charge goes through) or neither of them to happen (if your card is declined or you cancel at checkout). Defining both steps as one transaction keeps them as a unit; if one step fails, the other is canceled too. You can learn more details about transactions and PostgreSQL at https://www.postgresql.org/docs/current/static/tutorial-transactions.html. We can apply this transaction block technique to review changes a query makes and then decide whether to keep or discard them. Using the meat_poultry_egg_inspect table, let’s say we’re cleaning dirty data related to the company AGRO Merchants Oakland LLC. The table has three rows listing the company, but one row has an extra comma in the name: company --------------------------- AGRO Merchants Oakland LLC AGRO Merchants Oakland LLC Estadísticos e-Books & Papers AGRO Merchants Oakland, LLC We want the name to be consistent, so we’ll remove the comma from the third row", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 126 + }, + { + "text": "has three rows listing the company, but one row has an extra comma in the name: company --------------------------- AGRO Merchants Oakland LLC AGRO Merchants Oakland LLC Estadísticos e-Books & Papers AGRO Merchants Oakland, LLC We want the name to be consistent, so we’ll remove the comma from the third row using an UPDATE query, as we did earlier. But this time we’ll check the result of our update before we make it final (and we’ll purposely make a mistake we want to discard). Listing 9-24 shows how to do this using a transaction block: ➊ START TRANSACTION; UPDATE meat_poultry_egg_inspect ➋ SET company = 'AGRO Merchantss Oakland LLC' WHERE company = 'AGRO Merchants Oakland, LLC'; ➌ SELECT company FROM meat_poultry_egg_inspect WHERE company LIKE 'AGRO%' ORDER BY company; ➍ ROLLBACK; Listing 9-24: Demonstrating a transaction block We’ll run each statement separately, beginning with START TRANSACTION; ➊. The database responds with the message START TRANSACTION, letting you know that any succeeding changes you make to data will not be made permanent unless you issue a COMMIT command. Next, we run the UPDATE statement, which changes the company name in the row where it has an extra comma. I intentionally added an extra s in the name used in the SET clause ➋ to introduce a mistake. When we view the names of companies starting with the letters AGRO using the SELECT statement ➌, we see that, oops, one company name is misspelled now: company --------------------------- AGRO Merchants Oakland LLC AGRO Merchants Oakland LLC AGRO Merchantss Oakland LLC Instead of rerunning the UPDATE statement to fix the typo, we can simply discard the change by running the ROLLBACK; ➍ command. When we rerun Estadísticos e-Books & Papers the SELECT statement to view the company names, we’re back to where we started: company --------------------------- AGRO Merchants Oakland LLC AGRO Merchants Oakland LLC AGRO Merchants Oakland, LLC From here, you could correct your UPDATE statement by removing the extra s and rerun it, beginning with the START TRANSACTION statement again. If you’re happy with the changes, run COMMIT; to make them permanent. NOTE When you start a transaction, any changes you make to the data aren’t visible to other database users until you execute COMMIT. Transaction blocks are often used in more complex database systems. Here you’ve used them to try a query and either accept or reject the changes, saving you time and headaches. Next, let’s look at another way to save time when updating lots of data. Improving Performance When Updating Large Tables Because of how PostgreSQL works internally, adding a column to a table and filling it with values can quickly inflate the table’s size. The reason is that the database creates a new version of the existing row each time a value is updated, but it doesn’t delete the old row version. (You’ll learn how to clean up these old rows when I discuss database maintenance in “Recovering Unused Space with VACUUM” on page 314.) For small data sets, the increase is", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 127 + }, + { + "text": "new version of the existing row each time a value is updated, but it doesn’t delete the old row version. (You’ll learn how to clean up these old rows when I discuss database maintenance in “Recovering Unused Space with VACUUM” on page 314.) For small data sets, the increase is negligible, but for tables with hundreds of thousands or millions of rows, the time required to update rows and the resulting extra disk usage can be substantial. Instead of adding a column and filling it with values, we can save disk space by copying the entire table and adding a populated column during Estadísticos e-Books & Papers the operation. Then, we rename the tables so the copy replaces the original, and the original becomes a backup. Listing 9-25 shows how to copy meat_poultry_egg_inspect into a new table while adding a populated column. To do this, first drop the meat_poultry_egg_inspect_backup table we made earlier. Then run the CREATE TABLE statement. CREATE TABLE meat_poultry_egg_inspect_backup AS ➊ SELECT *, ➋ '2018-02-07'::date AS reviewed_date FROM meat_poultry_egg_inspect; Listing 9-25: Backing up a table while adding and filling a new column The query is a modified version of the backup script in Listing 9-8. Here, in addition to selecting all the columns using the asterisk wildcard ➊, we also add a column called reviewed_date by providing a value cast as a date data type ➋ and the AS keyword. That syntax adds and fills reviewed_date, which we might use to track the last time we checked the status of each plant. Then we use Listing 9-26 to swap the table names: ➊ ALTER TABLE meat_poultry_egg_inspect RENAME TO meat_poultry_egg_inspect_temp; ➋ ALTER TABLE meat_poultry_egg_inspect_backup RENAME TO meat_poultry_egg_inspect; ➌ ALTER TABLE meat_poultry_egg_inspect_temp RENAME TO meat_poultry_egg_inspect_backup; Listing 9-26: Swapping table names using ALTER TABLE Here we use ALTER TABLE with a RENAME TO clause to change a table name. Then we use the first statement to change the original table name to one that ends with _temp ➊. The second statement renames the copy we made with Listing 9-24 to the original name of the table ➋. Finally, we rename the table that ends with _temp to the ending _backup ➌. The original table is now called meat_poultry_egg_inspect_backup, and the copy with the added column is called meat_poultry_egg_inspect. By using this process, we avoid updating rows and having the database Estadísticos e-Books & Papers inflate the size of the table. When we eventually drop the _backup table, the remaining data table is smaller and does not require cleanup. Wrapping Up Gleaning useful information from data sometimes requires modifying the data to remove inconsistencies, fix errors, and make it more suitable for supporting an accurate analysis. In this chapter you learned some useful tools to help you assess dirty data and clean it up. In a perfect world, all data sets would arrive with everything clean and complete. But such a perfect world doesn’t exist, so the ability to alter, update, and delete data is indispensable. Let me restate the important", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 128 + }, + { + "text": "some useful tools to help you assess dirty data and clean it up. In a perfect world, all data sets would arrive with everything clean and complete. But such a perfect world doesn’t exist, so the ability to alter, update, and delete data is indispensable. Let me restate the important tasks of working safely. Be sure to back up your tables before you start making changes. Make copies of your columns, too, for an extra level of protection. When I discuss database maintenance for PostgreSQL later in the book, you’ll learn how to back up entire databases. These few steps of precaution will save you a world of pain. In the next chapter, we’ll return to math to explore some of SQL’s advanced statistical functions and techniques for analysis. TRY IT YOURSELF In this exercise, you’ll turn the meat_poultry_egg_inspect table into useful information. You need to answer two questions: how many of the plants in the table process meat, and how many process poultry? The answers to these two questions lie in the activities column. Unfortunately, the column contains an assortment of text with inconsistent input. Here’s an example of the kind of text you’ll find in the activities column: Estadísticos e-Books & Papers Poultry Processing, Poultry Slaughter Meat Processing, Poultry Processing Poultry Processing, Poultry Slaughter The mishmash of text makes it impossible to perform a typical count that would allow you to group processing plants by activity. However, you can make some modifications to fix this data. Your tasks are as follows: 1. Create two new columns called meat_processing and poultry_processing in your table. Each can be of the type boolean. 2. Using UPDATE, set meat_processing = TRUE on any row where the activities column contains the text Meat Processing. Do the same update on the poultry_processing column, but this time look for the text Poultry Processing in activities. 3. Use the data from the new, updated columns to count how many plants perform each type of activity. For a bonus challenge, count how many plants perform both activities. Estadísticos e-Books & Papers 10 STATISTICAL FUNCTIONS IN SQL A SQL database isn’t usually the first tool a data analyst chooses when performing statistical analysis that requires more than just calculating sums and averages. Typically, the software of choice would be full- featured statistics packages, such as SPSS or SAS, the programming languages R or Python, or even Excel. However, standard ANSI SQL, including PostgreSQL’s implementation, offers a handful of powerful stats functions that reveal a lot about your data without having to export your data set to another program. In this chapter, we’ll explore these SQL stats functions along with guidelines on when to use them. Statistics is a vast subject worthy of its own book, so we’ll only skim the surface here. Nevertheless, you’ll learn how to apply high-level statistical concepts to help you derive meaning from your data using a new data set from the U.S. Census Bureau. You’ll also learn to use SQL to create comparisons", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 129 + }, + { + "text": "vast subject worthy of its own book, so we’ll only skim the surface here. Nevertheless, you’ll learn how to apply high-level statistical concepts to help you derive meaning from your data using a new data set from the U.S. Census Bureau. You’ll also learn to use SQL to create comparisons using rankings and rates with FBI crime data as our subject. Creating a Census Stats Table Let’s return to one of my favorite data sources, the U.S. Census Bureau. In Chapters 4 and 5, you used the 2010 Decennial Census to import data Estadísticos e-Books & Papers and perform basic math and stats. This time you’ll use county data points compiled from the 2011–2015 American Community Survey (ACS) 5- Year Estimates, a separate survey administered by the Census Bureau. Use the code in Listing 10-1 to create the table acs_2011_2015_stats and import the CSV file acs_2011_2015_stats.csv. The code and data are available with all the book’s resources at https://www.nostarch.com/practicalSQL/. Remember to change C:\\YourDirectory\\ to the location of the CSV file. CREATE TABLE acs_2011_2015_stats ( ➊ geoid varchar(14) CONSTRAINT geoid_key PRIMARY KEY, county varchar(50) NOT NULL, st varchar(20) NOT NULL, ➋ pct_travel_60_min numeric(5,3) NOT NULL, pct_bachelors_higher numeric(5,3) NOT NULL, pct_masters_higher numeric(5,3) NOT NULL, median_hh_income integer, ➌ CHECK (pct_masters_higher <= pct_bachelors_higher) ); COPY acs_2011_2015_stats FROM 'C:\\YourDirectory\\acs_2011_2015_stats.csv' WITH (FORMAT CSV, HEADER, DELIMITER ','); ➍ SELECT * FROM acs_2011_2015_stats; Listing 10-1: Creating the Census 2011–2015 ACS 5-Year stats table and import data The acs_2011_2015_stats table has seven columns. The first three columns ➊ include a unique geoid that serves as the primary key, the name of the county, and the state name st. The next four columns display the following three percentages ➋ I derived for each county from raw data in the ACS release, plus one more economic indicator: pct_travel_60_min The percentage of workers ages 16 and older who commute more than 60 minutes to work. pct_bachelors_higher The percentage of people ages 25 and older whose level of education is a bachelor’s degree or higher. (In the United States, a bachelor’s degree is usually awarded upon completing a four- year college education.) Estadísticos e-Books & Papers pct_masters_higher The percentage of people ages 25 and older whose level of education is a master’s degree or higher. (In the United States, a master’s degree is the first advanced degree earned after completing a bachelor’s degree.) median_hh_income The county’s median household income in 2015 inflation-adjusted dollars. As you learned in Chapter 5, a median value is the midpoint in an ordered set of numbers, where half the values are larger than the midpoint and half are smaller. Because averages can be skewed by a few very large or very small values, government reporting on economic data, such as income, tends to use medians. In this column, we omit the NOT NULL constraint because one county had no data reported. We include the CHECK constraint ➌ you learned in Chapter 7 to check that the figures for the bachelor’s degree are equal to or higher than those", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 130 + }, + { + "text": "such as income, tends to use medians. In this column, we omit the NOT NULL constraint because one county had no data reported. We include the CHECK constraint ➌ you learned in Chapter 7 to check that the figures for the bachelor’s degree are equal to or higher than those for the master’s degree, because in the United States, a bachelor’s degree is earned before or concurrently with a master’s degree. A county showing the opposite could indicate data imported incorrectly or a column mislabeled. Our data checks out: upon import, there are no errors showing a violation of the CHECK constraint. We use the SELECT statement ➍ to view all 3,142 rows imported, each corresponding to a county surveyed in this Census release. Next, we’ll use statistics functions in SQL to better understand the relationships among the percentages. THE DECENNIAL U.S. CENSUS VS. THE AMERICAN COMMUNITY SURVEY Each U.S. Census data product has its own methodology. The Decennial Census is a full count of the U.S. population, conducted every 10 years via a form mailed to every household in the country. One of its primary purposes is to determine the number of seats each state Estadísticos e-Books & Papers holds in the U.S. House of Representatives. In contrast, the ACS is an ongoing annual survey of about 3.5 million U.S. households. It enquires into details about income, education, employment, ancestry, and housing. Private- sector and public-sector organizations alike use ACS data to track trends and make various decisions. Currently, the Census Bureau packages ACS data into two releases: a 1-year data set that provides estimates for geographies with populations of 20,000 or more, and a 5- year data set that includes all geographies. Because it’s a survey, ACS results are estimates and have a margin of error, which I’ve omitted for brevity but which you’ll see included in a full ACS data set. Measuring Correlation with corr(Y, X) Researchers often want to understand the relationships between variables, and one such measure of relationships is correlation. In this section, we’ll use the corr(Y, X) function to measure correlation and investigate what relationship exists, if any, between the percentage of people in a county who’ve attained a bachelor’s degree and the median household income in that county. We’ll also determine whether, according to our data, a better-educated population typically equates to higher income and how strong the relationship between education level and income is if it does. First, some background. The Pearson correlation coefficient (generally denoted as r) is a measure for quantifying the strength of a linear relationship between two variables. It shows the extent to which an increase or decrease in one variable correlates to a change in another variable. The r values fall between −1 and 1. Either end of the range indicates a perfect correlation, whereas values near zero indicate a random distribution with no correlation. A positive r value indicates a Estadísticos e-Books & Papers direct relationship: as one variable increases, the other does too. When", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 131 + }, + { + "text": "variable. The r values fall between −1 and 1. Either end of the range indicates a perfect correlation, whereas values near zero indicate a random distribution with no correlation. A positive r value indicates a Estadísticos e-Books & Papers direct relationship: as one variable increases, the other does too. When graphed on a scatterplot, the data points representing each pair of values in a direct relationship would slope upward from left to right. A negative r value indicates an inverse relationship: as one variable increases, the other decreases. Dots representing an inverse relationship would slope downward from left to right on a scatterplot. Table 10-1 provides general guidelines for interpreting positive and negative r values, although as always with statistics, different statisticians may offer different interpretations. Table 10-1: Interpreting Correlation Coefficients Correlation coefficient (+/−)What it could mean 0 No relationship .01 to .29 Weak relationship .3 to .59 Moderate relationship .6 to .99 Strong to nearly perfect relationship 1 Perfect relationship In standard ANSI SQL and PostgreSQL, we calculate the Pearson correlation coefficient using corr(Y, X). It’s one of several binary aggregate functions in SQL and is so named because these functions accept two inputs. In binary aggregate functions, the input Y is the dependent variable whose variation depends on the value of another variable, and X is the independent variable whose value doesn’t depend on another variable. NOTE Even though SQL specifies the Y and X inputs for the corr() function, correlation calculations don’t distinguish between dependent and independent variables. Switching the order of inputs in corr() produces the same result. However, for convenience and readability, these examples order the input Estadísticos e-Books & Papers variables according to dependent and independent. We’ll use the corr(Y, X) function to discover the relationship between education level and income. Enter the code in Listing 10-2 to use corr(Y, X) with the median_hh_income and pct_bachelors_higher variables as inputs: SELECT corr(median_hh_income, pct_bachelors_higher) AS bachelors_income_r FROM acs_2011_2015_stats; Listing 10-2: Using corr(Y, X) to measure the relationship between education and income Run the query; your result should be an r value of just above .68 given as the floating-point double precision data type: bachelors_income_r ------------------ 0.682185675451399 This positive r value indicates that as a county’s educational attainment increases, household income tends to increase. The relationship isn’t perfect, but the r value shows the relationship is fairly strong. We can visualize this pattern by plotting the variables on a scatterplot using Excel, as shown in Figure 10-1. Each data point represents one U.S. county; the data point’s position on the x-axis shows the percentage of the population ages 25 and older that have a bachelor’s degree or higher. The data point’s position on the y-axis represents the county’s median household income. Estadísticos e-Books & Papers Figure 10-1: A scatterplot showing the relationship between education and income Notice that although most of the data points are grouped together in the bottom-left corner of the graph, they do generally slope upward from left to right. Also, the points spread", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 132 + }, + { + "text": "county’s median household income. Estadísticos e-Books & Papers Figure 10-1: A scatterplot showing the relationship between education and income Notice that although most of the data points are grouped together in the bottom-left corner of the graph, they do generally slope upward from left to right. Also, the points spread out rather than strictly follow a straight line. If they were in a straight line sloping up from left to right, the r value would be 1, indicating a perfect positive linear relationship. Checking Additional Correlations Now let’s calculate the correlation coefficients for the remaining variable pairs using the code in Listing 10-3: SELECT ➊ round( corr(median_hh_income, pct_bachelors_higher)::numeric, 2 ) AS bachelors_income_r, round( corr(pct_travel_60_min, median_hh_income)::numeric, 2 ) AS income_travel_r, round( corr(pct_travel_60_min, pct_bachelors_higher)::numeric, 2 ) AS bachelors_travel_r FROM acs_2011_2015_stats; Estadísticos e-Books & Papers Listing 10-3: Using corr(Y, X) on additional variables This time we’ll make the output more readable by rounding off the decimal values. We’ll do this by wrapping the corr(Y, X) function inside SQL’s round() function ➊, which takes two inputs: the numeric value to be rounded and an integer value indicating the number of decimal places to round the first value. If the second parameter is omitted, the value is rounded to the nearest whole integer. Because corr(Y, X) returns a floating-point value by default, we’ll change it to the numeric type using the :: notation you learned in Chapter 3. Here’s the output: bachelors_income_r income_travel_r bachelors_travel_r ------------------ --------------- ------------------ 0.68 0.05 -0.14 The bachelors_income_r value is 0.68, which is the same as our first run but rounded to two decimal places. Compared to bachelors_income_r, the other two correlations are weak. The income_travel_r value shows that the correlation between income and the percentage of those who commute more than an hour to work is practically zero. This indicates that a county’s median household income bears little connection to how long it takes people to get to work. The bachelors_travel_r value shows that the correlation of bachelor’s degrees and commuting is also low at -0.14. The negative value indicates an inverse relationship: as education increases, the percentage of the population that travels more than an hour to work decreases. Although this is interesting, a correlation coefficient that is this close to zero indicates a weak relationship. When testing for correlation, we need to note some caveats. The first is that even a strong correlation does not imply causality. We can’t say that a change in one variable causes a change in the other, only that the changes move together. The second is that correlations should be subject to testing to determine whether they’re statistically significant. Those tests are beyond the scope of this book but worth studying on your own. Nevertheless, the SQL corr(Y, X) function is a handy tool for quickly checking correlations between variables. Estadísticos e-Books & Papers Predicting Values with Regression Analysis Researchers not only want to understand relationships between variables; they also want to predict values using available data. For example, let’s say 30 percent", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 133 + }, + { + "text": "own. Nevertheless, the SQL corr(Y, X) function is a handy tool for quickly checking correlations between variables. Estadísticos e-Books & Papers Predicting Values with Regression Analysis Researchers not only want to understand relationships between variables; they also want to predict values using available data. For example, let’s say 30 percent of a county’s population has a bachelor’s degree or higher. Given the trend in our data, what would we expect that county’s median household income to be? Likewise, for each percent increase in education, how much increase, on average, would we expect in income? We can answer both questions using linear regression. Simply put, the regression method finds the best linear equation, or straight line, that describes the relationship between an independent variable (such as education) and a dependent variable (such as income). Standard ANSI SQL and PostgreSQL include functions that perform linear regression. Figure 10-2 shows our previous scatterplot with a regression line added. Figure 10-2: Scatterplot with least squares regression line showing the relationship between education and income The straight line running through the middle of all the data points is called the least squares regression line, which approximates the “best fit” for Estadísticos e-Books & Papers a straight line that best describes the relationship between the variables. The equation for the regression line is like the slope-intercept formula you might remember from high school math but written using differently named variables: Y = bX + a. Here are the formula���s components: Y is the predicted value, which is also the value on the y-axis, or dependent variable. b is the slope of the line, which can be positive or negative. It measures how many units the y-axis value will increase or decrease for each unit of the x-axis value. X represents a value on the x-axis, or independent variable. a is the y-intercept, the value at which the line crosses the y-axis when the X value is zero. Let’s apply this formula using SQL. Earlier, we questioned what the expected median household income in a county would be if the percentage of people with a bachelor’s degree or higher in that county was 30 percent. In our scatterplot, the percentage with bachelor’s degrees falls along the x-axis, represented by X in the calculation. Let’s plug that value into the regression line formula in place of X: Y = b(30) + a To calculate Y, which represents the predicted median household income, we need the line’s slope, b, and the y-intercept, a. To get these values, we’ll use the SQL functions regr_slope(Y, X) and regr_intercept(Y, X), as shown in Listing 10-4: SELECT round( regr_slope(median_hh_income, pct_bachelors_higher)::numeric, 2 ) AS slope, round( regr_intercept(median_hh_income, pct_bachelors_higher)::numeric, 2 ) AS y_intercept FROM acs_2011_2015_stats; Listing 10-4: Regression slope and intercept functions Estadísticos e-Books & Papers Using the median_hh_income and pct_bachelors_higher variables as inputs for both functions, we’ll set the resulting value of the regr_slope(Y, X) function as slope and the output for the regr_intercept(Y, X) function as y_intercept. Run the query; the result should", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 134 + }, + { + "text": "Listing 10-4: Regression slope and intercept functions Estadísticos e-Books & Papers Using the median_hh_income and pct_bachelors_higher variables as inputs for both functions, we’ll set the resulting value of the regr_slope(Y, X) function as slope and the output for the regr_intercept(Y, X) function as y_intercept. Run the query; the result should show the following: slope y_intercept ------ ----------- 926.95 27901.15 The slope value shows that for every one-unit increase in bachelor’s degree percentage, we can expect a county’s median household income will increase by 926.95. Slope always refers to change per one unit of X. The y_intercept value shows that when the regression line crosses the y- axis, where the percentage with bachelor’s degrees is at 0, the y-axis value is 27901.15. Now let’s plug both values into the equation to get the Y value: Y = 926.95(30) + 27901.15 Y = 55709.65 Based on our calculation, in a county in which 30 percent of people age 25 and older have a bachelor’s degree or higher, we can expect a median household income in that county to be about $55,710. Of course, our data includes counties whose median income falls above and below that predicted value, but we expect this to be the case because our data points in the scatterplot don’t line up perfectly along the regression line. Recall that the correlation coefficient we calculated was 0.68, indicating a strong but not perfect relationship between education and income. Other factors probably contributed to variations in income as well. Finding the Effect of an Independent Variable with r- squared Earlier in the chapter, we calculated the correlation coefficient, r, to determine the direction and strength of the relationship between two Estadísticos e-Books & Papers variables. We can also calculate the extent that the variation in the x (independent) variable explains the variation in the y (dependent) variable by squaring the r value to find the coefficient of determination, better known as r-squared. An r-squared value is between zero and one and indicates the percentage of the variation that is explained by the independent variable. For example, if r-squared equals .1, we would say that the independent variable explains 10 percent of the variation in the dependent variable, or not much at all. To find r-squared, we use the regr_r2(Y, X) function in SQL. Let’s apply it to our education and income variables using the code in Listing 10-5: SELECT round( regr_r2(median_hh_income, pct_bachelors_higher)::numeric, 3 ) AS r_squared FROM acs_2011_2015_stats; Listing 10-5: Calculating the coefficient of determination, or r-squared This time we’ll round off the output to the nearest thousandth place and set the result to r_squared. The query should return the following result: r_squared --------- 0.465 The r-squared value of 0.465 indicates that about 47 percent of the variation in median household income in a county can be explained by the percentage of people with a bachelor’s degree or higher in that county. What explains the other 53 percent of the variation in household income? Any number of factors could explain the", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 135 + }, + { + "text": "that about 47 percent of the variation in median household income in a county can be explained by the percentage of people with a bachelor’s degree or higher in that county. What explains the other 53 percent of the variation in household income? Any number of factors could explain the rest of the variation, and statisticians will typically test numerous combinations of variables to determine what they are. But before you use these numbers in a headline or presentation, it’s worth revisiting the following points: 1. Correlation doesn’t prove causality. For verification, do a Google Estadísticos e-Books & Papers search on “correlation and causality.” Many variables correlate well but have no meaning. (See http://www.tylervigen.com/spurious- correlations for examples of correlations that don’t prove causality, including the correlation between divorce rate in Maine and margarine consumption.) Statisticians usually perform significance testing on the results to make sure values are not simply the result of randomness. 2. Statisticians also apply additional tests to data before accepting the results of a regression analysis, including whether the variables follow the standard bell curve distribution and meet other criteria for a valid result. Given these factors, SQL’s statistics functions are useful as a preliminary survey of your data before doing more rigorous analysis. If your work involves statistics, a full study on performing regression is worthwhile. Creating Rankings with SQL Rankings make the news often. You’ll see them used anywhere from weekend box office charts to a sports team’s league standings. You’ve already learned how to order query results based on values in a column, but SQL lets you go further and create numbered rankings. Rankings are useful for data analysis in several ways, such as tracking changes over time if you have several years’ worth of data. You can also simply use a ranking as a fact on its own in a report. Let’s explore how to create rankings using SQL. Ranking with rank() and dense_rank() Standard ANSI SQL includes several ranking functions, but we’ll just focus on two: rank() and dense_rank(). Both are window functions, which perform calculations across sets of rows we specify using the OVER clause. Unlike aggregate functions, which group rows while calculating results, Estadísticos e-Books & Papers window functions present results for each row in the table. The difference between rank() and dense_rank() is the way they handle the next rank value after a tie: rank() includes a gap in the rank order, but dense_rank() does not. This concept is easier to understand in action, so let’s look at an example. Consider a Wall Street analyst who covers the highly competitive widget manufacturing market. The analyst wants to rank companies by their annual output. The SQL statements in Listing 10-6 create and fill a table with this data and then rank the companies by widget output: CREATE TABLE widget_companies ( id bigserial, company varchar(30) NOT NULL, widget_output integer NOT NULL ); INSERT INTO widget_companies (company, widget_output) VALUES ('Morse Widgets', 125000), ('Springfield Widget Masters', 143000), ('Best Widgets', 196000), ('Acme Inc.', 133000),", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 136 + }, + { + "text": "create and fill a table with this data and then rank the companies by widget output: CREATE TABLE widget_companies ( id bigserial, company varchar(30) NOT NULL, widget_output integer NOT NULL ); INSERT INTO widget_companies (company, widget_output) VALUES ('Morse Widgets', 125000), ('Springfield Widget Masters', 143000), ('Best Widgets', 196000), ('Acme Inc.', 133000), ('District Widget Inc.', 201000), ('Clarke Amalgamated', 620000), ('Stavesacre Industries', 244000), ('Bowers Widget Emporium', 201000); SELECT company, widget_output, ➊ rank() OVER (ORDER BY widget_output DESC), ➋ dense_rank() OVER (ORDER BY widget_output DESC) FROM widget_companies; Listing 10-6: Using the rank() and dense_rank() window functions Notice the syntax in the SELECT statement that includes rank() ➊ and dense_rank() ➋. After the function names, we use the OVER clause and in parentheses place an expression that specifies the “window” of rows the function should operate on. In this case, we want both functions to work on all rows of the widget_output column, sorted in descending order. Here’s the output: company widget_output rank dense_rank -------------------------- ------------- ---- ---------- Estadísticos e-Books & Papers Clarke Amalgamated 620000 1 1 Stavesacre Industries 244000 2 2 Bowers Widget Emporium 201000 3 3 District Widget Inc. 201000 3 3 Best Widgets 196000 5 4 Springfield Widget Masters 143000 6 5 Acme Inc. 133000 7 6 Morse Widgets 125000 8 7 The columns produced by the rank() and dense_rank() functions show each company’s ranking based on the widget_output value from highest to lowest, with Clarke Amalgamated at number one. To see how rank() and dense_rank() differ, check the fifth row listing, Best Widgets. With rank(), Best Widgets is the fifth highest ranking company, showing there are four companies with more output and there is no company ranking in fourth place, because rank() allows a gap in the order when a tie occurs. In contrast, dense_rank(), which doesn’t allow a gap in the rank order, reflects the fact that Best Widgets has the fourth highest output number regardless of how many companies produced more. Therefore, Best Widgets ranks in fourth place using dense_rank(). Both ways of handling ties have merit, but in practice rank() is used most often. It’s also what I recommend using, because it more accurately reflects the total number of companies ranked, shown by the fact that Best Widgets has four companies ahead of it in total output, not three. Let’s look at a more complex ranking example. Ranking Within Subgroups with PARTITION BY The ranking we just did was a simple overall ranking based on widget output. But sometimes you’ll want to produce ranks within groups of rows in a table. For example, you might want to rank government employees by salary within each department or rank movies by box office earnings within each genre. To use window functions in this way, we’ll add PARTITION BY to the OVER clause. A PARTITION BY clause divides table rows according to values in a column we specify. Here’s an example using made-up data about grocery stores. Enter the Estadísticos e-Books & Papers code in Listing 10-7 to fill a table", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 137 + }, + { + "text": "functions in this way, we’ll add PARTITION BY to the OVER clause. A PARTITION BY clause divides table rows according to values in a column we specify. Here’s an example using made-up data about grocery stores. Enter the Estadísticos e-Books & Papers code in Listing 10-7 to fill a table called store_sales: CREATE TABLE store_sales ( store varchar(30), category varchar(30) NOT NULL, unit_sales bigint NOT NULL, CONSTRAINT store_category_key PRIMARY KEY (store, category) ); INSERT INTO store_sales (store, category, unit_sales) VALUES ('Broders', 'Cereal', 1104), ('Wallace', 'Ice Cream', 1863), ('Broders', 'Ice Cream', 2517), ('Cramers', 'Ice Cream', 2112), ('Broders', 'Beer', 641), ('Cramers', 'Cereal', 1003), ('Cramers', 'Beer', 640), ('Wallace', 'Cereal', 980), ('Wallace', 'Beer', 988); SELECT category, store, unit_sales, ➊ rank() OVER (PARTITION BY category ORDER BY unit_sales DESC) FROM store_sales; Listing 10-7: Applying rank() within groups using PARTITION BY In the table, each row includes a store’s product category and sales for that category. The final SELECT statement creates a result set showing how each store’s sales ranks within each category. The new element is the addition of PARTITION BY in the OVER clause ➊. In effect, the clause tells the program to create rankings one category at a time, using the store’s unit sales in descending order. Here’s the output: category store unit_sales rank --------- ------- ---------- ---- Beer Wallace 988 1 Beer Broders 641 2 Beer Cramers 640 3 Cereal Broders 1104 1 Cereal Cramers 1003 2 Cereal Wallace 980 3 Ice Cream Broders 2517 1 Ice Cream Cramers 2112 2 Ice Cream Wallace 1863 3 Notice that category names are ordered and grouped in the category Estadísticos e-Books & Papers column as a result of PARTITION BY in the OVER clause. Rows for each category are ordered by category unit sales with the rank column displaying the ranking. Using this table, we can see at a glance how each store ranks in a food category. For instance, Broders tops sales for cereal and ice cream, but Wallace wins in the beer category. You can apply this concept to many other scenarios: for example, for each auto manufacturer, finding the vehicle with the most consumer complaints; figuring out which month had the most rainfall in each of the last 20 years; finding the team with the most wins against left-handed pitchers; and so on. SQL offers additional window functions. Check the official PostgreSQL documentation at https://www.postgresql.org/docs/current/static/tutorial-window.html for an overview of window functions, and check https://www.postgresql.org/docs/current/static/functions-window.html for a listing of window functions. Calculating Rates for Meaningful Comparisons As helpful and interesting as they are, rankings based on raw counts aren’t always meaningful; in fact, they can actually be misleading. Consider this example of crime statistics: according to the U.S. Federal Bureau of Investigation (FBI), in 2015, New York City reported about 130,000 property crimes, which included burglary, larceny, motor vehicle thefts, and arson. Meanwhile, Chicago reported about 80,000 property crimes the same year. So, you’re more likely to find trouble in New York City, right? Not necessarily. In 2015, New York City had", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 138 + }, + { + "text": "(FBI), in 2015, New York City reported about 130,000 property crimes, which included burglary, larceny, motor vehicle thefts, and arson. Meanwhile, Chicago reported about 80,000 property crimes the same year. So, you’re more likely to find trouble in New York City, right? Not necessarily. In 2015, New York City had more than 8 million residents, whereas Chicago had 2.7 million. Given that context, just comparing the total numbers of property crimes in the two cities isn’t very meaningful. A more accurate way to compare these numbers is to turn them into rates. Analysts often calculate a rate per 1,000 people, or some multiple of that number, for apples-to-apples comparisons. For the property crimes Estadísticos e-Books & Papers in this example, the math is simple: divide the number of offenses by the population and then multiply that quotient by 1,000. For example, if a city has 80 vehicle thefts and a population of 15,000, you can calculate the rate of vehicle thefts per 1,000 people as follows: (80 / 15,000) × 1,000 = 5.3 vehicle thefts per thousand residents This is easy math with SQL, so let’s try it using select city-level data I compiled from the FBI’s 2015 Crime in the United States report available at https://ucr.fbi.gov/crime-in-the-u.s/2015/crime-in-the-u.s.-2015/home. Listing 10-8 contains the code to create and fill a table. Remember to point the script to the location in which you’ve saved the CSV file, which you can download at https://www.nostarch.com/practicalSQL/. CREATE TABLE fbi_crime_data_2015 ( st varchar(20), city varchar(50), population integer, violent_crime integer, property_crime integer, burglary integer, larceny_theft integer, motor_vehicle_theft integer, CONSTRAINT st_city_key PRIMARY KEY (st, city) ); COPY fbi_crime_data_2015 FROM 'C:\\YourDirectory\\fbi_crime_data_2015.csv' WITH (FORMAT CSV, HEADER, DELIMITER ','); SELECT * FROM fbi_crime_data_2015 ORDER BY population DESC; Listing 10-8: Creating and filling a 2015 FBI crime data table The fbi_crime_data_2015 table includes the state, city name, and population for that city. Next is the number of crimes reported by police in categories, including violent crime, vehicle thefts, and property crime. To calculate property crimes per 1,000 people in cities with more than 500,000 people and order them, we’ll use the code in Listing 10-9: SELECT city, st, Estadísticos e-Books & Papers population, property_crime, round( ➊ (property_crime::numeric / population) * 1000, 1 ) AS pc_per_1000 FROM fbi_crime_data_2015 WHERE population >= 500000 ORDER BY (property_crime::numeric / population) DESC; Listing 10-9: Finding property crime rates per thousand in cities with 500,000 or more people In Chapter 5, you learned that when dividing an integer by an integer, one of the values must be a numeric or decimal for the result to include decimal places. We do that in the rate calculation ➊ with PostgreSQL’s double-colon shorthand. Because we don’t need to see many decimal places, we wrap the statement in the round() function to round off the output to the nearest tenth. Then we give the calculated column an alias of pc_per_1000 for easy reference. Here’s a portion of the result set: Tucson, Arizona, has the highest rate of property crimes, followed by San Francisco, California.", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 139 + }, + { + "text": "wrap the statement in the round() function to round off the output to the nearest tenth. Then we give the calculated column an alias of pc_per_1000 for easy reference. Here’s a portion of the result set: Tucson, Arizona, has the highest rate of property crimes, followed by San Francisco, California. At the bottom is New York City, with a rate that’s one-fourth of Tucson’s. If we had compared the cities based solely on the raw numbers of property crimes, we’d have a far different result than the one we derived by calculating the rate per thousand. I’d be remiss not to point out that the FBI website at https://ucr.fbi.gov/ucr-statistics-their-proper-use/ discourages creating rankings from its crime data, stating that doing so creates “misleading perceptions which adversely affect geographic entities and their residents.” They point out that variations in crimes and crime rates across Estadísticos e-Books & Papers the country are often due to a number of factors ranging from population density to economic conditions and even the climate. Also, the FBI’s crime data has well-documented short​comings, including incomplete reporting by police agencies. That said, asking why a locality has higher or lower crime rates than others is still worth pursuing, and rates do provide some measure of comparison despite certain limitations. Wrapping Up That wraps up our exploration of statistical functions in SQL, rankings, and rates. Now your SQL analysis toolkit includes ways to find relationships among variables using statistics functions, create rankings from ordered data, and properly compare raw numbers by turning them into rates. That toolkit is starting to look impressive! Next, we’ll dive deeper into date and time data, using SQL functions to extract the information we need. TRY IT YOURSELF Test your new skills with the following questions: 1. In Listing 10-2, the correlation coefficient, or r value, of the variables pct_bachelors_higher and median_hh_income was about .68. Write a query using the same data set to show the correlation between pct_masters_higher and median_hh_income. Is the r value higher or lower? What might explain the difference? 2. In the FBI crime data, which cities with a population of 500,000 or more have the highest rates of motor vehicle thefts (column motor_vehicle_theft)? Which have the highest violent crime rates (column violent_crime)? 3. As a bonus challenge, revisit the libraries data in the table pls_fy2014_pupld14a in Chapter 8. Rank library agencies based on Estadísticos e-Books & Papers the rate of visits per 1,000 population (column popu_lsa), and limit the query to agencies serving 250,000 people or more. Estadísticos e-Books & Papers 11 WORKING WITH DATES AND TIMES Columns filled with dates and times can indicate when events happened or how long they took, and that can lead to interesting lines of inquiry. What patterns exist in the moments on a timeline? Which events were shortest or longest? What relationships exist between a particular activity and the time of day or season in which it occurred? In this chapter, we’ll explore these kinds of questions using SQL data types for dates", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 140 + }, + { + "text": "of inquiry. What patterns exist in the moments on a timeline? Which events were shortest or longest? What relationships exist between a particular activity and the time of day or season in which it occurred? In this chapter, we’ll explore these kinds of questions using SQL data types for dates and times and their related functions. We’ll start with a closer look at data types and functions related to dates and times. Then we’ll explore a data set that contains information on trips by New York City taxicabs to look for patterns and try to discover what, if any, story the data tells. We’ll also explore time zones using Amtrak data to calculate the duration of train trips across the United States. Data Types and Functions for Dates and Times Chapter 3 explored primary SQL data types, but to review, here are the four data types related to dates and times: date Records only the date. PostgreSQL accepts several date formats. For example, valid formats for adding the 21st day of September 2018 Estadísticos e-Books & Papers are September 21, 2018 or 9/21/2018. I recommend using YYYY-MM-DD (or 2018- 09-21), which is the ISO 8601 international standard format and also the default PostgreSQL date output. Using the ISO format helps avoid confusion when sharing data internationally. time Records only the time. Adding with time zone makes the column time zone aware. The ISO 8601 format is HH:MM:SS, where HH represents the hour, MM the minutes, and SS the seconds. You can add an optional time zone designator. For example, 2:24 PM in San Francisco during standard time in fall and winter would be 14:24 PST. timestamp Records the date and time. You can add with time zone to make the column time zone aware. The format timestamp with time zone is part of the SQL standard, but with PostgreSQL, you can use the shorthand timestamptz, which combines the date and time formats plus a time zone designator at the end: YYYY-MM-DD HH:MM:SS TZ. You can specify time zones in three different formats: its UTC offset, an area/location designator, or a standard abbreviation. interval Holds a value that represents a unit of time expressed in the format quantity unit. It doesn’t record the start or end of a period, only its duration. Examples include 12 days or 8 hours. The first three data types, date, time, and timestamp, are known as datetime types whose values are called datetimes. The interval value is an interval type whose values are intervals. All four data types can track the system clock and the nuances of the calendar. For example, date and timestamp recognize that June has 30 days. Therefore, June 31 is an invalid datetime value that causes the database to throw an error. Likewise, the date February 29 is valid only in a leap year, such as 2020. Manipulating Dates and Times We can use SQL functions to perform calculations on dates and times or extract components from them. For example, we can", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 141 + }, + { + "text": "datetime value that causes the database to throw an error. Likewise, the date February 29 is valid only in a leap year, such as 2020. Manipulating Dates and Times We can use SQL functions to perform calculations on dates and times or extract components from them. For example, we can retrieve the day of the week from a timestamp or extract just the month from a date. ANSI Estadísticos e-Books & Papers SQL outlines a handful of functions to do this, but many database managers (including MySQL and Microsoft SQL Server) deviate from the standard to implement their own date and time data types, syntax, and function names. If you’re using a database other than PostgreSQL, check its documentation. Let’s review how to manipulate dates and times using PostgreSQL functions. Extracting the Components of a timestamp Value It’s not unusual to need just one piece of a date or time value for analysis, particularly when you’re aggregating results by month, year, or even minute. We can extract these components using the PostgreSQL date_part() function. Its format looks like this: date_part(text, value) The function takes two inputs. The first is a string in text format that represents the part of the date or time to extract, such as hour, minute, or week. The second is the date, time, or timestamp value. To see the date_part() function in action, we’ll execute it multiple times on the same value using the code in Listing 11-1. In the listing, we format the string as a timestamp with time zone using the PostgreSQL-specific shorthand timestamptz. We also assign a column name to each with AS. SELECT date_part('year', '2019-12-01 18:37:12 EST'::timestamptz) AS \"year\", date_part('month', '2019-12-01 18:37:12 EST'::timestamptz) AS \"month\", date_part('day', '2019-12-01 18:37:12 EST'::timestamptz) AS \"day\", date_part('hour', '2019-12-01 18:37:12 EST'::timestamptz) AS \"hour\", date_part('minute', '2019-12-01 18:37:12 EST'::timestamptz) AS \"minute\", date_part('seconds', '2019-12-01 18:37:12 EST'::timestamptz) AS \"seconds\", date_part('timezone_hour', '2019-12-01 18:37:12 EST'::timestamptz) AS \"tz\", date_part('week', '2019-12-01 18:37:12 EST'::timestamptz) AS \"week\", date_part('quarter', '2019-12-01 18:37:12 EST'::timestamptz) AS \"quarter\", date_part('epoch', '2019-12-01 18:37:12 EST'::timestamptz) AS \"epoch\"; Listing 11-1: Extracting components of a timestamp value using date_part() Each column statement in this SELECT query first uses a string to name Estadísticos e-Books & Papers the component we want to extract: year, month, day, and so on. The second input uses the string 2019-12-01 18:37:12 EST cast as a timestamp with time zone with the PostgreSQL double-colon syntax and the timestamptz shorthand. In December, the United States is observing standard time, which is why we can designate the Eastern time zone using the Eastern Standard Time (EST) designation. Here’s the output as shown on my computer, which is located in the U.S. Eastern time zone. (The database converts the values to reflect your PostgreSQL time zone setting, so your output might be different; for example, if it’s set to the U.S. Pacific time zone, the hour will show as 15): Each column contains a single value that represents 6:37:12 PM on December 1, 2019, in the U.S. Eastern time zone. Even though you designated the time zone", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 142 + }, + { + "text": "so your output might be different; for example, if it’s set to the U.S. Pacific time zone, the hour will show as 15): Each column contains a single value that represents 6:37:12 PM on December 1, 2019, in the U.S. Eastern time zone. Even though you designated the time zone using EST in the string, PostgreSQL reports back the UTC offset of that time zone, which is the number of hours plus or minus from UTC. UTC refers to Coordinated Universal Time, a world time standard, as well as the value of UTC +/−00:00, the time zone that covers the United Kingdom and Western Africa. Here, the UTC offset is -5 (because EST is five hours behind UTC). NOTE You can derive the UTC offset from the time zone but not vice versa. Each UTC offset can refer to multiple named time zones plus standard and daylight saving time variants. The first seven values are easy to recognize from the original timestamp, but the last three are calculated values that deserve an explanation. The week column shows that December 1, 2019, falls in the 48th week of the year. This number is determined by ISO 8601 standards, which Estadísticos e-Books & Papers start each week on a Monday. That means a week at the end of a year can extend from December into January of the following year. The quarter column shows that our test date is part of the fourth quarter of the year. The epoch column shows a measurement, which is used in computer systems and programming languages, that represents the number of seconds elapsed before or after 12 AM, January 1, 1970, at UTC 0. A positive value designates a time since that point; a negative value designates a time before it. In this example, 1,575,243,432 seconds elapsed between January 1, 1970, and the timestamp. Epoch is useful if you need to compare two timestamps mathematically on an absolute scale. PostgreSQL also supports the SQL-standard extract() function, which parses datetimes in the same way as the date_part() function. I’ve featured date_part() here instead for two reasons. First, its name helpfully reminds us what it does. Second, extract() isn’t widely supported by database managers. Most notably, it’s absent in Microsoft’s SQL Server. Nevertheless, if you need to use extract(), the syntax takes this form: extract(text from value) To replicate the first date_part() example in Listing 11-1 where we pull the year from the timestamp, we’d set up the function like this: extract('year' from '2019-12-01 18:37:12 EST'::timestamptz) PostgreSQL provides additional components you can extract or calculate from dates and times. For the full list of functions, see the documentation at https://www.postgresql.org/docs/current/static/functions- datetime.html. Creating Datetime Values from timestamp Components It’s not unusual to come across a data set in which the year, month, and day exist in separate columns, and you might want to create a datetime value from these components. To perform calculations on a date, it’s Estadísticos e-Books & Papers helpful to combine and format those pieces correctly", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 143 + }, + { + "text": "not unusual to come across a data set in which the year, month, and day exist in separate columns, and you might want to create a datetime value from these components. To perform calculations on a date, it’s Estadísticos e-Books & Papers helpful to combine and format those pieces correctly into one column. You can use the following PostgreSQL functions to make datetime objects: make_date(year, month, day) Returns a value of type date make_time(hour, minute, seconds) Returns a value of type time without time zone make_timestamptz(year, month, day, hour, minute, second, time zone) Returns a timestamp with time zone The variables for these three functions take integer types as input, with two exceptions: seconds are of the type double precision because you can supply fractions of seconds, and time zones must be specified with a text string that names the time zone. Listing 11-2 shows examples of the three functions in action using components of February 22, 2018, for the date, and 6:04:30.3 PM in Lisbon, Portugal for the time: SELECT make_date(2018, 2, 22); SELECT make_time(18, 4, 30.3); SELECT make_timestamptz(2018, 2, 22, 18, 4, 30.3, 'Europe/Lisbon'); Listing 11-2: Three functions for making datetimes from components When I run each query in order, the output on my computer in the U.S. Eastern time zone is as follows. Again, yours may differ depending on your time zone setting: 2018-02-22 18:04:30.3 2018-02-22 13:04:30.3-05 Notice that the timestamp in the third line shows 13:04:30.3, which is Eastern Standard Time and is five hours behind (-05) the time input to the function: 18:04:30.3. In our discussion on time zone–enabled columns in “Dates and Times” on page 32, I noted that PostgreSQL displays times relative to the client’s time zone or the time zone set in the database session. This output reflects the appropriate time because my Estadísticos e-Books & Papers location is five hours behind Lisbon. We’ll explore working with time zones in more detail, and you’ll learn to adjust its display in “Working with Time Zones” on page 177. Retrieving the Current Date and Time If you need to record the current date or time as part of a query—when updating a row, for example—standard SQL provides functions for that too. The following functions record the time as of the start of the query: current_date Returns the date. current_time Returns the current time with time zone. current_timestamp Returns the current timestamp with time zone. A shorthand PostgreSQL-specific version is now(). localtime Returns the current time without time zone. localtimestamp Returns the current timestamp without time zone. Because these functions record the time at the start of the query (or a collection of queries grouped under a transaction, which I covered in Chapter 9), they’ll provide that same time throughout the execution of a query regardless of how long the query runs. So, if your query updates 100,000 rows and takes 15 seconds to run, any timestamp recorded at the start of the query will be applied to each row, and so each row will", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 144 + }, + { + "text": "provide that same time throughout the execution of a query regardless of how long the query runs. So, if your query updates 100,000 rows and takes 15 seconds to run, any timestamp recorded at the start of the query will be applied to each row, and so each row will receive the same timestamp. If, instead, you want the date and time to reflect how the clock changes during the execution of the query, you can use the PostgreSQL- specific clock_timestamp() function to record the current time as it elapses. That way, if you’re updating 100,000 rows and inserting a timestamp each time, each row gets the time the row updated rather than the time at the start of the query. Note that clock_timestamp() can slow large queries and may be subject to system limitations. Listing 11-3 shows current_timestamp and clock_timestamp() in action when inserting a row in a table: Estadísticos e-Books & Papers CREATE TABLE current_time_example ( time_id bigserial, ➊ current_timestamp_col timestamp with time zone, ➋ clock_timestamp_col timestamp with time zone ); INSERT INTO current_time_example (current_timestamp_col, clock_timestamp_col) ➌ (SELECT current_timestamp, clock_timestamp() FROM generate_series(1,1000)); SELECT * FROM current_time_example; Listing 11-3: Comparing current_timestamp and clock_timestamp() during row insert The code creates a table that includes two timestamp columns with a time zone. The first holds the result of the current_timestamp function ➊, which records the time at the start of the INSERT statement that adds 1,000 rows to the table. To do that, we use the generate_series() function, which returns a set of integers starting with 1 and ending with 1,000. The second column holds the result of the clock_timestamp() function ➋, which records the time of insertion of each row. You call both functions as part of the INSERT statement ➌. Run the query, and the result from the final SELECT statement should show that the time in the current_timestamp_col is the same for all rows, whereas the time in clock_timestamp_col increases with each row inserted. Working with Time Zones Time zone data lets the dates and times in your database reflect the location around the globe where those dates and times apply and their UTC offset. A timestamp of 1 PM is only useful, for example, if you know whether the value refers to local time in Asia, Eastern Europe, one of the 12 time zones of Antarctica, or anywhere else on the globe. Of course, very often you’ll receive data sets that contain no time zone data in their datetime columns. This isn’t always a deal breaker in terms of whether or not you should continue to use the data. If you know that every event in the data happened in the same location, having the time Estadísticos e-Books & Papers zone in the timestamp is less critical, and it’s relatively easy to modify all the timestamps of your data to reflect that single time zone. Let’s look at some strategies for working with time zones in your data. Finding Your Time Zone Setting When working with time", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 145 + }, + { + "text": "e-Books & Papers zone in the timestamp is less critical, and it’s relatively easy to modify all the timestamps of your data to reflect that single time zone. Let’s look at some strategies for working with time zones in your data. Finding Your Time Zone Setting When working with time zones in SQL, you first need know the time zone setting for your database server. If you installed PostgreSQL on your own computer, the default will be your local time zone. If you’re connecting to a PostgreSQL database elsewhere, perhaps on a network or a cloud provider such as Amazon Web Services, the time zone setting may be different than your own. To help avoid confusion, database administrators often set a shared server’s time zone to UTC. To find out the default time zone of your PostgreSQL server, use the SHOW command with timezone, as shown in Listing 11-4: SHOW timezone; Listing 11-4: Showing your PostgreSQL server’s default time zone Entering Listing 11-4 into pgAdmin and running it on my computer returns US/Eastern, one of several location names that falls into the Eastern time zone, which encompasses eastern Canada and the United States, the Caribbean, and parts of Mexico. NOTE You can use SHOW ALL; to see the settings of every parameter on your PostgreSQL server. You can also use the two commands in Listing 11-5 to list all time zone names, abbreviations, and their UTC offsets: SELECT * FROM pg_timezone_abbrevs; SELECT * FROM pg_timezone_names; Listing 11-5: Showing time zone abbreviations and names Estadísticos e-Books & Papers You can easily filter either of these SELECT statements with a WHERE clause to look up specific location names or time zones: SELECT * FROM pg_timezone_names WHERE name LIKE 'Europe%'; This code should return a table listing that includes the time zone name, abbreviation, UTC offset, and a boolean column is_dst that notes whether the time zone is currently observing daylight saving time: name abbrev utc_offset is_dst ---------------- ------ ---------- ------ Europe/Amsterdam CEST 02:00:00 t Europe/Andorra CEST 02:00:00 t Europe/Astrakhan +04 04:00:00 f Europe/Athens EEST 03:00:00 t Europe/Belfast BST 01:00:00 t --snip-- This is a faster way of looking up time zones than using Wikipedia. Now let’s look at how to set the time zone to a particular value. Setting the Time Zone When you installed PostgreSQL, the server’s default time zone was set as a parameter in postgresql.conf, a file that contains dozens of values read by PostgreSQL each time it starts. The location of postgresql.conf in your file system varies depending on your operating system and sometimes on the way you installed PostgreSQL. To make permanent changes to postgresql.conf, you need to edit the file and restart the server, which might be impossible if you’re not the owner of the machine. Changes to configurations might also have unintended consequences for other users or applications. I’ll cover working with postgresql.conf in more depth in Chapter 17. However, for now you can easily set the pgAdmin client’s time zone on a per-session basis,", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 146 + }, + { + "text": "be impossible if you’re not the owner of the machine. Changes to configurations might also have unintended consequences for other users or applications. I’ll cover working with postgresql.conf in more depth in Chapter 17. However, for now you can easily set the pgAdmin client’s time zone on a per-session basis, and the change should last as long as you’re connected to the server. This solution is handy when you want to specify how you view a particular table or handle timestamps in a query. To set and change the pgAdmin client’s time zone, we use the Estadísticos e-Books & Papers command SET timezone TO, as shown in Listing 11-6: ➊ SET timezone TO 'US/Pacific'; ➋ CREATE TABLE time_zone_test ( test_date timestamp with time zone ); ➌ INSERT INTO time_zone_test VALUES ('2020-01-01 4:00'); ➍ SELECT test_date FROM time_zone_test; ➎ SET timezone TO 'US/Eastern'; ➏ SELECT test_date FROM time_zone_test; ➐ SELECT test_date AT TIME ZONE 'Asia/Seoul' FROM time_zone_test; Listing 11-6: Setting the time zone for a client session First, we set the time zone to US/Pacific ➊, which designates the Pacific time zone that covers western Canada and the United States along with Baja California in Mexico. Second, we create a one-column table ➋ with a data type of timestamp with time zone and insert a single row to display a test result. Notice that the value inserted, 2020-01-01 4:00, is a timestamp with no time zone ➌. You’ll encounter timestamps with no time zone quite often, particularly when you acquire data sets restricted to a specific location. When executed, the first SELECT statement ➍ returns 2020-01-01 4:00 as a timestamp that now contains time zone data: test_date ---------------------- 2020-01-01 04:00:00-08 Recall from our discussion on data types in Chapter 3 that the -08 at the end of this timestamp is the UTC offset. In this case, the -08 shows that the Pacific time zone is eight hours behind UTC. Because we initially set the pgAdmin client’s time zone to US/Pacific for this session, any value we now enter into a column that is time zone aware will be in Estadísticos e-Books & Papers Pacific time and coded accordingly. However, it’s worth noting that on the server, the timestamp with time zone data type always stores data as UTC internally; the time zone setting governs how it’s displayed. Now comes some fun. We change the time zone for this session to the Eastern time zone using the SET command ➎ and the US/Eastern designation. Then, when we execute the SELECT statement ➏ again, the result should be as follows: test_date ---------------------- 2020-01-01 07:00:00-05 In this example, two components of the timestamp have changed: the time is now 07:00, and the UTC offset is -05 because we’re viewing the timestamp from the perspective of the Eastern time zone: 4 AM Pacific is 7 AM Eastern. The original Pacific time value remains unaltered in the table, and the database converts it to show the time in whatever time zone we set at ➎. Even", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 147 + }, + { + "text": "is -05 because we’re viewing the timestamp from the perspective of the Eastern time zone: 4 AM Pacific is 7 AM Eastern. The original Pacific time value remains unaltered in the table, and the database converts it to show the time in whatever time zone we set at ➎. Even more convenient is that we can view a timestamp through the lens of any time zone without changing the session setting. The final SELECT statement uses the AT TIME ZONE keywords ➐ to display the timestamp in our session as Korea standard time (KST) by specifying Asia/Seoul: timezone ------------------- 2020-01-01 21:00:00 Now we know that the database value of 4 AM in US/Pacific on January 1, 2020, is equivalent to 9 PM that same day in Asia/Seoul. Again, this syntax changes the output data type, but the data on the server remains unchanged. If the original value is a timestamp with time zone, the output removes the time zone. If the original value has no time zone, the output is timestamp with time zone. The ability of databases to track time zones is extremely important for accurate calculations of intervals, as you’ll see next. Estadísticos e-Books & Papers Calculations with Dates and Times We can perform simple arithmetic on datetime and interval types the same way we can on numbers. Addition, subtraction, multiplication, and division are all possible in PostgreSQL using the math operators +, -, *, and /. For example, you can subtract one date from another date to get an integer that represents the difference in days between the two dates. The following code returns an integer of 3: SELECT '9/30/1929'::date - '9/27/1929'::date; The result indicates that these two dates are exactly three days apart. Likewise, you can use the following code to add a time interval to a date to return a new date: SELECT '9/30/1929'::date + '5 years'::interval; This code adds five years to the date 9/30/1929 to return a timestamp value of 9/30/1934. You can find more examples of math functions you can use with dates and times in the PostgreSQL documentation at https://www.postgresql.org/docs/current/static/functions-datetime.html. Let’s explore some more practical examples using actual transportation data. Finding Patterns in New York City Taxi Data When I visit New York City, I usually take at least one ride in one of the 13,500 iconic yellow cars that ferry hundreds of thousands of people across the city’s five boroughs each day. The New York City Taxi and Limousine Commission releases data on monthly yellow taxi trips plus other for-hire vehicles. We’ll use this large, rich data set to put date functions to practical use. The yellow_tripdata_2016_06_01.csv file available from the book’s resources (at https://www.nostarch.com/practicalSQL/) holds one day of yellow taxi trip records from June 1, 2016. Save the file to your computer and execute the code in Listing 11-7 to build the Estadísticos e-Books & Papers nyc_yellow_taxi_trips_2016_06_01 table. Remember to change the file path in the COPY command to the location where you’ve saved the file and adjust the", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 148 + }, + { + "text": "taxi trip records from June 1, 2016. Save the file to your computer and execute the code in Listing 11-7 to build the Estadísticos e-Books & Papers nyc_yellow_taxi_trips_2016_06_01 table. Remember to change the file path in the COPY command to the location where you’ve saved the file and adjust the path format to reflect whether you’re using Windows, macOS, or Linux. ➊ CREATE TABLE nyc_yellow_taxi_trips_2016_06_01 ( trip_id bigserial PRIMARY KEY, vendor_id varchar(1) NOT NULL, tpep_pickup_datetime timestamp with time zone NOT NULL, tpep_dropoff_datetime timestamp with time zone NOT NULL, passenger_count integer NOT NULL, trip_distance numeric(8,2) NOT NULL, pickup_longitude numeric(18,15) NOT NULL, pickup_latitude numeric(18,15) NOT NULL, rate_code_id varchar(2) NOT NULL, store_and_fwd_flag varchar(1) NOT NULL, dropoff_longitude numeric(18,15) NOT NULL, dropoff_latitude numeric(18,15) NOT NULL, payment_type varchar(1) NOT NULL, fare_amount numeric(9,2) NOT NULL, extra numeric(9,2) NOT NULL, mta_tax numeric(5,2) NOT NULL, tip_amount numeric(9,2) NOT NULL, tolls_amount numeric(9,2) NOT NULL, improvement_surcharge numeric(9,2) NOT NULL, total_amount numeric(9,2) NOT NULL ); ➋ COPY nyc_yellow_taxi_trips_2016_06_01 ( vendor_id, tpep_pickup_datetime, tpep_dropoff_datetime, passenger_count, trip_distance, pickup_longitude, pickup_latitude, rate_code_id, store_and_fwd_flag, dropoff_longitude, dropoff_latitude, payment_type, fare_amount, extra, mta_tax, tip_amount, tolls_amount, improvement_surcharge, total_amount ) FROM 'C:\\YourDirectory\\yellow_tripdata_2016_06_01.csv' WITH (FORMAT CSV, HEADER, DELIMITER ','); ➌ CREATE INDEX tpep_pickup_idx Estadísticos e-Books & Papers ON nyc_yellow_taxi_trips_2016_06_01 (tpep_pickup_datetime); Listing 11-7: Creating a table and importing NYC yellow taxi data The code in Listing 11-7 builds the table ➊, imports the rows ➋, and creates an index ➌. In the COPY statement, we provide the names of columns because the input CSV file doesn’t include the trip_id column that exists in the target table. That column is of type bigserial, which you’ve learned is an auto-incrementing integer and will fill automatically. After your import is complete, you should have 368,774 rows, one for each yellow cab ride on June 1, 2016. You can check the number of rows in your table with a count using the following code: SELECT count(*) FROM nyc_yellow_taxi_trips_2016_06_01; Each row includes data on the number of passengers, the location of pickup and drop-off in latitude and longitude, and the fare and tips in U.S. dollars. The data dictionary that describes all columns and codes is available at http://www.nyc.gov/html/tlc/downloads/pdf/data_dictionary_trip_records_yellow .pdf. For these exercises, we’re most interested in the timestamp columns tpep_pickup_datetime and tpep_dropoff_datetime, which represent the start and end times of the ride. (The Technology Passenger Enhancements Project [TPEP] is a program that in part includes automated collection of data about taxi rides.) The values in both timestamp columns include the time zone provided by the Taxi and Limousine Commission. In all rows of the CSV file, the time zone included with the timestamp is shown as -4, which is the summertime UTC offset for the Eastern time zone when New York City and the rest of the U.S. East Coast observe daylight saving time. If you’re not or your PostgreSQL server isn’t located in Eastern time, I suggest setting your time zone using the following code so your results will match mine: SET timezone TO 'US/Eastern'; Estadísticos e-Books & Papers Now let’s explore the patterns we can identify in", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 149 + }, + { + "text": "Coast observe daylight saving time. If you’re not or your PostgreSQL server isn’t located in Eastern time, I suggest setting your time zone using the following code so your results will match mine: SET timezone TO 'US/Eastern'; Estadísticos e-Books & Papers Now let’s explore the patterns we can identify in the data related to these times. The Busiest Time of Day One question you might ask after viewing this data set is when taxis provide the most rides. Is it morning or evening rush hour, or is there another time—at least, on this day—when rides spiked? You can determine the answer with a simple aggregation query that uses date_part(). Listing 11-8 contains the query to count rides by hour using the pickup time as the input: SELECT ➊ date_part('hour', tpep_pickup_datetime) AS trip_hour, ➋ count(*) FROM nyc_yellow_taxi_trips_2016_06_01 GROUP BY trip_hour ORDER BY trip_hour; Listing 11-8: Counting taxi trips by hour In the query’s first column ➊, date_part() extracts the hour from tpep_pickup_datetime so we can group the number of rides by hour. Then we aggregate the number of rides in the second column via the count() function ➋. The rest of the query follows the standard patterns for grouping and ordering the results, which should return 24 rows, one for each hour of the day: trip_hour count --------- ----- 0 8182 1 5003 2 3070 3 2275 4 2229 5 3925 6 10825 7 18287 8 21062 9 18975 10 17367 11 17383 Estadísticos e-Books & Papers 12 18031 13 17998 14 19125 15 18053 16 15069 17 18513 18 22689 19 23190 20 23098 21 24106 22 22554 23 17765 Eyeballing the numbers, it’s apparent that on June 1, 2016, New York City taxis had the most passengers between 6 PM and 10 PM, possibly reflecting commutes home plus the plethora of city activities on a summer evening. But to see the overall pattern, it’s best to visualize the data. Let’s do this next. Exporting to CSV for Visualization in Excel Charting data with a tool such as Microsoft Excel makes it easier to understand patterns, so I often export query results to a CSV file and work up a quick chart. Listing 11-9 uses the query from the preceding example within a COPY ... TO statement, similar to Listing 4-9 on page 52: COPY (SELECT date_part('hour', tpep_pickup_datetime) AS trip_hour, count(*) FROM nyc_yellow_taxi_trips_2016_06_01 GROUP BY trip_hour ORDER BY trip_hour ) TO 'C:\\YourDirectory\\hourly_pickups_2016_06_01.csv' WITH (FORMAT CSV, HEADER, DELIMITER ','); Listing 11-9: Exporting taxi pickups per hour to a CSV file When I load the data into Excel and build a line graph, the day’s pattern becomes more obvious and thought-provoking, as shown in Figure 11-1. Estadísticos e-Books & Papers Figure 11-1: NYC yellow taxi pickups by hour Rides bottomed out in the wee hours of the morning before rising sharply between 5 AM and 8 AM. Volume remained relatively steady throughout the day and increased again for evening rush hour after 5 PM. But there was a dip between 3 PM", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 150 + }, + { + "text": "NYC yellow taxi pickups by hour Rides bottomed out in the wee hours of the morning before rising sharply between 5 AM and 8 AM. Volume remained relatively steady throughout the day and increased again for evening rush hour after 5 PM. But there was a dip between 3 PM and 4 PM—why? To answer that question, we would need to dig deeper to analyze data that spanned several days or even several months to see whether our data from June 1, 2016, is typical. We could use the date_part() function to compare trip volume on weekdays versus weekends by extracting the day of the week. To be even more ambitious, we could check weather reports and compare trips on rainy days versus sunny days. There are many different ways to slice a data set to derive conclusions. When Do Trips Take the Longest? Let’s investigate another interesting question: at which hour did taxi trips take the longest? One way to find an answer is to calculate the median trip time for each hour. The median is the middle value in an ordered set of values; it’s often more accurate than an average for making comparisons because a few very small or very large values in the set won’t skew the results as they would with the average. In Chapter 5, we used the percentile_cont() function to find medians. Estadísticos e-Books & Papers We use it again in Listing 11-10 to calculate median trip times: SELECT ➊ date_part('hour', tpep_pickup_datetime) AS trip_hour, ➋ percentile_cont(.5) ➌ WITHIN GROUP (ORDER BY tpep_dropoff_datetime - tpep_pickup_datetime) AS median_trip FROM nyc_yellow_taxi_trips_2016_06_01 GROUP BY trip_hour ORDER BY trip_hour; Listing 11-10: Calculating median trip time by hour We’re aggregating data by the hour portion of the timestamp column tpep_pickup_datetime again, which we extract using date_part() ➊. For the input to the percentile_cont() function ➋, we subtract the pickup time from the drop-off time in the WITHIN GROUP clause ➌. The results show that the 1 PM hour has the highest median trip time of 15 minutes: date_part median_trip --------- ----------- 0 00:10:04 1 00:09:27 2 00:08:59 3 00:09:57 4 00:10:06 5 00:07:37 6 00:07:54 7 00:10:23 8 00:12:28 9 00:13:11 10 00:13:46 11 00:14:20 12 00:14:49 13 00:15:00 14 00:14:35 15 00:14:43 16 00:14:42 17 00:14:15 18 00:13:19 19 00:12:25 20 00:11:46 21 00:11:54 22 00:11:37 23 00:11:14 As we would expect, trip times are shortest in the early morning hours. This result makes sense because less traffic in the early morning means Estadísticos e-Books & Papers passengers are more likely to get to their destinations faster. Now that we’ve explored ways to extract portions of the timestamp for analysis, let’s dig deeper into analysis that involves intervals. Finding Patterns in Amtrak Data Amtrak, the nationwide rail service in America, offers several packaged trips across the United States. The All American, for example, is a train that departs from Chicago and stops in New York, New Orleans, Los Angeles, San Francisco, and Denver before returning to Chicago. Using", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 151 + }, + { + "text": "Finding Patterns in Amtrak Data Amtrak, the nationwide rail service in America, offers several packaged trips across the United States. The All American, for example, is a train that departs from Chicago and stops in New York, New Orleans, Los Angeles, San Francisco, and Denver before returning to Chicago. Using data from the Amtrak website (http://www.amtrak.com/), we’ll build a table that shows information for each segment of the trip. The trip spans four time zones, so we’ll need to track the time zones each time we enter an arrival or departure time. Then we’ll calculate the duration of the journey at each segment and figure out the length of the entire trip. Calculating the Duration of Train Trips Let’s create a table that divides The All American train route into six segments. Listing 11-11 contains SQL to create and fill a table with the departure and arrival time for each leg of the journey: SET timezone TO 'US/Central';➊ CREATE TABLE train_rides ( trip_id bigserial PRIMARY KEY, segment varchar(50) NOT NULL, departure timestamp with time zone NOT NULL,➋ arrival timestamp with time zone NOT NULL ); INSERT INTO train_rides (segment, departure, arrival)➌ VALUES ('Chicago to New York', '2017-11-13 21:30 CST', '2017-11-14 18:23 EST'), ('New York to New Orleans', '2017-11-15 14:15 EST', '2017-11-16 19:32 CST'), ('New Orleans to Los Angeles', '2017-11-17 13:45 CST', '2017-11-18 9:00 PST'), ('Los Angeles to San Francisco', '2017-11-19 10:10 PST', '2017-11-19 21:24 PST'), ('San Francisco to Denver', '2017-11-20 9:10 PST', '2017-11-21 18:38 MST'), ('Denver to Chicago', '2017-11-22 19:10 MST', '2017-11-23 14:50 CST'); SELECT * FROM train_rides; Estadísticos e-Books & Papers Listing 11-11: Creating a table to hold train trip data First, we set the session to the Central time zone, the value for Chicago, using the US/Central designator ➊. We’ll use Central time as our reference when viewing the timestamps of the data we enter so that regardless of your and my machine’s default time zones, we’ll share the same view of the data. Next, we use the standard CREATE TABLE statement. Note that columns for departures and arrival times are set to timestamp with time zone ➋. Finally, we insert rows that represent the six legs of the trip ➌. Each timestamp input reflects the time zone of the departure and arrival city. Specifying the city’s time zone is the key to getting an accurate calculation of trip duration and accounting for time zone changes. It also accounts for annual changes to and from daylight saving time if they were to occur during the time span you’re examining. The final SELECT statement should return the contents of the table like this: All timestamps should now carry a UTC offset of -06, which is equivalent to the Central time zone in the United States during the month of November, after the nation had switched to standard time. Regardless of the time zone we supplied on insert, our view of the data is now in Central time, and the times are adjusted accordingly if they’re in another time zone.", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 152 + }, + { + "text": "time zone in the United States during the month of November, after the nation had switched to standard time. Regardless of the time zone we supplied on insert, our view of the data is now in Central time, and the times are adjusted accordingly if they’re in another time zone. Now that we’ve created segments corresponding to each leg of the trip, we’ll use Listing 11-12 to calculate the duration of each segment: SELECT segment, ➊ to_char(departure, 'YYYY-MM-DD HH12:MI a.m. TZ') AS departure, Estadísticos e-Books & Papers ➋ arrival - departure AS segment_time FROM train_rides; Listing 11-12: Calculating the length of each trip segment This query lists the trip segment, the departure time, and the duration of the segment journey. Before we look at the calculation, notice the additional code around the departure column ➊. These are PostgreSQL- specific formatting functions that specify how to format different components of the timestamp. In this case, the to_char() function turns the departure timestamp column into a string of characters formatted as YYYY- MM-DD HH12:MI a.m. TZ. The YYYY-MM-DD portion specifies the ISO format for the date, and the HH12:MI a.m. portion presents the time in hours and minutes. The HH12 portion specifies the use of a 12-hour clock rather than 24-hour military time. The a.m. portion specifies that we want to show morning or night times using lowercase characters separated by periods, and the TZ portion denotes the time zone. For a complete list of formatting functions, check out the PostgreSQL documentation at https://www.postgresql.org/docs/current/static/functions- formatting.html. Last, we subtract departure from arrival to determine the segment_time ➋. When you run the query, the output should look like this: Subtracting one timestamp from another produces an interval data type, which was introduced in Chapter 3. As long as the value is less than 24 hours, PostgreSQL presents the interval in the HH:MM:SS format. For values greater than 24 hours, it returns the format 1 day 08:28:00, as shown in the San Francisco to Denver segment. Estadísticos e-Books & Papers In each calculation, PostgreSQL accounts for the changes in time zones so we don’t inadvertently add or lose hours when subtracting. If we used a timestamp without time zone data type, we would end up with an incorrect trip length if a segment spanned multiple time zones. Calculating Cumulative Trip Time As it turns out, San Francisco to Denver is the longest leg of the All American train trip. But how long does the entire trip take? To answer this question, we’ll revisit window functions, which you learned about in “Ranking with rank() and dense_rank()” on page 164. Our prior query produced an interval, which we labeled segment_time. It would seem like the natural next step would be to write a query to add those values, creating a cumulative interval after each segment. And indeed, we can use sum() as a window function, combined with the OVER clause mentioned in Chapter 10, to create running totals. But when we do, the resulting values are odd.", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 153 + }, + { + "text": "step would be to write a query to add those values, creating a cumulative interval after each segment. And indeed, we can use sum() as a window function, combined with the OVER clause mentioned in Chapter 10, to create running totals. But when we do, the resulting values are odd. To see what I mean, run the code in Listing 11-13: SELECT segment, arrival - departure AS segment_time, sum(arrival - departure) OVER (ORDER BY trip_id) AS cume_time FROM train_rides; Listing 11-13: Calculating cumulative intervals using OVER In the third column, we sum the intervals generated when we subtract departure from arrival. The resulting running total in the cume_time column is accurate but formatted in an unhelpful way: segment segment_time cume_time ---------------------------- -------------- --------------- Chicago to New York 19:53:00 19:53:00 New York to New Orleans 1 day 06:17:00 1 day 26:10:00 New Orleans to Los Angeles 21:15:00 1 day 47:25:00 Los Angeles to San Francisco 11:14:00 1 day 58:39:00 San Francisco to Denver 1 day 08:28:00 2 days 67:07:00 Denver to Chicago 18:40:00 2 days 85:47:00 PostgreSQL creates one sum for the day portion of the interval and another for the hours and minutes. So, instead of a more understandable Estadísticos e-Books & Papers cumulative time of 5 days 13:47:00, the database reports 2 days 85:47:00. Both results amount to the same length of time, but 2 days 85:47:00 is harder to decipher. This is an unfortunate limitation of summing the database intervals using this syntax. As a workaround, we’ll use the code in Listing 11-14: SELECT segment, arrival - departure AS segment_time, sum(date_part➊('epoch', (arrival - departure))) OVER (ORDER BY trip_id) * interval '1 second'➋ AS cume_time FROM train_rides; Listing 11-14: Better formatting for cumulative trip time Recall from earlier in this chapter that epoch is the number of seconds that have elapsed since midnight on January 1, 1970, which makes it useful for calculating duration. In Listing 11-14, we use date_part() ➊ with the epoch setting to extract the number of seconds elapsed between the arrival and departure intervals. Then we multiply each sum with an interval of 1 second ➋ to convert those seconds to an interval value. The output is clearer using this method: segment segment_time cume_time ---------------------------- -------------- --------- Chicago to New York 19:53:00 19:53:00 New York to New Orleans 1 day 06:17:00 50:10:00 New Orleans to Los Angeles 21:15:00 71:25:00 Los Angeles to San Francisco 11:14:00 82:39:00 San Francisco to Denver 1 day 08:28:00 115:07:00 Denver to Chicago 18:40:00 133:47:00 The final cume_time, now in HH:MM:SS format, adds all the segments to return the total trip length of 133 hours and 47 minutes. That’s a long time to spend on a train, but I’m sure the scenery is well worth the ride. Wrapping Up Handling times and dates in SQL databases adds an intriguing dimension to your analysis, letting you answer questions about when an event occurred along with other temporal concerns in your data. With a solid Estadísticos e-Books & Papers grasp of time and", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 154 + }, + { + "text": "scenery is well worth the ride. Wrapping Up Handling times and dates in SQL databases adds an intriguing dimension to your analysis, letting you answer questions about when an event occurred along with other temporal concerns in your data. With a solid Estadísticos e-Books & Papers grasp of time and date formats, time zones, and functions to dissect the components of a timestamp, you can analyze just about any data set you come across. Next, we’ll look at advanced query techniques that help answer more complex questions. TRY IT YOURSELF Try these exercises to test your skills on dates and times. 1. Using the New York City taxi data, calculate the length of each ride using the pickup and drop-off timestamps. Sort the query results from the longest ride to the shortest. Do you notice anything about the longest or shortest trips that you might want to ask city officials about? 2. Using the AT TIME ZONE keywords, write a query that displays the date and time for London, Johannesburg, Moscow, and Melbourne the moment January 1, 2100, arrives in New York City. 3. As a bonus challenge, use the statistics functions in Chapter 10 to calculate the correlation coefficient and r-squared values using trip time and the total_amount column in the New York City taxi data, which represents the total amount charged to passengers. Do the same with the trip_distance and total_amount columns. Limit the query to rides that last three hours or less. Estadísticos e-Books & Papers 12 ADVANCED QUERY TECHNIQUES Sometimes data analysis requires advanced SQL techniques that go beyond a table join or basic SELECT query. For example, to find the story in your data, you might need to write a query that uses the results of other queries as inputs. Or you might need to reclassify numerical values into categories before counting them. Like other programming languages, SQL provides a collection of functions and options essential for solving more complex problems, and that is what we’ll explore in this chapter. For the exercises, I’ll introduce a data set of temperatures recorded in select U.S. cities and we’ll revisit data sets you’ve created in previous chapters. The code for the exercises is available, along with all the book’s resources, at https://www.nostarch.com/practicalSQL/. You’ll continue to use the analysis database you’ve already built. Let’s get started. Using Subqueries A subquery is nested inside another query. Typically, it’s used for a calculation or logical test that provides a value or set of data to be passed into the main portion of the query. Its syntax is not unusual: we just enclose the subquery in parentheses and use it where needed. For example, we can write a subquery that returns multiple rows and treat the Estadísticos e-Books & Papers results as a table in the FROM clause of the main query. Or we can create a scalar subquery that returns a single value and use it as part of an expression to filter rows via WHERE, IN, and HAVING clauses.", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 155 + }, + { + "text": "multiple rows and treat the Estadísticos e-Books & Papers results as a table in the FROM clause of the main query. Or we can create a scalar subquery that returns a single value and use it as part of an expression to filter rows via WHERE, IN, and HAVING clauses. These are the most common uses of subqueries. You first encountered a subquery in Chapter 9 in the ANSI SQL standard syntax for a table UPDATE, which is shown again here. Both the data for the update and the condition that specifies which rows to update are generated by subqueries that look for values that match the columns in table and table_b: UPDATE table ➊ SET column = (SELECT column FROM table_b WHERE table.column = table_b.column) ➋ WHERE EXISTS (SELECT column FROM table_b WHERE table.column = table_b.column); This example query has two subqueries that use the same syntax. We use the SELECT statement inside parentheses ➊ as the first subquery in the SET clause, which generates values for the update. Similarly, we use a second subquery in the WHERE EXISTS clause, again with a SELECT statement ➋ to filter the rows we want to update. Both subqueries are correlated subqueries and are so named because they depend on a value or table name from the main query that surrounds them. In this case, both subqueries depend on table from the main UPDATE statement. An uncorrelated subquery has no reference to objects in the main query. It’s easier to understand these concepts by working with actual data, so let’s look at some examples. We’ll revisit two data sets from earlier chapters: the Decennial 2010 Census table us_counties_2010 you created in Chapter 4 and the meat_poultry_egg_inspect table in Chapter 9. Filtering with Subqueries in a WHERE Clause You know that a WHERE clause lets you filter query results based on criteria you provide, using an expression such as WHERE quantity > 1000. But this Estadísticos e-Books & Papers requires that you already know the value to use for comparison. What if you don’t? That’s one way a subquery comes in handy: it lets you write a query that generates one or more values to use as part of an expression in a WHERE clause. Generating Values for a Query Expression Say you wanted to write a query to show which U.S. counties are at or above the 90th percentile, or top 10 percent, for population. Rather than writing two separate queries—one to calculate the 90th percentile and the other to filter by counties—you can do both at once using a subquery in a WHERE clause, as shown in Listing 12-1: SELECT geo_name, state_us_abbreviation, p0010001 FROM us_counties_2010 ➊ WHERE p0010001 >= ( SELECT percentile_cont(.9) WITHIN GROUP (ORDER BY p0010001) FROM us_counties_2010 ) ORDER BY p0010001 DESC; Listing 12-1: Using a subquery in a WHERE clause This query is standard in terms of what we’ve done so far except that the WHERE clause ➊, which filters by the total population column p0010001, doesn’t include", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 156 + }, + { + "text": "percentile_cont(.9) WITHIN GROUP (ORDER BY p0010001) FROM us_counties_2010 ) ORDER BY p0010001 DESC; Listing 12-1: Using a subquery in a WHERE clause This query is standard in terms of what we’ve done so far except that the WHERE clause ➊, which filters by the total population column p0010001, doesn’t include a value like it normally would. Instead, after the >= comparison operators, we provide a second query in parentheses. This second query uses the percentile_cont() function in Chapter 5 to generate one value: the 90th percentile cut-off point in the p0010001 column, which will then be used in the main query. NOTE Using percentile_cont() to filter with a subquery works only if you pass in a single input, as shown. If you pass in an array, as in Listing 5-12 on page 68, percentile_cont() returns an array, and the query will fail to evaluate the >= against an array type. Estadísticos e-Books & Papers If you run the subquery separately by highlighting it in pgAdmin, you should see the results of the subquery, a value of 197444.6. But you won’t see that number when you run the entire query in Listing 12-1, because the result of that subquery is passed directly to the WHERE clause to use in filtering the results. The entire query should return 315 rows, or about 10 percent of the 3,143 rows in us_counties_2010. geo_name state_us_abbreviation p0010001 ------------------ --------------------- -------- Los Angeles County CA 9818605 Cook County IL 5194675 Harris County TX 4092459 Maricopa County AZ 3817117 San Diego County CA 3095313 --snip-- Elkhart County IN 197559 Sangamon County IL 197465 The result includes all counties with a population greater than or equal to 197444.6, the value the subquery generated. Using a Subquery to Identify Rows to Delete Adding a subquery to a WHERE clause can be useful in query statements other than SELECT. For example, we can use a similar subquery in a DELETE statement to specify what to remove from a table. Imagine you have a table with 100 million rows that, because of its size, takes a long time to query. If you just want to work on a subset of the data (such as a particular state), you can make a copy of the table and delete what you don’t need from it. Listing 12-2 shows an example of this approach. It makes a copy of the census table using the method you learned in Chapter 9 and then deletes everything from that backup except the 315 counties in the top 10 percent of population: CREATE TABLE us_counties_2010_top10 AS SELECT * FROM us_counties_2010; Estadísticos e-Books & Papers DELETE FROM us_counties_2010_top10 WHERE p0010001 < ( SELECT percentile_cont(.9) WITHIN GROUP (ORDER BY p0010001) FROM us_counties_2010_top10 ); Listing 12-2: Using a subquery in a WHERE clause with DELETE Run the code in Listing 12-2, and then execute SELECT count(*) FROM us_counties_2010_top10; to count the remaining rows in the table. The result should be 315 rows, which is the original 3,143 minus the 2,828 the subquery", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 157 + }, + { + "text": "us_counties_2010_top10 ); Listing 12-2: Using a subquery in a WHERE clause with DELETE Run the code in Listing 12-2, and then execute SELECT count(*) FROM us_counties_2010_top10; to count the remaining rows in the table. The result should be 315 rows, which is the original 3,143 minus the 2,828 the subquery deleted. Creating Derived Tables with Subqueries If your subquery returns rows and columns of data, you can convert that data to a table by placing it in a FROM clause, the result of which is known as a derived table. A derived table behaves just like any other table, so you can query it or join it to other tables, even other derived tables. This approach is helpful when a single query can’t perform all the operations you need. Let’s look at a simple example. In Chapter 5, you learned the difference between average and median values. I explained that a median can often better indicate a data set’s central value because a few very large or small values (or outliers) can skew an average. For that reason, I often recommend comparing the average and median. If they’re close, the data probably falls in a normal distribution (the familiar bell curve), and the average is a good representation of the central value. If the average and median are far apart, some outliers might be having an effect or the distribution is skewed, not normal. Finding the average and median population of U.S. counties as well as the difference between them is a two-step process. We need to calculate the average and the median, and then we need to subtract the two. We can do both operations in one fell swoop with a subquery in the FROM clause, as shown in Listing 12-3. SELECT round(calcs.average, 0) AS average, calcs.median, Estadísticos e-Books & Papers round(calcs.average - calcs.median, 0) AS median_average_diff FROM ( ➊ SELECT avg(p0010001) AS average, percentile_cont(.5) WITHIN GROUP (ORDER BY p0010001)::numeric(10,1) AS median FROM us_counties_2010 ) ➋ AS calcs; Listing 12-3: Subquery as a derived table in a FROM clause The subquery ➊ is straightforward. We use the avg() and percentile_cont() functions to find the average and median of the census table’s p0010001 total population column and name each column with an alias. Then we name the subquery with an alias ➋ of calcs so we can reference it as a table in the main query. Subtracting the median from the average, both of which are returned by the subquery, is done in the main query; then the main query rounds the result and labels it with the alias median_average_diff. Run the query, and the result should be the following: average median median_average_diff ------- ------- ------------------- 98233 25857.0 72376 The difference between the median and average, 72,736, is nearly three times the size of the median. That helps show that a relatively small number of high-population counties push the average county size over 98,000, whereas the median of all counties is much less at 25,857. Joining Derived Tables Because derived", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 158 + }, + { + "text": "difference between the median and average, 72,736, is nearly three times the size of the median. That helps show that a relatively small number of high-population counties push the average county size over 98,000, whereas the median of all counties is much less at 25,857. Joining Derived Tables Because derived tables behave like regular tables, you can join them. Joining derived tables lets you perform multiple preprocessing steps before arriving at the result. For example, say we wanted to determine which states have the most meat, egg, and poultry processing plants per million population; before we can calculate that rate, we need to know the number of plants in each state and the population of each state. We start by counting producers by state using the Estadísticos e-Books & Papers meat_poultry_egg_inspect table in Chapter 9. Then we can use the us_counties_2010 table to count population by state by summing and grouping county values. Listing 12-4 shows how to write subqueries for both tasks and join them to calculate the overall rate. SELECT census.state_us_abbreviation AS st, census.st_population, plants.plant_count, ➊ round((plants.plant_count/census.st_population::numeric(10,1))*1000000, 1) AS plants_per_million FROM ( ➋ SELECT st, count(*) AS plant_count FROM meat_poultry_egg_inspect GROUP BY st ) AS plants JOIN ( ➌ SELECT state_us_abbreviation, sum(p0010001) AS st_population FROM us_counties_2010 GROUP BY state_us_abbreviation ) AS census ➍ ON plants.st = census.state_us_abbreviation ORDER BY plants_per_million DESC; Listing 12-4: Joining two derived tables You learned how to calculate rates in Chapter 10, so the math and syntax in the main query for finding plants_per_million ➊ should be familiar. We divide the number of plants by the population, and then multiply that quotient by 1 million. For the inputs, we use the values generated from derived tables using subqueries. The first subquery ➋ finds the number of plants in each state using the count() aggregate function and then groups them by state. We label this subquery with the plants alias for reference in the main part of the query. The second subquery ➌ finds the total population by state by using sum() on the p0010001 total population column and then groups those by state_us_abbreviation. We alias this derived table as census. Estadísticos e-Books & Papers Next, we join the derived tables ➍ by linking the st column in plants to the state_us_abbreviation column in census. We then list the results in descending order based on the calculated rates. Here’s a sample output of 51 rows showing the highest and lowest rates: st st_population plant_count plants_per_million -- ------------- ----------- ------------------ NE 1826341 110 60.2 IA 3046355 149 48.9 VT 625741 27 43.1 HI 1360301 47 34.6 ND 672591 22 32.7 --snip-- SC 4625364 55 11.9 LA 4533372 49 10.8 AZ 6392017 37 5.8 DC 601723 2 3.3 WY 563626 1 1.8 The results line up with what we might expect. The top states are well-known meat producers. For example, Nebraska is one of the nation’s top cattle exporters, and Iowa leads the United States in pork production. Washington, D.C., and Wyoming at the bottom of the", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 159 + }, + { + "text": "3.3 WY 563626 1 1.8 The results line up with what we might expect. The top states are well-known meat producers. For example, Nebraska is one of the nation’s top cattle exporters, and Iowa leads the United States in pork production. Washington, D.C., and Wyoming at the bottom of the list are among those states with the fewest plants per million. NOTE Your results will differ slightly if you didn’t add missing state values to the meat_poultry_egg_inspect table as noted in “Updating Rows Where Values Are Missing” on page 141. Generating Columns with Subqueries You can also generate new columns of data with subqueries by placing a subquery in the column list after SELECT. Typically, you would use a single value from an aggregate. For example, the query in Listing 12-5 selects the geo_name and total population column p0010001 from us_counties_2010, and then adds a subquery to add the median of all counties to each row in the new column us_median: Estadísticos e-Books & Papers SELECT geo_name, state_us_abbreviation AS st, p0010001 AS total_pop, (SELECT percentile_cont(.5) WITHIN GROUP (ORDER BY p0010001) FROM us_counties_2010) AS us_median FROM us_counties_2010; Listing 12-5: Adding a subquery to a column list The first rows of the result set should look like this: geo_name st total_pop us_median -------------- -- --------- --------- Autauga County AL 54571 25857 Baldwin County AL 182265 25857 Barbour County AL 27457 25857 Bibb County AL 22915 25857 Blount County AL 57322 25857 --snip-- On its own, that repeating us_median value isn’t very helpful because it’s the same each time. It would be more interesting and useful to generate values that indicate how much each county’s population deviates from the median value. Let’s look at how we can use the same subquery technique to do that. Listing 12-6 builds on Listing 12-5 by adding a subquery expression after SELECT that calculates the difference between the population and the median for each county: SELECT geo_name, state_us_abbreviation AS st, p0010001 AS total_pop, (SELECT percentile_cont(.5) WITHIN GROUP (ORDER BY p0010001) FROM us_counties_2010) AS us_median, ➊ p0010001 - (SELECT percentile_cont(.5) WITHIN GROUP (ORDER BY p0010001) FROM us_counties_2010) AS diff_from_median FROM us_counties_2010 ➋ WHERE (p0010001 - (SELECT percentile_cont(.5) WITHIN GROUP (ORDER BY p0010001) FROM us_counties_2010)) BETWEEN -1000 AND 1000; Listing 12-6: Using a subquery expression in a calculation The added subquery ➊ is part of a column definition that subtracts the subquery’s result from p0010001, the total population. It puts that new data in a column with an alias of diff_from_median. To make this query even Estadísticos e-Books & Papers more useful, we can narrow the results further to show only counties whose population falls within 1,000 of the median. This would help us identify which counties in America have close to the median county population. To do this, we repeat the subquery expression in the WHERE clause ➋ and filter results using the BETWEEN -1000 AND 1000 expression. The outcome should reveal 71 counties with a population relatively close to the U.S. median. Here are the first five", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 160 + }, + { + "text": "have close to the median county population. To do this, we repeat the subquery expression in the WHERE clause ➋ and filter results using the BETWEEN -1000 AND 1000 expression. The outcome should reveal 71 counties with a population relatively close to the U.S. median. Here are the first five rows of the results: Bear in mind that subqueries add to overall query execution time; therefore, if we were working with millions of rows, we could simplify Listing 12-6 by eliminating the subquery that displays the column us_median. I’ve left it in this example for your reference. Subquery Expressions You can also use subqueries to filter rows by evaluating whether a condition evaluates as true or false. For this, we can use several standard ANSI SQL subquery expressions, which are a combination of a keyword with a subquery and are generally used in WHERE clauses to filter rows based on the existence of values in another table. The PostgreSQL documentation at https://www.postgresql.org/docs/current/static/functions-subquery.html lists available subquery expressions, but here we’ll examine the syntax for just two of them. Generating Values for the IN Operator Estadísticos e-Books & Papers The subquery expression IN (subquery) is like the IN comparison operator in Chapter 2 except we use a subquery to provide the list of values to check against rather than having to manually provide one. In the following example, we use a subquery to generate id values from a retirees table, and then use that list for the IN operator in the WHERE clause. The NOT IN expression does the opposite to find employees whose id value does not appear in retirees. SELECT first_name, last_name FROM employees WHERE id IN ( SELECT id FROM retirees); We would expect the output to show the names of employees who have id values that match those in retirees. NOTE The presence of NULL values in a subquery result set will cause a query with a NOT IN expression to return no rows. If your data contains NULL values, use the WHERE NOT EXISTS expression described in the next section. Checking Whether Values Exist Another subquery expression, EXISTS (subquery), is a true/false test. It returns a value of true if the subquery in parentheses returns at least one row. If it returns no rows, EXISTS evaluates to false. In the following example, the query returns all names from an employees table as long as the subquery finds at least one value in id in a retirees table. SELECT first_name, last_name FROM employees WHERE EXISTS ( SELECT id FROM retirees); Rather than return all names from employees, we instead could mimic the behavior of IN and limit names to where the subquery after EXISTS finds Estadísticos e-Books & Papers at least one corresponding id value in retirees. The following is a correlated subquery because the table named in the main query is referenced in the subquery. SELECT first_name, last_name FROM employees WHERE EXISTS ( SELECT id FROM retirees WHERE id = employees.id); This approach is particularly helpful if", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 161 + }, + { + "text": "Papers at least one corresponding id value in retirees. The following is a correlated subquery because the table named in the main query is referenced in the subquery. SELECT first_name, last_name FROM employees WHERE EXISTS ( SELECT id FROM retirees WHERE id = employees.id); This approach is particularly helpful if you need to join on more than one column, which you can’t do with the IN expression. You can also use the NOT keyword with EXISTS. For example, to find employees with no corresponding record in retirees, you would run this query: SELECT first_name, last_name FROM employees WHERE NOT EXISTS ( SELECT id FROM retirees WHERE id = employees.id); The technique of using NOT with EXISTS is helpful for assessing whether a data set is complete. Common Table Expressions Earlier in this chapter, you learned how to create derived tables by placing subqueries in a FROM clause. A second approach to creating temporary tables for querying uses the Common Table Expression (CTE), a relatively recent addition to standard SQL that’s informally called a “WITH clause.” Using a CTE, you can define one or more tables up front with subqueries. Then you can query the table results as often as needed in a main query that follows. Listing 12-7 shows a simple CTE called large_counties based on our census data, followed by a query of that table. The code determines how many counties in each state have 100,000 people or more. Let’s walk Estadísticos e-Books & Papers through the example. ➊ WITH large_counties (geo_name, st, p0010001) AS ( ➋ SELECT geo_name, state_us_abbreviation, p0010001 FROM us_counties_2010 WHERE p0010001 >= 100000 ) ➌ SELECT st, count(*) FROM large_counties GROUP BY st ORDER BY count(*) DESC; Listing 12-7: Using a simple CTE to find large counties The WITH ... AS block ➊ defines the CTE’s temporary table large_counties. After WITH, we name the table and list its column names in parentheses. Unlike column definitions in a CREATE TABLE statement, we don’t need to provide data types, because the temporary table inherits those from the subquery ➋, which is enclosed in parentheses after AS. The subquery must return the same number of columns as defined in the temporary table, but the column names don’t need to match. Also, the column list is optional if you’re not renaming columns, although including the list is still a good idea for clarity even if you don’t rename columns. The main query ➌ counts and groups the rows in large_counties by st, and then orders by the count in descending order. The top five rows of the results should look like this: st count -- ----- TX 39 CA 35 FL 33 PA 31 OH 28 --snip-- As you can see, Texas, California, and Florida are among the states with the highest number of counties with a population of 100,000 or Estadísticos e-Books & Papers more. You could find the same results using a SELECT query instead of a CTE, as shown here: SELECT state_us_abbreviation, count(*) FROM us_counties_2010 WHERE", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 162 + }, + { + "text": "can see, Texas, California, and Florida are among the states with the highest number of counties with a population of 100,000 or Estadísticos e-Books & Papers more. You could find the same results using a SELECT query instead of a CTE, as shown here: SELECT state_us_abbreviation, count(*) FROM us_counties_2010 WHERE p0010001 >= 100000 GROUP BY state_us_abbreviation ORDER BY count(*) DESC; So why use a CTE? One reason is that by using a CTE, you can pre- stage subsets of data to feed into a larger query for more complex analysis. Also, you can reuse each table defined in a CTE in multiple places in the main query, which means you don’t have to repeat the SELECT query each time. Another commonly cited advantage is that the code is more readable than if you performed the same operation with subqueries. Listing 12-8 uses a CTE to rewrite the join of derived tables in Listing 12-4 (finding the states that have the most meat, egg, and poultry processing plants per million population) into a more readable format: WITH ➊ counties (st, population) AS (SELECT state_us_abbreviation, sum(population_count_100_percent) FROM us_counties_2010 GROUP BY state_us_abbreviation), ➋ plants (st, plants) AS (SELECT st, count(*) AS plants FROM meat_poultry_egg_inspect GROUP BY st) SELECT counties.st, population, plants, round((plants/population::numeric(10,1)) * 1000000, 1) AS per_million ➌ FROM counties JOIN plants ON counties.st = plants.st ORDER BY per_million DESC; Listing 12-8: Using CTEs in a table join Following the WITH keyword, we define two tables using subqueries. The first subquery, counties ➊, returns the population of each state. The second, plants ➋, returns the number of plants per state. With those tables Estadísticos e-Books & Papers defined, we join them ➌ on the st column in each table and calculate the rate per million. The results are identical to the joined derived tables in Listing 12-4, but Listing 12-8 is easier to read. As another example, you can use a CTE to simplify queries with redundant code. For example, in Listing 12-6, we used a subquery with the percentile_cont() function in three different locations to find median county population. In Listing 12-9, we can write that subquery just once as a CTE: ➊ WITH us_median AS (SELECT percentile_cont(.5) WITHIN GROUP (ORDER BY p0010001) AS us_median_pop FROM us_counties_2010) SELECT geo_name, state_us_abbreviation AS st, p0010001 AS total_pop, ➋ us_median_pop, ➌ p0010001 - us_median_pop AS diff_from_median ➍ FROM us_counties_2010 CROSS JOIN us_median ➎ WHERE (p0010001 - us_median_pop) BETWEEN -1000 AND 1000; Listing 12-9: Using CTEs to minimize redundant code After the WITH keyword, we define us_median ➊ as the result of the same subquery used in Listing 12-6, which finds the median population using percentile_cont(). Then we reference the us_median_pop column on its own ➋, as part of a calculated column ➌, and in a WHERE clause ➎. To make the value available to every row in the us_counties_2010 table during SELECT, we use the CROSS JOIN query ➍ you learned in Chapter 6. This query provides identical results to those in Listing 12-6, but", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 163 + }, + { + "text": "as part of a calculated column ➌, and in a WHERE clause ➎. To make the value available to every row in the us_counties_2010 table during SELECT, we use the CROSS JOIN query ➍ you learned in Chapter 6. This query provides identical results to those in Listing 12-6, but we only had to write the subquery once to find the median. Not only does this save time, but it also lets you revise the query more easily. For example, to find counties whose population is close to the 90th percentile, you can substitute .9 for .5 as input to percentile_cont() in just one place. Cross Tabulations Estadísticos e-Books & Papers Cross tabulations provide a simple way to summarize and compare variables by displaying them in a table layout, or matrix. In a matrix, rows represent one variable, columns represent another variable, and each cell where a row and column intersects holds a value, such as a count or percentage. You’ll often see cross tabulations, also called pivot tables or crosstabs, used to report summaries of survey results or to compare sets of variables. A frequent example happens during every election when candidates’ votes are tallied by geography: candidate ward 1 ward 2 ward 3 --------- ------ ------ ------ Dirk 602 1,799 2,112 Pratt 599 1,398 1,616 Lerxst 911 902 1,114 In this case, the candidates’ names are one variable, the wards (or city districts) are another variable, and the cells at the intersection of the two hold the vote totals for that candidate in that ward. Let’s look at how to generate cross tabulations. Installing the crosstab() Function Standard ANSI SQL doesn’t have a crosstab function, but PostgreSQL does as part of a module you can install easily. Modules include PostgreSQL extras that aren’t part of the core application; they include functions related to security, text search, and more. You can find a list of PostgreSQL modules at https://www.postgresql.org/docs/current/static/contrib.html. PostgreSQL’s crosstab() function is part of the tablefunc module. To install tablefunc in the pgAdmin Query Tool, execute this command: CREATE EXTENSION tablefunc; PostgreSQL should return the message CREATE EXTENSION when it’s done installing. (If you’re working with another database management system, check the documentation to see whether it offers a similar functionality. Estadísticos e-Books & Papers For example, Microsoft SQL Server has the PIVOT command.) Next, we’ll create a basic crosstab so you can learn the syntax, and then we’ll handle a more complex case. Tabulating Survey Results Let’s say your company needs a fun employee activity, so you coordinate an ice cream social at your three offices in the city. The trouble is, people are particular about ice cream flavors. To choose flavors people will like, you decide to conduct a survey. The CSV file ice_cream_survey.csv contains 200 responses to your survey. You can download this file, along with all the book’s resources, at https://www.nostarch.com/practicalSQL/. Each row includes a response_id, office, and flavor. You’ll need to count how many people chose each flavor at each office and present", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 164 + }, + { + "text": "to conduct a survey. The CSV file ice_cream_survey.csv contains 200 responses to your survey. You can download this file, along with all the book’s resources, at https://www.nostarch.com/practicalSQL/. Each row includes a response_id, office, and flavor. You’ll need to count how many people chose each flavor at each office and present the results in a readable way to your colleagues. In your analysis database, use the code in Listing 12-10 to create a table and load the data. Make sure you change the file path to the location on your computer where you saved the CSV file. CREATE TABLE ice_cream_survey ( response_id integer PRIMARY KEY, office varchar(20), flavor varchar(20) ); COPY ice_cream_survey FROM 'C:\\YourDirectory\\ice_cream_survey.csv' WITH (FORMAT CSV, HEADER); Listing 12-10: Creating and filling the ice_cream_survey table If you want to inspect the data, run the following to view the first five rows: SELECT * FROM ice_cream_survey LIMIT 5; The data should look like this: Estadísticos e-Books & Papers response_id office flavor ----------- -------- ---------- 1 Uptown Chocolate 2 Midtown Chocolate 3 Downtown Strawberry 4 Uptown Chocolate 5 Midtown Chocolate It looks like chocolate is in the lead! But let’s confirm this choice by using the code in Listing 12-11 to generate a crosstab from the table: SELECT * ➊ FROM crosstab('SELECT ➋office, ➌flavor, ➍count(*) FROM ice_cream_survey GROUP BY office, flavor ORDER BY office', ➎ 'SELECT flavor FROM ice_cream_survey GROUP BY flavor ORDER BY flavor') ➏ AS (office varchar(20), chocolate bigint, strawberry bigint, vanilla bigint); Listing 12-11: Generating the ice cream survey crosstab The query begins with a SELECT * statement that selects everything from the contents of the crosstab() function ➊. We place two subqueries inside the crosstab() function. The first subquery generates the data for the crosstab and has three required columns. The first column, office ➋, supplies the row names for the crosstab, and the second column, flavor ➌, supplies the category columns. The third column supplies the values for each cell where row and column intersect in the table. In this case, we want the intersecting cells to show a count() ➍ of each flavor selected at each office. This first subquery on its own creates a simple aggregated list. The second subquery ➎ produces the set of category names for the columns. The crosstab() function requires that the second subquery return Estadísticos e-Books & Papers only one column, so here we use SELECT to retrieve the flavor column, and we use GROUP BY to return that column’s unique values. Then we specify the names and data types of the crosstab’s output columns following the AS keyword ➏. The list must match the row and column names in the order the subqueries generate them. For example, because the second subquery that supplies the category columns orders the flavors alphabetically, the output column list does as well. When we run the code, our data displays in a clean, readable crosstab: office chocolate strawberry vanilla -------- --------- ---------- ------- Downtown 23 32 19 Midtown 41 23 Uptown 22 17 23", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 165 + }, + { + "text": "second subquery that supplies the category columns orders the flavors alphabetically, the output column list does as well. When we run the code, our data displays in a clean, readable crosstab: office chocolate strawberry vanilla -------- --------- ---------- ------- Downtown 23 32 19 Midtown 41 23 Uptown 22 17 23 It’s easy to see at a glance that the Midtown office favors chocolate but has no interest in strawberry, which is represented by a NULL value showing that strawberry received no votes. But strawberry is the top choice Downtown, and the Uptown office is more evenly split among the three flavors. Tabulating City Temperature Readings Let’s create another crosstab, but this time we’ll use real data. The temperature_readings.csv file, also available with all the book’s resources at https://www.nostarch.com/practicalSQL/, contains a year’s worth of daily temperature readings from three observation stations around the United States: Chicago, Seattle, and Waikiki, a neighborhood on the south shore of the city of Honolulu. The data come from the U.S. National Oceanic and Atmospheric Administration (NOAA) at https://www.ncdc.noaa.gov/cdo-web/datatools/findstation/. Each row in the CSV file contains four values: the station name, the date, the day’s maximum temperature, and the day’s minimum temperature. All temperatures are in Fahrenheit. For each month in each city, we want to calculate the median high temperature so we can compare climates. Listing 12-12 contains the code to create the Estadísticos e-Books & Papers temperature_readings table and import the CSV file: CREATE TABLE temperature_readings ( reading_id bigserial, station_name varchar(50), observation_date date, max_temp integer, min_temp integer ); COPY temperature_readings (station_name, observation_date, max_temp, min_temp) FROM 'C:\\YourDirectory\\temperature_readings.csv' WITH (FORMAT CSV, HEADER); Listing 12-12: Creating and filling a temperature_readings table The table contains the four columns from the CSV file along with an added reading_id of type bigserial that we use as a surrogate primary key. If you perform a quick count on the table, you should have 1,077 rows. Now, let’s see what cross tabulating the data does using Listing 12-13: SELECT * FROM crosstab('SELECT ➊ station_name, ➋ date_part(''month'', observation_date), ➌ percentile_cont(.5) WITHIN GROUP (ORDER BY max_temp) FROM temperature_readings GROUP BY station_name, date_part(''month'', observation_date) ORDER BY station_name', 'SELECT month FROM ➍generate_series(1,12) month') AS (station varchar(50), jan numeric(3,0), feb numeric(3,0), mar numeric(3,0), apr numeric(3,0), may numeric(3,0), jun numeric(3,0), jul numeric(3,0), aug numeric(3,0), sep numeric(3,0), oct numeric(3,0), nov numeric(3,0), dec numeric(3,0) ); Estadísticos e-Books & Papers Listing 12-13: Generating the temperature readings crosstab The structure of the crosstab is the same as in Listing 12-11. The first subquery inside the crosstab() function generates the data for the crosstab, calculating the median maximum temperature for each month. It supplies the three required columns. The first column, station_name ➊, names the rows. The second column uses the date_part() function ➋ you learned in Chapter 11 to extract the month from observation_date, which provides the crosstab columns. Then we use percentile_cont(.5) ➌ to find the 50th percentile, or the median, of the max_temp. We group by station name and month so we have a median max_temp for each month at", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 166 + }, + { + "text": "➋ you learned in Chapter 11 to extract the month from observation_date, which provides the crosstab columns. Then we use percentile_cont(.5) ➌ to find the 50th percentile, or the median, of the max_temp. We group by station name and month so we have a median max_temp for each month at each station. As in Listing 12-11, the second subquery produces the set of category names for the columns. I’m using a function called generate_series() ➍ in a manner noted in the official PostgreSQL documentation to create a list of numbers from 1 to 12 that match the month numbers date_part() extracts from observation_date. Following AS, we provide the names and data types for the crosstab’s output columns. Each is a numeric type, matching the output of the percentile function. The following output is practically poetry: We’ve transformed a raw set of daily readings into a compact table showing the median maximum temperature each month for each station. You can see at a glance that the temperature in Waikiki is consistently balmy, whereas Chicago’s median high temperatures vary from just above freezing to downright pleasant. Seattle falls between the two. Crosstabs do take time to set up, but viewing data sets in a matrix often makes comparisons easier than viewing the same data in a vertical list. Keep in mind that the crosstab() function is CPU-intensive, so tread carefully when querying sets that have millions or billions of rows. Estadísticos e-Books & Papers Reclassifying Values with CASE The ANSI Standard SQL CASE statement is a conditional expression, meaning it lets you add some “if this, then . . .” logic to a query. You can use CASE in multiple ways, but for data analysis, it’s handy for reclassifying values into categories. You can create categories based on ranges in your data and classify values according to those categories. The CASE syntax follows this pattern: ➊ CASE WHEN condition THEN result ➋ WHEN another_condition THEN result ➌ ELSE result ➍ END We give the CASE keyword ➊, and then provide at least one WHEN condition THEN result clause, where condition is any expression the database can evaluate as true or false, such as county = 'Dutchess County' or date > '1995-08- 09'. If the condition is true, the CASE statement returns the result and stops checking any further conditions. The result can be any valid data type. If the condition is false, the database moves on to evaluate the next condition. To evaluate more conditions, we can add optional WHEN ... THEN clauses ➋. We can also provide an optional ELSE clause ➌ to return a result in case no condition evaluates as true. Without an ELSE clause, the statement would return a NULL when no conditions are true. The statement finishes with an END keyword ➍. Listing 12-14 shows how to use the CASE statement to reclassify the temperature readings data into descriptive groups (named according to my own bias against cold weather): SELECT max_temp, CASE WHEN max_temp >= 90 THEN", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 167 + }, + { + "text": "a NULL when no conditions are true. The statement finishes with an END keyword ➍. Listing 12-14 shows how to use the CASE statement to reclassify the temperature readings data into descriptive groups (named according to my own bias against cold weather): SELECT max_temp, CASE WHEN max_temp >= 90 THEN 'Hot' WHEN max_temp BETWEEN 70 AND 89 THEN 'Warm' WHEN max_temp BETWEEN 50 AND 69 THEN 'Pleasant' WHEN max_temp BETWEEN 33 AND 49 THEN 'Cold' WHEN max_temp BETWEEN 20 AND 32 THEN 'Freezing' ELSE 'Inhumane' END AS temperature_group FROM temperature_readings; Estadísticos e-Books & Papers Listing 12-14: Reclassifying temperature data with CASE We create five ranges for the max_temp column in temperature_readings, which we define using comparison operators. The CASE statement evaluates each value to find whether any of the five expressions are true. If so, the statement outputs the appropriate text. Note that the ranges account for all possible values in the column, leaving no gaps. If none of the statements is true, then the ELSE clause assigns the value to the category Inhumane. The way I’ve structured the ranges, this happens only when max_temp is below 20 degrees. Alternatively, we could replace ELSE with a WHEN clause that looks for temperatures less than or equal to 19 degrees by using max_temp <= 19. Run the code; the first five rows of output should look like this: max_temp temperature_group -------- ----------------- 31 Freezing 34 Cold 32 Freezing 32 Freezing 34 Cold --snip-- Now that we’ve collapsed the data set into six categories, let’s use those categories to compare climate among the three cities in the table. Using CASE in a Common Table Expression The operation we performed with CASE on the temperature data in the previous section is a good example of a preprocessing step you would use in a CTE. Now that we’ve grouped the temperatures in categories, let’s count the groups by city in a CTE to see how many days of the year fall into each temperature category. Listing 12-15 shows the code for reclassifying the daily maximum temperatures recast to generate a temps_collapsed CTE and then use it for an analysis: Estadísticos e-Books & Papers ➊ WITH temps_collapsed (station_name, max_temperature_group) AS (SELECT station_name, CASE WHEN max_temp >= 90 THEN 'Hot' WHEN max_temp BETWEEN 70 AND 89 THEN 'Warm' WHEN max_temp BETWEEN 50 AND 69 THEN 'Pleasant' WHEN max_temp BETWEEN 33 AND 49 THEN 'Cold' WHEN max_temp BETWEEN 20 AND 32 THEN 'Freezing' ELSE 'Inhumane' END FROM temperature_readings) ➋ SELECT station_name, max_temperature_group, count(*) FROM temps_collapsed GROUP BY station_name, max_temperature_group ORDER BY station_name, count(*) DESC; Listing 12-15: Using CASE in a CTE This code reclassifies the temperatures, and then counts and groups by station name to find general climate classifications of each city. The WITH keyword defines the CTE of temps_collapsed ➊, which has two columns: station_name and max_temperature_group. We then run a SELECT query on the CTE ➋, performing straightforward count(*) and GROUP BY operations on both columns. The results should look like this: station_name max_temperature_group count", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 168 + }, + { + "text": "climate classifications of each city. The WITH keyword defines the CTE of temps_collapsed ➊, which has two columns: station_name and max_temperature_group. We then run a SELECT query on the CTE ➋, performing straightforward count(*) and GROUP BY operations on both columns. The results should look like this: station_name max_temperature_group count ------------------------------ --------------------- ----- CHICAGO NORTHERLY ISLAND IL US Warm 133 CHICAGO NORTHERLY ISLAND IL US Cold 92 CHICAGO NORTHERLY ISLAND IL US Pleasant 91 CHICAGO NORTHERLY ISLAND IL US Freezing 30 CHICAGO NORTHERLY ISLAND IL US Inhumane 8 CHICAGO NORTHERLY ISLAND IL US Hot 8 SEATTLE BOEING FIELD WA US Pleasant 198 SEATTLE BOEING FIELD WA US Warm 98 SEATTLE BOEING FIELD WA US Cold 50 SEATTLE BOEING FIELD WA US Hot 3 WAIKIKI 717.2 HI US Warm 361 WAIKIKI 717.2 HI US Hot 5 Using this classification scheme, the amazingly consistent Waikiki weather, with Warm maximum temperatures 361 days of the year, confirms its appeal as a vacation destination. From a temperature standpoint, Seattle looks good too, with nearly 300 days of high temps categorized as Pleasant or Warm (although this belies Seattle’s legendary rainfall). Chicago, with 30 days of Freezing max temps and 8 days Inhumane, probably isn’t for Estadísticos e-Books & Papers me. Wrapping Up In this chapter, you learned to make queries work harder for you. You can now add subqueries in multiple locations to provide finer control over filtering or preprocessing data before analyzing it in a main query. You also can visualize data in a matrix using cross tabulations and reclassify data into groups; both techniques give you more ways to find and tell stories using your data. Great work! Throughout the next chapters, we’ll dive into SQL techniques that are more specific to PostgreSQL. We’ll begin by working with and searching text and strings. TRY IT YOURSELF Here are two tasks to help you become more familiar with the concepts introduced in the chapter: 1. Revise the code in Listing 12-15 to dig deeper into the nuances of Waikiki’s high temperatures. Limit the temps_collapsed table to the Waikiki maximum daily temperature observations. Then use the WHEN clauses in the CASE statement to reclassify the temperatures into seven groups that would result in the following text output: '90 or more' '88-89' '86-87' '84-85' '82-83' '80-81' '79 or less' In which of those groups does Waikiki’s daily maximum temperature fall most often? Estadísticos e-Books & Papers 2. Revise the ice cream survey crosstab in Listing 12-11 to flip the table. In other words, make flavor the rows and office the columns. Which elements of the query do you need to change? Are the counts different? Estadísticos e-Books & Papers 13 MINING TEXT TO FIND MEANINGFUL DATA Although it might not be obvious at first glance, you can extract data and even quantify data from text in speeches, reports, press releases, and other documents. Even though most text exists as unstructured or semi-structured data, which is not organized in rows and columns, as in a table,", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 169 + }, + { + "text": "DATA Although it might not be obvious at first glance, you can extract data and even quantify data from text in speeches, reports, press releases, and other documents. Even though most text exists as unstructured or semi-structured data, which is not organized in rows and columns, as in a table, you can use SQL to derive meaning from it. One way to do this is to transform the text into structured data. You search for and extract elements such as dates or codes from the text, load them into a table, and analyze them. Another way to find meaning from textual data is to use advanced text analysis features, such as PostgreSQL’s full text search. Using these techniques, ordinary text can reveal facts or trends that might otherwise remain hidden. In this chapter, you’ll learn how to use SQL to analyze and transform text. You’ll start with simple text wrangling using string formatting and pattern matching before moving on to more advanced analysis functions. We’ll use two data sets as examples: a small collection of crime reports from a sheriff’s department near Washington, D.C., and a set of State of the Union addresses delivered by former U.S. presidents. Formatting Text Using String Functions Estadísticos e-Books & Papers Whether you’re looking for data in text or simply want to change how it looks in a report, you first need to transform it into a format you can use. PostgreSQL has more than 50 built-in string functions that handle routine but necessary tasks, such as capitalizing letters, combining strings, and removing unwanted spaces. Some are part of the ANSI SQL standard, and others are specific to PostgreSQL. You’ll find a complete list of string functions at https://www.postgresql.org/docs/current/static/functions-string.html, but in this section we’ll examine several that you’ll likely use most often. You can use these functions inside a variety of queries. Let’s try one now using a simple query that places a function after SELECT and runs it in the pgAdmin Query Tool, like this: SELECT upper('hello');. Examples of each function plus code for all the listings in this chapter are available at https://www.nostarch.com/practicalSQL/. Case Formatting The capitalization functions format the text’s case. The upper(string) function capitalizes all alphabetical characters of a string passed to it. Nonalphabet characters, such as numbers, remain unchanged. For example, upper('Neal7') returns NEAL7. The lower(string) function lowercases all alphabetical characters while keeping nonalphabet characters unchanged. For example, lower('Randy') returns randy. The initcap(string) function capitalizes the first letter of each word. For example, initcap('at the end of the day') returns At The End Of The Day. This function is handy for formatting titles of books or movies, but because it doesn’t recognize acronyms, it’s not always the perfect solution. For example, initcap('Practical SQL') would return Practical Sql, because it doesn’t recognize SQL as an acronym. The upper() and lower() functions are ANSI SQL standard commands, but initcap() is PostgreSQL-specific. These three functions give you enough options to rework a column of text into the case you prefer. Note that", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 170 + }, + { + "text": "For example, initcap('Practical SQL') would return Practical Sql, because it doesn’t recognize SQL as an acronym. The upper() and lower() functions are ANSI SQL standard commands, but initcap() is PostgreSQL-specific. These three functions give you enough options to rework a column of text into the case you prefer. Note that capitalization does not work with all locales or languages. Estadísticos e-Books & Papers Character Information Several functions return data about the string rather than transforming it. These functions are helpful on their own or combined with other functions. For example, the char_length(string) function returns the number of characters in a string, including any spaces. For example, char_length(' Pat ') returns a value of 5, because the three letters in Pat and the spaces on either end total five characters. You can also use the non- ANSI SQL function length(string) to count strings, which has a variant that lets you count the length of binary strings. NOTE The length() function can return a different value than char_length() when used with multibyte encodings, such as character sets covering the Chinese, Japanese, or Korean languages. The position(substring in string) function returns the location of the substring characters in the string. For example, position(', ' in 'Tan, Bella') returns 4, because the comma and space characters (, ) specified in the substring passed as the first parameter start at the fourth index position in the main string Tan, Bella. Both char_length() and position() are in the ANSI SQL standard. Removing Characters The trim(characters from string) function removes unwanted characters from strings. To declare one or more characters to remove, add them to the function followed by the keyword from and the main string you want to change. Options to remove leading characters (at the front of the string), trailing characters (at the end of the string), or both make this function super flexible. For example, trim('s' from 'socks') removes all s characters and returns ock. To remove only the s at the end of the string, add the trailing Estadísticos e-Books & Papers keyword before the character to trim: trim(trailing 's' from 'socks') returns sock. If you don’t specify any characters to remove, trim() removes any spaces in the string by default. For example, trim(' Pat ') returns Pat without the leading or trailing spaces. To confirm the length of the trimmed string, we can nest trim() inside char_length() like this: SELECT char_length(trim(' Pat ')); This query should return 3, the number of letters in Pat, which is the result of trim(' Pat '). The ltrim(string, characters) and rtrim(string, characters) functions are PostgreSQL-specific variations of the trim() function. They remove characters from the left or right ends of a string. For example, rtrim('socks', 's') returns sock by removing only the s on the right end of the string. Extracting and Replacing Characters The left(string, number) and right(string, number) functions, both ANSI SQL standard, extract and return selected characters from a string. For example, to get just the 703 area code from the phone number", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 171 + }, + { + "text": "returns sock by removing only the s on the right end of the string. Extracting and Replacing Characters The left(string, number) and right(string, number) functions, both ANSI SQL standard, extract and return selected characters from a string. For example, to get just the 703 area code from the phone number 703-555-1212, use left('703-555-1212', 3) to specify that you want the first three characters of the string starting from the left. Likewise, right('703-555-1212', 8) returns eight characters from the right: 555-1212. To substitute characters in a string, use the replace(string, from, to) function. To change bat to cat, for example, you would use replace('bat', 'b', 'c') to specify that you want to replace the b in bat with a c. Now that you know basic functions for manipulating strings, let’s look at how to match more complex patterns in text and turn those patterns into data we can analyze. Matching Text Patterns with Regular Expressions Estadísticos e-Books & Papers Regular expressions (or regex) are a type of notational language that describes text patterns. If you have a string with a noticeable pattern (say, four digits followed by a hyphen and then two more digits), you can write a regular expression that describes the pattern. You can then use the notation in a WHERE clause to filter rows by the pattern or use regular expression functions to extract and wrangle text that contains the same pattern. Regular expressions can seem inscrutable to beginning programmers; they take practice to comprehend because they use single-character symbols that aren’t intuitive. Getting an expression to match a pattern can involve trial and error, and each programming language has subtle differences in the way it handles regular expressions. Still, learning regular expressions is a good investment because you gain superpower- like abilities to search text using many programming languages, text editors, and other applications. In this section, I’ll provide enough regular expression basics to work through the exercises. To learn more, I recommend interactive online code testers, such as https://regexr.com/ or http://www.regexpal.com/, which have notation references. Regular Expression Notation Matching letters and numbers using regular expression notation is straightforward because letters and numbers (and certain symbols) are literals that indicate the same characters. For example, Al matches the first two characters in Alicia. For more complex patterns, you’ll use combinations of the regular expression elements in Table 13-1. Table 13-1: Regular Expression Notation Basics ExpressionDescription . A dot is a wildcard that finds any character except a newline. [FGz] Any character in the square brackets. Here, F, G, or z. Estadísticos e-Books & Papers [a-z] A range of characters. Here, lowercase a to z. [^a-z] The caret negates the match. Here, not lowercase a to z. \\w Any word character or underscore. Same as [A-Za-z0-9_]. \\d Any digit. \\s A space. \\t Tab character. \\n Newline character. \\r Carriage return character. ^ Match at the start of a string. $ Match at the end of a string. ? Get the preceding match zero or one time. *", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 172 + }, + { + "text": "Any word character or underscore. Same as [A-Za-z0-9_]. \\d Any digit. \\s A space. \\t Tab character. \\n Newline character. \\r Carriage return character. ^ Match at the start of a string. $ Match at the end of a string. ? Get the preceding match zero or one time. * Get the preceding match zero or more times. + Get the preceding match one or more times. {m} Get the preceding match exactly m times. {m,n} Get the preceding match between m and n times. a|b The pipe denotes alternation. Find either a or b. ( ) Create and report a capture group or set precedence. (?: ) Negate the reporting of a capture group. Using these basic regular expressions, you can match various kinds of characters and also indicate how many times and where to match them. For example, placing characters inside square brackets ([]) lets you match any single character or a range. So, [FGz] matches a single F, G, or z, whereas [A-Za-z] will match any uppercase or lowercase letter. The backslash (\\) precedes a designator for special characters, such as a tab (\\t), digit (\\d), or newline (\\n), which is a line ending character in text files. Estadísticos e-Books & Papers There are several ways to indicate how many times to match a character. Placing a number inside curly brackets indicates you want to match it that many times. For example, \\d{4} matches four digits in a row, and \\d{1,4} matches a digit between one and four times. The ?, *, and + characters provide a useful shorthand notation for the number of matches. For example, the plus sign (+) after a character indicates to match it one or more times. So, the expression a+ would find the aa characters in the string aardvark. Additionally, parentheses indicate a capture group, which you can use to specify just a portion of the matched text to display in the query results. This is useful for reporting back just a part of a matched expression. For example, if you were hunting for an HH:MM:SS time format in text and wanted to report only the hour, you could use an expression such as (\\d{2}):\\d{2}:\\d{2}. This looks for two digits (\\d{2}) of the hour followed by a colon, another two digits for the minutes and a colon, and then the two- digit seconds. By placing the first \\d{2} inside parentheses, you can extract only those two digits, even though the entire expression matches the full time. Table 13-2 shows examples of combining regular expressions to capture different portions of the sentence “The game starts at 7 p.m. on May 2, 2019.” Table 13-2: Regular Expression Matching Examples ExpressionWhat it matches Result .+ Any character one or more times The game starts at 7 p.m. on May 2, 2019. \\d{1,2} (?:a.m.|p.m.) One or two digits followed by a space and a.m. or p.m. in a noncapture group 7 p.m. ^\\w+ One or more word characters at the start The \\w+.$ One or", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 173 + }, + { + "text": ".+ Any character one or more times The game starts at 7 p.m. on May 2, 2019. \\d{1,2} (?:a.m.|p.m.) One or two digits followed by a space and a.m. or p.m. in a noncapture group 7 p.m. ^\\w+ One or more word characters at the start The \\w+.$ One or more word characters followed by any character at the end 2019. May|June Either of the words May or June May Estadísticos e-Books & Papers \\d{4} Four digits 2019 May \\d, \\d{4} May followed by a space, digit, comma, space, and four digits May 2, 2019 These results show the usefulness of regular expressions for selecting only the parts of the string that interest us. For example, to find the time, we use the expression \\d{1,2} (?:a.m.|p.m.) to look for either one or two digits because the time could be a single or double digit followed by a space. Then we look for either a.m. or p.m.; the pipe symbol separating the terms indicates the either-or condition, and placing them in parentheses separates the logic from the rest of the expression. We need the ?: symbol to indicate that we don’t want to treat the terms inside the parentheses as a capture group, which would report a.m. or p.m. only. The ?: ensures that the full match will be returned. You can use any of these regular expressions in pgAdmin by placing the text and regular expression inside the substring(string from pattern) function to return the matched text. For example, to find the four-digit year, use the following query: SELECT substring('The game starts at 7 p.m. on May 2, 2019.' from '\\d{4}'); This query should return 2019, because we specified that the pattern should look for any digit that is four characters long, and 2019 is the only digit in this string that matches these criteria. You can check out sample substring() queries for all the examples in Table 13-2 in the book’s code resources at https://www.nostarch.com/practicalSQL/. The lesson here is that if you can identify a pattern in the text, you can use a combination of regular expression symbols to locate it. This technique is particularly useful when you have repeating patterns in text that you want to turn into a set of data to analyze. Let’s practice how to use regular expression functions using a real-world example. Turning Text to Data with Regular Expression Estadísticos e-Books & Papers Functions A sheriff’s department in one of the Washington, D.C., suburbs publishes daily reports that detail the date, time, location, and description of incidents the department investigates. These reports would be great to analyze, except they post the information in Microsoft Word documents saved as PDF files, which is not the friendliest format for importing into a database. If I copy and paste incidents from the PDF into a text editor, the result is blocks of text that look something like Listing 13-1: ➊ 4/16/17-4/17/17 ➋ 2100-0900 hrs. ➌ 46000 Block Ashmere Sq. ➍ Sterling ➎ Larceny: ➏The victim reported that", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 174 + }, + { + "text": "friendliest format for importing into a database. If I copy and paste incidents from the PDF into a text editor, the result is blocks of text that look something like Listing 13-1: ➊ 4/16/17-4/17/17 ➋ 2100-0900 hrs. ➌ 46000 Block Ashmere Sq. ➍ Sterling ➎ Larceny: ➏The victim reported that a bicycle was stolen from their opened garage door during the overnight hours. ➐ C0170006614 04/10/17 1605 hrs. 21800 block Newlin Mill Rd. Middleburg Larceny: A license plate was reported stolen from a vehicle. SO170006250 Listing 13-1: Crime reports text Each block of text includes dates ➊, times ➋, a street address ➌, city or town ➍, the type of crime ➎, and a description of the incident ➏. The last piece of information is a code ➐ that might be a unique ID for the incident, although we’d have to check with the sheriff’s department to be sure. There are slight inconsistencies. For example, the first block of text has two dates (4/16/17-4/17/17) and two times (2100-0900 hrs.), meaning the exact time of the incident is unknown and likely occurred within that time span. The second block has one date and time. If you compile these reports regularly, you can expect to find some good insights that could answer important questions: Where do crimes Estadísticos e-Books & Papers tend to occur? Which crime types occur most frequently? Do they happen more often on weekends or weekdays? Before you can start answering these questions, you’ll need to extract the text into table columns using regular expressions. Creating a Table for Crime Reports I’ve collected five of the crime incidents into a file named crime_reports.csv that you can download at https://www.nostarch.com/practicalSQL/. Download the file and save it on your computer. Then use the code in Listing 13-2 to build a table that has a column for each data element you can parse from the text using a regular expression. CREATE TABLE crime_reports ( crime_id bigserial PRIMARY KEY, date_1 timestamp with time zone, date_2 timestamp with time zone, street varchar(250), city varchar(100), crime_type varchar(100), description text, case_number varchar(50), original_text text NOT NULL ); COPY crime_reports (original_text) FROM 'C:\\YourDirectory\\crime_reports.csv' WITH (FORMAT CSV, HEADER OFF, QUOTE '\"'); Listing 13-2: Creating and loading the crime_reports table Run the CREATE TABLE statement in Listing 13-2, and then use COPY to load the text into the column original_text. The rest of the columns will be NULL until we fill them. When you run SELECT original_text FROM crime_reports; in pgAdmin, the results grid should display five rows and the first several words of each report. When you hover your cursor over any cell, pgAdmin shows all the text in that row, as shown in Figure 13-1. Estadísticos e-Books & Papers Figure 13-1: Displaying additional text in the pgAdmin results grid Now that you’ve loaded the text you’ll be parsing, let’s explore this data using PostgreSQL regular expression functions. Matching Crime Report Date Patterns The first piece of data we want to extract from the report original_text is the", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 175 + }, + { + "text": "& Papers Figure 13-1: Displaying additional text in the pgAdmin results grid Now that you’ve loaded the text you’ll be parsing, let’s explore this data using PostgreSQL regular expression functions. Matching Crime Report Date Patterns The first piece of data we want to extract from the report original_text is the date or dates of the crime. Most of the reports have one date, although one has two. The reports also have associated times, and we’ll combine the extracted date and time into a timestamp. We’ll fill date_1 with the first (or only) date and time in each report. In cases where a second date or second time exists, we’ll create a timestamp and add it to date_2. For extracting data, we’ll use the regexp_match(string, pattern) function, which is similar to substring() with a few exceptions. One is that it returns each match as text in an array. Also, if there are no matches, it returns NULL. As you might recall from Chapter 5, arrays are a list of elements; in one exercise, you used an array to pass a list of values into the percentile_cont() function to calculate quartiles. I’ll show you how to work with results that come back as an array when we parse the crime reports. NOTE The regexp_match() function was introduced in PostgreSQL 10 and is not Estadísticos e-Books & Papers available in earlier versions. To start, let’s use regexp_match() to find dates in each of the five incidents in crime_reports. The general pattern to match is MM/DD/YY, although there may be one or two digits for both the month and date. Here’s a regular expression that matches the pattern: \\d{1,2}\\/\\d{1,2}\\/\\d{2} In this expression, \\d{1,2} indicates the month. The numbers inside the curly brackets specify that you want at least one digit and at most two digits. Next, you want to look for a forward slash (/), but because a forward slash can have special meaning in regular expressions, you must escape that character by placing a backslash (\\) in front of it, like this \\/. Escaping a character in this context simply means we want to treat it as a literal rather than letting it take on special meaning. So, the combination of the backslash and forward slash (\\/) indicates you want a forward slash. Another \\d{1,2} follows for a single- or double-digit day of the month. The expression ends with a second escaped forward slash and \\d{2} to indicate the two-digit year. Let���s pass the expression \\d{1,2}\\/\\d{1,2}\\/\\d{2} to regexp_match(), as shown in Listing 13-3: SELECT crime_id, regexp_match(original_text, '\\d{1,2}\\/\\d{1,2}\\/\\d{2}') FROM crime_reports; Listing 13-3: Using regexp_match() to find the first date Run that code in pgAdmin, and the results should look like this: crime_id regexp_match -------- ------------ 1 {4/16/17} 2 {4/8/17} 3 {4/4/17} 4 {04/10/17} 5 {04/09/17} Note that each row shows the first date listed for the incident, because regexp_match() returns the first match it finds by default. Also note that each Estadísticos e-Books & Papers date is enclosed in curly brackets. That’s PostgreSQL indicating", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 176 + }, + { + "text": "------------ 1 {4/16/17} 2 {4/8/17} 3 {4/4/17} 4 {04/10/17} 5 {04/09/17} Note that each row shows the first date listed for the incident, because regexp_match() returns the first match it finds by default. Also note that each Estadísticos e-Books & Papers date is enclosed in curly brackets. That’s PostgreSQL indicating that regexp_match() returns each result in an array, or list of elements. In “Extracting Text from the regexp_match() Result” on page 224, I’ll show you how to access those elements from the array. You can also read more about using arrays in PostgreSQL at https://www.postgresql.org/docs/current/static/arrays.html. Matching the Second Date When Present We’ve successfully extracted the first date from each report. But recall that one of the five incidents has a second date. To find and display all the dates in the text, you must use the related regexp_matches() function and pass in an option in the form of the flag g, as shown in Listing 13-4. SELECT crime_id, regexp_matches(original_text, '\\d{1,2}\\/\\d{1,2}\\/\\d{2}', 'g'➊) FROM crime_reports; Listing 13-4: Using the regexp_matches() function with the 'g' flag The regexp_matches() function, when supplied the g flag ➊, differs from regexp_match() by returning each match the expression finds as a row in the results rather than returning just the first match. Run the code again with this revision; you should now see two dates for the incident that has a crime_id of 1, like this: crime_id regexp_matches -------- -------------- 1 {4/16/17} 1 {4/17/17} 2 {4/8/17} 3 {4/4/17} 4 {04/10/17} 5 {04/09/17} Any time a crime report has a second date, we want to load it and the associated time into the date_2 column. Although adding the g flag shows us all the dates, to extract just the second date in a report, we can use the pattern we always see when two dates exist. In Listing 13-1, the first block of text showed the two dates separated by a hyphen, like this: Estadísticos e-Books & Papers 4/16/17-4/17/17 This means you can switch back to regexp_match() and write a regular expression to look for a hyphen followed by a date, as shown in Listing 13-5. SELECT crime_id, regexp_match(original_text, '-\\d{1,2}\\/\\d{1,2}\\/\\d{2}') FROM crime_reports; Listing 13-5: Using regexp_match() to find the second date Although this query finds the second date in the first item (and returns a NULL for the rest), there’s an unintended consequence: it displays the hyphen along with it. crime_id regexp_match -------- ------------ 1 {-4/17/17} 2 3 4 5 You don’t want to include the hyphen, because it’s an invalid format for the timestamp data type. Fortunately, you can specify the exact part of the regular expression you want to return by placing parentheses around it to create a capture group, like this: -(\\d{1,2}/\\d{1,2}/\\d{1,2}) This notation returns only the part of the regular expression you want. Run the modified query in Listing 13-6 to report only the data in parentheses. SELECT crime_id, regexp_match(original_text, '-(\\d{1,2}\\/\\d{1,2}\\/\\d{1,2})') FROM crime_reports; Listing 13-6: Using a capture group to return only the date The query in Listing 13-6 should return just the second", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 177 + }, + { + "text": "only the part of the regular expression you want. Run the modified query in Listing 13-6 to report only the data in parentheses. SELECT crime_id, regexp_match(original_text, '-(\\d{1,2}\\/\\d{1,2}\\/\\d{1,2})') FROM crime_reports; Listing 13-6: Using a capture group to return only the date The query in Listing 13-6 should return just the second date without the leading hyphen, as shown here: Estadísticos e-Books & Papers crime_id regexp_match -------- ------------ 1 {4/17/17} 2 3 4 5 The process you’ve just completed is typical. You start with text to analyze, and then write and refine the regular expression until it finds the data you want. So far, we’ve created regular expressions to match the first date and a second date, if it exists. Now, let’s use regular expressions to extract additional data elements. Matching Additional Crime Report Elements In this section, we’ll capture times, addresses, crime type, description, and case number from the crime reports. Here are the expressions for capturing this information: First hour \\/\\d{2}\\n(\\d{4}) The first hour, which is the hour the crime was committed or the start of the time range, always follows the date in each crime report, like this: 4/16/17-4/17/17 2100-0900 hrs. To find the first hour, we start with an escaped forward slash and \\d{2}, which represents the two-digit year preceding the first date (17). The \\n character indicates the newline because the hour always starts on a new line, and \\d{4} represents the four-digit hour (2100). Because we just want to return the four digits, we put \\d{4} inside parentheses as a capture group. Second hour \\/\\d{2}\\n\\d{4}-(\\d{4}) If the second hour exists, it will follow a hyphen, so we add a hyphen and another \\d{4} to the expression we just created for the first hour. Again, the second \\d{4} goes inside a capture group, because 0900 is Estadísticos e-Books & Papers the only hour we want to return. Street hrs.\\n(\\d+ .+(?:Sq.|Plz.|Dr.|Ter.|Rd.)) In this data, the street always follows the time’s hrs. designation and a newline (\\n), like this: 04/10/17 1605 hrs. 21800 block Newlin Mill Rd. The street address always starts with some number that varies in length and ends with an abbreviated suffix of some kind. To describe this pattern, we use \\d+ to match any digit that appears one or more times. Then we specify a space and look for any character one or more times using the dot wildcard and plus sign (.+) notation. The expression ends with a series of terms separated by the alternation pipe symbol that looks like this: (?:Sq.|Plz.|Dr.|Ter.|Rd.). The terms are inside parentheses, so the expression will match one or another of those terms. When we group terms like this, if we don’t want the parentheses to act as a capture group, we need to add ?: to negate that effect. NOTE In a large data set, it’s likely roadway names would end with suffixes beyond the five in our regular expression. After making an initial pass at extracting the street, you can run a query to check for unmatched", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 178 + }, + { + "text": "capture group, we need to add ?: to negate that effect. NOTE In a large data set, it’s likely roadway names would end with suffixes beyond the five in our regular expression. After making an initial pass at extracting the street, you can run a query to check for unmatched rows to find additional suffixes to match. City (?:Sq.|Plz.|Dr.|Ter.|Rd.)\\n(\\w+ \\w+|\\w+)\\n Because the city always follows the street suffix, we reuse the terms separated by the alternation symbol we just created for the street. We follow that with a newline (\\n) and then use a capture group to look for two words or one word (\\w+ \\w+|\\w+) before a final newline, because a town or city name can be more than a single word. Estadísticos e-Books & Papers Crime type \\n(?:\\w+ \\w+|\\w+)\\n(.*): The type of crime always precedes a colon (the only time a colon is used in each report) and might consist of one or more words, like this: --snip-- Middleburg Larceny: A license plate was reported stolen from a vehicle. SO170006250 --snip-- To create an expression that matches this pattern, we follow a newline with a nonreporting capture group that looks for the one- or two-word city. Then we add another newline and match any character that occurs zero or more times before a colon using (.*):. Description :\\s(.+)(?:C0|SO) The crime description always comes between the colon after the crime type and the case number. The expression starts with the colon, a space character (\\s), and then a capture group to find any character that appears one or more times using the .+ notation. The nonreporting capture group (?:C0|SO) tells the program to stop looking when it encounters either C0 or SO, the two character pairs that start each case number (a C followed by a zero, and an S followed by a capital O). We have to do this because the description might have one or more line breaks. Case number (?:C0|SO)[0-9]+ The case number starts with either C0 or SO, followed by a set of digits. To match this pattern, the expression looks for either C0 or SO in a nonreporting capture group followed by any digit from 0 to 9 that occurs one or more times using the [0-9] range notation. Now let’s pass these regular expressions to regexp_match() to see them in action. Listing 13-7 shows a sample regexp_match() query that retrieves the case number, first date, crime type, and city: Estadísticos e-Books & Papers SELECT regexp_match(original_text, '(?:C0|SO)[0-9]+') AS case_number, regexp_match(original_text, '\\d{1,2}\\/\\d{1,2}\\/\\d{2}') AS date_1, regexp_match(original_text, '\\n(?:\\w+ \\w+|\\w+)\\n(.*):') AS crime_type, regexp_match(original_text, '(?:Sq.|Plz.|Dr.|Ter.|Rd.)\\n(\\w+ \\w+|\\w+)\\n') AS city FROM crime_reports; Listing 13-7: Matching case number, date, crime type, and city Run the code, and the results should look like this: After all that wrangling, we’ve transformed the text into a structure that is more suitable for analysis. Of course, you would have to include many more incidents to count the frequency of crime type by city or the number of crimes per month to identify any trends. To load", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 179 + }, + { + "text": "this: After all that wrangling, we’ve transformed the text into a structure that is more suitable for analysis. Of course, you would have to include many more incidents to count the frequency of crime type by city or the number of crimes per month to identify any trends. To load each parsed element into the table’s columns, we’ll create an UPDATE query. But before you can insert the text into a column, you’ll need to learn how to extract the text from the array that regexp_match() returns. Extracting Text from the regexp_match() Result In “Matching Crime Report Date Patterns” on page 218, I mentioned that regexp_match() returns an array containing text values. Two clues reveal that these are text values. The first is that the data type designation in the column header shows text[] instead of text. The second is that each result is surrounded by curly brackets. Figure 13-2 shows how pgAdmin displays the results of the query in Listing 13-7. Estadísticos e-Books & Papers Figure 13-2: Array values in the pgAdmin results grid The crime_reports columns we want to update are not array types, so rather than passing in the array values returned by regexp_match(), we need to extract the values from the array first. We do this by using array notation, as shown in Listing 13-8. SELECT crime_id, ➊ (regexp_match(original_text, '(?:C0|SO)[0-9]+'))[1]➋ AS case_number FROM crime_reports; Listing 13-8: Retrieving a value from within an array First, we wrap the regexp_match() function ➊ in parentheses. Then, at the end, we provide a value of 1, which represents the first element in the array, enclosed in square brackets ➋. The query should produce the following results: crime_id case_number -------- ----------- 1 C0170006614 2 C0170006162 3 C0170006079 4 SO170006250 5 SO170006211 Now the data type designation in the pgAdmin column header should show text instead of text[], and the values are no longer enclosed in curly brackets. We can now insert these values into crime_reports using an UPDATE query. Estadísticos e-Books & Papers Updating the crime_reports Table with Extracted Data With each element currently available as text, we can update columns in the crime_reports table with the appropriate data from the original crime report. To start, Listing 13-9 combines the extracted first date and time into a single timestamp value for the column date_1. UPDATE crime_reports ➊ SET date_1 = ( ➋ (regexp_match(original_text, '\\d{1,2}\\/\\d{1,2}\\/\\d{2}'))[1] ➌ || ' ' || ➍ (regexp_match(original_text, '\\/\\d{2}\\n(\\d{4})'))[1] ➎ ||' US/Eastern' ➏ )::timestamptz; SELECT crime_id, date_1, original_text FROM crime_reports; Listing 13-9: Updating the crime_reports date_1 column Because the date_1 column is of type timestamp, we must provide an input in that data type. To do that, we’ll use the PostgreSQL double-pipe (||) concatenation operator to combine the extracted date and time in a format that’s acceptable for timestamp with time zone input. In the SET clause ➊, we start with the regex pattern that matches the first date ➋. Next, we concatenate the date with a space using two single quotes ➌ and repeat the concatenation operator.", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 180 + }, + { + "text": "extracted date and time in a format that’s acceptable for timestamp with time zone input. In the SET clause ➊, we start with the regex pattern that matches the first date ➋. Next, we concatenate the date with a space using two single quotes ➌ and repeat the concatenation operator. This step combines the date with a space before connecting it to the regex pattern that matches the time ➍. Then we include the time zone for the Washington, D.C., area by concatenating that at the end of the string ➎ using the US/Eastern designation. Concatenating these elements creates a string in the pattern of MM/DD/YY HHMM TIMEZONE, which is acceptable as a timestamp input. We cast the string to a timestamp with time zone data type ➏ using the PostgreSQL double-colon shorthand and the timestamptz abbreviation. When you run the UPDATE portion of the code, PostgreSQL should return the message UPDATE 5. Running the SELECT statement in pgAdmin Estadísticos e-Books & Papers should show the now-filled date_1 column alongside a portion of the original_text column, like this: At a glance, you can see that date_1 accurately captures the first date and time that appears in the original text and puts it into a useable format that we can analyze. Note that if you’re not in the Eastern time zone, the timestamps will instead reflect your pgAdmin client’s time zone. As you learned in “Setting the Time Zone” on page 178, you can use the command SET timezone TO 'US/Eastern'; to change the client to reflect Eastern time. Using CASE to Handle Special Instances You could write an UPDATE statement for each remaining data element, but combining those statements into one would be more efficient. Listing 13- 10 updates all the crime_reports columns using a single statement while handling inconsistent values in the data. UPDATE crime_reports SET date_1➊ = ( (regexp_match(original_text, '\\d{1,2}\\/\\d{1,2}\\/\\d{2}'))[1] || ' ' || (regexp_match(original_text, '\\/\\d{2}\\n(\\d{4})'))[1] ||' US/Eastern' )::timestamptz, date_2➋ = CASE➌ WHEN➍ (SELECT regexp_match(original_text, '-(\\d{1,2}\\/\\d{1,2}\\/\\d{1,2})') IS NULL➎) AND (SELECT regexp_match(original_text, '\\/\\d{2}\\n\\d{4}-(\\d{4})') IS NOT NULL➏) Estadísticos e-Books & Papers THEN➐ ((regexp_match(original_text, '\\d{1,2}\\/\\d{1,2}\\/\\d{2}'))[1] || ' ' || (regexp_match(original_text, '\\/\\d{2}\\n\\d{4}-(\\d{4})'))[1] ||' US/Eastern' )::timestamptz WHEN➑ (SELECT regexp_match(original_text, '-(\\d{1,2}\\/\\d{1,2}\\/\\d{1,2})') IS NOT NULL) AND (SELECT regexp_match(original_text, '\\/\\d{2}\\n\\d{4}-(\\d{4})') IS NOT NULL) THEN ((regexp_match(original_text, '-(\\d{1,2}\\/\\d{1,2}\\/\\d{1,2})'))[1] || ' ' || (regexp_match(original_text, '\\/\\d{2}\\n\\d{4}-(\\d{4})'))[1] ||' US/Eastern' )::timestamptz ELSE NULL➒ END, street = (regexp_match(original_text, 'hrs.\\n(\\d+ .+ (?:Sq.|Plz.|Dr.|Ter.|Rd.))'))[1], city = (regexp_match(original_text, '(?:Sq.|Plz.|Dr.|Ter.|Rd.)\\n(\\w+ \\w+|\\w+)\\n'))[1], crime_type = (regexp_match(original_text, '\\n(?:\\w+ \\w+|\\w+)\\n(.*):'))[1], description = (regexp_match(original_text, ':\\s(.+)(?:C0|SO)'))[1], case_number = (regexp_match(original_text, '(?:C0|SO)[0-9]+'))[1]; Listing 13-10: Updating all crime_reports columns This UPDATE statement might look intimidating, but it’s not if we break it down by column. First, we use the same code from Listing 13-9 to update the date_1 column ➊. But to update date_2 ➋, we need to account for the inconsistent presence of a second date and time. In our limited data set, there are three possibilities: 1. A second hour exists but not a second date. This occurs when a report covers a range of hours on one date. 2. A second date and second", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 181 + }, + { + "text": "to account for the inconsistent presence of a second date and time. In our limited data set, there are three possibilities: 1. A second hour exists but not a second date. This occurs when a report covers a range of hours on one date. 2. A second date and second hour exist. This occurs when a report covers more than one date. 3. Neither a second date nor a second hour exists. To insert the correct value in date_2 for each scenario, we use the CASE statement syntax you learned in “Reclassifying Values with CASE” on page 207 to test for each possibility. After the CASE keyword ➌, we use a series of WHEN ... THEN statements to check for the first two conditions and Estadísticos e-Books & Papers provide the value to insert; if neither condition exists, we use an ELSE keyword to provide a NULL. The first WHEN statement ➍ checks whether regexp_match() returns a NULL ➎ for the second date and a value for the second hour (using IS NOT NULL ➏). If that condition evaluates as true, the THEN statement ➐ concatenates the first date with the second hour to create a timestamp for the update. The second WHEN statement ➑ checks that regexp_match() returns a value for the second hour and second date. If true, the THEN statement concatenates the second date with the second hour to create a timestamp. If neither of the two WHEN statements returns true, the ELSE statement ➒ provides a NULL for the update because there is only a first date and first time. NOTE The WHEN statements handle the possibilities that exist in our small sample data set. If you are working with more data, you might need to handle additional variations, such as a second date but not a second time. When we run the full query in Listing 13-10, PostgreSQL should report UPDATE 5. Success! Now that we’ve updated all the columns with the appropriate data while accounting for elements that have additional data, we can examine all the columns of the table and find the parsed elements from original_text. Listing 13-11 queries four of the columns: SELECT date_1, street, city, crime_type FROM crime_reports; Listing 13-11: Viewing selected crime data The results of the query should show a nicely organized set of data that looks something like this: Estadísticos e-Books & Papers You’ve successfully transformed raw text into a table that can answer questions and reveal storylines about crime in this area. The Value of the Process Writing regular expressions and coding a query to update a table can take time, but there is value to identifying and collecting data this way. In fact, some of the best data sets you’ll encounter are those you build yourself. Everyone can download the same data sets, but the ones you build are yours alone. You get to be first person to find and tell the story behind the data. Also, after you set up your database and queries, you", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 182 + }, + { + "text": "best data sets you’ll encounter are those you build yourself. Everyone can download the same data sets, but the ones you build are yours alone. You get to be first person to find and tell the story behind the data. Also, after you set up your database and queries, you can use them again and again. In this example, you could collect crime reports every day (either by hand or by automating downloads using a programming language such as Python) for an ongoing data set that you can mine continually for trends. In the next section, we’ll finish our exploration of regular expressions using additional PostgreSQL functions. Using Regular Expressions with WHERE You’ve filtered queries using LIKE and ILIKE in WHERE clauses. In this section, you’ll learn to use regular expressions in WHERE clauses so you can perform more complex matches. We use a tilde (~) to make a case-sensitive match on a regular expression and a tilde-asterisk (~*) to perform a case-insensitive match. You can negate either expression by adding an exclamation point in front. For example, !~* indicates to not match a regular expression that is case- insensitive. Listing 13-12 shows how this works using the 2010 Census Estadísticos e-Books & Papers table us_counties_2010 from previous exercises: SELECT geo_name FROM us_counties_2010 ➊ WHERE geo_name ~* '(.+lade.+|.+lare.+)' ORDER BY geo_name; SELECT geo_name FROM us_counties_2010 ➋ WHERE geo_name ~* '.+ash.+' AND geo_name !~ 'Wash.+' ORDER BY geo_name; Listing 13-12: Using regular expressions in a WHERE clause The first WHERE clause ➊ uses the tilde-asterisk (~*) to perform a case- insensitive match on the regular expression (.+lade.+|.+lare.+) to find any county names that contain either the letters lade or lare between other characters. The results should show eight rows: geo_name ------------------- Bladen County Clare County Clarendon County Glades County Langlade County Philadelphia County Talladega County Tulare County As you can see, the county names include the letters lade or lare between other characters. The second WHERE clause ➋ uses the tilde-asterisk (~*) as well as a negated tilde (!~) to find county names containing the letters ash but excluding those starting with Wash. This query should return the following: geo_name -------------- Nash County Wabash County Wabash County Wabasha County All four counties in this output have names that contain the letters ash Estadísticos e-Books & Papers but don’t start with Wash. These are fairly simple examples, but you can do more complex matches using regular expressions that you wouldn’t be able to perform with the wildcards available with just LIKE and ILIKE. Additional Regular Expression Functions Let’s look at three more regular expression functions you might find useful when working with text. Listing 13-13 shows several regular expression functions that replace and split text: ➊ SELECT regexp_replace('05/12/2018', '\\d{4}', '2017'); ➋ SELECT regexp_split_to_table('Four,score,and,seven,years,ago', ','); ➌ SELECT regexp_split_to_array('Phil Mike Tony Steve', ','); Listing 13-13: Regular expression functions to replace and split text The regexp_replace(string, pattern, replacement text) function lets you substitute a matched pattern with replacement text. In the example at ➊, we’re", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 183 + }, + { + "text": "and split text: ➊ SELECT regexp_replace('05/12/2018', '\\d{4}', '2017'); ➋ SELECT regexp_split_to_table('Four,score,and,seven,years,ago', ','); ➌ SELECT regexp_split_to_array('Phil Mike Tony Steve', ','); Listing 13-13: Regular expression functions to replace and split text The regexp_replace(string, pattern, replacement text) function lets you substitute a matched pattern with replacement text. In the example at ➊, we’re searching the date string 05/12/2018 for any set of four digits in a row using \\d{4}. When found, we replace them with the replacement text 2017. The result of that query is 05/12/2017 returned as text. The regexp_split_to_table(string, pattern) function splits delimited text into rows. Listing 13-13 uses this function to split the string 'Four,score,and,seven,years,ago' on commas ➋, resulting in a set of rows that has one word in each row: regexp_split_to_table --------------------- Four score and seven years ago Keep this function in mind as you tackle the “Try It Yourself” exercises at the end of the chapter. Estadísticos e-Books & Papers The regexp_split_to_array(string, pattern) function splits delimited text into an array. The example splits the string Phil Mike Tony Steve on spaces ➌, returning a text array that should look like this in pgAdmin: regexp_split_to_array ---------------------- {Phil,Mike,Tony,Steve} The text[] notation in pgAdmin’s column header along with curly brackets around the results confirms that this is indeed an array type, which provides another means of analysis. For example, you could then use a function such as array_length() to count the number of words, as shown in Listing 13-14. SELECT array_length(regexp_split_to_array('Phil Mike Tony Steve', ' '), 1); Listing 13-14: Finding an array length The query should return 4 because four elements are in this array. You can read more about array_length() and other array functions at https://www.postgresql.org/docs/current/static/functions-array.html. Full Text Search in PostgreSQL PostgreSQL comes with a powerful full text search engine that gives you more options when searching for information in large amounts of text. You’re familiar with Google or other web search engines and similar technology that powers search on news websites or research databases, such as LexisNexis. Although the implementation and capability of full text search demands several chapters, here I’ll walk you through a simple example of setting up a table for text search and functions for searching using PostgreSQL. For this example, I assembled 35 speeches by former U.S. presidents who served after World War II through the Gerald R. Ford administration. Consisting mostly of State of the Union addresses, these public texts are available through the Internet Archive at Estadísticos e-Books & Papers https://archive.org/ and the University of California’s American Presidency Project at http://www.presidency.ucsb.edu/ws/index.php/. You can find the data in the sotu-1946-1977.csv file along with the book’s resources at https://www.nostarch.com/practicalSQL/. Let’s start with the data types unique to full text search. Text Search Data Types PostgreSQL’s implementation of text search includes two data types. The tsvector data type represents the text to be searched and to be stored in an optimized form. The tsquery data type represents the search query terms and operators. Let’s look at the details of both. Storing Text as Lexemes with tsvector", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 184 + }, + { + "text": "implementation of text search includes two data types. The tsvector data type represents the text to be searched and to be stored in an optimized form. The tsquery data type represents the search query terms and operators. Let’s look at the details of both. Storing Text as Lexemes with tsvector The tsvector data type reduces text to a sorted list of lexemes, which are units of meaning in language. Think of lexemes as words without the variations created by suffixes. For example, the tsvector format would store the words washes, washed, and washing as the lexeme wash while noting each word’s position in the original text. Converting text to tsvector also removes small stop words that usually don’t play a role in search, such as the or it. To see how this data type works, let’s convert a string to tsvector format. Listing 13-15 uses the PostgreSQL search function to_tsvector(), which normalizes the text “I am walking across the sitting room to sit with you” to lexemes: SELECT to_tsvector('I am walking across the sitting room to sit with you.'); Listing 13-15: Converting text to tsvector data Execute the code, and it should return the following output in tsvector format: 'across':4 'room':7 'sit':6,9 'walk':3 Estadísticos e-Books & Papers The to_tsvector() function reduces the number of words from eleven to four, eliminating words such as I, am, and the, which are not helpful search terms. The function removes suffixes, changing walking to walk and sitting to sit. It also orders the words alphabetically, and the number following each colon indicates its position in the original string, taking stop words into account. Note that sit is recognized as being in two positions, one for sitting and one for sit. Creating the Search Terms with tsquery The tsquery data type represents the full text search query, again optimized as lexemes. It also provides operators for controlling the search. Examples of operators include the ampersand (&) for AND, the pipe symbol (|) for OR, and the exclamation point (!) for NOT. A special <-> operator lets you search for adjacent words or words a certain distance apart. Listing 13-16 shows how the to_tsquery() function converts search terms to the tsquery data type. SELECT to_tsquery('walking & sitting'); Listing 13-16: Converting search terms to tsquery data After running the code, you should see that the resulting tsquery data type has normalized the terms into lexemes, which match the format of the data to search: 'walk' & 'sit' Now you can use terms stored as tsquery to search text optimized as tsvector. Using the @@ Match Operator for Searching With the text and search terms converted to the full text search data types, you can use the double at sign (@@) match operator to check whether a query matches text. The first query in Listing 13-17 uses Estadísticos e-Books & Papers to_tsquery() to search for the words walking and sitting, which we combine with the & operator. It returns a Boolean value of true because both walking and", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 185 + }, + { + "text": "at sign (@@) match operator to check whether a query matches text. The first query in Listing 13-17 uses Estadísticos e-Books & Papers to_tsquery() to search for the words walking and sitting, which we combine with the & operator. It returns a Boolean value of true because both walking and sitting are present in the text converted by to_tsvector(). SELECT to_tsvector('I am walking across the sitting room') @@ to_tsquery('walking & sitting'); SELECT to_tsvector('I am walking across the sitting room') @@ to_tsquery('walking & running'); Listing 13-17: Querying a tsvector type with a tsquery However, the second query returns false because both walking and running are not present in the text. Now let’s build a table for searching the speeches. Creating a Table for Full Text Search Let’s start by creating a table to hold the speech text. The code in Listing 13-18 creates and fills president_speeches so it contains a column for the original speech text as well as a column of type tsvector. The reason is that we need to convert the original speech text into that tsvector column to optimize it for searching. We can’t easily do that conversion during import, so let’s handle that as a separate step. Be sure to change the file path to match the location of your saved CSV file: CREATE TABLE president_speeches ( sotu_id serial PRIMARY KEY, president varchar(100) NOT NULL, title varchar(250) NOT NULL, speech_date date NOT NULL, speech_text text NOT NULL, search_speech_text tsvector ); COPY president_speeches (president, title, speech_date, speech_text) FROM 'C:\\YourDirectory\\sotu-1946-1977.csv' WITH (FORMAT CSV, DELIMITER '|', HEADER OFF, QUOTE '@'); Listing 13-18: Creating and filling the president_speeches table After executing the query, run SELECT * FROM president_speeches; to see the data. In pgAdmin, hover your mouse over any cell to see extra words not visible in the results grid. You should see a sizeable amount of text in each Estadísticos e-Books & Papers row of the speech_text column. Next, we copy the contents of speech_text to the tsvector column search_speech_text and transform it to that data type at the same time. The UPDATE query in Listing 13-19 handles the task: UPDATE president_speeches ➊ SET search_speech_text = to_tsvector('english', speech_text); Listing 13-19: Converting speeches to tsvector in the search_speech_text column The SET clause ➊ fills search_speech_text with the output of to_tsvector(). The first argument in the function specifies the language for parsing the lexemes. We’re using the default of english here, but you can substitute spanish, german, french, or whatever language you want to use (some languages may require you to find and install additional dictionaries). The second argument is the name of the input column. Run the code to fill the column. Finally, we want to index the search_speech_text column to speed up searches. You learned about indexing in Chapter 7, which focused on PostgreSQL’s default index type, B-Tree. For full text search, the PostgreSQL documentation recommends using the Generalized Inverted Index (GIN; see https://www.postgresql.org/docs/current/static/textsearch- indexes.html). You can add a GIN index using CREATE INDEX in Listing 13-20: CREATE INDEX search_idx", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 186 + }, + { + "text": "speed up searches. You learned about indexing in Chapter 7, which focused on PostgreSQL’s default index type, B-Tree. For full text search, the PostgreSQL documentation recommends using the Generalized Inverted Index (GIN; see https://www.postgresql.org/docs/current/static/textsearch- indexes.html). You can add a GIN index using CREATE INDEX in Listing 13-20: CREATE INDEX search_idx ON president_speeches USING gin(search_speech_text); Listing 13-20: Creating a GIN index for text search The GIN index contains an entry for each lexeme and its location, allowing the database to find matches more quickly. NOTE Another way to set up a column for search is to create an index on a text column using the to_tsvector() function. See https://www.postgresql.org/docs/current/static/textsearch- tables.html for details. Estadísticos e-Books & Papers Now you’re ready to use search functions. Searching Speech Text Thirty-two years’ worth of presidential speeches is fertile ground for exploring history. For example, the query in Listing 13-21 lists the speeches in which the president mentioned Vietnam: SELECT president, speech_date FROM president_speeches ➊ WHERE search_speech_text @@ to_tsquery('Vietnam') ORDER BY speech_date; Listing 13-21: Finding speeches containing the word Vietnam In the WHERE clause, the query uses the double at sign (@@) match operator ➊ between the search_speech_text column (of data type tsvector) and the query term Vietnam, which to_tsquery() transforms into tsquery data. The results should list 10 speeches, showing that the first mention of Vietnam came up in a 1961 special message to Congress by John F. Kennedy and became a recurring topic starting in 1966 as America’s involvement in the Vietnam War escalated. president speech_date ----------------- ----------- John F. Kennedy 1961-05-25 Lyndon B. Johnson 1966-01-12 Lyndon B. Johnson 1967-01-10 Lyndon B. Johnson 1968-01-17 Lyndon B. Johnson 1969-01-14 Richard M. Nixon 1970-01-22 Richard M. Nixon 1972-01-20 Richard M. Nixon 1973-02-02 Gerald R. Ford 1975-01-15 Gerald R. Ford 1977-01-12 Before we try more searches, let’s add a method for showing the location of our search term in the text. Showing Search Result Locations Estadísticos e-Books & Papers To see where our search terms appear in text, we can use the ts_headline() function. It displays one or more highlighted search terms surrounded by adjacent words. Options for this function give you flexibility in how to format the display. Listing 13-22 highlights how to display a search for a specific instance of Vietnam using ts_headline(): SELECT president, speech_date, ➊ ts_headline(speech_text, to_tsquery('Vietnam'), ➋ 'StartSel = <, StopSel = >, MinWords=5, MaxWords=7, MaxFragments=1') FROM president_speeches WHERE search_speech_text @@ to_tsquery('Vietnam'); Listing 13-22: Displaying search results with ts_headline() To declare ts_headline() ➊, we pass the original speech_text column rather than the tsvector column we used in the search and relevance functions as the first argument. Then, as the second argument, we pass a to_tsquery() function that specifies the word to highlight. We follow this with a third argument that lists optional formatting parameters ➋ separated by commas. Here, we specify the characters to identify the start and end of the highlighted word (StartSel and StopSel). We also set the minimum and maximum number of words to display (MinWords and MaxWords), plus", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 187 + }, + { + "text": "We follow this with a third argument that lists optional formatting parameters ➋ separated by commas. Here, we specify the characters to identify the start and end of the highlighted word (StartSel and StopSel). We also set the minimum and maximum number of words to display (MinWords and MaxWords), plus the maximum number of fragments to show using MaxFragments. These settings are optional, and you can adjust them according to your needs. The results of this query should show at most seven words per speech, highlighting the word Vietnam: Estadísticos e-Books & Papers Using this technique, we can quickly see the context of the term we searched. You might also use this function to provide flexible display options for a search feature on a web application. Let’s continue trying forms of searches. Using Multiple Search Terms As another example, we could look for speeches in which a president mentioned the word transportation but didn’t discuss roads. We might want to do this to find speeches that focused on broader policy rather than a specific roads program. To do this, we use the syntax in Listing 13-23: SELECT president, speech_date ➊ ts_headline(speech_text, to_tsquery('transportation & !roads'), 'StartSel = <, StopSel = >, MinWords=5, MaxWords=7, MaxFragments=1') FROM president_speeches ➋ WHERE search_speech_text @@ to_tsquery('transportation & !roads'); Listing 13-23: Finding speeches with the word transportation but not roads Again, we use ts_headline() ➊ to highlight the terms our search finds. In the to_tsquery() function in the WHERE clause ➋, we pass transportation and roads, combining them with the ampersand (&) operator. We use the exclamation point (!) in front of roads to indicate that we want speeches that do not contain this word. This query should find eight speeches that fit the criteria. Here are the first four rows: Estadísticos e-Books & Papers Notice that the highlighted words in the ts_headline column include transportation and transport. The reason is that the to_tsquery() function converted transportation to the lexeme transport for the search term. This database behavior is extremely useful in helping to find relevant related words. Searching for Adjacent Words Finally, we’ll use the distance (<->) operator, which consists of a hyphen between the less than and greater than signs, to find adjacent words. Alternatively, you can place a number between the signs to find terms that many words apart. For example, Listing 13-24 searches for any speeches that include the word military immediately followed by defense: SELECT president, speech_date, ts_headline(speech_text, to_tsquery('military <-> defense'), 'StartSel = <, StopSel = >, MinWords=5, MaxWords=7, MaxFragments=1') FROM president_speeches WHERE search_speech_text @@ to_tsquery('military <-> defense'); Listing 13-24: Finding speeches where defense follows military This query should find four speeches, and because to_tsquery() converts the search terms to lexemes, the words identified in the speeches should include plurals, such as military defenses. The following shows the four speeches that have the adjacent terms: Estadísticos e-Books & Papers If you changed the query terms to military <2> defense, the database would return matches where the terms are exactly two words apart,", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 188 + }, + { + "text": "words identified in the speeches should include plurals, such as military defenses. The following shows the four speeches that have the adjacent terms: Estadísticos e-Books & Papers If you changed the query terms to military <2> defense, the database would return matches where the terms are exactly two words apart, as in the phrase “our military and defense commitments.” Ranking Query Matches by Relevance You can also rank search results by relevance using two of PostgreSQL’s full text search functions. These functions are helpful when you’re trying to understand which piece of text, or speech in this case, is most relevant to your particular search terms. One function, ts_rank(), generates a rank value (returned as a variable- precision real data type) based on how often the lexemes you’re searching for appear in the text. The other function, ts_rank_cd(), considers how close the lexemes searched are to each other. Both functions can take optional arguments to take into account document length and other factors. The rank value they generate is an arbitrary decimal that’s useful for sorting but doesn’t have any inherent meaning. For example, a value of 0.375 generated during one query isn’t directly comparable to the same value generated during a different query. As an example, Listing 13-25 uses ts_rank() to rank speeches containing all the words war, security, threat, and enemy: SELECT president, speech_date, ➊ ts_rank(search_speech_text, to_tsquery('war & security & threat & enemy')) AS score FROM president_speeches ➋ WHERE search_speech_text @@ to_tsquery('war & security & threat & enemy') ORDER BY score DESC/ LIMIT 5 Listing 13-25: Scoring relevance with ts_rank() Estadísticos e-Books & Papers In this query, the ts_rank() function ➊ takes two arguments: the search_speech_text column and the output of a to_tsquery() function containing the search terms. The output of the function receives the alias score. In the WHERE clause ➋ we filter the results to only those speeches that contain the search terms specified. Then we order the results in score in descending order and return just five of the highest-ranking speeches. The results should be as follows: president speech_date score -------------------- ----------- --------- Harry S. Truman 1946-01-21 0.257522 Lyndon B. Johnson 1968-01-17 0.186296 Dwight D. Eisenhower 1957-01-10 0.140851 Harry S. Truman 1952-01-09 0.0982469 Richard M. Nixon 1972-01-20 0.0973585 Harry S. Truman’s 1946 State of the Union message, just four months after the end of World War II, contains the words war, security, threat, and enemy more often than the other speeches. However, it also happens to be the longest speech in the table (which you can determine by using char_length(), as you learned earlier in the chapter). The length of the speeches influences these rankings because ts_rank() factors in the number of matching terms in a given text. Lyndon B. Johnson’s 1968 State of the Union address, delivered at the height of the Vietnam War, comes in second. It would be ideal to compare frequencies between speeches of identical lengths to get a more accurate ranking, but this isn’t always possible. However, we can factor", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 189 + }, + { + "text": "a given text. Lyndon B. Johnson’s 1968 State of the Union address, delivered at the height of the Vietnam War, comes in second. It would be ideal to compare frequencies between speeches of identical lengths to get a more accurate ranking, but this isn’t always possible. However, we can factor in the length of each speech by adding a normalization code as a third parameter of the ts_rank() function, as shown in Listing 13-26: SELECT president, speech_date, ts_rank(search_speech_text, to_tsquery('war & security & threat & enemy'), 2➊)::numeric AS score FROM president_speeches WHERE search_speech_text @@ to_tsquery('war & security & threat & enemy') ORDER BY score DESC LIMIT 5; Estadísticos e-Books & Papers Listing 13-26: Normalizing ts_rank() by speech length Adding the optional code 2 ➊ instructs the function to divide the score by the length of the data in the search_speech_text column. This quotient then represents a score normalized by the document length, giving an apples-to-apples comparison among the speeches. The PostgreSQL documentation at https://www.postgresql.org/docs/current/static/textsearch- controls.html lists all the options available for text search, including using the document length and dividing by the number of unique words. After running the code in Listing 13-26, the rankings should change: president speech_date score -------------------- ----------- ------------ Lyndon B. Johnson 1968-01-17 0.0000728288 Dwight D. Eisenhower 1957-01-10 0.0000633609 Richard M. Nixon 1972-01-20 0.0000497998 Harry S. Truman 1952-01-09 0.0000365366 Dwight D. Eisenhower 1958-01-09 0.0000355315 In contrast to the ranking results in Listing 13-25, Johnson’s 1968 speech now tops the rankings, and Truman’s 1946 message falls out of the top five. This might be a more meaningful ranking than the first sample output, because we normalized it by length. But four of the five top-ranked speeches are the same between the two sets, and you can be reasonably certain that each of these four is worthy of closer examination to understand more about wartime presidential speeches. Wrapping Up Far from being boring, text offers abundant opportunities for data analysis. In this chapter, you’ve learned valuable techniques for turning ordinary text into data you can extract, quantify, search, and rank. In your work or studies, keep an eye out for routine reports that have facts buried inside chunks of text. You can use regular expressions to dig them out, turn them into structured data, and analyze them to find trends. You can also use search functions to analyze the text. In the next chapter, you’ll learn how PostgreSQL can help you analyze Estadísticos e-Books & Papers geographic information. TRY IT YOURSELF Use your new text-wrangling skills to tackle these tasks: 1. The style guide of a publishing company you’re writing for wants you to avoid commas before suffixes in names. But there are several names like Alvarez, Jr. and Williams, Sr. in your database. Which functions can you use to remove the comma? Would a regular expression function help? How would you capture just the suffixes to place them into a separate column? 2. Using any one of the State of the Union addresses, count the number of unique", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 190 + }, + { + "text": "Williams, Sr. in your database. Which functions can you use to remove the comma? Would a regular expression function help? How would you capture just the suffixes to place them into a separate column? 2. Using any one of the State of the Union addresses, count the number of unique words that are five characters or more. (Hint: You can use regexp_split_to_table() in a subquery to create a table of words to count.) Bonus: Remove commas and periods at the end of each word. 3. Rewrite the query in Listing 13-25 using the ts_rank_cd() function instead of ts_rank(). According to the PostgreSQL documentation, ts_rank_cd() computes cover density, which takes into account how close the lexeme search terms are to each other. Does using the ts_rank_cd() function significantly change the results? Estadísticos e-Books & Papers 14 ANALYZING SPATIAL DATA WITH POSTGIS These days, mobile apps can provide a list of coffee shops near you within seconds. They can do that because they’re powered by a geographic information system (GIS), which is any system that allows for storing, editing, analyzing, and displaying spatial data. As you can imagine, GIS has many practical applications today, from helping city planners decide where to build schools based on population patterns to finding the best detour around a traffic jam. Spatial data refers to information about the location and shape of objects, which can be two and three dimensional. For example, the spatial data we’ll use in this chapter contains coordinates describing geometric shapes, such as points, lines, and polygons. These shapes in turn represent features you would find on a map, such as roads, lakes, or countries. Conveniently, you can use PostgreSQL to store and analyze spatial data, which allows you to calculate the distance between points, compute the size of areas, and identify whether two objects intersect. However, to enable spatial analysis and store spatial data types in PostgreSQL, you need to install an open source extension called PostGIS. The PostGIS extension also provides additional functions and operators that work specifically with spatial data. Estadísticos e-Books & Papers In this chapter, you’ll learn to use PostGIS to analyze roadways in Santa Fe, New Mexico as well as the location of farmers’ markets across the United States. You’ll learn how to construct and query spatial data types and how to work with different geographic data formats you might encounter when you obtain data from public and private data sources. You’ll also learn about map projections and grid systems. The goal is to give you tools to glean information from spatial data, similar to how you’ve analyzed numbers and text. We’ll begin by setting up PostGIS so we can explore different types of spatial data. All code and data for the exercises are available with the book’s resources at https://www.nostarch.com/practicalSQL/. Installing PostGIS and Creating a Spatial Database PostGIS is a free, open source project created by the Canadian geospatial company Refractions Research and maintained by an international team of developers under the Open Source Geospatial Foundation.", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 191 + }, + { + "text": "code and data for the exercises are available with the book’s resources at https://www.nostarch.com/practicalSQL/. Installing PostGIS and Creating a Spatial Database PostGIS is a free, open source project created by the Canadian geospatial company Refractions Research and maintained by an international team of developers under the Open Source Geospatial Foundation. You’ll find documentation and updates at http://postgis.net/. If you’re using Windows or macOS and have installed PostgreSQL following the steps in the book’s Introduction, PostGIS should be on your machine. It’s also often installed on PostgreSQL on cloud providers, such as Amazon Web Services. But if you’re using Linux or if you installed PostgreSQL some other way on Windows or macOS, follow the installation instructions at http://postgis.net/install/. Let’s create a database and enable PostGIS. The process is similar to the one you used to create your first database in Chapter 1 but with a few extra steps. Follow these steps in pgAdmin to make a database called gis_analysis: 1. In the pgAdmin object browser (left pane), connect to your server and expand the Databases node by clicking the plus sign. 2. Click once on the analysis database you’ve used for past exercises. 3. Choose Tools ▸ Query Tool. Estadísticos e-Books & Papers 4. In the Query Tool, run the code in Listing 14-1. CREATE DATABASE gis_analysis; Listing 14-1: Creating a gis_analysis database PostgreSQL will create the gis_analysis database, which is no different than others you’ve made. To enable PostGIS extensions on it, follow these steps: 1. Close the Query Tool tab. 2. In the object browser, right-click Databases and select Refresh. 3. Click the new gis_analysis database in the list to highlight it. 4. Open a new Query Tool tab by selecting Tools ▸ Query Tool. The gis_analysis database should be listed at the top of the editing pane. 5. In the Query Tool, run the code in Listing 14-2. CREATE EXTENSION postgis; Listing 14-2: Loading the PostGIS extension You’ll see the message CREATE EXTENSION. Your database has now been updated to include spatial data types and dozens of spatial analysis functions. Run SELECT postgis_full_version(); to display the version number of PostGIS along with its installed components. The version won’t match the PostgreSQL version installed, but that’s okay. The Building Blocks of Spatial Data Before you learn to query spatial data, let’s look at how it’s described in GIS and related data formats (although if you want to dive straight into queries, you can skip to “Analyzing Farmers’ Markets Data” on page 250 and return here later). A point on a grid is the smallest building block of spatial data. The grid might be marked with x- and y-axes, or longitude and latitude if Estadísticos e-Books & Papers we’re using a map. A grid could be flat, with two dimensions, or it could describe a three-dimensional space such as a cube. In some data formats, such as the JavaScript-based GeoJSON, a point might have a location on the grid as well as attributes providing additional information. For example, a grocery", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 192 + }, + { + "text": "a map. A grid could be flat, with two dimensions, or it could describe a three-dimensional space such as a cube. In some data formats, such as the JavaScript-based GeoJSON, a point might have a location on the grid as well as attributes providing additional information. For example, a grocery store could be described by a point containing its longitude and latitude as well as attributes showing the store’s name and hours of operation. Two-Dimensional Geometries To create more complex spatial data, you connect multiple points using lines. The International Organization for Standardization (ISO) and the Open Geospatial Consortium (OGC) have created a simple feature standard for building and accessing two- and three-dimensional shapes, sometimes referred to as geometries. PostGIS supports the standard. The most commonly used simple features you’ll encounter when querying or creating spatial data with PostGIS include the following: Point A single location in a two- or three-dimensional plane. On maps, a Point is usually represented by a dot marking a longitude and latitude. LineString Two or more points connected by a straight line. With LineStrings, you can represent features such as a road, hiking trail, or stream. Polygon A two-dimensional shape, like a triangle or a square, that has three or more straight sides, each constructed from a LineString. In geographic analysis, Polygons represent objects such as nations, states, buildings, and bodies of water. A Polygon also can have one or more interior Polygons that act as holes inside the larger Polygon. MultiPoint A set of Points. For example, you can represent multiple locations of a retailer with a single MultiPoint object that contains each store’s latitude and longitude. Estadísticos e-Books & Papers MultiLineString A set of LineStrings. You can represent, for example, an object such as a road with several noncontinuous segments. MultiPolygon A set of Polygons. For example, you can represent a parcel of land that is divided into two parts by a road: you can group them in one MultiPolygon object rather than using separate polygons. Figure 14-1 shows an example of each feature. Figure 14-1: Visual examples of geometries Using PostGIS functions, you can create your own spatial data by constructing these objects using points or other geometries. Or, you can use PostGIS functions to perform calculations on existing spatial data. Generally, to create a spatial object, the functions require input of a well- known text (WKT) string, which is text that represents a geometry, plus an optional Spatial Reference System Identifier (SRID) that specifies the grid on which to place the objects. I’ll explain the SRID shortly, but first, let’s look at examples of WKT strings and then build some geometries using Estadísticos e-Books & Papers them. Well-Known Text Formats The OGC standard’s WKT format includes the geometry type and its coordinates inside one or more sets of parentheses. The number of coordinates and parentheses varies depending on the geometry you want to create. Table 14-1 shows examples of the more frequently used geometry types and their WKT formats. Here, I", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 193 + }, + { + "text": "OGC standard’s WKT format includes the geometry type and its coordinates inside one or more sets of parentheses. The number of coordinates and parentheses varies depending on the geometry you want to create. Table 14-1 shows examples of the more frequently used geometry types and their WKT formats. Here, I show longitude/latitude pairs for the coordinates, but you might encounter grid systems that use other measures. NOTE WKT accepts coordinates in the order of longitude, latitude, which is backward from Google Maps and some other software. Tom MacWright, formerly of the Mapbox software company, notes at https://macwright.org/lonlat/ that neither order is “right” and catalogs the “frustrating inconsistency” in which mapping-related code handles the order of coordinates. Table 14-1: Well-Known Text Formats for Geometries Geometry Format Notes Point POINT (-74.9 42.7) A coordinate pair marking a point at −74.9 longitude and 42.7 latitude. LineString LINESTRING (-74.9 42.7, -75.1 42.7) A straight line with endpoints marked by two coordinate pairs. Polygon POLYGON ((-74.9 42.7, -75.1 42.7, -75.1 42.6, -74.9 42.7)) A triangle outlined by three different pairs of coordinates. Although listed twice, the first Estadísticos e-Books & Papers and last pair are the same coordinates, closing the shape. MultiPoint MULTIPOINT (-74.9 42.7, -75.1 42.7) Two Points, one for each pair of coordinates. MultiLineStringMULTILINESTRING ((-76.27 43.1, -76.06 43.08), (-76.2 43.3, -76.2 43.4, -76.4 43.1)) Two LineStrings. The first has two points; the second has three. MultiPolygon MULTIPOLYGON (((-74.92 42.7, -75.06 42.71, -75.07 42.64, -74.92 42.7), (-75.0 42.66, -75.0 42.64, -74.98 42.64, -74.98 42.66, -75.0 42.66))) Two Polygons. The first is a triangle, and the second is a rectangle. Although these examples create simple shapes, in practice, complex geometries could comprise thousands of coordinates. A Note on Coordinate Systems Representing the Earth’s spherical surface on a two-dimensional map is not easy. Imagine peeling the outer layer of the Earth from the globe and trying to spread it on a table while keeping all pieces of the continents and oceans connected. Inevitably, some areas of the map would stretch. This is what occurs when cartographers create a map projection with its own projected coordinate system that flattens the Earth’s round surface to a two-dimensional plane. Some projections represent the entire world; others are specific to regions or purposes. For example, the Mercator projection is commonly used for navigation in apps, such as Google Maps. The math behind its transformation distorts land areas close to the North and South Poles, making them appear much larger than reality. The Albers projection is the one you would most likely see displayed on TV screens in the United States as votes are tallied on election night. It’s also used by the U.S. Census Bureau. Projections are derived from geographic coordinate systems, which define Estadísticos e-Books & Papers the grid of latitude, longitude, and height of any point on the globe along with factors including the Earth’s shape. Whenever you obtain geographic data, it’s critical to know the coordinate systems it references to check whether your calculations are accurate. Often,", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 194 + }, + { + "text": "coordinate systems, which define Estadísticos e-Books & Papers the grid of latitude, longitude, and height of any point on the globe along with factors including the Earth’s shape. Whenever you obtain geographic data, it’s critical to know the coordinate systems it references to check whether your calculations are accurate. Often, the coordinate system or projection is named in user documentation. Spatial Reference System Identifier When using PostGIS (and many GIS applications), you need to specify the coordinate system you’re using via its SRID. When you enabled the PostGIS extension at the beginning of this chapter, the process created the table spatial_ref_sys, which contains SRIDs as its primary key. The table also contains the column srtext, which includes a WKT representation of the spatial reference system as well as other metadata. In this chapter, we’ll frequently use SRID 4326, the ID for the geographic coordinate system WGS 84. It’s the most recent World Geodetic System (WGS) standard used by GPS, and you’ll encounter it often if you acquire spatial data. You can see the WKT representation for WGS 84 by running the code in Listing 14-3 that looks for its SRID, 4326: SELECT srtext FROM spatial_ref_sys WHERE srid = 4326; Listing 14-3: Retrieving the WKT for SRID 4326 Run the query and you should get the following result, which I’ve indented for readability: GEOGCS[\"WGS 84\", DATUM[\"WGS_1984\", SPHEROID[\"WGS 84\",6378137,298.257223563, AUTHORITY[\"EPSG\",\"7030\"]], AUTHORITY[\"EPSG\",\"6326\"]], PRIMEM[\"Greenwich\",0, AUTHORITY[\"EPSG\",\"8901\"]], UNIT[\"degree\",0.0174532925199433, AUTHORITY[\"EPSG\",\"9122\"]], AUTHORITY[\"EPSG\",\"4326\"]] Estadísticos e-Books & Papers You don’t need to use this information for any of this chapter’s exercises, but it’s helpful to know some of the variables and how they define the projection. The GEOGCS keyword provides the geographic coordinate system in use. Keyword PRIMEM specifies the location of the Prime Meridian, or longitude 0. To see definitions of all the variables, check the reference at http://docs.geotools.org/stable/javadocs/org/opengis/referencing/doc- files/WKT.html. Conversely, if you ever need to find the SRID associated with a coordinate system, you can query the srtext column in spatial_ref_sys to find it. PostGIS Data Types Installing PostGIS adds five data types to your database. The two data types we’ll use in the exercises are geography and geometry. Both types can store spatial data, such as the points, lines, polygons, SRIDs, and so on you just learned about, but they have important distinctions: geography A data type based on a sphere, using the round-earth coordinate system (longitude and latitude). All calculations occur on the globe, taking its curvature into account. That makes the math complicated and limits the number of functions available to work with the geography type. But because the Earth’s curvature is factored in, calculations for distance are more precise; you should use the geography data type when handling data that spans large areas. Also, the results from calculations on the geography type will be expressed in meters. geometry A data type based on a plane, using the Euclidean coordinate system. Calculations occur on straight lines as opposed to along the curvature of a sphere, making calculations for geographical distance less precise than with the", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 195 + }, + { + "text": "the results from calculations on the geography type will be expressed in meters. geometry A data type based on a plane, using the Euclidean coordinate system. Calculations occur on straight lines as opposed to along the curvature of a sphere, making calculations for geographical distance less precise than with the geography data type; the results of calculations are expressed in units of whichever coordinate system you’ve designated. Estadísticos e-Books & Papers The PostGIS documentation at https://postgis.net/docs/using_postgis_dbmanagement.html offers guidance on when to use one or the other type. In short, if you’re working strictly with longitude/latitude data or if your data covers a large area, such as a continent or the globe, use the geography type, even though it limits the functions you can use. If your data covers a smaller area, the geometry type provides more functions and better performance. You can also change one type to the other using CAST. With the background you have now, we can start working with spatial objects. Creating Spatial Objects with PostGIS Functions PostGIS has more than three dozen constructor functions that build spatial objects using WKT or coordinates. You can find a list at https://postgis.net/docs/reference.html#Geometry_Constructors, but the following sections explain several that you’ll use in the exercises. Most PostGIS functions begin with the letters ST, which is an ISO naming standard that means spatial type. Creating a Geometry Type from Well-Known Text The ST_GeomFromText(WKT, SRID) function creates a geometry data type from an input of a WKT string and an optional SRID. Listing 14-4 shows simple SELECT statements that generate geometry data types for each of the simple features described in Table 14-1. Running these SELECT statements is optional, but it’s important to know how to construct each simple feature. SELECT ST_GeomFromText(➊'POINT(-74.9233606 42.699992)', ➋4326); SELECT ST_GeomFromText('LINESTRING(-74.9 42.7, -75.1 42.7)', 4326); SELECT ST_GeomFromText('POLYGON((-74.9 42.7, -75.1 42.7, -75.1 42.6, -74.9 42.7))', 4326); SELECT ST_GeomFromText('MULTIPOINT (-74.9 42.7, -75.1 42.7)', 4326); SELECT ST_GeomFromText('MULTILINESTRING((-76.27 43.1, -76.06 43.08), (-76.2 43.3, -76.2 43.4, -76.4 43.1))', 4326); SELECT ST_GeomFromText('MULTIPOLYGON➌(( Estadísticos e-Books & Papers (-74.92 42.7, -75.06 42.71, -75.07 42.64, -74.92 42.7)➍, (-75.0 42.66, -75.0 42.64, -74.98 42.64, -74.98 42.66, -75.0 42.66)))', 4326); Listing 14-4: Using ST_GeomFromText() to create spatial objects For each example, we give coordinates as the first input and the SRID 4326 as the second. In the first example, we create a point by inserting the WKT POINT string ➊ as the first argument to ST_GeomFromText() with the SRID ➋ as the optional second argument. We use the same format in the rest of the examples. Note that we don’t have to indent the coordinates. I only do so here to make the coordinate pairs more readable. Be sure to keep track of the number of parentheses that segregate objects, particularly in complex structures, such as the MultiPolygon. For example, we need to use two opening parentheses ➌ and enclose each polygon’s coordinates within another set of parentheses ➍. Executing each statement should return the geometry data type encoded in a string of characters that looks something like", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 196 + }, + { + "text": "segregate objects, particularly in complex structures, such as the MultiPolygon. For example, we need to use two opening parentheses ➌ and enclose each polygon’s coordinates within another set of parentheses ➍. Executing each statement should return the geometry data type encoded in a string of characters that looks something like this truncated example: 0101000020E61000008EDA0E5718BB52C017BB7D5699594540 ... This result shows how the data is stored in a table. Typically, you won’t be reading that string of code. Instead, you’ll use geometry (or geography) columns as inputs to functions. Creating a Geography Type from Well-Known Text To create a geography data type, you can use ST_GeogFromText(WKT) to convert a WKT or ST_GeogFromText(EWKT) to convert a PostGIS-specific variation called extended WKT that includes the SRID. Listing 14-5 shows how to pass in the SRID as part of the extended WKT string to create a MultiPoint geography object with three points: SELECT ST_GeogFromText('SRID=4326;MULTIPOINT(-74.9 42.7, -75.1 42.7, -74.924 42.6)') Estadísticos e-Books & Papers Listing 14-5: Using ST_GeogFromText() to create spatial objects Along with the all-purpose ST_GeomFromText() and ST_GeogFromText() functions, PostGIS includes several that are specific to creating certain spatial objects. I’ll cover those briefly next. Point Functions The ST_PointFromText() and ST_MakePoint() functions will turn a WKT POINT into a geometry data type. Points mark coordinates, such as longitude and latitude, which you would use to identify locations or use as building blocks of other objects, such as LineStrings. Listing 14-6 shows how these functions work: SELECT ➊ST_PointFromText('POINT(-74.9233606 42.699992)', 4326); SELECT ➋ST_MakePoint(-74.9233606, 42.699992); SELECT ➌ST_SetSRID(ST_MakePoint(-74.9233606, 42.699992), 4326); Listing 14-6: Functions specific to making Points The ST_PointFromText(WKT, SRID) ➊ function creates a point geometry type from a WKT POINT and an optional SRID as the second input. The PostGIS docs note that the function includes validation of coordinates that makes it slower than the ST_GeomFromText() function. The ST_MakePoint(x, y, z, m) ➋ function creates a point geometry type on a two-, three-, and four-dimensional grid. The first two parameters, x and y in the example, represent longitude and latitude coordinates. You can use the optional z to represent altitude and m to represent a fourth- dimensional measure, such as time. That would allow you to mark a location at a certain time, for example. The ST_MakePoint() function is faster than ST_GeomFromText() and ST_PointFromText(), but if you want to specify an SRID, you’ll need to designate one by wrapping it inside the ST_SetSRID() ➌ function. LineString Functions Estadísticos e-Books & Papers Now let’s examine some functions we use specifically for creating LineString geometry data types. Listing 14-7 shows how they work: SELECT ➊ST_LineFromText('LINESTRING(-105.90 35.67,-105.91 35.67)', 4326); SELECT ➋ST_MakeLine(ST_MakePoint(-74.9, 42.7), ST_MakePoint(-74.1, 42.4)); Listing 14-7: Functions specific to making LineStrings The ST_LineFromText(WKT, SRID) ➊ function creates a LineString from a WKT LINESTRING and an optional SRID as its second input. Like ST_PointFromText() earlier, this function includes validation of coordinates that makes it slower than ST_GeomFromText(). The ST_MakeLine(geom, geom) ➋ function creates a LineString from inputs that must be of the geometry data type. In Listing 14-7, the example uses two ST_MakePoint()", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 197 + }, + { + "text": "LINESTRING and an optional SRID as its second input. Like ST_PointFromText() earlier, this function includes validation of coordinates that makes it slower than ST_GeomFromText(). The ST_MakeLine(geom, geom) ➋ function creates a LineString from inputs that must be of the geometry data type. In Listing 14-7, the example uses two ST_MakePoint() functions as inputs to create the start and endpoint of the line. You can also pass in an ARRAY object with multiple points, perhaps generated by a subquery, to generate a more complex line. Polygon Functions Let’s look at three Polygon functions: ST_PolygonFromText(), ST_MakePolygon(), and ST_MPolyFromText(). All create geometry data types. Listing 14-8 shows how you can create Polygons with each: SELECT ➊ST_PolygonFromText('POLYGON((-74.9 42.7, -75.1 42.7, -75.1 42.6, -74.9 42.7))', 4326); SELECT ➋ST_MakePolygon( ST_GeomFromText('LINESTRING(-74.92 42.7, -75.06 42.71, -75.07 42.64, -74.92 42.7)', 4326)); SELECT ➌ST_MPolyFromText('MULTIPOLYGON(( (-74.92 42.7, -75.06 42.71, -75.07 42.64, -74.92 42.7), (-75.0 42.66, -75.0 42.64, -74.98 42.64, -74.98 42.66, -75.0 42.66) ))', 4326); Listing 14-8: Functions specific to making Polygons Estadísticos e-Books & Papers The ST_PolygonFromText(WKT, SRID) ➊ function creates a Polygon from a WKT POLYGON and an optional SRID. As with the similarly named functions for creating points and lines, it includes a validation step that makes it slower than ST_GeomFromText(). The ST_MakePolygon(linestring) ➋ function creates a Polygon from a LineString that must open and close with the same coordinates, ensuring the object is closed. This example uses ST_GeomFromText() to create the LineString geometry using a WKT LINESTRING. The ST_MPolyFromText(WKT, SRID) ➌ function creates a MultiPolygon from a WKT and an optional SRID. Now you have the building blocks to analyze spatial data. Next, we’ll use them to explore a set of data. Analyzing Farmers’ Markets Data The National Farmers’ Market Directory from the U.S. Department of Agriculture catalogs the location and offerings of more than 8,600 “markets that feature two or more farm vendors selling agricultural products directly to customers at a common, recurrent physical location,” according to https://www.ams.usda.gov/local-food-directories/farmersmarkets/. Attending these markets makes for an enjoyable weekend activity, so it would help to find those within a reasonable traveling distance. We can use SQL spatial queries to find the closest markets. The farmers_markets.csv file contains a portion of the USDA data on each market, and it’s available along with the book’s resources at https://www.nostarch.com/practicalSQL/. Save the file to your computer and run the code in Listing 14-9 to create and load a farmers_markets table. Make sure you’re connected to the gis_analysis database you made earlier in this chapter, and change the COPY statement file path to match your file’s location. CREATE TABLE farmers_markets ( fmid bigint PRIMARY KEY, market_name varchar(100) NOT NULL, Estadísticos e-Books & Papers street varchar(180), city varchar(60), county varchar(25), st varchar(20) NOT NULL, zip varchar(10), longitude numeric(10,7), latitude numeric(10,7), organic varchar(1) NOT NULL ); COPY farmers_markets FROM 'C:\\YourDirectory\\farmers_markets.csv' WITH (FORMAT CSV, HEADER); Listing 14-9: Creating and loading the farmers_markets table The table contains routine address data plus the longitude and latitude for most markets. Twenty-nine of the markets were missing those values when I", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 198 + }, + { + "text": "varchar(10), longitude numeric(10,7), latitude numeric(10,7), organic varchar(1) NOT NULL ); COPY farmers_markets FROM 'C:\\YourDirectory\\farmers_markets.csv' WITH (FORMAT CSV, HEADER); Listing 14-9: Creating and loading the farmers_markets table The table contains routine address data plus the longitude and latitude for most markets. Twenty-nine of the markets were missing those values when I downloaded the file from the USDA. An organic column indicates whether the market offers organic products; a hyphen (-) in that column indicates an unknown value. After you import the data, count the rows using SELECT count(*) FROM farmers_markets;. If everything imported correctly, you should have 8,681 rows. Creating and Filling a Geography Column To perform spatial queries on the markets’ longitude and latitude, we need to convert those coordinates into a single column of a spatial data type. Because we’re working with locations spanning the entire United States and an accurate measurement of a large spherical distance is important, we’ll use the geography type. After creating the column, we can update it using Points derived from the coordinates, and then apply an index to speed up queries. Listing 14-10 contains the statements for doing these tasks: ➊ ALTER TABLE farmers_markets ADD COLUMN geog_point geography(POINT,4326); UPDATE farmers_markets SET geog_point = ➋ST_SetSRID( ➌ST_MakePoint(longitude,latitude),4326 )➍::geography; Estadísticos e-Books & Papers ➎ CREATE INDEX market_pts_idx ON farmers_markets USING GIST (geog_point); SELECT longitude, latitude, geog_point, ➏ ST_AsText(geog_point) FROM farmers_markets WHERE longitude IS NOT NULL LIMIT 5; Listing 14-10: Creating and indexing a geography column The ALTER TABLE statement ➊ you learned in Chapter 9 with the ADD COLUMN option creates a column of the geography type called geog_point that will hold points and reference the WSG 84 coordinate system, which we denote using SRID 4326. Next, we run a standard UPDATE statement to fill the geog_point column. Nested inside a ST_SetSRID() ➋ function, the ST_MakePoint() ➌ function takes as input the longitude and latitude columns from the table. The output, which is the geometry type by default, must be cast to geography to match the geog_point column type. To do this, we use the PostgreSQL-specific double-colon syntax (::) ➍ for casting data types. Adding a GiST Index Before you start analysis, it’s wise to add an index to the new column to speed up calculations. In Chapter 7, you learned about PostgreSQL’s default index, the B-Tree. A B-Tree index is useful for data that you can order and search using equality and range operators, but it’s less useful for spatial objects. The reason is that you cannot easily sort GIS data along one axis. For example, the application has no way to determine which of these coordinate pairs is greatest: (0,0), (0,1), or (1,0). Instead, for spatial data, the makers of PostGIS recommend using the Generalized Search Tree (GiST) index. PostgreSQL core team member Bruce Momjian describes GiST as “a general indexing framework designed to allow indexing of complex data types,” including geometries. The CREATE INDEX statement ➎ in Listing 14-10 adds a GiST index to Estadísticos e-Books & Papers geog_point. We can then use", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 199 + }, + { + "text": "Generalized Search Tree (GiST) index. PostgreSQL core team member Bruce Momjian describes GiST as “a general indexing framework designed to allow indexing of complex data types,” including geometries. The CREATE INDEX statement ➎ in Listing 14-10 adds a GiST index to Estadísticos e-Books & Papers geog_point. We can then use the SELECT statement to view the geography data to show the newly encoded geog_points column. To view the WKT version of geog_point, we wrap it in a ST_AsText() function ➏. The results should look similar to this, with geog_point truncated for brevity: Now we’re ready to perform calculations on the points. Finding Geographies Within a Given Distance While in Iowa in 2014 to report a story on farming, I visited the massive Downtown Farmers’ Market in Des Moines. With hundreds of vendors, the market spans several city blocks in the Iowa capital. Farming is big business in Iowa, and even though the downtown market is huge, it’s not the only one in the area. Let’s use PostGIS to find more farmers’ markets within a short distance from the downtown Des Moines market. The PostGIS function ST_DWithin() returns a Boolean value of true if one spatial object is within a specified distance of another object. If you’re working with the geography data type, as we are here, you need to use meters as the distance unit. If you’re using the geometry type, use the distance unit specified by the SRID. NOTE PostGIS distance measurements are on a straight line for geometry data, whereas for geography data, they’re on a sphere. Be careful not to confuse either with driving distance along roadways, which is usually farther from point to point. To perform calculations related to driving distances, check out the extension pgRouting at http://pgrouting.org/. Estadísticos e-Books & Papers Listing 14-11 uses the ST_DWithin() function to filter farmers_markets to show markets within 10 kilometers of the Downtown Farmers’ Market in Des Moines: SELECT market_name, city, st FROM farmers_markets WHERE ST_DWithin(➊geog_point, ➋ST_GeogFromText('POINT(-93.6204386 41.5853202)'), ➌10000) ORDER BY market_name; Listing 14-11: Using ST_DWithin() to locate farmers’ markets within 10 kilometers of a point The first input for ST_DWithin() is geog_point ➊, which holds the location of each row’s market in the geography data type. The second input is the ST_GeogFromText() function ➋ that returns a point geography from WKT. The coordinates -93.6204386 and 41.5853202 represent the longitude and latitude of the Downtown Farmers’ Market in Des Moines. The final input is 10000 ➌, which is the number of meters in 10 kilometers. The database calculates the distance between each market in the table and the downtown market. If a market is within 10 kilometers, it is included in the results. We’re using points here, but this function works with any geography or geometry type. If you’re working with objects such as polygons, you can use the related ST_DFullyWithin() function to find objects that are completely within a specified distance. Run the query; it should return nine rows: market_name city st --------------------------------------- --------------- ---- Beaverdale Farmers Market", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 200 + }, + { + "text": "function works with any geography or geometry type. If you’re working with objects such as polygons, you can use the related ST_DFullyWithin() function to find objects that are completely within a specified distance. Run the query; it should return nine rows: market_name city st --------------------------------------- --------------- ---- Beaverdale Farmers Market Des Moines Iowa Capitol Hill Farmers Market Des Moines Iowa Downtown Farmers' Market - Des Moines Des Moines Iowa Drake Neighborhood Farmers Market Des Moines Iowa Eastside Farmers Market Des Moines Iowa Highland Park Farmers Market Des Moines Iowa Historic Valley Junction Farmers Market West Des Moines Iowa LSI Global Greens Farmers' Market Des Moines Iowa Valley Junction Farmers Market West Des Moines Iowa Estadísticos e-Books & Papers One of these nine markets is the Downtown Farmers’ Market in Des Moines, which makes sense because its location is at the point used for comparison. The rest are other markets in Des Moines or in nearby West Des Moines. This operation should be familiar because it’s a standard feature on many online maps and product apps that let you locate stores or points of interest near you. Although this list of nearby markets is helpful, it would be even more helpful to know the exact distance of markets from downtown. We’ll use another function to report that. Finding the Distance Between Geographies The ST_Distance() function returns the minimum distance between two spatial objects. It also returns meters for geographies and SRID units for geometries. For example, Listing 14-12 calculates the distance in miles from Yankee Stadium in New York City’s Bronx borough to Citi Field in Queens, home of the New York Mets: SELECT ST_Distance( ST_GeogFromText('POINT(-73.9283685 40.8296466)'), ST_GeogFromText('POINT(-73.8480153 40.7570917)') ) / 1609.344 AS mets_to_yanks; Listing 14-12: Using ST_Distance() to calculate the miles between Yankee Stadium and Citi Field (Mets) In this example, to see the result in miles, we divide the result of the ST_Distance() function by 1609.344 (the number of meters in a mile) to convert the unit of distance from meters to miles. The result is about 6.5 miles: mets_to_yanks ---------------- 6.54386182787521 Let’s apply this technique for finding distance between points to the farmers’ market data using the code in Listing 14-13. We’ll display all farmers’ markets within 10 kilometers of the Downtown Farmers’ Market Estadísticos e-Books & Papers in Des Moines and show the distance in miles: SELECT market_name, city, ➊round( (ST_Distance(geog_point, ST_GeogFromText('POINT(-93.6204386 41.5853202)') ) / 1609.344)➋::numeric(8,5), 2 ) AS miles_from_dt FROM farmers_markets ➌ WHERE ST_DWithin(geog_point, ST_GeogFromText('POINT(-93.6204386 41.5853202)'), 10000) ORDER BY miles_from_dt ASC; Listing 14-13: Using ST_Distance() for each row in farmers_markets The query is similar to Listing 14-11, which used ST_DWithin() to find markets 10 kilometers or closer to downtown, but adds the ST_Distance() function as a column to calculate and display the distance from downtown. I’ve wrapped the function inside round() ➊ to trim the output. We provide ST_Distance() with the same two inputs we gave ST_DWithin() in Listing 14-11: geog_point and the ST_GeogFromText() function. The ST_Distance() function then calculates the distance between the points specified", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 201 + }, + { + "text": "column to calculate and display the distance from downtown. I’ve wrapped the function inside round() ➊ to trim the output. We provide ST_Distance() with the same two inputs we gave ST_DWithin() in Listing 14-11: geog_point and the ST_GeogFromText() function. The ST_Distance() function then calculates the distance between the points specified by both inputs, returning the result in meters. To convert to miles, we divide by 1609.344 ➋, which is the approximate number of meters in a mile. Then, to provide the round() function with the correct input data type, we cast the column result to type numeric. The WHERE clause ➌ uses the same ST_DWithin() function and inputs as in Listing 14-11. You should see the following results, ordered by distance in ascending order: Estadísticos e-Books & Papers Again, this is the type of list you see every day on your phone or computer when you’re searching online for a nearby store or address. You might also find it helpful for many other analysis scenarios, such as finding all the schools within a certain distance of a known source of pollution or all the houses within five miles of an airport. NOTE Another type of distance measurement supported by PostGIS, K-Nearest Neighbor, provides the ability to quickly find the closest point or shape to one you specify. For a lengthy overview of how it works, see http://workshops.boundlessgeo.com/postgis-intro/knn.html. So far, you’ve learned how to build spatial objects from WKT. Next, I’ll show you a common data format used in GIS called the shapefile and how to bring it into PostGIS for analysis. Working with Census Shapefiles A shapefile is a GIS data format developed by Esri, a U.S. company known for its ArcGIS mapping visualization and analysis platform. In addition to serving as the standard file format for GIS platforms—such as ArcGIS and the open source QGIS—governments, corporations, nonprofits, and technical organizations use shapefiles to display, analyze, and distribute Estadísticos e-Books & Papers data that includes a variety of geographic features, such as buildings, roads, and territorial boundaries. Shapefiles contain the information describing the shape of a feature (such as a county, a road, or a lake) as well as a database containing attributes about them. Those attributes might include their name and other descriptors. A single shapefile can contain only one type of shape, such as polygons or points, and when you load a shapefile into a GIS platform that supports visualization, you can view the shapes and query their attributes. PostgreSQL, with the PostGIS extension, doesn’t visualize the shapefile data, but it does allow you to run complex queries on the spatial data in the shapefile, which we’ll do in “Exploring the Census 2010 Counties Shapefile” on page 259 and “Performing Spatial Joins” on page 262. First, let’s examine the structure and contents of shapefiles. Contents of a Shapefile A shapefile refers to a collection of files with different extensions, and each serves a different purpose. Usually, when you download a shapefile from a source, it comes in a", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 202 + }, + { + "text": "and “Performing Spatial Joins” on page 262. First, let’s examine the structure and contents of shapefiles. Contents of a Shapefile A shapefile refers to a collection of files with different extensions, and each serves a different purpose. Usually, when you download a shapefile from a source, it comes in a compressed archive, such as .zip. You’ll need to unzip it to access the individual files. Per ArcGIS documentation, these are the most common extensions you’ll encounter: .shp Main file that stores the feature geometry. .shx Index file that stores the index of the feature geometry. .dbf Database table (in dBASE format) that stores the attribute information of features. .xml XML-format file that stores metadata about the shapefile. .prj Projection file that stores the coordinate system information. You can open this file with a text editor to view the geographic coordinate system and projection. Estadísticos e-Books & Papers According to the documentation, files with the first three extensions include necessary data required for working with a shapefile. The other file types are optional. You can load a shapefile into PostGIS to access its spatial objects and the attributes for each. Let’s do that next and explore some additional analysis functions. Loading Shapefiles via the GUI Tool There are two ways to load shapefiles into your database. The PostGIS suite includes a Shapefile Import/Export Manager with a simple graphical user interface (GUI), which users may prefer. Alternately, you can use the command line application shp2pgsql, which is described in “Loading Shapefiles with shp2pgsql” on page 311. Let’s start with a look at how to work with the GUI tool. Windows Shapefile Importer/Exporter On Windows, if you followed the installation steps in the book’s Introduction, you should find the Shapefile Import/Export Manager by selecting Start ▸ PostGIS Bundle x.y for PostgreSQL x64 x.y ▸ PostGIS 2.0 Shapefile and DBF Loader Exporter. Whatever you see in place of x.y should match the version of the software you downloaded. You can skip ahead to “Connecting to the Database and Loading a Shapefile” on page 258. macOS and Linux Shapefile Importer/Exporter On macOS, the postgres.app installation outlined in the book’s Introduction doesn’t include the GUI tool, and as of this writing the only macOS version of the tool available (from the geospatial firm Boundless) doesn’t work with macOS High Sierra. I’ll update the status at the book’s resources at https://www.nostarch.com/practicalSQL/ if that changes. In the meantime, follow the instructions found in “Loading Shapefiles with shp2pgsql” on page 311. Then move on to “Exploring the Census 2010 Counties Shapefile” on page 259. Estadísticos e-Books & Papers For Linux users, pgShapeLoader is available as the application shp2pgsql-gui. Visit http://postgis.net/install/ and follow the instructions for your Linux distribution. Now, you can connect to the database and load a shapefile. Connecting to the Database and Loading a Shapefile Let’s connect the Shapefile Import/Export Manager to your database and then load a shapefile. I’ve included several shapefiles with the resources for this chapter at https://www.nostarch.com/practicalSQL/. We’ll start with TIGER/Line Shapefiles from", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 203 + }, + { + "text": "you can connect to the database and load a shapefile. Connecting to the Database and Loading a Shapefile Let’s connect the Shapefile Import/Export Manager to your database and then load a shapefile. I’ve included several shapefiles with the resources for this chapter at https://www.nostarch.com/practicalSQL/. We’ll start with TIGER/Line Shapefiles from the U.S. Census that contain the boundaries for each county or county equivalent, such as parish or borough, as of the 2010 Decennial Census. You can learn more about this series of shapefiles at https://www.census.gov/geo/maps-data/data/tiger- line.html. NOTE Many organizations provide data in shapefile format. Start with your national or local government agencies or check the Wikipedia entry “List of GIS data sources.” Save tl_2010_us_county10.zip to your computer and unzip it; the archive should contain five files with the extensions I listed earlier on page 257. Then open the Shapefile and DBF Loader Exporter app. First, you need to establish a connection between the app and your gis_analysis database. To do that, follow these steps: 1. Click View connection details. 2. In the dialog that opens, enter postgres for the Username, and enter a password if you added one for the server during initial setup. 3. Ensure that Server Host has localhost and 5432 by default. Leave those as is unless you’re on a different server or port. 4. Enter gis_analysis for the Database name. Figure 14-2 shows a Estadísticos e-Books & Papers screenshot of what the connection should look like. 5. Click OK. You should see the message Connection Succeeded in the log window. Figure 14-2: Establishing the PostGIS connection in the shapefile loader Now that you’ve successfully established the PostGIS connection, you can load your shapefile: 1. Under Options, change DBF file character encoding to Latin1—we do this because the shapefile attributes include county names with characters that require this encoding. Keep the default checked boxes, including the one to create an index on the spatial column. Click OK. 2. Click Add File and select tl_2010_us_county10.shp from the location you saved it. Click Open. The file should appear in the Shapefile list in the loader, as shown in Figure 14-3. Estadísticos e-Books & Papers Figure 14-3: Specifying upload details in the shapefile loader 3. In the Table column, double-click to select the table name. Replace it with us_counties_2010_shp. 4. In the SRID column, double-click and enter 4269. That’s the ID for the North American Datum 1983 coordinate system, which is often used by U.S. federal agencies including the Census Bureau. 5. Click Import. In the log window, you should see a message that ends with the following message: Shapefile type: Polygon PostGIS type: MULTIPOLYGON[2] Shapefile import completed. Switch to pgAdmin, and in the object browser, expand the gis_analysis node and continue expanding by selecting Schemas ▸ public ▸ Tables. Refresh your tables by right-clicking Tables and selecting Refresh from the pop-up menu. You should see us_counties_2010_shp listed. Congrats! You’ve loaded your shapefile into a table. As part of the import, the shapefile loader also indexed the geom column. Exploring", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 204 + }, + { + "text": "and continue expanding by selecting Schemas ▸ public ▸ Tables. Refresh your tables by right-clicking Tables and selecting Refresh from the pop-up menu. You should see us_counties_2010_shp listed. Congrats! You’ve loaded your shapefile into a table. As part of the import, the shapefile loader also indexed the geom column. Exploring the Census 2010 Counties Shapefile Estadísticos e-Books & Papers The us_counties_2010_shp table contains columns including each county’s name as well as the Federal Information Processing Standards (FIPS) codes uniquely assigned to each state and county. The geom column contains the spatial data on each county’s boundary. To start, let’s check what kind of spatial object geom contains using the ST_AsText() function. Use the code in Listing 14-14 to show the WKT representation of the first geom value in the table. SELECT ST_AsText(geom) FROM us_counties_2010_shp LIMIT 1; Listing 14-14: Checking the geom column’s WKT representation The result is a MultiPolygon with hundreds of coordinate pairs that outline the boundary of the county. Here’s a portion of the output: MULTIPOLYGON(((-162.637688 54.801121,-162.641178 54.795317,-162.644046 54.789099,-162.653751 54.780339,-162.666629 54.770215,-162.677799 54.762716,- 162.692356 54.758771,-162.70676 54.754987,-162.722965 54.753155,-162.740178 54.753102,-162.76206 54.757968,-162.783454 54.765285,-162.797004 54.772181, -162.802591 54.775817,-162.807411 54.779871,-162.811898 54.786852, --snip-- ))) Each coordinate pair marks a point on the boundary of the county. Now, you’re ready to analyze the data. Finding the Largest Counties in Square Miles The census data leads us to a natural question: which county has the largest area? To calculate the county area, Listing 14-15 uses the ST_Area() function, which returns the area of a Polygon or MultiPolygon object. If you’re working with a geography data type, ST_Area() returns the result in square meters. With a geometry data type, the function returns the area in SRID units. Typically, the units are not useful for practical analysis, but you can cast the geometry data to geography to obtain square meters. That’s what we’ll do here. This is a more intensive calculation than others we’ve done so far, so if you’re using an older computer, expect extra time for the query to complete. SELECT name10, Estadísticos e-Books & Papers statefp10 AS st, round( ( ST_Area(➊geom::geography) / ➋2589988.110336 )::numeric, 2 ) AS ➌square_miles FROM us_counties_2010_shp ORDER BY square_miles ➍DESC LIMIT 5; Listing 14-15: Finding the largest counties by area using ST_Area() The geom column is data type geometry, so to find the area in square meters, we cast the geom column as a geography data type using the double- colon syntax ➊. Then, to get square miles, we divide the area by 2589988.110336, which is the number of square meters in a square mile ➋. To make the result easier to read, I’ve wrapped it in a round() function and named the resulting column square_miles ➌. Finally, we list the results in descending order from the largest area to the smallest ➍ and use LIMIT 5 to show only the first five results, which should look like this: name10 st square_miles ---------------- -- ------------ Yukon-Koyukuk 02 147805.08 North Slope 02 94796.21 Bethel 02 45504.36 Northwest Arctic 02 40748.95 Valdez-Cordova 02 40340.08 The", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 205 + }, + { + "text": "descending order from the largest area to the smallest ➍ and use LIMIT 5 to show only the first five results, which should look like this: name10 st square_miles ---------------- -- ------------ Yukon-Koyukuk 02 147805.08 North Slope 02 94796.21 Bethel 02 45504.36 Northwest Arctic 02 40748.95 Valdez-Cordova 02 40340.08 The five counties with the largest areas are all in Alaska, denoted by the state FIPS code 02. Yukon-Koyukuk, located in the heart of Alaska, is more than 147,800 square miles. (Keep that information in mind for the “Try It Yourself” exercise at the end of the chapter.) Finding a County by Longitude and Latitude If you’ve ever wondered how website ads seem to know where you live (“You won’t believe what this Boston man did with his old shoes!”), it’s thanks to geolocation services that use various means, such as your phone’s GPS, to find your longitude and latitude. Once your coordinates are known, they can be used in a spatial query to find which geography contains that point. Estadísticos e-Books & Papers You can do the same using your census shapefile and the ST_Within() function, which returns true if one geometry is inside another. Listing 14- 16 shows an example using the longitude and latitude of downtown Hollywood: SELECT name10, statefp10 FROM us_counties_2010_shp WHERE ST_Within('SRID=4269;POINT(-118.3419063 34.0977076)'::geometry, geom); Listing 14-16: Using ST_Within() to find the county belonging to a pair of coordinates The ST_Within() function inside the WHERE clause requires two geometry inputs and checks whether the first is inside the second. For the function to work properly, both geometry inputs must have the same SRID. In this example, the first input is an extended WKT representation of a Point that includes the SRID 4269 (same as the census data), which is then cast as a geometry type. The ST_Within() function doesn’t accept a separate SRID input, so to set it for the supplied WKT, you must prefix it to the string like this: 'SRID=4269;POINT(-118.3419063 34.0977076)'. The second input is the geom column from the table. Run the query; you should see the following result: name10 statefp10 ----------- --------- Los Angeles 06 The query shows that the Point you supplied is within Los Angeles county in California (state FIPS 06). This information is very handy, because by joining additional data to this table you can tell a person about demographics or points of interest near them. Try supplying other longitude and latitude pairs to see which U.S. county they fall in. If you provide coordinates outside the United States, the query should return no results because the shapefile only contains U.S. areas. Performing Spatial Joins In Chapter 6, you learned about SQL joins, which involved linking Estadísticos e-Books & Papers related tables via columns where values match or where an expression is true. You can perform joins using spatial data columns too, which opens up interesting opportunities for analysis. For example, you could join a table of coffee shops (which includes their longitude and latitude) to the counties table to", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 206 + }, + { + "text": "related tables via columns where values match or where an expression is true. You can perform joins using spatial data columns too, which opens up interesting opportunities for analysis. For example, you could join a table of coffee shops (which includes their longitude and latitude) to the counties table to find out how many shops exist in each county based on their location. Or, you can use a spatial join to append data from one table to another for analysis, again based on location. In this section, we’ll explore spatial joins with a detailed look at roads and waterways using census data. Exploring Roads and Waterways Data Much of the year, the Santa Fe River, which cuts through the New Mexico state capital, is a dry riverbed better described as an intermittent stream. According to the Santa Fe city website, the river is susceptible to flash flooding and was named the nation’s most endangered river in 2007. If you were an urban planner, it would help to know where the river crosses roadways so you could plan for emergency response when it floods. You can determine these locations using another set of U.S. Census TIGER/Line shapefiles, which has details on roads and waterways in Santa Fe County. These shapefiles are also included with the book’s resources. Download and unzip tl_2016_35049_linearwater.zip and tl_2016_35049_roads.zip, and then launch the Shapefile and DBF Loader Exporter. Following the same steps in “Loading Shapefiles via the GUI Tool” on page 257, import both shapefiles to gis_analysis. Name the water table santafe_linearwater_2016 and the roads table santafe_roads_2016. Next, refresh your database and run a quick SELECT * FROM query on both tables to view the data. You should have 12,926 rows in the roads table and 1,198 in the linear water table. As with the counties shapefile you imported via the loader GUI, both tables have an indexed geom column of type geometry. It’s helpful to check the type of spatial object in the column so you know the type of spatial feature you’re querying. You can do that using the ST_AsText() function Estadísticos e-Books & Papers you learned in Listing 14-14 or using ST_GeometryType(), as shown in Listing 14-17: SELECT ST_GeometryType(geom) FROM santafe_linearwater_2016 LIMIT 1; SELECT ST_GeometryType(geom) FROM santafe_roads_2016 LIMIT 1; Listing 14-17: Using ST_GeometryType() to determine geometry Both queries should return one row with the same value: ST_MultiLineString. That value indicates that waterways and roads are stored as MultiLineString objects, which are a series of points connected by straight lines. Joining the Census Roads and Water Tables To find all the roads in Santa Fe that cross the Santa Fe River, we’ll join the tables using the JOIN ... ON syntax you learned in Chapter 6. Rather than looking for values that match in columns in both tables as usual, we’ll write a query that tells us where objects overlap. We’ll do this using the ST_Intersects() function, which returns a Boolean true if two spatial objects contact each other. Inputs can be either geometry or", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 207 + }, + { + "text": "6. Rather than looking for values that match in columns in both tables as usual, we’ll write a query that tells us where objects overlap. We’ll do this using the ST_Intersects() function, which returns a Boolean true if two spatial objects contact each other. Inputs can be either geometry or geography types. Listing 14-18 joins the tables: ➊ SELECT water.fullname AS waterway, roads.rttyp, roads.fullname AS road ➋ FROM santafe_linearwater_2016 water JOIN santafe_roads_2016 roads ➌ ON ST_Intersects(water.geom, roads.geom) WHERE water.fullname = ➍'Santa Fe Riv' ORDER BY roads.fullname; Listing 14-18: Spatial join with ST_Intersects() to find roads crossing the Santa Fe River The SELECT column list ➊ includes the fullname column from the santafe_linearwater_2016 table, which gets water as its alias in the FROM ➋ clause. The column list includes the rttyp code, which represents the Estadísticos e-Books & Papers route type, and fullname columns from the santafe_roads_2016 table, aliased as roads. In the ON portion ➌ of the JOIN clause, we use the ST_Intersects() function with the geom columns from both tables as inputs. This is an example of using the ON clause with an expression that evaluates to a Boolean result, as noted in “Linking Tables Using JOIN” on page 74. Then we use fullname to filter the results to show only those that have the full string 'Santa Fe Riv' ➍, which is how the Santa Fe River is listed in the water table. The query should return 54 rows; here are the first five: waterway rttyp road ------------ ----- ---------------- Santa Fe Riv M Baca Ranch Ln Santa Fe Riv M Cam Alire Santa Fe Riv M Cam Carlos Rael Santa Fe Riv M Cam Dos Antonios Santa Fe Riv M Cerro Gordo Rd --snip-- Each road in the results intersects with a portion of the Santa Fe River. The route type code for each of the first results is M, which indicates that the road name shown is its common name as opposed to a county or state recognized name, for example. Other road names in the complete results carry route types of C, S, or U (for unknown). The full route type code list is available at https://www.census.gov/geo/reference/rttyp.html. Finding the Location Where Objects Intersect We successfully identified all the roads that intersect the Santa Fe River. This is a good start, but it would help our survey of flood-danger areas more to know precisely where each intersection occurs. We can modify the query to include the ST_Intersection() function, which returns the location of the place where objects cross. I’ve added it as a column in Listing 14-19: SELECT water.fullname AS waterway, roads.rttyp, roads.fullname AS road, Estadísticos e-Books & Papers ➊ST_AsText(ST_Intersection(➋water.geom, roads.geom)) FROM santafe_linearwater_2016 water JOIN santafe_roads_2016 roads ON ST_Intersects(water.geom, roads.geom) WHERE water.fullname = 'Santa Fe Riv' ORDER BY roads.fullname; Listing 14-19: Using ST_Intersection() to show where roads cross the river The function returns a geometry object, so to get its WKT representation, we must wrap it in ST_AsText() ➊. The ST_Intersection() function takes two inputs: the", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 208 + }, + { + "text": "roads ON ST_Intersects(water.geom, roads.geom) WHERE water.fullname = 'Santa Fe Riv' ORDER BY roads.fullname; Listing 14-19: Using ST_Intersection() to show where roads cross the river The function returns a geometry object, so to get its WKT representation, we must wrap it in ST_AsText() ➊. The ST_Intersection() function takes two inputs: the geom columns ➋ from both the water and roads tables. Run the query, and the results should now include the exact coordinate location, or locations, where the river crosses the roads: You can probably think of more ideas for analyzing spatial data. For example, if you obtained a shapefile showing buildings, you could find those close to the river and in danger of flooding during heavy rains. Governments and private organizations regularly use these techniques as part of their planning process. Wrapping Up Mapping features is a powerful analysis tool, and the techniques you learned in this chapter provide you with a strong start toward exploring more with PostGIS. You might also want to look at the open source mapping application QGIS (http://www.qgis.org/), which provides tools for visualizing geographic data and working in depth with shapefiles. QGIS also works quite well with PostGIS, letting you add data from your tables directly onto a map. Estadísticos e-Books & Papers You’ve now added working with geographic data to your analysis skills. In the remaining chapters, I’ll give you additional tools and tips for working with PostgreSQL and related tools to continue to increase your skills. TRY IT YOURSELF Use the spatial data you’ve imported in this chapter to try additional analysis: 1. Earlier, you found which U.S. county has the largest area. Now, aggregate the county data to find the area of each state in square miles. (Use the statefp10 column in the us_counties_2010_shp table.) How many states are bigger than the Yukon-Koyukuk area? 2. Using ST_Distance(), determine how many miles separate these two farmers’ markets: the Oakleaf Greenmarket (9700 Argyle Forest Blvd, Jacksonville, Florida) and Columbia Farmers Market (1701 West Ash Street, Columbia, Missouri). You’ll need to first find the coordinates for both in the farmers_markets table. (Hint: You can also write this query using the Common Table Expression syntax you learned in Chapter 12.) 3. More than 500 rows in the farmers_markets table are missing a value in the county column, which is an example of dirty government data. Using the us_counties_2010_shp table and the ST_Intersects() function, perform a spatial join to find the missing county names based on the longitude and latitude of each market. Because geog_point in farmers_markets is of the geography type and its SRID is 4326, you’ll need to cast geom in the census table to the geography type and change its SRID using ST_SetSRID(). Estadísticos e-Books & Papers 15 SAVING TIME WITH VIEWS, FUNCTIONS, AND TRIGGERS One of the advantages of using a programming language is that it allows us to automate repetitive, boring tasks. For example, if you have to run the same query every month to update the same table, sooner or later", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 209 + }, + { + "text": "e-Books & Papers 15 SAVING TIME WITH VIEWS, FUNCTIONS, AND TRIGGERS One of the advantages of using a programming language is that it allows us to automate repetitive, boring tasks. For example, if you have to run the same query every month to update the same table, sooner or later you’ll search for a shortcut to accomplish the task. The good news is that shortcuts exist! In this chapter, you’ll learn techniques to encapsulate queries and logic into reusable PostgreSQL database objects that will speed up your workflow. As you read through this chapter, keep in mind the DRY programming principle: Don’t Repeat Yourself. Avoiding repetition saves time and prevents unnecessary mistakes. You’ll begin by learning to save queries as reusable database views. Next, you’ll explore how to create your own functions to perform operations on your data. You’ve already used functions, such as round() and upper(), to transform data; now, you’ll make functions to perform operations you specify. Then you’ll set up triggers to run functions automatically when certain events occur on a table. Using these techniques, you can reduce repetitive work and help maintain the integrity of your data. We’ll use tables created from examples in earlier chapters to practice Estadísticos e-Books & Papers these techniques. If you connected to the gis_analysis database in pgAdmin while working through Chapter 14, follow the instructions in that chapter to return to the analysis database. All the code for this chapter is available for download along with the book’s resources at https://www.nostarch.com/practicalSQL/. Let’s get started. Using Views to Simplify Queries A view is a virtual table you can create dynamically using a saved query. For example, every time you access the view, the saved query runs automatically and displays the results. Similar to a regular table, you can query a view, join a view to regular tables (or other views), and use the view to update or insert data into the table it’s based on, albeit with some caveats. In this section, we’ll look at regular views with a PostgreSQL syntax that is largely in line with the ANSI SQL standard. These views execute their underlying query each time you access the view, but they don’t store data the way a table does. A materialized view, which is specific to PostgreSQL, Oracle, and a limited number of other database systems, caches data created by the view, and you can later update that cached data. We won’t explore materialized views here, but you can browse to https://www.postgresql.org/docs/current/static/sql-creatematerializedview.html to learn more. Views are especially useful because they allow you to: Avoid duplicate effort by letting you write a query once and access the results when needed Reduce complexity for yourself or other database users by showing only columns relevant to your needs Provide security by limiting access to only certain columns in a table Estadísticos e-Books & Papers NOTE To ensure data security and fully prevent users from seeing sensitive information, such as the underlying salary data in the employees table, you must", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 210 + }, + { + "text": "users by showing only columns relevant to your needs Provide security by limiting access to only certain columns in a table Estadísticos e-Books & Papers NOTE To ensure data security and fully prevent users from seeing sensitive information, such as the underlying salary data in the employees table, you must restrict access by setting account permissions in PostgreSQL. Typically, a database administrator handles this function for an organization, but if you want to explore this issue further, read the PostgreSQL documentation on user roles at https://www.postgresql.org/docs/current/static/sql- createrole.html and the GRANT command at https://www.postgresql.org/docs/current/static/sql-grant.html. Views are easy to create and maintain. Let’s work through several examples to see how they work. Creating and Querying Views In this section, we’ll use data in the Decennial U.S. Census us_counties_2010 table you imported in Chapter 4. Listing 15-1 uses this data to create a view called nevada_counties_pop_2010 that displays only four out of the original 16 columns, showing data on just Nevada counties: ➊ CREATE OR REPLACE VIEW nevada_counties_pop_2010 AS ➋ SELECT geo_name, state_fips, county_fips, p0010001 AS pop_2010 FROM us_counties_2010 WHERE state_us_abbreviation = 'NV' ➌ ORDER BY county_fips; Listing 15-1: Creating a view that displays Nevada 2010 counties Here, we define the view using the keywords CREATE OR REPLACE VIEW ➊, followed by the view’s name and AS. Next is a standard SQL query SELECT ➋ that fetches the total population (the p0010001 column) for each Nevada county from the us_counties_2010 table. Then we order the data by the county’s FIPS (Federal Information Processing Standards) code ➌, which Estadísticos e-Books & Papers is a standard designator the Census Bureau and other federal agencies use to specify each county and state. Notice the OR REPLACE keywords after CREATE, which tell the database that if a view with this name already exists, replace it with the definition here. But here’s a caveat according to the PostgreSQL documentation: the query that generates the view ➋ must have the columns with the same names and same data types in the same order as the view it’s replacing. However, you can add columns at the end of the column list. Run the code in Listing 15-1 using pgAdmin. The database should respond with the message CREATE VIEW. To find the view you created, in pgAdmin’s object browser, right-click the analysis database and choose Refresh. Choose Schemas ▸ public ▸ Views to see the new view. When you right-click the view and choose Properties, you should see the query under the Definition tab in the dialog that opens. NOTE As with other database objects, you can delete a view using the DROP command. In this example, the syntax would be DROP VIEW nevada_counties_pop_2010;. After creating the view, you can use the view in the FROM clause of a SELECT query the same way you would use an ordinary table. Enter the code in Listing 15-2, which retrieves the first five rows from the view: SELECT * FROM nevada_counties_pop_2010 LIMIT 5; Listing 15-2: Querying the nevada_counties_pop_2010 view Aside from the five-row", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 211 + }, + { + "text": "the view in the FROM clause of a SELECT query the same way you would use an ordinary table. Enter the code in Listing 15-2, which retrieves the first five rows from the view: SELECT * FROM nevada_counties_pop_2010 LIMIT 5; Listing 15-2: Querying the nevada_counties_pop_2010 view Aside from the five-row limit, the result should be the same as if you had run the SELECT query used to create the view in Listing 15-1: geo_name state_fips county_fips pop_2010 ---------------- ---------- ----------- -------- Churchill County 32 001 24877 Estadísticos e-Books & Papers Clark County 32 003 1951269 Douglas County 32 005 46997 Elko County 32 007 48818 Esmeralda County 32 009 783 This simple example isn’t very useful unless quickly listing Nevada county population is a task you’ll perform frequently. So, let’s imagine a question data-minded analysts in a political research organization might ask often: what was the percent change in population for each county in Nevada (or any other state) from 2000 to 2010? We wrote a query to answer this question in Listing 6-13 (see “Performing Math on Joined Table Columns” on page 88). It wasn’t onerous to create, but it did require joining tables on two columns and using a percent change formula that involved rounding and type casting. To avoid repeating that work, we can save a query similar to the one in Listing 6-13 as a view. Listing 15-3 does this using a modified version of the earlier code in Listing 15-1: ➊ CREATE OR REPLACE VIEW county_pop_change_2010_2000 AS ➋ SELECT c2010.geo_name, c2010.state_us_abbreviation AS st, c2010.state_fips, c2010.county_fips, c2010.p0010001 AS pop_2010, c2000.p0010001 AS pop_2000, ➌ round( (CAST(c2010.p0010001 AS numeric(8,1)) - c2000.p0010001) / c2000.p0010001 * 100, 1 ) AS pct_change_2010_2000 ➍ FROM us_counties_2010 c2010 INNER JOIN us_counties_2000 c2000 ON c2010.state_fips = c2000.state_fips AND c2010.county_fips = c2000.county_fips ORDER BY c2010.state_fips, c2010.county_fips; Listing 15-3: Creating a view showing population change for U.S. counties We start the view definition with CREATE OR REPLACE VIEW ➊, followed by the name of the view and AS. The SELECT query ➋ names columns from the census tables and includes a column definition with a percent change calculation ➌ that you learned about in Chapter 5. Then we join the Census 2010 and 2000 tables ➍ using the state and county FIPS codes. Run the code, and the database should again respond with CREATE VIEW. Now that we’ve created the view, we can use the code in Listing 15-4 Estadísticos e-Books & Papers to run a simple query against the new view that retrieves data for Nevada counties: SELECT geo_name, st, pop_2010, ➊ pct_change_2010_2000 FROM county_pop_change_2010_2000 ➋ WHERE st = 'NV' LIMIT 5; Listing 15-4: Selecting columns from the county_pop_change_2010_2000 view In Listing 15-2, in the query against the first view we created, we retrieved every column in the view by using the asterisk wildcard after the SELECT keyword. Listing 15-4 shows that, as with a query on a table, we can name specific columns when querying a view. Here, we specify four of the county_pop_change_2010_2000", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 212 + }, + { + "text": "query against the first view we created, we retrieved every column in the view by using the asterisk wildcard after the SELECT keyword. Listing 15-4 shows that, as with a query on a table, we can name specific columns when querying a view. Here, we specify four of the county_pop_change_2010_2000 view’s seven columns. One is pct_change_2010_2000 ➊, which returns the result of the percent change calculation we’re looking for. As you can see, it’s much simpler to write the column name like this than the whole formula! We’re also filtering the results using a WHERE clause ➋, similar to how we would filter any query instead of returning all rows. After querying the four columns from the view, the results should look like this: geo_name st pop_2010 pct_change_2010_2000 ---------------- -- -------- -------------------- Churchill County NV 24877 3.7 Clark County NV 1951269 41.8 Douglas County NV 46997 13.9 Elko County NV 48818 7.8 Esmeralda County NV 783 -19.4 Now we can revisit this view as often as we like to pull data for presentations or to answer questions about the percent change in population for each county in Nevada (or any other state) from 2000 to 2010. Looking at just these five rows, you can see that a couple of interesting stories emerge: the effect of the 2000s’ housing boom on Clark County, Estadísticos e-Books & Papers which includes the city of Las Vegas, as well as a sharp drop in population in Esmeralda County, which has one of the lowest population densities in the United States. Inserting, Updating, and Deleting Data Using a View You can update or insert data in the underlying table that a view queries as long as the view meets certain conditions. One requirement is that the view must reference a single table. If the view’s query joins tables, as with the population change view we just built in the previous section, then you can’t perform inserts or updates directly. Also, the view’s query can’t contain DISTINCT, GROUP BY, or other clauses. (See a complete list of restrictions at https://www.postgresql.org/docs/current/static/sql- createview.html.) You already know how to directly insert and update data on a table, so why do it through a view? One reason is that with a view you can exercise more control over which data a user can update. Let’s work through an example to see how this works. Creating a View of Employees In the Chapter 6 lesson on joins, we created and filled departments and employees tables with four rows about people and where they work (if you skipped that section, you can revisit Listing 6-1 on page 75). Running a quick SELECT * FROM employees; query shows the table’s contents, as you can see here: emp_id first_name last_name salary dept_id ------ ---------- --------- ------ ------- 1 Nancy Jones 62500 1 2 Lee Smith 59300 1 3 Soo Nguyen 83000 2 4 Janet King 95000 2 Let’s say we want to give users in the Tax Department (its dept_id is 1) the ability", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 213 + }, + { + "text": "can see here: emp_id first_name last_name salary dept_id ------ ---------- --------- ------ ------- 1 Nancy Jones 62500 1 2 Lee Smith 59300 1 3 Soo Nguyen 83000 2 4 Janet King 95000 2 Let’s say we want to give users in the Tax Department (its dept_id is 1) the ability to add, remove, or update their employees’ names without letting them change salary information or data of employees in another Estadísticos e-Books & Papers department. To do this, we can set up a view using Listing 15-5: CREATE OR REPLACE VIEW employees_tax_dept AS SELECT emp_id, first_name, last_name, dept_id FROM employees ➊ WHERE dept_id = 1 ORDER BY emp_id ➋ WITH LOCAL CHECK OPTION; Listing 15-5: Creating a view on the employees table Similar to the views we’ve created so far, we’re selecting only the columns we want to show from the employees table and using WHERE to filter the results on dept_id = 1 ➊ to list only Tax Department staff. To restrict inserts or updates to Tax Department employees only, we add the WITH LOCAL CHECK OPTION ➋, which rejects any insert or update that does not meet the criteria of the WHERE clause. For example, the option won’t allow anyone to insert or update a row in the underlying table where the employee’s dept_id is 3. Create the employees_tax_dept view by running the code in Listing 15-5. Then run SELECT * FROM employees_tax_dept;, which should provide these two rows: emp_id first_name last_name dept_id ------ ---------- --------- ------- 1 Nancy Jones 1 2 Lee Smith 1 The result shows the employees who work in the Tax Department; they’re two of the four rows in the entire employees table. Now, let’s look at how inserts and updates work via this view. Inserting Rows Using the employees_tax_dept View We can also use a view to insert or update data, but instead of using the table name in the INSERT or UPDATE statement, we substitute the view name. After we add or change data using a view, the change is applied to the Estadísticos e-Books & Papers underlying table, which in this case is employees. The view then reflects the change via the query it runs. Listing 15-6 shows two examples that attempt to add new employee records via the employees_tax_dept view. The first succeeds, but the second fails. ➊ INSERT INTO employees_tax_dept (first_name, last_name, dept_id) VALUES ('Suzanne', 'Legere', 1); ➋ INSERT INTO employees_tax_dept (first_name, last_name, dept_id) VALUES ('Jamil', 'White', 2); ➌ SELECT * FROM employees_tax_dept; ➍ SELECT * FROM employees; Listing 15-6: Successful and rejected inserts via the employees_tax_dept view In the first INSERT ➊, which follows the insert format you learned in Chapter 1, we supply the first and last names of Suzanne Legere plus her dept_id. Because the dept_id is 1, the value satisfies the LOCAL CHECK in the view, and the insert succeeds when it executes. But when we run the second INSERT ➋ to add an employee named Jamil White using a dept_id of 2, the operation", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 214 + }, + { + "text": "names of Suzanne Legere plus her dept_id. Because the dept_id is 1, the value satisfies the LOCAL CHECK in the view, and the insert succeeds when it executes. But when we run the second INSERT ➋ to add an employee named Jamil White using a dept_id of 2, the operation fails with the error message new row violates check option for view \"employees_tax_dept\". The reason is that when we created the view in Listing 15-5, we used the WHERE clause to show only rows with dept_id = 1. The dept_id of 2 does not pass the LOCAL CHECK in the view, and it’s prevented from being inserted. Run the SELECT statement ➌ on the view to check that Suzanne Legere was successfully added: emp_id first_name last_name dept_id ------ ---------- --------- ------- 1 Nancy Jones 1 2 Lee Smith 1 5 Suzanne Legere 1 We can also query the employees table ➍ to see that, in fact, Suzanne Legere was added to the full table. The view queries the employees table Estadísticos e-Books & Papers each time we access it. emp_id first_name last_name salary dept_id ------ ---------- --------- ------ ------- 1 Nancy Jones 62500 1 2 Lee Smith 59300 1 3 Soo Nguyen 83000 2 4 Janet King 95000 2 5 Suzanne Legere 1 As you can see from the addition of “Suzanne Legere,” the data we add using a view is also added to the underlying table. However, because the view doesn’t include the salary column, its value in her row is NULL. If you attempt to insert a salary value using this view, you would receive the error message column \"salary\" of relation \"employees_tax_dept\" does not exist. The reason is that even though the salary column exists in the underlying employees table, it’s not referenced in the view. Again, this is one way to limit access to sensitive data. Check the links I provided in the note on page 268 to learn more about granting permissions to users if you plan to take on database administrator responsibilities. Updating Rows Using the employees_tax_dept View The same restrictions on accessing data in an underlying table apply when we make updates on data in the employees_tax_dept view. Listing 15-7 shows a standard query to update the spelling of Suzanne’s last name using UPDATE (as a person with more than one uppercase letter in his last name, I can confirm misspelling names isn’t unusual). UPDATE employees_tax_dept SET last_name = 'Le Gere' WHERE emp_id = 5; SELECT * FROM employees_tax_dept; Listing 15-7: Updating a row via the employees_tax_dept view Run the code, and the result from the SELECT query should show the updated last name, which occurs in the underlying employees table: emp_id first_name last_name dept_id ------ ---------- --------- ------- Estadísticos e-Books & Papers 1 Nancy Jones 1 2 Lee Smith 1 5 Suzanne Le Gere 1 Suzanne’s last name is now correctly spelled as “Le Gere,” not “Legere.” However, if we try to update the name of an employee who is not in", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 215 + }, + { + "text": "last_name dept_id ------ ---------- --------- ------- Estadísticos e-Books & Papers 1 Nancy Jones 1 2 Lee Smith 1 5 Suzanne Le Gere 1 Suzanne’s last name is now correctly spelled as “Le Gere,” not “Legere.” However, if we try to update the name of an employee who is not in the Tax Department, the query fails just as it did when we tried to insert Jamil White in Listing 15-6. In addition, trying to use this view to update the salary of an employee—even one in the Tax Department—will fail with the same error I noted in the previous section. If the view doesn’t reference a column in the underlying table, you cannot access that column through the view. Again, the fact that updates on views are restricted in this way offers ways to ensure privacy and security for certain pieces of data. Deleting Rows Using the employees_tax_dept View Now, let’s explore how to delete rows using a view. The restrictions on which data you can affect apply here as well. For example, if Suzanne Le Gere in the Tax Department gets a better offer from another firm and decides to join the other company, you could remove her from the employees table through the employees_tax_dept view. Listing 15-8 shows the query in the standard DELETE syntax: DELETE FROM employees_tax_dept WHERE emp_id = 5; Listing 15-8: Deleting a row via the employees_tax_dept view Run the query, and PostgreSQL should respond with DELETE 1. However, when you try to delete a row for an employee in a department other than the Tax Department, PostgreSQL won’t allow it and will report DELETE 0. In summary, views not only give you control over access to data, but also shortcuts for working with data. Next, let’s explore how to use functions to save more time. Estadísticos e-Books & Papers Programming Your Own Functions You’ve used plenty of functions throughout the book, whether to capitalize letters with upper() or add numbers with sum(). Behind these functions is a significant amount of (sometimes complex) programming that takes an input, transforms it or initiates an action, and returns a response. You saw that extent of code in Listing 5-14 on page 69 when you created a median() function, which uses 30 lines of code to find the middle value in a group of numbers. PostgreSQL’s built-in functions and other functions database programmers develop to automate processes can use even more lines of code, including links to external code written in another language, such as C. We won’t write complicated code here, but we’ll work through some examples of building functions that you can use as a launching pad for your own ideas. Even simple, user-created functions can help you avoid repeating code when you’re analyzing data. The code in this section is specific to PostgreSQL and is not part of the ANSI SQL standard. In some databases, notably Microsoft SQL Server and MySQL, implementing reusable code happens in a stored procedure. If you’re using another database management", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 216 + }, + { + "text": "help you avoid repeating code when you’re analyzing data. The code in this section is specific to PostgreSQL and is not part of the ANSI SQL standard. In some databases, notably Microsoft SQL Server and MySQL, implementing reusable code happens in a stored procedure. If you’re using another database management system, check its documentation for specifics. Creating the percent_change() Function To learn the syntax for creating a function, let’s write a function to simplify calculating the percent change of two values, which is a staple of data analysis. In Chapter 5, you learned that the percent change formula can be expressed this way: percent change = (New Number – Old Number) / Old Number Rather than writing that formula each time we need it, we can create a function called percent_change() that takes the new and old numbers as inputs and returns the result rounded to a user-specified number of decimal places. Let’s walk through the code in Listing 15-9 to see how to Estadísticos e-Books & Papers declare a simple SQL function: ➊ CREATE OR REPLACE FUNCTION ➋ percent_change(new_value numeric, old_value numeric, decimal_places integer ➌DEFAULT 1) ➍ RETURNS numeric AS ➎ 'SELECT round( ((new_value - old_value) / old_value) * 100, decimal_places );' ➏ LANGUAGE SQL ➐ IMMUTABLE ➑ RETURNS NULL ON NULL INPUT; Listing 15-9: Creating a percent_change() function A lot is happening in this code, but it’s not as complicated as it looks. We start with the command CREATE OR REPLACE FUNCTION ➊, followed by the name of the function ➋ and, in parentheses, a list of arguments that are the function’s inputs. Each argument has a name and data type. For example, we specify that new_value and old_value are numeric, whereas decimal_places (which specifies the number of places to round results) is integer. For decimal_places, we specify 1 as the DEFAULT ➌ value to indicate that we want the results to display only one decimal place. Because we set a default value, the argument will be optional when we call the function later. We then use the keywords RETURNS numeric AS ➍ to tell the function to return its calculation as type numeric. If this were a function to concatenate strings, we might return text. Next, we write the meat of the function that performs the calculation. Inside single quotes, we place a SELECT query ➎ that includes the percent change calculation nested inside a round() function. In the formula, we use the function’s argument names instead of numbers. We then supply a series of keywords that define the function’s attributes and behavior. The LANGUAGE ➏ keyword specifies that we’ve written this function using plain SQL, which is one of several languages PostgreSQL supports in functions. Another common option is a Estadísticos e-Books & Papers PostgreSQL-specific procedural language called PL/pgSQL that, in addition to providing the means to create functions, adds features not found in standard SQL, such as logical control structures (IF ... THEN ... ELSE). PL/pgSQL is the default procedural language installed with PostgreSQL, but", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 217 + }, + { + "text": "common option is a Estadísticos e-Books & Papers PostgreSQL-specific procedural language called PL/pgSQL that, in addition to providing the means to create functions, adds features not found in standard SQL, such as logical control structures (IF ... THEN ... ELSE). PL/pgSQL is the default procedural language installed with PostgreSQL, but you can install others, such as PL/Perl and PL/Python, to use the Perl and Python programming languages in your database. Later in this chapter, I’ll show examples of PL/pgSQL and Python. Next, the IMMUTABLE keyword ➐ indicates that the function won’t be making any changes to the database, which can improve performance. The line RETURNS NULL ON NULL INPUT ➑ guarantees that the function will supply a NULL response if any input that is not supplied by default is a NULL. Run the code using pgAdmin to create the percent_change() function. The server should respond with the message CREATE FUNCTION. Using the percent_change() Function To test the new percent_change() function, run it by itself using SELECT, as shown in Listing 15-10: SELECT percent_change(110, 108, 2); Listing 15-10: Testing the percent_change() function This example uses a value of 110 for the new number, 108 for the old number, and 2 as the desired number of decimal places to round the result. Run the code; the result should look like this: percent_change -------------- 1.85 The result indicates that there is a 1.85 percent increase between 108 and 110. You can experiment with other numbers to see how the results change. Also, try changing the decimal_places argument to values including 0, or omit it, to see how that affects the output. You should see results that have more or fewer numbers after the decimal point, based on your Estadísticos e-Books & Papers input. Of course, we created this function to avoid having to write the full percent change formula in queries. Now let’s use it to calculate the percent change using a version of the Decennial Census population change query we wrote in Chapter 6, as shown in Listing 15-11: SELECT c2010.geo_name, c2010.state_us_abbreviation AS st, c2010.p0010001 AS pop_2010, ➊ percent_change(c2010.p0010001, c2000.p0010001) AS pct_chg_func, ➋ round( (CAST(c2010.p0010001 AS numeric(8,1)) - c2000.p0010001) / c2000.p0010001 * 100, 1 ) AS pct_chg_formula FROM us_counties_2010 c2010 INNER JOIN us_counties_2000 c2000 ON c2010.state_fips = c2000.state_fips AND c2010.county_fips = c2000.county_fips ORDER BY pct_chg_func DESC LIMIT 5; Listing 15-11: Testing percent_change() on census data Listing 15-11 uses the original query in Listing 6-13 and adds the percent_change() function ➊ as a column before the formula ➋ so we can compare results. As inputs, we use the 2010 total population column (c2010.p0010001) as the new number and the 2000 total population as the old (c2000.p0010001). When you run the query, the results should display the five counties with the greatest percent change in population, and the results from the function should match the results from the formula entered directly into the query ➋. Each result displays one decimal place, the function’s default value, because we didn’t provide the optional third argument when", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 218 + }, + { + "text": "should display the five counties with the greatest percent change in population, and the results from the function should match the results from the formula entered directly into the query ➋. Each result displays one decimal place, the function’s default value, because we didn’t provide the optional third argument when we called Estadísticos e-Books & Papers the function. Now that we know the function works as intended, we can use percent_change() any time we need to solve that calculation. Using a function is much faster than having to write a formula each time we need to use it! Updating Data with a Function We can also use a function to simplify routine updates to data. In this section, we’ll write a function that assigns the correct number of personal days available to a teacher (in addition to vacation) based on their hire date. We’ll use the teachers table from the first lesson in Chapter 1, “Creating a Table” on page 5. If you skipped that section, you can return to it to create the table and insert the data using the example code in Listing 1-2 on page 6 and Listing 1-3 on page 8. Let’s start by adding a column to teachers to hold the personal days using the code in Listing 15-12: ALTER TABLE teachers ADD COLUMN personal_days integer; SELECT first_name, last_name, hire_date, personal_days FROM teachers; Listing 15-12: Adding a column to the teachers table and seeing the data Listing 15-12 updates the teachers table using ALTER and adds the personal_days column using the keywords ADD COLUMN. Run the SELECT statement to view the data. When both queries finish, you should see the following six rows: first_name last_name hire_date personal_days ---------- --------- ---------- ------------- Janet Smith 2011-10-30 Lee Reynolds 1993-05-22 Samuel Cole 2005-08-01 Samantha Bush 2011-10-30 Betty Diaz 2005-08-30 Kathleen Roush 2010-10-22 The personal_days column holds NULL values because we haven’t provided Estadísticos e-Books & Papers any values yet. Now, let’s create a function called update_personal_days() that updates the personal_days column with the correct personal days based on the teacher’s hire date. We’ll use the following rules to update the data in the personal_days column: Less than five years since hire: 3 personal days Between five and 10 years since hire: 4 personal days More than 10 years since hire: 5 personal days The code in Listing 15-13 is similar to the code we used to create the percent_change() function, but this time we’ll use the PL/pgSQL language instead of plain SQL. Let’s walk through some differences. CREATE OR REPLACE FUNCTION update_personal_days() ➊ RETURNS void AS ➋$$ ➌ BEGIN UPDATE teachers SET personal_days = ➍ CASE WHEN (now() - hire_date) BETWEEN '5 years'::interval AND '10 years'::interval THEN 4 WHEN (now() - hire_date) > '10 years'::interval THEN 5 ELSE 3 END; ➎ RAISE NOTICE 'personal_days updated!'; END; ➏ $$ LANGUAGE plpgsql; Listing 15-13: Creating an update_personal_days() function We begin with CREATE OR REPLACE FUNCTION, followed by the function’s name. This time, we provide no arguments because no user input", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 219 + }, + { + "text": "WHEN (now() - hire_date) > '10 years'::interval THEN 5 ELSE 3 END; ➎ RAISE NOTICE 'personal_days updated!'; END; ➏ $$ LANGUAGE plpgsql; Listing 15-13: Creating an update_personal_days() function We begin with CREATE OR REPLACE FUNCTION, followed by the function’s name. This time, we provide no arguments because no user input is required. The function operates on predetermined columns with set rules for calculating intervals. Also, we use RETURNS void ➊ to note that the function returns no data; it simply updates the personal_days column. Often, when writing PL/pgSQL-based functions, the PostgreSQL convention is to use the non-ANSI SQL standard dollar-quote ($$) ➋ to mark the start and end of the string that contains all the function’s commands. (As with the percent_change() function earlier, you could use Estadísticos e-Books & Papers single quote marks to enclose the string, but then any single quotes in the string would need to be doubled, and that looks messy.) So, everything between the pairs of $$ is the code that does the work. You can also add some text between the dollar signs, like $namestring$, to create a unique pair of beginning and ending quotes. This is useful, for example, if you need to quote a query inside the function. Right after the first $$ we start a BEGIN ... END; ➌ block to denote the function; inside it we place an UPDATE statement that uses a CASE statement ➍ to determine the number of days each teacher gets. We subtract the hire_date from the current date, which is retrieved from the server by the now() function. Depending on which range now() - hire_date falls into, the CASE statement returns the correct number of days off corresponding to the range. We use RAISE NOTICE ➎ to display a message in pgAdmin that the function is done. At the end, we use the LANGUAGE ➏ keyword to specify that we’ve written this function using PL/pgSQL. Run the code in Listing 15-13 to create the update_personal_days() function. Then use the following line to run it in pgAdmin: SELECT update_personal_days(); Now when you rerun the SELECT statement in Listing 15-12, you should see that each row of the personal_days column is filled with the appropriate values. Note that your results may vary depending on when you run this function, because the result of now() is constantly updated with the passage of time. first_name last_name hire_date personal_days ---------- --------- ---------- ------------- Janet Smith 2011-10-30 4 Lee Reynolds 1993-05-22 5 Samuel Cole 2005-08-01 5 Samantha Bush 2011-10-30 4 Betty Diaz 2005-08-30 5 Kathleen Roush 2010-10-22 4 You could use the update_personal_days() function to regularly update data manually after performing certain tasks, or you could use a task scheduler such as pgAgent (a separate open source tool) to run it Estadísticos e-Books & Papers automatically. You can learn about pgAgent and other tools in “PostgreSQL Utilities, Tools, and Extensions” on page 334. Using the Python Language in a Function Previously, I mentioned that PL/pgSQL is the default procedural language within PostgreSQL,", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 220 + }, + { + "text": "pgAgent (a separate open source tool) to run it Estadísticos e-Books & Papers automatically. You can learn about pgAgent and other tools in “PostgreSQL Utilities, Tools, and Extensions” on page 334. Using the Python Language in a Function Previously, I mentioned that PL/pgSQL is the default procedural language within PostgreSQL, but the database also supports creating functions using open source languages, such as Perl and Python. This support allows you to take advantage of those languages’ features as well as related modules within functions you create. For example, with Python, you can use the pandas library for data analysis. The documentation at https://www.postgresql.org/docs/current/static/server- programming.html provides a comprehensive review of the available languages, but here I’ll show you a very simple function using Python. To enable PL/Python, you must add the extension using the code in Listing 15-14. If you get an error, such as could not access file \"$libdir/plpython2\", that means PL/Python wasn’t included when you installed PostgreSQL. Refer back to the troubleshooting links for each operating system in “Installing PostgreSQL” on page xxviii. CREATE EXTENSION plpythonu; Listing 15-14: Enabling the PL/Python procedural language NOTE The extension plpythonu currently installs Python version 2.x. If you want to use Python 3.x, install the extension plpython3u instead. However, available versions might vary based on PostgreSQL distribution. After enabling the extension, create a function following the same syntax you just learned in Listing 15-9 and Listing 15-13, but use Python for the body of the function. Listing 15-15 shows how to use PL/Python to create a function called trim_county() that removes the word “County” from the end of a string. We’ll use this function to clean up names of Estadísticos e-Books & Papers counties in the census data. CREATE OR REPLACE FUNCTION trim_county(input_string text) ➊ RETURNS text AS $$ ➋ import re ➌ cleaned = re.sub(r' County', '', input_string) return cleaned ➍ $$ LANGUAGE plpythonu; Listing 15-15: Using PL/Python to create the trim_county() function The structure should look familiar with some exceptions. Unlike the example in Listing 15-13, we don’t follow the $$ ➊ with a BEGIN ... END; block. That is a PL/pgSQL–specific requirement that we don’t need in PL/Python. Instead, we get straight to the Python code by starting with a statement to import the Python regular expressions module, re ➋. Even if you don’t know much about Python, you can probably deduce that the next two lines of code ➌ set a variable called cleaned to the results of a Python regular expression function called sub(). That function looks for a space followed by the word County in the input_string passed into the function and substitutes an empty string, which is denoted by two apostrophes. Then the function returns the content of the variable cleaned. To end, we specify LANGUAGE plpythonu ➍ to note we’re writing the function with PL/Python. Run the code to create the function, and then execute the SELECT statement in Listing 15-16 to see it in action. SELECT geo_name, trim_county(geo_name) FROM us_counties_2010 ORDER BY state_fips, county_fips", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 221 + }, + { + "text": "content of the variable cleaned. To end, we specify LANGUAGE plpythonu ➍ to note we’re writing the function with PL/Python. Run the code to create the function, and then execute the SELECT statement in Listing 15-16 to see it in action. SELECT geo_name, trim_county(geo_name) FROM us_counties_2010 ORDER BY state_fips, county_fips LIMIT 5; Listing 15-16: Testing the trim_county() function We use the geo_name column in the us_counties_2010 table as input to trim_county(). That should return these results: geo_name trim_county -------------- ----------- Autauga County Autauga Estadísticos e-Books & Papers Baldwin County Baldwin Barbour County Barbour Bibb County Bibb Blount County Blount As you can see, the trim_county() function evaluated each value in the geo_name column and removed a space and the word County when present. Although this is a trivial example, it shows how easy it is to use Python— or one of the other supported procedural languages—inside a function. Next, you’ll learn how to use triggers to automate your database. Automating Database Actions with Triggers A database trigger executes a function whenever a specified event, such as an INSERT, UPDATE, or DELETE, occurs on a table or a view. You can set a trigger to fire before, after, or instead of the event, and you can also set it to fire once for each row affected by the event or just once per operation. For example, let’s say you delete 20 rows from a table. You could set the trigger to fire once for each of the 20 rows deleted or just one time. We’ll work through two examples. The first example keeps a log of changes made to grades at a school. The second automatically classifies temperatures each time we collect a reading. Logging Grade Updates to a Table Let’s say we want to automatically track changes made to a student grades table in our school’s database. Every time a row is updated, we want to record the old and new grade plus the time the change occurred (search for “David Lightman and grades” and you’ll see why this might be worth tracking). To handle this task automatically, we’ll need three items: A grades_history table to record the changes to grades in a grades table A trigger to run a function every time a change occurs in the grades table, which we’ll name grades_update The function the trigger will execute; we’ll call this function Estadísticos e-Books & Papers record_if_grade_changed() Creating Tables to Track Grades and Updates Let’s start by making the tables we need. Listing 15-17 includes the code to first create and fill grades and then create grades_history: ➊ CREATE TABLE grades ( student_id bigint, course_id bigint, course varchar(30) NOT NULL, grade varchar(5) NOT NULL, PRIMARY KEY (student_id, course_id) ); ➋ INSERT INTO grades VALUES (1, 1, 'Biology 2', 'F'), (1, 2, 'English 11B', 'D'), (1, 3, 'World History 11B', 'C'), (1, 4, 'Trig 2', 'B'); ➌ CREATE TABLE grades_history ( student_id bigint NOT NULL, course_id bigint NOT NULL, change_time timestamp with time zone NOT NULL, course varchar(30) NOT", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 222 + }, + { + "text": "); ➋ INSERT INTO grades VALUES (1, 1, 'Biology 2', 'F'), (1, 2, 'English 11B', 'D'), (1, 3, 'World History 11B', 'C'), (1, 4, 'Trig 2', 'B'); ➌ CREATE TABLE grades_history ( student_id bigint NOT NULL, course_id bigint NOT NULL, change_time timestamp with time zone NOT NULL, course varchar(30) NOT NULL, old_grade varchar(5) NOT NULL, new_grade varchar(5) NOT NULL, PRIMARY KEY (student_id, course_id, change_time) ); Listing 15-17: Creating the grades and grades_history tables These commands are straightforward. We use CREATE to make a grades table ➊ and add four rows using INSERT ➋, where each row represents a student’s grade in a class. Then we use CREATE TABLE to make the grades_history table ➌ to hold the data we log each time an existing grade is altered. The grades_history table has columns for the new grade, old grade, and the time of the change. Run the code to create the tables and fill the grades table. We insert no data into grades_history here because the trigger process will handle that task. Creating the Function and Trigger Estadísticos e-Books & Papers Next, let’s write the record_if_grade_changed() function the trigger will execute. We must write the function before naming it in the trigger. Let’s go through the code in Listing 15-18: CREATE OR REPLACE FUNCTION record_if_grade_changed() ➊ RETURNS trigger AS $$ BEGIN ➋ IF NEW.grade <> OLD.grade THEN INSERT INTO grades_history ( student_id, course_id, change_time, course, old_grade, new_grade) VALUES (OLD.student_id, OLD.course_id, now(), OLD.course, ➌ OLD.grade, ➍ NEW.grade); END IF; ➎ RETURN NEW; END; $$ LANGUAGE plpgsql; Listing 15-18: Creating the record_if_grade_changed() function The record_if_grade_changed() function follows the pattern of earlier examples in the chapter but with exceptions specific to working with triggers. First, we specify RETURNS trigger ➊ instead of a data type or void. Because record_if_grade_changed() is a PL/pgSQL function, we place the procedure inside the BEGIN ... END; block. We start the procedure using an IF ... THEN statement ➋, which is one of the control structures PL/pgSQL provides. We use it here to run the INSERT statement only if the updated grade is different from the old grade, which we check using the <> operator. When a change occurs to the grades table, the trigger (which we’ll create next) will execute. For each row that is changed, the trigger will pass two collections of data into record_if_grade_changed(). The first is the row values before they were changed, noted with the prefix OLD. The Estadísticos e-Books & Papers second is the row values after they were changed, noted with the prefix NEW. The function can access the original row values and the updated row values, which it will use for a comparison. If the IF ... THEN statement evaluates as true, which means that the old and new grade values are different, we use INSERT to add a row to grades_history that contains both OLD.grade ➌ and NEW.grade ➍. A trigger must have a RETURN statement ➎, although the PostgreSQL documentation at https://www.postgresql.org/docs/current/static/plpgsql- trigger.html details the scenarios in which a", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 223 + }, + { + "text": "true, which means that the old and new grade values are different, we use INSERT to add a row to grades_history that contains both OLD.grade ➌ and NEW.grade ➍. A trigger must have a RETURN statement ➎, although the PostgreSQL documentation at https://www.postgresql.org/docs/current/static/plpgsql- trigger.html details the scenarios in which a trigger return value actually matters (sometimes it is ignored). The documentation also explains that you can use statements to return a NULL or raise an exception in case of error. Run the code in Listing 15-18 to create the function. Next, add the grades_update trigger to the grades table using Listing 15-19: ➊ CREATE TRIGGER grades_update ➋ AFTER UPDATE ON grades ➌ FOR EACH ROW ➍ EXECUTE PROCEDURE record_if_grade_changed(); Listing 15-19: Creating the grades_update trigger In PostgreSQL, the syntax for creating a trigger follows the ANSI SQL standard (although the contents of the trigger function do not). The code begins with a CREATE TRIGGER ➊ statement, followed by clauses that control when the trigger runs and how it behaves. We use AFTER UPDATE ➋ to specify that we want the trigger to fire after the update occurs on the grades row. We could also use the keywords BEFORE or INSTEAD OF depending on the situation. We write FOR EACH ROW ➌ to tell the trigger to execute the procedure once for each row updated in the table. For example, if someone ran an update that affected three rows, the procedure would run three times. The alternate (and default) is FOR EACH STATEMENT, which runs the procedure once. If we didn’t care about capturing changes to each row and simply Estadísticos e-Books & Papers wanted to record that grades were changed at a certain time, we could use that option. Finally, we use EXECUTE PROCEDURE ➍ to name record_if_grade_changed() as the function the trigger should run. Create the trigger by running the code in Listing 15-19 in pgAdmin. The database should respond with the message CREATE TRIGGER. Testing the Trigger Now that we’ve created the trigger and the function it should run, let’s make sure they work. First, when you run SELECT * FROM grades_history;, you’ll see that the table is empty because we haven’t made any changes to the grades table yet and there’s nothing to track. Next, when you run SELECT * FROM grades; you should see the grade data, as shown here: student_id course_id course grade ---------- --------- ----------------- ----- 1 1 Biology 2 F 1 2 English 11B D 1 3 World History 11B C 1 4 Trig 2 B That Biology 2 grade doesn’t look very good. Let’s update it using the code in Listing 15-20: UPDATE grades SET grade = 'C' WHERE student_id = 1 AND course_id = 1; Listing 15-20: Testing the grades_update trigger When you run the UPDATE, pgAdmin doesn’t display anything to let you know that the trigger executed in the background. It just reports UPDATE 1, meaning the row with grade F was updated. But our trigger did run, which we", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 224 + }, + { + "text": "AND course_id = 1; Listing 15-20: Testing the grades_update trigger When you run the UPDATE, pgAdmin doesn’t display anything to let you know that the trigger executed in the background. It just reports UPDATE 1, meaning the row with grade F was updated. But our trigger did run, which we can confirm by examining columns in grades_history using this SELECT query: SELECT student_id, change_time, course, old_grade, new_grade FROM grades_history; Estadísticos e-Books & Papers When you run this query, you should see that the grades_history table, which contains all changes to grades, now has one row: This row displays the old Biology 2 grade of F, the new value C, and change_time, showing the time of the change made (your result should reflect your date and time). Note that the addition of this row to grades_history happened in the background without the knowledge of the person making the update. But the UPDATE event on the table caused the trigger to fire, which executed the record_if_grade_changed() function. If you’ve used a content management system, such as WordPress or Drupal, this sort of revision tracking might be familiar. It provides a helpful record of changes made to content for reference and auditing purposes, and, unfortunately, can lead to occasional finger-pointing. Regardless, the ability to trigger actions on a database automatically gives you more control over your data. Automatically Classifying Temperatures In Chapter 12, we used the SQL CASE statement to reclassify temperature readings into descriptive categories. The CASE statement (with a slightly different syntax) is also part of the PL/pgSQL procedural language, and we can use its capability to assign values to variables to automatically store those category names in a table each time we add a temperature reading. If we’re routinely collecting temperature readings, using this technique to automate the classification spares us from having to handle the task manually. We’ll follow the same steps we used for logging the grade changes: we first create a function to classify the temperatures, and then create a trigger to run the function each time the table is updated. Use Listing 15- 21 to create a temperature_test table for the exercise: Estadísticos e-Books & Papers CREATE TABLE temperature_test ( station_name varchar(50), observation_date date, max_temp integer, min_temp integer, max_temp_group varchar(40), PRIMARY KEY (station_name, observation_date) ); Listing 15-21: Creating a temperature_test table In Listing 15-21, the temperature_test table contains columns to hold the name of the station and date of the temperature observation. Let’s imagine that we have some process to insert a row once a day that provides the maximum and minimum temperature for that location, and we need to fill the max_temp_group column with a descriptive classification of the day’s high reading to provide text to a weather forecast we’re distributing. To do this, we first make a function called classify_max_temp(), as shown in Listing 15-22: CREATE OR REPLACE FUNCTION classify_max_temp() RETURNS trigger AS $$ BEGIN ➊ CASE WHEN NEW.max_temp >= 90 THEN NEW.max_temp_group := 'Hot';➋ WHEN NEW.max_temp BETWEEN 70 AND 89 THEN NEW.max_temp_group :=", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 225 + }, + { + "text": "to a weather forecast we’re distributing. To do this, we first make a function called classify_max_temp(), as shown in Listing 15-22: CREATE OR REPLACE FUNCTION classify_max_temp() RETURNS trigger AS $$ BEGIN ➊ CASE WHEN NEW.max_temp >= 90 THEN NEW.max_temp_group := 'Hot';➋ WHEN NEW.max_temp BETWEEN 70 AND 89 THEN NEW.max_temp_group := 'Warm'; WHEN NEW.max_temp BETWEEN 50 AND 69 THEN NEW.max_temp_group := 'Pleasant'; WHEN NEW.max_temp BETWEEN 33 AND 49 THEN NEW.max_temp_group := 'Cold'; WHEN NEW.max_temp BETWEEN 20 AND 32 THEN NEW.max_temp_group := 'Freezing'; ELSE NEW.max_temp_group := 'Inhumane'; END CASE; RETURN NEW; END; $$ LANGUAGE plpgsql; Listing 15-22: Creating the classify_max_temp() function By now, these functions should look familiar. What is new here is the PL/pgSQL version of the CASE syntax ➊, which differs slightly from the Estadísticos e-Books & Papers SQL syntax in that the PL/pgSQL syntax includes a semicolon after each WHEN ... THEN clause ➋. Also new is the assignment operator (:=), which we use to assign the descriptive name to the NEW.max_temp_group column based on the outcome of the CASE function. For example, the statement NEW.max_temp_group := 'Cold' assigns the string 'Cold' to NEW.max_temp_group when the temperature value is between 33 and 49 degrees Fahrenheit, and when the function returns the NEW row to be inserted in the table, it will include the string value Cold. Run the code to create the function. Next, using the code in Listing 15-23, create a trigger to execute the function each time a row is added to temperature_test: CREATE TRIGGER temperature_insert ➊ BEFORE INSERT ON temperature_test ➋ FOR EACH ROW ➌ EXECUTE PROCEDURE classify_max_temp(); Listing 15-23: Creating the temperature_insert trigger In this example, we classify max_temp and create a value for max_temp_group prior to inserting the row into the table. Doing so is more efficient than performing a separate update after the row is inserted. To specify that behavior, we set the temperature_insert trigger to fire BEFORE INSERT ➊. We also want the trigger to fire FOR EACH ROW inserted ➋ because we want each max_temp recorded in the table to get a descriptive classification. The final EXECUTE PROCEDURE statement names the classify_max_temp() function ➌ we just created. Run the CREATE TRIGGER statement in pgAdmin, and then test the setup using Listing 15-24: INSERT INTO temperature_test (station_name, observation_date, max_temp, min_temp) VALUES ('North Station', '1/19/2019', 10, -3), ('North Station', '3/20/2019', 28, 19), ('North Station', '5/2/2019', 65, 42), ('North Station', '8/9/2019', 93, 74); SELECT * FROM temperature_test; Listing 15-24: Inserting rows to test the temperature_insert trigger Estadísticos e-Books & Papers Here we insert four rows into temperature_test, and we expect the temperature_insert trigger to fire for each row—and it does! The SELECT statement in the listing should display these results: Due to the trigger and function we created, each max_temp inserted automatically receives the appropriate classification in the max_temp_group column. This temperature example and the earlier grade-change auditing example are rudimentary, but they give you a glimpse of how useful triggers and functions can be in simplifying data maintenance. Wrapping Up Although the techniques", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 226 + }, + { + "text": "and function we created, each max_temp inserted automatically receives the appropriate classification in the max_temp_group column. This temperature example and the earlier grade-change auditing example are rudimentary, but they give you a glimpse of how useful triggers and functions can be in simplifying data maintenance. Wrapping Up Although the techniques you learned in this chapter begin to merge with those of a database administrator, you can apply the concepts to reduce the amount of time you spend repeating certain tasks. I hope these approaches will help you free up more time to find interesting stories in your data. This chapter concludes our discussion of analysis techniques and the SQL language. The next two chapters offer workflow tips to help you increase your command of PostgreSQL. They include how to connect to a database and run queries from your computer’s command line, and how to maintain your database. TRY IT YOURSELF Review the concepts in the chapter with these exercises: Estadísticos e-Books & Papers 1. Create a view that displays the number of New York City taxi trips per hour. Use the taxi data in Chapter 11 and the query in Listing 11-8 on page 182. 2. In Chapter 10, you learned how to calculate rates per thousand. Turn that formula into a rates_per_thousand() function that takes three arguments to calculate the result: observed_number, base_number, and decimal_places. 3. In Chapter 9, you worked with the meat_poultry_egg_inspect table that listed food processing facilities. Write a trigger that automatically adds an inspection date each time you insert a new facility into the table. Use the inspection_date column added in Listing 9-19 on page 146, and set the date to be six months from the current date. You should be able to describe the steps needed to implement a trigger and how the steps relate to each other. Estadísticos e-Books & Papers 16 USING POSTGRESQL FROM THE COMMAND LINE Before computers featured a graphical user interface (GUI), which lets you use menus, icons, and buttons to navigate applications, the main way to issue instructions to them was by entering commands on the command line. The command line—also called a command line interface, console, shell, or terminal—is a text-based interface where you enter names of programs or other commands to perform tasks, such as editing files or listing the contents of a file directory. When I was in college, to edit a file, I had to enter commands into a terminal connected to an IBM mainframe computer. The reams of text that then scrolled onscreen were reminiscent of the green characters that define the virtual world portrayed in The Matrix. It felt mysterious and as though I had attained new powers. Even today, movies portray fictional hackers by showing them entering cryptic, text-only commands on a computer. In this chapter, I’ll show you how to access this text-only world. Here are some advantages of working from the command line instead of a GUI, such as pgAdmin: You can often work faster by entering short commands", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 227 + }, + { + "text": "fictional hackers by showing them entering cryptic, text-only commands on a computer. In this chapter, I’ll show you how to access this text-only world. Here are some advantages of working from the command line instead of a GUI, such as pgAdmin: You can often work faster by entering short commands instead of Estadísticos e-Books & Papers clicking through layers of menu items. You gain access to some functions that only the command line provides. If command line access is all you have to work with (for example, when you’ve connected to a remote computer), you can still get work done. We’ll use psql, a command line tool in PostgreSQL that lets you run queries, manage database objects, and interact with the computer’s operating system via text command. You’ll first learn how to set up and access your computer’s command line, and then launch psql. It takes time to learn how to use the command line, and even experienced experts often resort to documentation to recall the available command line options. But learning to use the command line greatly enhances your work efficiency. Setting Up the Command Line for psql To start, we’ll access the command line on your operating system and set an environment variable called PATH that tells your system where to find psql. Environment variables hold parameters that specify system or application configurations, such as where to store temporary files, or allow you to enable or disable options. Setting PATH, which stores the names of one or more directories containing executable programs, tells the command line interface the location of psql, avoiding the hassle of having to enter its full directory path each time you launch it. Windows psql Setup On Windows, you’ll run psql within Command Prompt, the application that provides that system’s command line interface. Let’s start by using PATH to tell Command Prompt where to find psql.exe, which is the full name of the psql application on Windows, as well as other PostgreSQL command line utilities. Estadísticos e-Books & Papers Adding psql and Utilities to the Windows PATH The following steps assume that you installed PostgreSQL according to the instructions described in “Windows Installation” on page xxix. (If you installed PostgreSQL another way, use the Windows File Explorer to search your C: drive to find the directory that holds psql.exe, and then replace C:\\Program Files\\PostgreSQL\\x.y\\bin in steps 5 and 6 with your own path.) 1. Open the Windows Control Panel. Enter Control Panel in the search box on the Windows taskbar, and then click the Control Panel icon. 2. Inside the Control Panel app, enter Environment in the search box at the top right. In the list of search results displayed, click Edit the System Environment Variables. A System Properties dialog should appear. 3. In the System Properties dialog, on the Advanced tab, click Environment Variables. The dialog that opens should have two sections: User variables and System variables. In the User variables section, if you don’t see a PATH variable, continue to step a", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 228 + }, + { + "text": "Environment Variables. A System Properties dialog should appear. 3. In the System Properties dialog, on the Advanced tab, click Environment Variables. The dialog that opens should have two sections: User variables and System variables. In the User variables section, if you don’t see a PATH variable, continue to step a to create a new one. If you do see an existing PATH variable, continue to step b to modify it. 1. If you don’t see PATH in the User variables section, click New to open a New User Variable dialog, shown in Figure 16-1. Figure 16-1: Creating a new PATH environment variable in Windows 10 In the Variable name box, enter PATH. In the Variable value box, enter C:\\Program Files\\PostgreSQL\\x.y\\bin, where x.y is the version of PostgreSQL you’re using. Click OK to close all the Estadísticos e-Books & Papers dialogs. 2. If you do see an existing PATH variable in the User variables section, highlight it and click Edit. In the list of variables that displays, click New and enter C:\\Program Files\\PostgreSQL\\x.y\\bin, where x.y is the version of PostgreSQL you’re using. It should look like the highlighted line in Figure 16-2. When you’re finished, click OK to close all the dialogs. Figure 16-2: Editing existing PATH environment variables in Windows 10 Now when you launch Command Prompt, the PATH should include the directory. Note that any time you make changes to the PATH, you must close and reopen Command Prompt for the changes to take effect. Next, let’s set up Command Prompt. Estadísticos e-Books & Papers Launching and Configuring the Windows Command Prompt Command Prompt is an executable file named cmd.exe. To launch it, select Start ▸ Windows System ▸ Command Prompt. When the application opens, you should see a window with a black background that displays version and copyright information along with a prompt showing your current directory. On my Windows 10 system, Command Prompt opens to my default user directory and displays C:\\Users\\Anthony>, as shown in Figure 16-3. Figure 16-3: My Command Prompt in Windows 10 NOTE For fast access to Command Prompt, you can add it to your Windows taskbar. When Command Prompt is running, right-click its icon on the taskbar and then select Pin to taskbar. The line C:\\Users\\Anthony> indicates that Command Prompt’s current working directory is my C: drive, which is typically the main hard drive on a Windows system, and the \\Users\\Anthony directory on that drive. The right arrow (>) indicates the area where you type your commands. You can customize the font and colors plus access other settings by clicking the Command Prompt icon at the left of its window bar and selecting Properties from the menu. To make Command Prompt more suited for query output, I recommend setting the window size (on the Layout tab) to a width of 80 and a height of 25. My preferred font is Lucida Console 14, but experiment to find one you like. Estadísticos e-Books & Papers Entering Instructions on Windows Command Prompt Now you’re ready", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 229 + }, + { + "text": "for query output, I recommend setting the window size (on the Layout tab) to a width of 80 and a height of 25. My preferred font is Lucida Console 14, but experiment to find one you like. Estadísticos e-Books & Papers Entering Instructions on Windows Command Prompt Now you’re ready to enter instructions in Command Prompt. Enter help at the prompt, and press ENTER on your keyboard to see a list of available commands. You can view information about a particular command by typing its name after help. For example, enter help time to display information on using the time command to set or view the system time. Exploring the full workings of Command Prompt is beyond the scope of this book; however, you should try some of the commands in Table 16-1, which contains frequently used commands you’ll find immediately useful but are not necessary for the exercises in this chapter. Also, check out Command Prompt cheat sheets online for more information. Table 16-1: Useful Windows Commands CommandFunction Example Action cd Change directory cd C:\\my-stuff Change to the my- stuff directory on the C: drive copy Copy a file copy C:\\my-stuff\\song.mp3 C:\\Music\\song_favorite.mp3Copy the song.mp3 file from my-stuff to a new file called song_favorite.mp3 in the Music directory del Delete del *.jpg Delete all files with a .jpg extension in the current directory (asterisk wildcard) dir List directory contents dir /p Show directory contents one screen at a time (using the /p option) findstr Find strings in findstr \"peach\" *.txt Search for the text Estadísticos e-Books & Papers text files matching a regular expression “peach” in all .txt files in the current directory mkdir Make a new directory makedir C:\\my-stuff\\Salad Create a Salad directory inside the my-stuff directory move Move a file move C:\\my-stuff\\song.mp3 C:\\Music\\ Move the file song.mp3 to the C:\\Music directory With your Command Prompt open and configured, you’re ready to roll. Skip ahead to “Working with psql” on page 299. macOS psql Setup On macOS, you’ll run psql within Terminal, the application that provides access to that system’s command line via a shell program called bash. Shell programs on Unix- or Linux-based systems, including macOS, provide not only the command prompt where users enter instructions, but also their own programming language for automating tasks. For example, you can use bash commands to write a program to log in to a remote computer, transfer files, and log out. Let’s start by telling bash where to find psql and other PostgreSQL command line utilities by setting the PATH environment variable. Then we’ll launch Terminal. Adding psql and Utilities to the macOS PATH Before Terminal loads the bash shell, it checks for the presence of several optional text files that can supply configuration information. We’ll place our PATH information inside .bash_profile, which is one of these optional text files. Then, whenever we open Terminal, the startup process should read .bash_profile and obtain the PATH value. Estadísticos e-Books & Papers NOTE You can also use .bash_profile to set your command line’s", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 230 + }, + { + "text": "can supply configuration information. We’ll place our PATH information inside .bash_profile, which is one of these optional text files. Then, whenever we open Terminal, the startup process should read .bash_profile and obtain the PATH value. Estadísticos e-Books & Papers NOTE You can also use .bash_profile to set your command line’s colors, automatically run programs, and create shortcuts, among other tasks. See https://natelandau.com/my-mac-osx-bash_profile/ for a great example of customizing the file. On Unix- or Linux-based systems, files that begin with a period are called dot files and are hidden by default. We’ll need to edit .bash_profile to add PATH. Using the following steps, unhide .bash_profile so it appears in the macOS Finder: 1. Launch Terminal by navigating to Applications ▸ Utilities ▸ Terminal. 2. At the command prompt, which displays your username and computer name followed by a dollar sign ($), enter the following text and then press RETURN: defaults write com.apple.finder AppleShowAllFiles YES 3. Quit Terminal (⌘-Q). Then, while holding down the OPTION key, right-click the Finder icon on your Mac dock, and select Relaunch. Follow these steps to edit or create .bash_profile: 1. Using the macOS Finder, navigate to your user directory by opening the Finder and clicking Macintosh HD then Users. 2. Open your user directory (it should have a house icon). Because you changed the setting to show hidden files, you should now see grayed- out files and directories, which are normally hidden, along with regular files and directories. 3. Check for an existing .bash_profile file. If one exists, right-click and open it with your preferred text editor or use the macOS TextEdit app. If .bash_profile doesn’t exist, open TextEdit to create and save a Estadísticos e-Books & Papers file with that name to your user directory. Next, we’ll add a PATH statement to .bash_profile. These instructions assume you installed PostgreSQL using Postgres.app, as outlined in “macOS Installation” on page xxx. To add to the path, place the following line in .bash_profile: export PATH=\"/Applications/Postgres.app/Contents/Versions/latest/bin:$PATH\" Save and close the file. If Terminal is open, close and relaunch it before moving on to the next section. Launching and Configuring the macOS Terminal Launch Terminal by navigating to Applications ▸ Utilities ▸ Terminal. When it opens, you should see a window that displays the date and time of your last login followed by a prompt that includes your computer name, current working directory, and username, ending with a dollar sign ($). On my Mac, the prompt displays ad:~ anthony$, as shown in Figure 16- 4. Estadísticos e-Books & Papers Figure 16-4: Terminal command line in macOS The tilde (~) indicates that Terminal is currently working in my home directory, which is /Users/anthony. Terminal doesn’t display the full directory path, but you can see that information at any time by entering the pwd command (short for “print working directory”) and pressing RETURN on your keyboard. The area after the dollar sign is where you type commands. NOTE For fast access to Terminal, add it to your macOS Dock. While Terminal is", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 231 + }, + { + "text": "you can see that information at any time by entering the pwd command (short for “print working directory”) and pressing RETURN on your keyboard. The area after the dollar sign is where you type commands. NOTE For fast access to Terminal, add it to your macOS Dock. While Terminal is running, right-click its icon and select Options ▸ Keep in Dock. If you’ve never used Terminal, its default black and white color scheme might seem boring. You can change fonts, colors, and other settings by selecting Terminal ▸ Preferences. To make Terminal bigger Estadísticos e-Books & Papers to better fit the query output display, I recommend setting the window size (on the Window tab) to a width of 80 columns and a height of 25 rows. My preferred font (on the Text tab) is Monaco 14, but experiment to find one you like. Exploring the full workings of Terminal and related commands is beyond the scope of this book, but take some time to try several commands. Table 16-2 lists commonly used commands you’ll find immediately useful but not necessary for the exercises in this chapter. Enter man (short for “manual”) followed by a command name to get help on any command. For example, you can use man ls to find out how to use the ls command to list directory contents. Table 16-2: Useful Terminal Commands CommandFunction Example Action cd Change directory cd /Users/pparker/my- stuff/ Change to the my-stuff directory cp Copy files cp song.mp3 song_backup.mp3 Copy the file song.mp3 to song_backup.mp3 in the current directory grep Find strings in a text file matching a regular expression grep 'us_counties_2010' *.sql Find all lines in files with a .sql extension that have the text “us_counties_2010” ls List directory contents ls -al List all files and directories (including hidden) in “long” format mkdir Make a new directory mkdir resumes Make a directory named resumes under the current working directory mv mv song.mp3 Estadísticos e-Books & Papers Move a file /Users/pparker/songsMove the file song.mp3 from the current directory to a /songs directory under a user directory rm Remove (delete) files rm *.jpg Delete all files with a .jpg extension in the current directory (asterisk wildcard) With your Terminal open and configured, you’re ready to roll. Skip ahead to “Working with psql” on page 299. Linux psql Setup Recall from “Linux Installation” on page xxxi that methods for installing PostgreSQL vary according to your Linux distribution. Nevertheless, psql is part of the standard PostgreSQL install, and you probably already ran psql commands as part of the installation process via your distribution’s command line terminal application. Even if you didn’t, standard Linux installations of PostgreSQL will automatically add psql to your PATH, so you should be able to access it. Launch a terminal application. On some distributions, such as Ubuntu, you can open a terminal by pressing CTRL-ALT-T. Also note that the macOS Terminal commands in Table 16-2 apply to Linux as well and may be useful to you. With your terminal open, you’re", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 232 + }, + { + "text": "should be able to access it. Launch a terminal application. On some distributions, such as Ubuntu, you can open a terminal by pressing CTRL-ALT-T. Also note that the macOS Terminal commands in Table 16-2 apply to Linux as well and may be useful to you. With your terminal open, you’re ready to roll. Proceed to the next section, “Working with psql.” Working with psql Now that you’ve identified your command line interface and set it up to recognize the location of psql, let’s launch psql and connect to a database Estadísticos e-Books & Papers on your local installation of PostgreSQL. Then we’ll explore executing queries and special commands for retrieving database information. Launching psql and Connecting to a Database Regardless of the operating system you’re using, you start psql in the same way. Open your command line interface (Command Prompt on Windows, Terminal on macOS or Linux). To launch psql, we use the following pattern at the command prompt: psql -d database_name -U user_name Following the psql application name, we provide the database name after a -d argument and a username after -U. For the database name, we’ll use analysis, which is where we created the majority of our tables for the book’s exercises. For username, we’ll use postgres, which is the default user created during installation. For example, to connect your local machine to the analysis database, you would enter this: psql -d analysis -U postgres You can connect to a database on a remote server by specifying the -h argument followed by the host name. For example, you would use the following line if you were connecting to a computer on a server called example.com: psql -d analysis -U postgres -h example.com If you set a password during installation, you should receive a password prompt when psql launches. If so, enter your password and press ENTER. You should then see a prompt that looks like this: psql (10.1) Type \"help\" for help. analysis=# Estadísticos e-Books & Papers Here, the first line lists the version number of psql and the server you’re connected to. Your version will vary depending on when you installed PostgreSQL. The prompt where you’ll enter commands is analysis=#, which refers to the name of the database, followed by an equal sign (=) and a hash mark (#). The hash mark indicates that you’re logged in with superuser privileges, which give you unlimited ability to access and create objects and set up accounts and security. If you’re logged in as a user without superuser privileges, the last character of the prompt will be a greater-than sign (>). As you can see, the user account you logged in with here (postgres) is a superuser. NOTE PostgreSQL installations create a default superuser account called postgres. If you’re running postgres.app on macOS, that installation created an additional superuser account that has your system username and no password. Getting Help At the psql prompt, you can easily get help with psql commands and SQL commands. Table 16-3 lists commands you", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 233 + }, + { + "text": "create a default superuser account called postgres. If you’re running postgres.app on macOS, that installation created an additional superuser account that has your system username and no password. Getting Help At the psql prompt, you can easily get help with psql commands and SQL commands. Table 16-3 lists commands you can type at the psql prompt and shows the information they’ll display. Table 16-3: Help Commands Within psql CommandDisplays \\? Commands available within psql, such as \\dt to list tables. \\? options Options for use with the psql command, such as -U to specify a username. \\? variables Variables for use with psql, such as VERSION for the current psql version. \\h List of SQL commands. Add a command name to see detailed Estadísticos e-Books & Papers help for it (for example, \\h INSERT). Even experienced users often need a refresher on commands and options, and having the details in the psql application is handy. Let’s move on and explore some commands. Changing the User and Database Connection You can use a series of meta-commands, which are preceded by a backslash, to issue instructions to psql rather than the database. For example, to connect to a different database or switch the user account you’re connected to, you can use the \\c meta-command. To switch to the gis_analysis database we created in Chapter 14, enter \\c followed by the name of the database at the psql prompt: analysis=# \\c gis_analysis The application should respond with the following message: You are now connected to database \"gis_analysis\" as user \"postgres\". gis_analysis=# To log in as a different user, for example, using a username the macOS installation created for me, I could add that username after the database name. On my Mac, the syntax looks like this: analysis-# \\c gis_analysis anthony The response should be as follows: You are now connected to database \"gis_analysis\" as user \"anthony\". gis_analysis=# You might have various reasons to use multiple user accounts like this. For example, you might want to create a user account with limited permissions for colleagues or for a database application. You can learn more about creating and managing user roles by reading the PostgreSQL documentation at https://www.postgresql.org/docs/current/static/sql- Estadísticos e-Books & Papers createrole.html. Let’s switch back to the analysis database using the \\c command. Next, we’ll enter SQL commands at the psql prompt. Running SQL Queries on psql We’ve configured psql and connected to a database, so now let’s run some SQL queries, starting with a single-line query and then a multiline query. To enter SQL into psql, you can type it directly at the prompt. For example, to see a few rows from the 2010 Census table we’ve used throughout the book, enter a query at the prompt, as shown in Listing 16-1: analysis=# SELECT geo_name FROM us_counties_2010 LIMIT 3; Listing 16-1: Entering a single-line query in psql Press ENTER to execute the query, and psql should display the following results in text including the number of rows returned: geo_name ---------------- Autauga County Baldwin", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 234 + }, + { + "text": "query at the prompt, as shown in Listing 16-1: analysis=# SELECT geo_name FROM us_counties_2010 LIMIT 3; Listing 16-1: Entering a single-line query in psql Press ENTER to execute the query, and psql should display the following results in text including the number of rows returned: geo_name ---------------- Autauga County Baldwin County Barbour County (3 rows) analysis=# Below the result, you can see the analysis=# prompt again, ready for further input from the user. Press the up and down arrows on your keyboard to you scroll through recent queries to avoid having to retype them. Or you can simply enter a new query. Entering a Multiline Query You’re not limited to single-line queries. For example, you can press ENTER each time you want to enter a new line. Note that psql won’t execute the query until you provide a line that ends with a semicolon. To Estadísticos e-Books & Papers see an example, reenter the query in Listing 16-1 using the format shown in Listing 16-2: analysis=# SELECT geo_name analysis-# FROM us_counties_2010 analysis-# LIMIT 3; Listing 16-2: Entering a multiline query in psql Note that when your query extends past one line, the symbol between the database name and the hash mark changes from an equal sign (=) to a hyphen (-). This multiline query executes only when you press ENTER after the final line, which ends with a semicolon. Checking for Open Parentheses in the psql Prompt Another helpful feature of psql is that it shows when you haven’t closed a pair of parentheses. Listing 16-3 shows this in action: analysis=# CREATE TABLE wineries ( analysis(# id bigint, analysis(# winery_name varchar(100) analysis(# ); CREATE TABLE Listing 16-3: Showing open parentheses in the psql prompt Here, you create a simple table called wineries that has two columns. After entering the first line of the CREATE TABLE statement and an open parenthesis, the prompt then changes from analysis=# to analysis(# to include an open parenthesis that reminds you an open parenthesis needs closing. The prompt maintains that configuration until you add the closing parenthesis. NOTE If you have a lengthy query saved in a text file, such as one from this book’s resources, you can copy it to your computer clipboard and paste it into psql (CTRL-V on Windows, ⌘-V on macOS, and SHIFT-CTRL-V on Linux). That saves you from typing the whole query. After you paste the query text into Estadísticos e-Books & Papers psql, press ENTER to execute it. Editing Queries If you’re working with a query in psql and want to modify it, you can edit it using the \\e or \\edit meta-command. Enter \\e to open the last-executed query in a text editor. Which editor psql uses by default depends on your operating system. On Windows, psql defaults to Notepad, a simple GUI text editor. On macOS and Linux, psql uses a command line application called vim, which is a favorite among programmers but can seem inscrutable for beginners. Check out a helpful vim cheat sheet at https://vim.rtorr.com/.", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 235 + }, + { + "text": "default depends on your operating system. On Windows, psql defaults to Notepad, a simple GUI text editor. On macOS and Linux, psql uses a command line application called vim, which is a favorite among programmers but can seem inscrutable for beginners. Check out a helpful vim cheat sheet at https://vim.rtorr.com/. For now, you can use the following steps to make simple edits: When vim opens the query in an editing window, press I to activate insert mode. Make your edits to the query. Press ESC and then SHIFT+: to display a colon command prompt at the bottom left of the vim screen, which is where you enter commands to control vim. Enter wq (for “write, quit”) and press ENTER to save your changes. Now when you exit to the psql prompt, it should execute your revised query. Press the up arrow key to see the revised text. Navigating and Formatting Results The query you ran in Listings 16-1 and 16-2 returned only one column and a handful of rows, so its output was contained nicely in your command line interface. But for queries with more columns or rows, the output can take up more than one screen, making it difficult to navigate. Fortunately, you can use formatting options using the \\pset meta- command to tailor the output into a format you prefer. Estadísticos e-Books & Papers Setting Paging of Results You can adjust the output format by specifying how psql displays lengthy query results. For example, Listing 16-4 shows the change in output format when we remove the LIMIT clause from the query in Listing 16-1 and execute it at the psql prompt: analysis=# SELECT geo_name FROM us_counties_2010; geo_name ----------------------------------- Autauga County Baldwin County Barbour County Bibb County Blount County Bullock County Butler County Calhoun County Chambers County Cherokee County Chilton County Choctaw County Clarke County Clay County Cleburne County Coffee County Colbert County : Listing 16-4: A query with scrolling results Recall that this table has 3,143 rows. Listing 16-4 shows only the first 17 on the screen with a colon at the bottom (the number of visible rows depends on your terminal configuration). The colon indicates that there are more results than shown; press the down arrow key to scroll through them. Scrolling through this many rows can take a while. Press Q at any time to exit the scrolling results and return to the psql prompt. You can have your results immediately scroll to the end by changing the pager setting using the \\pset pager meta-command. Run that command at your psql prompt, and it should return the message Pager usage is off. Now when you rerun the query in Listing 16-3 with the pager setting turned off, you should see something like this: --snip-- Niobrara County Estadísticos e-Books & Papers Park County Platte County Sheridan County Sublette County Sweetwater County Teton County Uinta County Washakie County Weston County (3143 rows) analysis=# You’re immediately taken to the end of the results without having to scroll. To turn", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 236 + }, + { + "text": "off, you should see something like this: --snip-- Niobrara County Estadísticos e-Books & Papers Park County Platte County Sheridan County Sublette County Sweetwater County Teton County Uinta County Washakie County Weston County (3143 rows) analysis=# You’re immediately taken to the end of the results without having to scroll. To turn paging back on, run \\pset pager again. Formatting the Results Grid You can also use the \\pset meta-command with the following options to format how the results look: border int Use this option to specify whether the results grid has no border (0), internal lines dividing columns (1), or lines around all cells (2). For example, \\pset border 2 sets lines around all cells. format unaligned Use the option \\pset format unaligned to display the results in lines separated by a delimiter rather than in columns, similar to what you would see in a CSV file. The separator defaults to a pipe symbol (|). You can set a different separator using the fieldsep command. For example, to set a comma as the separator, run \\pset fieldsep ','. To revert to a column view, run \\pset format aligned. You can use the psql meta-command \\a to toggle between aligned and unaligned views. footer Use this option to toggle the results footer, which displays the result row count, on or off. null Use this option to set how null values are displayed. By default, they show as blanks. You can run \\pset null 'NULL' to replace blanks with all-caps NULL when the column value is NULL. You can explore additional options in the PostgreSQL documentation Estadísticos e-Books & Papers at https://www.postgresql.org/docs/current/static/app-psql.html. In addition, it’s possible to set up a .psqlrc file on macOS or Linux or a psqlrc.conf file on Windows to hold your configuration preferences and load them each time psql starts. A good example is provided at https://www.citusdata.com/blog/2017/07/16/customizing-my-postgres-shell- using-psqlrc/. Viewing Expanded Results Sometimes, it’s helpful to view results as a vertical block listing rather than in rows and columns, particularly when data is too big to fit onscreen in the normal horizontal results grid. Also, I often employ this format when I want an easy-to-scan way to review the values in columns on a row-by-row basis. In psql, you can switch to this view using the \\x (for expanded) meta-command. The best way to understand the difference between normal and expanded view is by looking at an example. Listing 16-5 shows the normal display you see when querying the grades table in Chapter 15 using psql: analysis=# SELECT * FROM grades; student_id | course_id | course | grade ------------+-----------+-------------------+------- 1 | 2 | English 11B | D 1 | 3 | World History 11B | C 1 | 4 | Trig 2 | B 1 | 1 | Biology 2 | C (4 rows) Listing 16-5: Normal display of the grades table query To change to the expanded view, enter \\x at the psql prompt, which should display the Expanded display is on message. Then, when you run the same", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 237 + }, + { + "text": "Trig 2 | B 1 | 1 | Biology 2 | C (4 rows) Listing 16-5: Normal display of the grades table query To change to the expanded view, enter \\x at the psql prompt, which should display the Expanded display is on message. Then, when you run the same query again, you should see the expanded results, as shown in Listing 16-6: analysis=# SELECT * FROM grades; -[ RECORD 1 ]----------------- student_id | 1 course_id | 2 course | English 11B grade | D -[ RECORD 2 ]----------------- Estadísticos e-Books & Papers student_id | 1 course_id | 3 course | World History 11B grade | C -[ RECORD 3 ]----------------- student_id | 1 course_id | 4 course | Trig 2 grade | B -[ RECORD 4 ]----------------- student_id | 1 course_id | 1 course | Biology 2 grade | C Listing 16-6: Expanded display of the grades table query The results appear in vertical blocks separated by record numbers. Depending on your needs and the type of data you’re working with, this format might be easier to read. You can revert to column display by entering \\x again at the psql prompt. In addition, setting \\x auto will make PostgreSQL automatically display the results in a table or expanded view based on the size of the output. Next, let’s explore how to use psql to dig into database information. Meta-Commands for Database Information In addition to writing queries from the command line, you can also use psql to display details about tables and other objects and functions in your database. To do this, you use a series of meta-commands that start with \\d and append a plus sign (+) to expand the output. You can also supply an optional pattern to filter the output. For example, you can enter \\dt+ to list all tables in the database and their size. Here’s a snippet of the output on my system: Estadísticos e-Books & Papers This result lists all tables in the current database alphabetically. You can filter the output by adding a pattern to match using a regular expression. For example, use \\dt+ us* to show only tables whose names begin with us (the asterisk acts as a wildcard). The results should look like this: Table 16-4 shows several additional \\d commands you might find helpful. Table 16-4: Examples of psql \\d Commands Command Displays \\d [pattern] Columns, data types, plus other information on objects \\di [pattern]Indexes and their associated tables \\dt [pattern]Tables and the account that owns them \\du [pattern]User accounts and their attributes \\dv [pattern]Views and the account that owns them \\dx [pattern]Installed extensions Estadísticos e-Books & Papers The entire list of \\d commands is available in the PostgreSQL documentation at https://www.postgresql.org/docs/current/static/app- psql.html, or you can see details by using the \\? command noted earlier. Importing, Exporting, and Using Files Now let’s explore how to get data in and out of tables or save information when you’re working on a remote server. The psql command line tool offers", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 238 + }, + { + "text": "PostgreSQL documentation at https://www.postgresql.org/docs/current/static/app- psql.html, or you can see details by using the \\? command noted earlier. Importing, Exporting, and Using Files Now let’s explore how to get data in and out of tables or save information when you’re working on a remote server. The psql command line tool offers one meta-command for importing and exporting data (\\copy) and another for copying query output to a file (\\o). We’ll start with the \\copy command. Using \\copy for Import and Export In Chapter 4, you learned how to use the SQL COPY command to import and export data. It’s a straightforward process, but there is one significant limitation: the file you’re importing or exporting must be on the same machine as the PostgreSQL server. That’s fine if you’re working on your local machine, as you’ve been doing with these exercises. But if you’re connecting to a database on a remote computer, you might not have access to the file system to provide a file to import or to fetch a file you’ve exported. You can get around this restriction by using the \\copy meta- command in psql. The \\copy meta-command works just like the SQL COPY command except when you execute it at the psql prompt, it can route data from your local machine to a remote server if that’s what you’re connected to. We won’t actually connect to a remote server to try this, but you can still learn the syntax. In Listing 16-7, we use psql to DROP the small state_regions table you created in Chapter 9, and then re-create the table and import data using \\copy. You’ll need to change the file path to match the location of the file on your computer. analysis=# DROP TABLE state_regions; DROP TABLE Estadísticos e-Books & Papers analysis=# CREATE TABLE state_regions ( analysis(# st varchar(2) CONSTRAINT st_key PRIMARY KEY, analysis(# region varchar(20) NOT NULL analysis(# ); CREATE TABLE analysis=# \\copy state_regions FROM 'C:\\YourDirectory\\state_regions.csv' WITH (FORMAT CSV, HEADER); COPY 56 Listing 16-7: Importing data using \\copy The DROP TABLE and CREATE TABLE statements in Listing 16-7 are straightforward. We first delete the state_regions table if it exists, and then re-create it. Then, to load the table, we use \\copy with the same syntax used with SQL COPY, naming a FROM clause that includes the file path on your machine, and a WITH clause that specifies the file is a CSV and has a header row. When you execute the statement, the server should respond with COPY 56, letting you know the rows have been successfully imported. If you were connected via psql to a remote server, you would use the same \\copy syntax, and the command would just route your local file to the remote server for importing. In this example, we used \\copy FROM to import a file. We could also use \\copy TO for exporting. Let’s look at another way to export output to a file. Saving Query Output to a File It’s sometimes helpful to save the query results and", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 239 + }, + { + "text": "the remote server for importing. In this example, we used \\copy FROM to import a file. We could also use \\copy TO for exporting. Let’s look at another way to export output to a file. Saving Query Output to a File It’s sometimes helpful to save the query results and messages generated during a psql session to a file, whether to keep a history of your work or to use the output in a spreadsheet or other application. To send query output to a file, you can use the \\o meta-command along with the full path and name of the output file. NOTE On Windows, file paths for the \\o command must either use Linux-style forward slashes, such as C:/my-stuff/my-file.txt, or double backslashes, such as C:\\\\my-stuff\\\\my-file.txt. Estadísticos e-Books & Papers For example, one of my favorite tricks is to set the output format to unaligned with a comma as a field separator and no row count in the footer, similar but not identical to a CSV output. (It’s not identical because a true CSV file, as you learned in Chapter 4, can include a character to quote values that contain a delimiter. Still, this trick works for simple CSV-like output.) Listing 16-8 shows the sequence of commands at the psql prompt: ➊ analysis=# \\a \\f , \\pset footer Output format is unaligned. Field separator is \",\". Default footer is off. analysis=# SELECT * FROM grades; ➋ student_id,course_id,course,grade 1,2,English 11B,D 1,3,World History 11B,C 1,4,Trig 2,B 1,1,Biology 2,C ➌ analysis=# \\o 'C:/YourDirectory/query_output.csv' analysis=# SELECT * FROM grades; ➍ analysis=# Listing 16-8: Saving query output to a file First, set the output format ➊ using the meta-commands \\a, \\f, and \\pset footer for unaligned, comma-separated data with no footer. When you run a simple SELECT query on the grades table, the output ➋ should return as values separated by commas. Next, to send that data to a file the next time you run the query, use the \\o meta-command and then provide a complete path to a file called query_output.csv ➌. When you run the SELECT query again, there should be no output to the screen ➍. Instead, you’ll find a file with the contents of the query in the directory specified at ➌. Note that every time you run a query from this point, the output is appended to the same file specified after the \\o command. To stop saving output to that file, you can either specify a new file or enter \\o with no filename to resume having results output to the screen. Estadísticos e-Books & Papers Reading and Executing SQL Stored in a File You can run SQL stored in a text file by executing psql on the command line and supplying the file name after an -f argument. This syntax lets you quickly run a query or table update from the command line or in conjunction with a system scheduler to run a job at regular intervals. Let’s say you saved the SELECT * FROM grades; query in", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 240 + }, + { + "text": "line and supplying the file name after an -f argument. This syntax lets you quickly run a query or table update from the command line or in conjunction with a system scheduler to run a job at regular intervals. Let’s say you saved the SELECT * FROM grades; query in a file called display- grades.sql. To run the saved query, use the following psql syntax at your command line: psql -d analysis -U postgres -f display-grades.sql When you press ENTER, psql should launch, run the stored query in the file, display the results, and exit. For repetitive tasks, this workflow can save you considerable time because you avoid launching pgAdmin or rewriting a query. You also can stack multiple queries in the file so they run in succession, which, for example, you might do if you want to run multiple updates on your database. Additional Command Line Utilities to Expedite Tasks PostgreSQL includes additional command line utilities that come in handy if you’re connected to a remote server or just want to save time by using the command line instead of launching pgAdmin or another GUI. You can enter these commands in your command line interface without launching psql. A listing is available at https://www.postgresql.org/docs/current/static/reference-client.html, and I’ll explain several in Chapter 17 that are specific to database maintenance. But here I’ll cover two that are particularly useful: creating a database at the command line with the createdb utility and loading shapefiles into a PostGIS database via the shp2pgsql utility. Adding a Database with createdb The first SQL statement you learned in Chapter 1 was CREATE DATABASE, Estadísticos e-Books & Papers which you used to add the database analysis to your PostgreSQL server. Rather than launching pgAdmin and writing a CREATE DATABASE statement, you can perform a similar action using the createdb command line utility. For example, to create a new database on your server named box_office, run the following at your command line: createdb -U postgres -e box_office The -U argument tells the command to connect to the PostgreSQL server using the postgres account. The -e argument (for “echo”) tells the command to print the SQL statement to the screen. Running this command generates the response CREATE DATABASE box_office; in addition to creating the database. You can then connect to the new database via psql using the following line: psql -d box_office -U postgres The createdb command accepts arguments to connect to a remote server (just like psql does) and to set options for the new database. A full list of arguments is available at https://www.postgresql.org/docs/current/static/app-createdb.html. Again, the createdb command is a time-saver that comes in handy when you don’t have access to a GUI. Loading Shapefiles with shp2pgsql In Chapter 14, you learned to import a shapefile into a database with the Shapefile Import/Export Manager included in the PostGIS suite. That tool’s GUI is easy to navigate, but importing a shapefile using the PostGIS command line tool shp2pgsql lets you accomplish the same thing using a single text command.", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 241 + }, + { + "text": "Chapter 14, you learned to import a shapefile into a database with the Shapefile Import/Export Manager included in the PostGIS suite. That tool’s GUI is easy to navigate, but importing a shapefile using the PostGIS command line tool shp2pgsql lets you accomplish the same thing using a single text command. To import a shapefile into a new table from the command line, use the following syntax: shp2pgsql -I -s SRID -W encoding shapefile_name table_name | psql -d database -U user Estadísticos e-Books & Papers A lot is happening in this single line. Here’s a breakdown of the arguments (if you skipped Chapter 14, you might need to review it now): -I Adds a GiST index on the new table’s geometry column. -s Lets you specify an SRID for the geometric data. -W Lets you specify encoding. (Recall that we used Latin1 for census shapefiles.) shapefile_name The name (including full path) of the file ending with the .shp extension. table_name The name of the table the shapefile is imported to. Following these arguments, you place a pipe symbol (|) to direct the output of shp2pgsql to psql, which has the arguments for naming the database and user. For example, to load the tl_2010_us_county10.shp shapefile into a us_counties_2010_shp table in the gis_analysis database, as you did in Chapter 14, you can simply run the following command. Note that although this command wraps onto two lines here, it should be entered as one line in the command line: shp2pgsql -I -s 4269 -W Latin1 tl_2010_us_county10.shp us_counties_2010_shp | psql -d gis_analysis -U postgres The server should respond with a number of SQL INSERT statements before creating the index and returning you to the command line. It might take some time to construct the entire set of arguments the first time around. But after you’ve done one, subsequent imports should take less time because you can simply substitute file and table names into the syntax you already wrote. Wrapping Up Are you feeling mysterious and powerful yet? Indeed, when you delve into a command line interface and make the computer do your bidding using text commands, you enter a world of computing that resembles a Estadísticos e-Books & Papers sci-fi movie sequence. Not only does working from the command line save you time, but it also helps you overcome barriers you encounter when you’re working in environments that don’t support graphical tools. In this chapter, you learned the basics of working with the command line plus PostgreSQL specifics. You discovered your operating system’s command line application and set it up to work with psql. Then you connected psql to a database and learned how to run SQL queries via the command line. Many experienced computer users prefer to use the command line for its simplicity and speed once they become familiar with using it. You might, too. In Chapter 17, we’ll review common database maintenance tasks including backing up data, changing server settings, and managing the growth of your database. These tasks will give you", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 242 + }, + { + "text": "users prefer to use the command line for its simplicity and speed once they become familiar with using it. You might, too. In Chapter 17, we’ll review common database maintenance tasks including backing up data, changing server settings, and managing the growth of your database. These tasks will give you more control over your working environment and help you better manage your data analysis projects. TRY IT YOURSELF To reinforce the techniques in this chapter, choose an example from an earlier chapter and try working through it using only the command line. Chapter 14 is a good choice because it gives you the opportunity to work with psql and the shapefile loader shp2pgsql. But choose any example that you think you would benefit from reviewing. Estadísticos e-Books & Papers 17 MAINTAINING YOUR DATABASE To wrap up our exploration of SQL, we’ll look at key database maintenance tasks and options for customizing PostgreSQL. In this chapter, you’ll learn how to track and conserve space in your databases, how to change system settings, and how to back up and restore databases. How often you’ll need to perform these tasks depends on your current role and interests. But if you want to be a database administrator or a backend developer, the topics covered here are vital to both jobs. It’s worth noting that database maintenance and performance tuning are often the subjects of entire books, and this chapter mainly serves as an introduction to a handful of essentials. If you want to learn more, a good place to begin is with the resources in the Appendix. Let’s start with the PostgreSQL VACUUM feature, which lets you shrink the size of tables by removing unused rows. Recovering Unused Space with VACUUM To prevent database files from growing out of control, you can use the PostgreSQL VACUUM command. In “Improving Performance When Updating Large Tables” on page 151, you learned that the size of PostgreSQL tables can grow as a result of routine operations. For Estadísticos e-Books & Papers example, when you update a value in a row, the database creates a new version of that row that includes the updated value, but it doesn’t delete the old version of the row. (PostgreSQL documentation refers to these leftover rows that you can’t see as “dead” rows.) Similarly, when you delete a row, even though the row is no longer visible, it lives on as a dead row in the table. The database uses dead rows to provide certain features in environments where multiple transactions are occurring and old versions of rows might be needed by transactions other than the current one. Running VACUUM designates the space occupied by dead rows as available for the database to use again. But VACUUM doesn’t return the space to your system’s disk. Instead, it just flags that space as available for the database to use for its next operation. To return unused space to your disk, you must use the VACUUM FULL option, which creates a new version of", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 243 + }, + { + "text": "to use again. But VACUUM doesn’t return the space to your system’s disk. Instead, it just flags that space as available for the database to use for its next operation. To return unused space to your disk, you must use the VACUUM FULL option, which creates a new version of the table that doesn’t include the freed-up dead row space. Although you can run VACUUM on demand, by default PostgreSQL runs the autovacuum background process that monitors the database and runs VACUUM as needed. Later in this chapter I’ll show you how to monitor autovacuum as well as run the VACUUM command manually. But first, let’s look at how a table grows as a result of updates and how you can track this growth. Tracking Table Size We’ll create a small test table and monitor its growth in size as we fill it with data and perform an update. The code for this exercise, as with all resources for the book, is available at https://www.nostarch.com/practicalSQL/. Creating a Table and Checking Its Size Listing 17-1 creates a vacuum_test table with a single column to hold an integer. Run the code, and then we’ll measure the table’s size. CREATE TABLE vacuum_test ( Estadísticos e-Books & Papers integer_column integer ); Listing 17-1: Creating a table to test vacuuming Before we fill the table with test data, let’s check how much space it occupies on disk to establish a reference point. We can do so in two ways: check the table properties via the pgAdmin interface, or run queries using PostgreSQL administrative functions. In pgAdmin, click once on a table to highlight it, and then click the Statistics tab. Table size is one of about two dozen indicators in the list. I’ll focus on running queries here because knowing them is helpful if for some reason pgAdmin isn’t available or you’re using another GUI. For example, Listing 17-2 shows how to check the vacuum_test table size using PostgreSQL functions: SELECT ➊pg_size_pretty( ➋pg_total_relation_size('vacuum_test') ); Listing 17-2: Determining the size of vacuum_test The outermost function, pg_size_pretty() ➊, converts bytes to a more easily understandable format in kilobytes, megabytes, or gigabytes. Wrapped inside pg_size_pretty() is the pg_total_relation_size() function ➋, which reports how many bytes a table, its indexes, and offline compressed data takes up on disk. Because the table is empty at this point, running the code in pgAdmin should return a value of 0 bytes, like this: pg_size_pretty -------------- 0 bytes You can get the same information using the command line. Launch psql as you learned in Chapter 16. Then, at the prompt, enter the command \\dt+ vacuum_test, which should display the following information including table size: Estadísticos e-Books & Papers Again, the current size of the vacuum_test table should display 0 bytes. Checking Table Size After Adding New Data Let’s add some data to the table and then check its size again. We’ll use the generate_series() function introduced in Chapter 11 to fill the table’s integer_column with 500,000 rows. Run the code in Listing 17-3", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 244 + }, + { + "text": "the vacuum_test table should display 0 bytes. Checking Table Size After Adding New Data Let’s add some data to the table and then check its size again. We’ll use the generate_series() function introduced in Chapter 11 to fill the table’s integer_column with 500,000 rows. Run the code in Listing 17-3 to do this: INSERT INTO vacuum_test SELECT * FROM generate_series(1,500000); Listing 17-3: Inserting 500,000 rows into vacuum_test This standard INSERT INTO statement adds the results of generate_series(), which is a series of values from 1 to 500,000, as rows to the table. After the query completes, rerun the query in Listing 17-2 to check the table size. You should see the following output: pg_size_pretty -------------- 17 MB The query reports that the vacuum_test table, now with a single column of 500,000 integers, uses 17MB of disk space. Checking Table Size After Updates Now, let’s update the data to see how that affects the table size. We’ll use the code in Listing 17-4 to update every row in the vacuum_test table by adding 1 to the integer_column values, replacing the existing value with a number that’s one greater. UPDATE vacuum_test SET integer_column = integer_column + 1; Estadísticos e-Books & Papers Listing 17-4: Updating all rows in vacuum_test Run the code, and then test the table size again. pg_size_pretty -------------- 35 MB The table size has doubled from 17MB to 35MB! The increase seems excessive, because the UPDATE simply replaced existing numbers with values of a similar size. But as you might have guessed, the reason for this increase in table size is that for every updated value, PostgreSQL creates a new row, and the old row (a “dead” row) remains in the table. So even though you only see 500,000 rows, the table has double that number of rows. Consequently, if you’re working with a database that is frequently updated, it will grow even if you’re not adding rows. This can surprise database owners who don’t monitor disk space because the drive eventually fills up and leads to server errors. You can use VACUUM to avoid this scenario. We’ll look at how using VACUUM and VACUUM FULL affects the table’s size on disk. But first, let’s review the process that runs VACUUM automatically as well as how to check on statistics related to table vacuums. Monitoring the autovacuum Process PostgreSQL’s autovacuum process monitors the database and launches VACUUM automatically when it detects a large number of dead rows in a table. Although autovacuum is enabled by default, you can turn it on or off and configure it using the settings I’ll cover in “Changing Server Settings” on page 318. Because autovacuum runs in the background, you won’t see any immediately visible indication that it’s working, but you can check its activity by running a query. PostgreSQL has its own statistics collector that tracks database activity and usage. You can look at the statistics by querying one of several views the system provides. (See a complete list of views for monitoring the", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 245 + }, + { + "text": "indication that it’s working, but you can check its activity by running a query. PostgreSQL has its own statistics collector that tracks database activity and usage. You can look at the statistics by querying one of several views the system provides. (See a complete list of views for monitoring the state Estadísticos e-Books & Papers of the system at https://www.postgresql.org/docs/current/static/monitoring- stats.html). To check the activity of autovacuum, query a view called pg_stat_all_tables using the code in Listing 17-5: SELECT ➊relname, ➋last_vacuum, ➌last_autovacuum, ➍vacuum_count, ➎autovacuum_count FROM pg_stat_all_tables WHERE relname = 'vacuum_test'; Listing 17-5: Viewing autovacuum statistics for vacuum_test The pg_stat_all_tables view shows relname ➊, which is the name of the table, plus statistics related to index scans, rows inserted and deleted, and other data. For this query, we’re interested in last_vacuum ➋ and last_autovacuum ➌, which contain the last time the table was vacuumed manually and automatically, respectively. We also ask for vacuum_count ➍ and autovacuum_count ➎, which show the number of times the vacuum was run manually and automatically. By default, autovacuum checks tables every minute. So, if a minute has passed since you last updated vacuum_test, you should see details of vacuum activity when you run the query in Listing 17-5. Here’s what my system shows (note that I’ve removed seconds from the time to save space here): The table shows the date and time of the last autovacuum, and the autovacuum_count column shows one occurrence. This result indicates that autovacuum executed a VACUUM command on the table once. However, because we’ve not vacuumed manually, the last_vacuum column is empty and the vacuum_count is 0. NOTE Estadísticos e-Books & Papers The autovacuum process also runs the ANALYZE command, which gathers data on the contents of tables. PostgreSQL stores this information and uses it to execute queries efficiently in the future. You can run ANALYZE manually if needed. Recall that VACUUM designates dead rows as available for the database to reuse but doesn’t reduce the size of the table on disk. You can confirm this by rerunning the code in Listing 17-2, which shows the table remains at 35MB even after the automatic vacuum. Running VACUUM Manually Depending on the server you’re using, you can turn off autovacuum. (I’ll show you how to view that setting in “Locating and Editing postgresql.conf” on page 319.) If autovacuum is off or if you simply want to run VACUUM manually, you can do so using a single line of code, as shown in Listing 17-6: VACUUM vacuum_test; Listing 17-6: Running VACUUM manually After you run this command, it should return the message VACUUM from the server. Now when you fetch statistics again using the query in Listing 17-5, you should see that the last_vacuum column reflects the date and time of the manual vacuum you just ran and the number in the vacuum_count column should increase by one. In this example, we executed VACUUM on our test table. But you can also run VACUUM on the entire database by omitting the table name.", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 246 + }, + { + "text": "last_vacuum column reflects the date and time of the manual vacuum you just ran and the number in the vacuum_count column should increase by one. In this example, we executed VACUUM on our test table. But you can also run VACUUM on the entire database by omitting the table name. In addition, you can add the VERBOSE keyword to provide more detailed information, such as the number of rows found in a table and the number of rows removed, among other information. Reducing Table Size with VACUUM FULL Estadísticos e-Books & Papers Next, we’ll run VACUUM with the FULL option. Unlike the default VACUUM, which only marks the space held by dead rows as available for future use, the FULL option returns space back to disk. As mentioned, VACUUM FULL creates a new version of a table, discarding dead rows in the process. Although this frees space on your system’s disk, there are a couple of caveats to keep in mind. First, VACUUM FULL takes more time to complete than VACUUM. Second, it must have exclusive access to the table while rewriting it, which means that no one can update data during the operation. The regular VACUUM command can run while updates and other operations are happening. To see how VACUUM FULL works, run the command in Listing 17-7: VACUUM FULL vacuum_test; Listing 17-7: Using VACUUM FULL to reclaim disk space After the command executes, test the table size again. It should be back down to 17MB, which is the size it was when we first inserted data. It’s never prudent or safe to run out of disk space, so minding the size of your database files as well as your overall system space is a worthwhile routine to establish. Using VACUUM to prevent database files from growing bigger than they have to is a good start. Changing Server Settings It’s possible to alter dozens of settings for your PostgreSQL server by editing values in postgresql.conf, one of several configuration text files that control server settings. Other files include pg_hba.conf, which controls connections to the server, and pg_ident.conf, which database administrators can use to map usernames on a network to usernames in PostgreSQL. See the PostgreSQL documentation on these files for details. For our purposes, we’ll use the postgresql.conf file because it contains settings we’re most interested in. Most of the values in the file are set to defaults you won’t ever need to adjust, but it’s worth exploring in case Estadísticos e-Books & Papers you want to change them to suit your needs. Let’s start with the basics. Locating and Editing postgresql.conf Before you can edit postgresql.conf, you’ll need to find its location, which varies depending on your operating system and install method. You can run the command in Listing 17-8 to locate the file: SHOW config_file; Listing 17-8: Showing the location of postgresql.conf When I run the command on a Mac, it shows the path to the file as: /Users/anthony/Library/Application Support/Postgres/var-10/postgresql.conf To edit postgresql.conf, navigate to the directory", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 247 + }, + { + "text": "system and install method. You can run the command in Listing 17-8 to locate the file: SHOW config_file; Listing 17-8: Showing the location of postgresql.conf When I run the command on a Mac, it shows the path to the file as: /Users/anthony/Library/Application Support/Postgres/var-10/postgresql.conf To edit postgresql.conf, navigate to the directory displayed by SHOW config_file; in your system, and open the file using a plain text editor, not a rich text editor like Microsoft Word. NOTE It’s a good idea to save a copy of postgresql.conf for reference in case you make a change that breaks the system and you need to revert to the original version. When you open the file, the first several lines should read as follows: # ----------------------------- # PostgreSQL configuration file # ----------------------------- # # This file consists of lines of the form: # # name = value The postgresql.conf file is organized into sections that specify settings for file locations, security, logging of information, and other processes. Many lines begin with a hash mark (#), which indicates the line is Estadísticos e-Books & Papers commented out and the setting shown is the active default. For example, in the postgresql.conf file section “Autovacuum Parameters,” the default is for autovacuum to be turned on. The hash mark (#) in front of the line means that the line is commented out and the default is in effect: #autovacuum = on # Enable autovacuum subprocess? 'on' To turn off autovacuum, you remove the hash mark at the beginning of the line and change the value to off: autovacuum = off # Enable autovacuum subprocess? 'on' Listing 17-9 shows some other settings you might want to explore, which are excerpted from the postgresql.conf section “Client Connection Defaults.” Use your text editor to search the file for the following settings. ➊ datestyle = 'iso, mdy' ➋ timezone = 'US/Eastern' ➌ default_text_search_config = 'pg_catalog.english' Listing 17-9: Sample postgresql.conf settings You can use the datestyle setting ➊ to specify how PostgreSQL displays dates in query results. This setting takes two parameters: the output format and the ordering of month, day, and year. The default for the output format is the ISO format (YYYY-MM-DD) we’ve used throughout this book, which I recommend you use for cross-national portability. However, you can also use the traditional SQL format (MM/DD/YYYY), the expanded Postgres format (Mon Nov 12 22:30:00 2018 EST), or the German format (DD.MM.YYYY) with dots between the date, month, and year. To specify the format using the second parameter, arrange m, d, and y in the order you prefer. The timezone parameter ➋ sets the (you guessed it) server time zone. Estadísticos e-Books & Papers Listing 17-9 shows the value US/Eastern, which reflects the time zone on my machine when I installed PostgreSQL. Yours should vary based on your location. When setting up PostgreSQL for use as the backend to a database application or on a network, administrators often set this value to UTC and use that as a standard on machines across multiple locations.", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 248 + }, + { + "text": "zone on my machine when I installed PostgreSQL. Yours should vary based on your location. When setting up PostgreSQL for use as the backend to a database application or on a network, administrators often set this value to UTC and use that as a standard on machines across multiple locations. The default_text_search_config value ➌ sets the language used by the full text search operations. Here, mine is set to english. Depending on your needs, you can set this to spanish, german, russian, or another language of your choice. These three examples represent only a handful of settings available for adjustment. Unless you end up deep in system tuning, you probably won’t have to tweak much else. Also, use caution when changing settings on a network server used by multiple people or applications; changes can have unintended consequences, so it’s worth communicating with colleagues first. After you make changes to postgresql.conf, you must save the file and then reload settings using the pg_ctl PostgreSQL command to apply the new settings. Let’s look at how to do that next. Reloading Settings with pg_ctl The command line utility pg_ctl allows you to perform actions on a PostgreSQL server, such as starting and stopping it, and checking its status. Here, we’ll use the utility to reload the settings files so changes we make will take effect. Running the command reloads all settings files at once. You’ll need to open and configure a command line prompt the same way you did in Chapter 16 when you learned how to set up and use psql. After you launch a command prompt, use one of the following commands to reload: On Windows, use: pg_ctl reload -D \"C:\\path\\to\\data\\directory\\\" Estadísticos e-Books & Papers On macOS or Linux, use: pg_ctl reload -D '/path/to/data/directory/' To find the location of your PostgreSQL data directory, run the query in Listing 17-10: SHOW data_directory; Listing 17-10: Showing the location of the data directory You place the path between double quotes on Windows and single quotes on macOS or Linux after the -D argument. You run this command on your system’s command prompt, not inside the psql application. Enter the command and press ENTER; it should respond with the message server signaled. The settings files will be reloaded and changes should take effect. Some settings, such as memory allocations, require a restart of the server. PostgreSQL will warn you if that’s the case. Backing Up and Restoring Your Database When you cleaned up the “dirty” USDA food producer data in Chapter 9, you learned how to create a backup copy of a table. However, depending on your needs, you might want to back up your entire database regularly either for safekeeping or for transferring data to a new or upgraded server. PostgreSQL offers command line tools that make backup and restore operations easy. The next few sections show examples of how to create a backup of a database or a single table, as well as how to restore them. Using pg_dump to Back Up a", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 249 + }, + { + "text": "to a new or upgraded server. PostgreSQL offers command line tools that make backup and restore operations easy. The next few sections show examples of how to create a backup of a database or a single table, as well as how to restore them. Using pg_dump to Back Up a Database or Table The PostgreSQL command line tool pg_dump creates an output file that contains all the data from your database, SQL commands for re-creating tables, and other database objects, as well as loading the data into tables. You can also use pg_dump to save only selected tables in your database. By Estadísticos e-Books & Papers default, pg_dump outputs a plain text file; I’ll discuss a custom compressed format first and then discuss other options. To back up the analysis database we’ve used for our exercises, run the command in Listing 17-11 at your system’s command prompt (not in psql): pg_dump -d analysis -U user_name -Fc > analysis_backup.sql Listing 17-11: Backing up the analysis database with pg_dump Here, we start the command with pg_dump, the -d argument, and name of the database to back up, followed by the -U argument and your username. Next, we use the -Fc argument to specify that we want to generate this backup in a custom PostgreSQL compressed format. Then we place a greater-than symbol (>) to redirect the output of pg_dump to a text file named analysis_backup.sql. To place the file in a directory other than the one your terminal prompt is currently open to, you can specify the complete directory path before the filename. When you execute the command by pressing ENTER, depending on your installation, you might see a password prompt. Fill in that password, if prompted. Then, depending on the size of your database, the command could take a few minutes to complete. The operation doesn’t output any messages to the screen while it’s working, but when it’s done, it should return you to a new command prompt and you should see a file named analysis_backup.sql in your current directory. To limit the backup to one or more tables that match a particular name, use the -t argument followed by the name of the table in single quotes. For example, to back up just the train_rides table, use the following command: pg_dump -t 'train_rides' -d analysis -U user_name -Fc > train_backup.sql Now let’s look at how to restore a backup, and then we’ll explore additional pg_dump options. Estadísticos e-Books & Papers Restoring a Database Backup with pg_restore After you’ve backed up your database using pg_dump, it’s very easy to restore it using the pg_restore utility. You might need to restore your database when migrating data to a new server or when upgrading to a new version of PostgreSQL. To restore the analysis database (assuming you’re on a server where analysis doesn’t exist), run the command in Listing 17-12 at the command prompt: pg_restore -C -d postgres -U user_name analysis_backup.sql Listing 17-12: Restoring the analysis database with pg_restore After pg_restore, you add", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 250 + }, + { + "text": "upgrading to a new version of PostgreSQL. To restore the analysis database (assuming you’re on a server where analysis doesn’t exist), run the command in Listing 17-12 at the command prompt: pg_restore -C -d postgres -U user_name analysis_backup.sql Listing 17-12: Restoring the analysis database with pg_restore After pg_restore, you add the -C argument, which tells the utility to create the analysis database on the server. (It gets the database name from the backup file.) Then, as you saw previously, the -d argument specifies the name of the database to connect to, followed by the -U argument and your username. Press ENTER and the restore will begin. When it’s done, you should be able to view your restored database via psql or in pgAdmin. Additional Backup and Restore Options You can configure pg_dump with multiple options to include or exclude certain database objects, such as tables matching a name pattern, or to specify the output format. Also, when we backed up the analysis database in “Using pg_dump to Back Up a Database or Table” on page 321, we specified the -Fc option with pg_dump to generate a custom PostgreSQL compressed format. The utility supports additional format options, including plain text. For details, check the full pg_dump documentation at https://www.postgresql.org/docs/current/static/app-pgdump.html. For corresponding restore options, check the pg_restore documentation at https://www.postgresql.org/docs/current/static/app-pgrestore.html. Wrapping Up Estadísticos e-Books & Papers In this chapter, you learned how to track and conserve space in your databases using the VACUUM feature in PostgreSQL. You also learned how to change system settings as well as back up and restore databases using other command line tools. You may not need to perform these tasks every day, but the maintenance tricks you learned here can help enhance the performance of your databases. Note that this is not a comprehensive overview of the topic; see the Appendix for more resources on database maintenance. In the next and final chapter of this book, I’ll share guidelines for identifying hidden trends and telling an effective story using your data. TRY IT YOURSELF Using the techniques you learned in this chapter, back up and restore the gis_analysis database you made in Chapter 14. After you back up the full database, you’ll need to delete the original to be able to restore it. You might also try backing up and restoring individual tables. In addition, use a text editor to explore the backup file created by pg_dump. Examine how it organizes the statements to create objects and insert data. Estadísticos e-Books & Papers 18 IDENTIFYING AND TELLING THE STORY BEHIND YOUR DATA Although learning SQL can be fun in and of itself, it serves a greater purpose: it helps uncover the hidden stories in your data. As you learned in this book, SQL gives you the tools to find interesting trends, insights, or anomalies in your data and then make smart decisions based on what you’ve learned. But how do you identify these trends just from a collection of rows and columns? And how can you glean", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 251 + }, + { + "text": "you learned in this book, SQL gives you the tools to find interesting trends, insights, or anomalies in your data and then make smart decisions based on what you’ve learned. But how do you identify these trends just from a collection of rows and columns? And how can you glean meaningful insights from these trends after identifying them? Identifying trends in your data set and creating a narrative of your findings sometimes requires considerable experimentation and enough fortitude to weather the occasional dead end. In this chapter, I outline a process I’ve used as an investigative journalist to discover stories in data and communicate my findings. I start with how to generate ideas by asking good questions as well as gathering and exploring data. Then I explain the analysis process, which culminates in presenting your findings clearly. These tips are less of a checklist and more of a general guideline that can help you avoid certain mistakes. Start with a Question Estadísticos e-Books & Papers Curiosity, intuition, or sometimes just dumb luck can often spark ideas for data analysis. If you’re a keen observer of your surroundings, you might notice changes in your community over time and wonder if you can measure that change. Consider your local real estate market as an example. If you see more “For Sale” signs popping up around town than usual, you might start asking questions. Is there a dramatic increase in home sales this year compared to last year? If so, by how much? Which neighborhoods are affected? These questions create a great opportunity for data analysis. If you’re a journalist, you might find a story. If you run a business, you might discover a new marketing opportunity. Likewise, if you surmise that a trend is occurring in your industry, confirming it might provide you with a business opportunity. For example, if you suspect that sales of a particular product have become sluggish, you can use data analysis to confirm the hunch and adjust inventory or marketing efforts appropriately. Keep track of these ideas and prioritize them according to their potential value. Analyzing data to satisfy your curiosity is perfectly fine, but if the answers can make your institution more effective or your company more profitable, that’s a sign they’re worth pursuing. Document Your Process Before you delve into analysis, consider how to make your process transparent and reproducible. For the sake of credibility, others in your organization as well as those outside it should be able to reproduce your work. In addition, make sure you document enough information so that if you set the project aside for several weeks, you won’t have a problem picking it up again. There isn’t one right way to document your work. Taking notes on research or creating step-by-step SQL queries that another person could use to replicate your data import, cleaning, and analysis can make it easier for others to verify your findings. Some analysts store notes and code in a text file. Others use version control", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 252 + }, + { + "text": "to document your work. Taking notes on research or creating step-by-step SQL queries that another person could use to replicate your data import, cleaning, and analysis can make it easier for others to verify your findings. Some analysts store notes and code in a text file. Others use version control systems, such as GitHub. The Estadísticos e-Books & Papers important factor is that you create your own system of documentation and use it consistently. Gather Your Data After you’ve hatched an idea for analysis, the next step is to find data that relates to the trend or question. If you’re working in an organization that already has its own data on the topic, lucky you—you’re set! In that case, you might be able to tap into internal marketing or sales databases, customer relationship management (CRM) systems, or subscriber or event registration data. But if your topic encompasses broader issues involving demographics, the economy, or industry-specific subjects, you’ll need to do some digging. A good place to start is to ask experts about the sources they use. Analysts, government decision-makers, and academics can often point you to available data and its usefulness. Federal, state, and local governments, as you’ve seen throughout the book, produce volumes of data on all kinds of topics. In the United States, check out the federal government’s data catalog site at https://www.data.gov/ or individual agency sites, such as the National Center for Education Statistics (NCES) at https://nces.ed.gov/. You can also browse local government websites. Any time you see a form for users to fill out or a report formatted in rows and columns, those are signs that structured data might be available for analysis. But all is not lost if you only have access to unstructured data. As you learned in Chapter 13, you can even mine unstructured data, such as text files. If the data you want to analyze was collected over multiple years, I recommend examining five or 10 years, or more, instead of just one or two, if possible. Although analyzing a snapshot of data collected over a month or a year can yield interesting results, many trends play out over a longer period of time and may not be evident if you look at a single year of data. I discuss this further in “Identify Key Indicators and Trends over Time” on page 329. Estadísticos e-Books & Papers No Data? Build Your Own Database Sometimes, no one has the data you need in a format you can use. But if you have time, patience, and a methodology, you might be able to build your own data set. That is what my USA TODAY colleague, Robert Davis, and I did when we wanted to study issues related to the deaths of college students on campuses in the United States. Not a single organization—not the schools or state or federal officials—could tell us how many college students were dying each year from accidents, overdoses, or illnesses on campus. We decided to collect our own data", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 253 + }, + { + "text": "study issues related to the deaths of college students on campuses in the United States. Not a single organization—not the schools or state or federal officials—could tell us how many college students were dying each year from accidents, overdoses, or illnesses on campus. We decided to collect our own data and structure the information into tables in a database. We started by researching news articles, police reports, and lawsuits related to student deaths. After finding reports of more than 600 student deaths from 2000 to 2005, we followed up with interviews with education experts, police, school officials, and parents. From each report, we cataloged details such as each student’s age, school, cause of death, year in school, and whether drugs or alcohol played a role. Our findings led to the publication of the article “In College, First Year Is by Far the Riskiest” in USA TODAY in 2006. The story featured the key finding from the analysis of our SQL database: freshmen were particularly vulnerable and accounted for the highest percentage of the student deaths we studied. You too can create a database if you lack the data you need. The key is to identify the pieces of information that matter, and then systematically collect them. Assess the Data’s Origins After you’ve identified a data set, find as much information about its origins and maintenance methods as you can. Governments and institutions gather data in all sorts of ways, and some methods produce data that is more credible and standardized than others. For example, you’ve already seen that USDA food producer data Estadísticos e-Books & Papers includes the same company names spelled in multiple ways. It’s worth knowing why. (Perhaps the data is manually copied from a written form to a computer.) Similarly, the New York City taxi data you analyzed in Chapter 11 records the start and end times of each trip. This begs the question, does the timer start when the passenger gets in and out of the vehicle, or is there some other trigger? You should know these details not only to draw better conclusions from analysis but also to pass them along to others who might be interpreting your analysis. The origins of a data set might also affect how you analyze the data and report your findings. For example, with U.S. Census data, it’s important to know that the Decennial Census conducted every 10 years is a complete count of the population, whereas the American Community Survey (ACS) is drawn from only a sample of households. As a result, ACS counts have a margin of error, but the Decennial Census doesn’t. It would be irresponsible to report on the ACS without considering how the margin of error could make differences between numbers insignificant. Interview the Data with Queries Once you have your data, understand its origins, and have loaded it into your database, you can explore it with queries. Throughout the book, I call this step “interviewing data,” which is what you should do to", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 254 + }, + { + "text": "of error could make differences between numbers insignificant. Interview the Data with Queries Once you have your data, understand its origins, and have loaded it into your database, you can explore it with queries. Throughout the book, I call this step “interviewing data,” which is what you should do to find out more about the contents of your data and whether they contain any red flags. A good place to start is with aggregates. Counts, sums, sorting, and grouping by column values should reveal minimum and maximum values, potential issues with duplicate entries, and a sense of the general scope of your data. If your database contains multiple, related tables, try joins to make sure you understand how the tables relate. Using LEFT JOIN and RIGHT JOIN, as you learned in Chapter 6, should show whether key values from one table are missing in another. That may or may not be a concern, but at least you’ll be able to identify potential problems you might want to address. Jot down a list of questions or concerns you have, and then move Estadísticos e-Books & Papers on to the next step. Consult the Data’s Owner After exploring your database and forming early conclusions about the quality and trends you observed, take some time to bring any questions or concerns you have to a person who knows the data well. That person could work at the agency or firm that gave you the data, or the person might be an analyst who has worked with the data before. This step is your chance to clarify your understanding of the data, verify initial findings, and discover whether the data has any issues that make it unsuitable for your needs. For example, if you’re querying a table and notice values in columns that seem to be gross outliers (such as dates in the future for events that were supposed to have happened in the past), you should ask about that discrepancy. Or, if you expect to find someone’s name in a table (perhaps even your own name), and it’s not there, that should prompt another question. Is it possible you don’t have the whole data set, or is there a problem with data collection? The goal is to get expert help to do the following: Understand the limits of the data. Make sure you know what the data includes, what it excludes, and any caveats about content that might affect how you perform your analysis. Make sure you have a complete data set. Verify that you have all the records you should expect to see and that if any data is missing, you understand why. Determine whether the data set suits your needs. Consider looking elsewhere for more reliable data if your source acknowledges problems with the data’s quality. Every data set and situation is unique, but consulting another user or owner of the data can help you avoid unnecessary missteps. Estadísticos e-Books & Papers Identify Key Indicators and Trends over Time When you’re", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 255 + }, + { + "text": "looking elsewhere for more reliable data if your source acknowledges problems with the data’s quality. Every data set and situation is unique, but consulting another user or owner of the data can help you avoid unnecessary missteps. Estadísticos e-Books & Papers Identify Key Indicators and Trends over Time When you’re satisfied that you understand the data and are confident in its trustworthiness, completeness, and appropriateness to your analysis, the next step is to run queries to identify key indicators and, if possible, trends over time. Your goal is to unearth data that you can summarize in a sentence or present as a slide in a presentation. An example finding would be something like this: “After five years of declines, the number of people enrolling in Widget University has increased by 5 percent for two consecutive semesters.” To identify this type of trend, you’ll follow a two-step process: 1. Choose an indicator to track. In U.S. Census data, it might be the percentage of the population that is over age 60. Or in the New York City taxi data, it could be the median number of weekday trips over the span of one year. 2. Track that indicator over multiple years to see how it has changed, if at all. In fact, these are the steps we used in Chapter 6 to apply percent change calculations to multiple years of census data contained in joined tables. In that case, we looked at the change in population in counties between 2000 and 2010. The population count was the key indicator, and the percent change showed the trend over the 10-year span for each county. One caveat about measuring change over time: even when you see a dramatic change between any two years, it’s worth digging into as many years’ worth of data as possible to understand the shorter-term change in the context of a long-term trend. Although a year-to-year change might seem dramatic, seeing it in context of multiyear activity can help you assess its true significance. For example, the U.S. National Center for Health Statistics releases data on the number of babies born each year. As a data nerd, I like to Estadísticos e-Books & Papers keep tabs on indicators like these, because births often reflect broader trends in culture or the economy. Figure 18-1 shows the annual number of births from 1910 to 2016. Figure 18-1: U.S. births from 1910 to 2016. Source: U.S. National Center for Health Statistics Looking at only the last five years of this graph (shaded in gray), we see that the number of births hovered steadily at approximately 3.9 million with small decreases in the last two years. Although the recent drops seem noteworthy (likely reflecting continuing decreases in birth rates for teens and women in their 20s), in the long-term context, they’re less interesting given that the number of births has remained near or over 4 million for the last 20 years. In fact, U.S. births have seen far more dramatic increases and decreases.", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 256 + }, + { + "text": "reflecting continuing decreases in birth rates for teens and women in their 20s), in the long-term context, they’re less interesting given that the number of births has remained near or over 4 million for the last 20 years. In fact, U.S. births have seen far more dramatic increases and decreases. One example you can see in Figure 18- 1 is the major rise in the mid-1940s following World War II, which signaled the start of the Baby Boom generation. By identifying key indicators and looking at change over time, both short term and long term, you might uncover one or more findings worth presenting to others or acting on. NOTE Estadísticos e-Books & Papers Any time you work with data from a survey, poll, or other sample, it’s important to test for statistical significance. Are the results actually a trend or just the result of chance? Significance testing is a statistical concept beyond the scope of this book but one that data analysts should know. See the Appendix for PostgreSQL resources for advanced statistics. Ask Why Data analysis can tell you what happened, but it doesn’t usually indicate why something happened. To learn why something happened, it’s worth revisiting the data with experts in the topic or the owners of the data. In the U.S. births data, it’s easy to calculate year-to-year percent change from those numbers. But the data doesn’t tell us why births steadily increased from the early 1980s to 1990. For that information, you might need to consult a demographer who would most likely explain that the rise in births during those years coincided with more Baby Boomers entering their childbearing years. When you share your findings and methodology with experts, ask them to note anything that seems unlikely or worthy of further examination. For the findings they can corroborate, ask them to help you understand the forces behind those findings. If they’re willing to be cited, you can use their comments to supplement your report or presentation. This is a standard approach journalists often use to quote experts’ reactions to data trends. Communicate Your Findings How you share the results of your analysis depends on your role. A student might present their results in a paper or dissertation. A person who works in a corporate setting might present their findings using PowerPoint, Keynote, or Google Slides. A journalist might write a story or produce a data visualization. Regardless of the end product, here are my tips for presenting the information well (using a fictional home sales Estadísticos e-Books & Papers analysis as an example): Identify an overarching theme based on your findings. Make the theme the title of your presentation, paper, or visualization. For example, for a presentation on real estate, you might use, “Home sales rise in suburban neighborhoods, fall in cities.” Present overall numbers to show the general trend. Highlight the key findings from your analysis. For example, “All suburban neighborhoods saw sales up 5 percent each of the last two years, reversing three", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 257 + }, + { + "text": "a presentation on real estate, you might use, “Home sales rise in suburban neighborhoods, fall in cities.” Present overall numbers to show the general trend. Highlight the key findings from your analysis. For example, “All suburban neighborhoods saw sales up 5 percent each of the last two years, reversing three years of declines. Meanwhile, city neighborhoods saw a decline of 2 percent.” Highlight specific examples that support the trend. Describe one or two relevant cases. For example, “In Smithtown, home sales increased 15 percent following the relocation of XYZ Corporation’s headquarters last year.” Acknowledge examples counter to the overall trend. Use one or two relevant cases here as well. For example, “Two city neighborhoods did show growth in home sales: Arvis (up 4.5 percent) and Zuma (up 3 percent).” Stick to the facts. Avoid distorting or exaggerating any findings. Provide expert opinion. Use quotes or citations. Visualize numbers using bar charts or line charts. Tables are helpful for giving your audience specific numbers, but it’s easier to understand trends from a visualization. Cite the source of the data and what your analysis includes or omits. Provide dates covered, the name of the provider, and any distinctions that affect the analysis. For example, “Based on Walton County tax filings in 2015 and 2016. Excludes commercial properties.” Share your data. Post data online for download, including the queries you used. Nothing says transparency more than sharing the data you analyzed with others so they can perform their own analysis and corroborate your findings. Estadísticos e-Books & Papers Generally, a short presentation that communicates your findings clearly and succinctly, and then invites dialogue from your audience thereafter, works best. Of course, you can follow your own preferred pattern for working with data and presenting your conclusions. But over the years, these steps have helped me avoid bad data and mistaken assumptions. Wrapping Up At last, you’ve reached the end of our practical exploration of SQL! Thank you for reading this book, and I welcome your suggestions and feedback on my website at https://www.anthonydebarros.com/contact/. At the end of this book is an appendix that lists additional PostgreSQL-related tools you might want to try. I hope you’ve come away with data analysis skills you can start using immediately on the data you encounter. More importantly, I hope you’ve seen that each data set has a story, or several stories, to tell. Identifying and telling these stories is what makes working with data worthwhile; it’s more than just combing through a collection of rows and columns. I look forward to hearing about what you discover! TRY IT YOURSELF It’s your turn to find and tell a story using the SQL techniques we’ve covered. Using the process outlined in this chapter, consider a local or national topic and search for available data. Assess its quality, the questions it might answer, and its timeliness. Consult with an expert who knows the data and the topic well. Load the data into PostgreSQL and interview it using aggregate queries and", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 258 + }, + { + "text": "outlined in this chapter, consider a local or national topic and search for available data. Assess its quality, the questions it might answer, and its timeliness. Consult with an expert who knows the data and the topic well. Load the data into PostgreSQL and interview it using aggregate queries and filters. What trends can you discover? Summarize your findings in a short presentation. Estadísticos e-Books & Papers Estadísticos e-Books & Papers ADDITIONAL POSTGRESQL RESOURCES This appendix contains some resources to help you stay informed about PostgreSQL developments, find additional software, and get help. Because software resources are likely to change, I’ll maintain a copy of this appendix at the GitHub repository that contains all the book’s resources. You can find a link via https://www.nostarch.com/practicalSQL/. PostgreSQL Development Environments Throughout the book, we’ve used the graphical user interface pgAdmin to connect to PostgreSQL, run queries, and view database objects. Although pgAdmin is free, open source, and popular, it’s not your only choice for working with PostgreSQL. You can read the entry called “Community Guide to PostgreSQL GUI Tools,” which catalogs many alternatives, on the PostgreSQL wiki at https://wiki.postgresql.org/wiki/Community_Guide_to_PostgreSQL_GUI_Tool s. The following list contains information on several tools I’ve tried, including free and paid options. The free tools work well for general analysis work. But if you wade deeper into database development, you might want to upgrade to the paid options, which typically offer advanced features and support: Estadísticos e-Books & Papers DataGrip A SQL development environment that offers code completion, bug detection, and suggestions for streamlining code, among many other features. It’s a paid product, but the company, JetBrains, offers discounts and free versions for students, educators, and non​profits (see http://www.jetbrains.com/datagrip/). Navicat A richly featured SQL development environment with versions that support PostgreSQL as well as other databases, including MySQL, Oracle, and Microsoft SQL Server. Navicat is a paid version only, but the company offers a 14-day free trial (see https://www.navicat.com/). pgManage A free, open source GUI client for Windows, macOS, and Linux, formerly known as Postage (see https://github.com/pgManage/pgManage/). Postico A macOS-only client from the maker of Postgres.app that looks like it takes its cues from Apple design. The full version is paid, but a restricted-feature version is available with no time limit (see https://eggerapps.at/postico/). PSequel Also macOS-only, PSequel is a free PostgreSQL client that is decidedly minimalist (see http://www.psequel.com/). A trial version can help you decide whether the product is right for you. PostgreSQL Utilities, Tools, and Extensions You can expand the capabilities of PostgreSQL via numerous third-party utilities, tools, and extensions. These range from additional backup and import/export options to improved formatting for the command line to powerful statistics packages. You’ll find a curated list online at https://github.com/dhamaniasad/awesome-postgres/, but here are several to highlight: Devart Excel Add-In for PostgreSQL An add-in that lets you load Estadísticos e-Books & Papers and edit data from PostgreSQL directly in Excel workbooks (see https://www.devart.com/excel-addins/postgresql.html). MADlib A machine learning and analytics library for large data sets (see http://madlib.apache.org/). pgAgent A job manager that lets you", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 259 + }, + { + "text": "are several to highlight: Devart Excel Add-In for PostgreSQL An add-in that lets you load Estadísticos e-Books & Papers and edit data from PostgreSQL directly in Excel workbooks (see https://www.devart.com/excel-addins/postgresql.html). MADlib A machine learning and analytics library for large data sets (see http://madlib.apache.org/). pgAgent A job manager that lets you run queries at scheduled times, among other tasks (see https://www.pgadmin.org/docs/pgadmin4/dev/pgagent.html). pgcli A replacement for psql that includes improved formatting when writing queries and viewing output (see https://github.com/dbcli/pgcli/). PL/R A loadable procedural language that provides the ability to use the R statistical programming language within PostgreSQL functions and triggers (see http://www.joeconway.com/plr.html). SciPy A collection of Python science and engineering libraries you can use with the PL/Python procedural language in PostgreSQL (see https://www.scipy.org/). PostgreSQL News Now that you’re a bona fide PostgreSQL user, it’s wise to stay on top of community news. The PostgreSQL development team releases new versions of the software on a regular basis, and its ecosystem spawns constant innovation and related products. Updates to PostgreSQL might impact code you’ve written or even offer new opportunities for analysis. Here’s a collection of online resources you can use to stay informed: EDB Blog Posts from the team at EnterpriseDB, a PostgreSQL services company that provides the Windows installer referenced in this book (see https://www.enterprisedb.com/blog/). Planet PostgreSQL A collection of blog posts and announcements from the database community (see https://planet.postgresql.org/). Postgres Weekly An email newsletter that rounds up Estadísticos e-Books & Papers announcements, blog posts, and product announcements (see https://postgresweekly.com/). PostgreSQL Mailing Lists These lists are useful for asking questions of community experts. The pgsql-novice and pgsql-general lists are particularly good for beginners, although note that email volume can be heavy (see https://www.postgresql.org/list/). PostgreSQL News Archive Official news from the Postgres team (see https://www.postgresql.org/about/newsarchive/). PostGIS Blog Announcements and updates on the PostGIS extension covered in Chapter 14 (see http://postgis.net/blog/). Additionally, I recommend paying attention to developer notes for any of the PostgreSQL-related software you use, such as pgAdmin. Documentation Throughout this book, I’ve made frequent reference to pages in the official PostgreSQL documentation. You can find documentation for each version of the software along with an FAQ and wiki on the main page at https://www.postgresql.org/docs/. It’s worth reading through various sections of the manual as you learn more about a particular topic, such as indexes, or search for all the options that come with functions. In particular, the Preface, Tutorial, and SQL Language sections cover much of the material presented in the book’s chapters. Other good resources for documentation are the Postgres Guide at http://postgresguide.com/ and Stack Overflow, where you can find questions and answers posted by developers at https://stackoverflow.com/questions/tagged/postgresql/. You can also check out the Q&A site for PostGIS at https://gis.stackexchange.com/questions/tagged/postgis/. Estadísticos e-Books & Papers INDEX Symbols + (addition operator), 56, 57 & (ampersand operator), 232, 236 * (asterisk) as multiplication operator, 56, 57 as wildcard in SELECT, 12 \\ (backslash), 42–43, 215 escaping characters with, 219 , (comma), 40 ||/ (cube root operator), 56, 58 {} (curly brackets), 215 denoting", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 260 + }, + { + "text": "Estadísticos e-Books & Papers INDEX Symbols + (addition operator), 56, 57 & (ampersand operator), 232, 236 * (asterisk) as multiplication operator, 56, 57 as wildcard in SELECT, 12 \\ (backslash), 42–43, 215 escaping characters with, 219 , (comma), 40 ||/ (cube root operator), 56, 58 {} (curly brackets), 215 denoting an array, 68 <-> (distance operator), 232, 236 @@ (double at sign match operator), 232 :: (double-colon CAST operator), 36 $$ (double-dollar quoting), 280 || (double-pipe concatenation operator), 143, 225 \" (double quote), 41, 94 = (equals comparison operator), 18 ! (exclamation point) as factorial operator, 56, 59 as negation, 228, 232, 236 ^ (exponentiation operator), 56, 58 / (forward slash) as division operator, 56, 57 in macOS file paths, 42 Estadísticos e-Books & Papers > (greater than comparison operator), 18 >= (greater than or equals comparison operator), 18 - (hyphen subtraction operator), 56, 57 < (less than comparison operator), 18 <= (less than or equals comparison operator), 18 != (not equal comparison operator), 18 <> (not equal comparison operator), 18 () (parentheses), 6, 8 to designate order of operations, 20 to specify columns for importing, 50 % (percent sign) as modulo operator, 56, 57 wildcard for pattern matching, 19 | (pipe character) as delimiter, 26, 43 to redirect output, 311 ; (semicolon), 3 ' (single quote), 8, 42 |/ (square root operator), 56, 58 ~* (tilde-asterisk case-insensitive matching operator), 228 ~ (tilde case-sensitive matching operator), 228 _ (underscore wildcard for pattern matching), 19 A adding numbers, 57 across columns, 60 addition operator (+), 56, 57 aggregate functions, 64, 117 avg(), 64 Estadísticos e-Books & Papers binary (two-input), 158 count(), 117–119, 131 filtering with HAVING, 127 interviewing data, 131 max(), 119–120 min(), 119–120 PostgreSQL documentation, 117 sum(), 64, 124–125 using GROUP BY clause, 120–123 aliases for table names, 86, 125 ALTER COLUMN statement, 107 ALTER TABLE statement, 137 ADD COLUMN, 137, 252 ADD CONSTRAINT, 107 ALTER COLUMN, 137 DROP COLUMN, 137, 148 table constraints, adding and removing, 107 American National Standards Institute (ANSI), xxiv ampersand operator (&), 232, 236 ANALYZE keyword with EXPLAIN command, 109 with VACUUM command, 317 AND operator, 20 ANSI (American National Standards Institute), xxiv antimeridian, 46 array, 68 array_length() function, 230 functions, 68 notation in query, 224 passing into ST_MakePoint(), 250 returned from regexp_match(), 219, 224 Estadísticos e-Books & Papers type indicated in results grid, 224 unnest() function, 68 with curly brackets, 68, 220 array_length() function, 230 AS keyword declaring table aliases with, 86, 90 renaming columns in query results with, 60, 61, 205 ASC keyword, 15 asterisk (*) as multiplication operator, 56, 57 as wildcard in SELECT statement, 12 attribute, 5 auto-incrementing integers, 27 as surrogate primary key, 101 gaps in sequence, 28 identity column SQL standard, 27 autovacuum, 316 editing server setting, 319 time of last vacuum, 317 average, 64 vs. median, 65, 194 avg() function, 64, 195 B backslash (\\), 42–43, 215 escaping characters with, 219 backups column, 140 improving performance when updating tables, 151–152 restoring from copied table, 142 Estadísticos e-Books &", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 261 + }, + { + "text": "SQL standard, 27 autovacuum, 316 editing server setting, 319 time of last vacuum, 317 average, 64 vs. median, 65, 194 avg() function, 64, 195 B backslash (\\), 42–43, 215 escaping characters with, 219 backups column, 140 improving performance when updating tables, 151–152 restoring from copied table, 142 Estadísticos e-Books & Papers tables, 139 BETWEEN comparison operator, 18, 198 inclusive property, 19 bigint integer data type, 27 bigserial integer data type, 6, 27, 101 as surrogate primary key, 102 binary aggregate functions, 158 BINARY file format, 42 birth data, U.S., 330 Boolean value, 74 B-Tree (balanced tree) index, 108 C camel case, 10, 94 caret symbol (^) exponentiation operator, 58 carriage return, 43 Cartesian Product as result of CROSS JOIN, 82 CASCADE keyword, 104 case sensitivity with ILIKE operator, 19 with LIKE operator, 19 CASE statement, 207 ELSE clause, 208 in Common Table Expression, 209–210 in UPDATE statement, 226 syntax, 207 WHEN clause, 208, 288 with trigger, 286 Estadísticos e-Books & Papers CAST() function, 35 shortcut notation, 36 categorizing data, 207 char character string type, 24 character set, 16 character string types, 24–26 char, 24 functional difference from number types, 26 performance in PostgreSQL, 25 text, 25 varchar, 24 character varying data type. See varchar data type char_length() function, 212 CHECK constraint, 104–105 classify_max_temp() user function, 287 clock_timestamp() function, 176 Codd, Edgar F., xxiv, 73 coefficient of determination. See r-squared collation setting, 16 column, 5 adding numbers in, 64 alias, 60 alter data type, 137 averaging values in, 64 avoiding spaces in name, 95 deleting, 148 indexes, 110 naming, 94 populating new during backup, 151 retrieving in queries, 13 updating values, 138 Estadísticos e-Books & Papers comma (,), 40 comma-delimited files. See CSV (comma-separated values) command line, 291 advantages of using, 292 createdb command, 310 psql application, 299 setup, 292 macOS, 296 PATH environment variable, 292, 296 Windows, 292 shell programs, 296 comma-separated values (CSV). See CSV comments in code, xxvii COMMIT statement, 149 Common Table Expression (CTE), 200 advantages, 201 CASE statement example, 209 definition, 200 comparison operators, 18 combining with AND and OR, 20 concatenation, 143 conditional expression, 207 constraints, 6, 96–97 adding and removing, 107 CHECK, 104–105, 157 column vs. table, 97 CONSTRAINT keyword, 76 foreign key, 102–103 NOT NULL, 106–107 PRIMARY KEY, 99 primary keys, 75, 97 Estadísticos e-Books & Papers UNIQUE, 76, 105–106 violations when altering table, 138 constructor, 68 Coordinated Universal Time (UTC), 33 COPY statement DELIMITER option, 43 description of, 39 exporting data, 25, 51–52 FORMAT option, 42 FROM keyword, 42 HEADER option, 43 importing data, 42–43 naming file paths, 25 QUOTE option, 43 specifying file formats, 42 TO, 51, 183 WITH keyword, 42 correlated subquery, 192, 199 corr() function, 157 correlation vs. causation, 163 count() function, 117, 131, 196 distinct values, 118 on multiple columns, 123 values present in a column, 118 with GROUP BY, 122 counting distinct values, 118 missing values displayed, 133 rows, 117 using pgAdmin, 118 CREATE DATABASE statement, 3 Estadísticos e-Books & Papers createdb utility, 310 CREATE EXTENSION statement,", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 262 + }, + { + "text": "163 count() function, 117, 131, 196 distinct values, 118 on multiple columns, 123 values present in a column, 118 with GROUP BY, 122 counting distinct values, 118 missing values displayed, 133 rows, 117 using pgAdmin, 118 CREATE DATABASE statement, 3 Estadísticos e-Books & Papers createdb utility, 310 CREATE EXTENSION statement, 203 CREATE FUNCTION statement, 276 CREATE INDEX statement, 108, 110 CREATE TABLE statement, 6 backing up a table with, 139 declaring data types, 24 TEMPORARY TABLE, 50 CREATE TRIGGER statement, 285 CREATE VIEW statement, 269 CROSS JOIN keywords, 82, 202 crosstab() function, 203, 205, 207 with tablefunc module, 203 cross tabulations, 203 CSV (comma-separated values), 40 header row, 41 CTE. See Common Table Expression (CTE) cube root operator (||/), 58 curly brackets ({}), 215 denoting an array, 68 current_date function, 175 current_time function, 175 current_timestamp function, 176 cut points, 66 D data identifying and telling stories in, 325 spatial, 241 Estadísticos e-Books & Papers structured and unstructured, 211 database backup and restore, 321 connecting to, 4, 5 create from command line, 310 creation, 1, 3–5 importing data with COPY, 42–43 maintenance, 313 server, 3 using consistent names, 94 database management system, 3 data dictionary, 23 data types, 5, 23 bigint, 27 bigserial, 6, 101 char, 24 character string types, 24–26 date, 5, 32, 172 date and time types, 32–34 decimal, 29 declaring with CREATE TABLE, 24 double precision, 29 full text search, 231 geography, 247 geometry, 247 importance of using appropriate type, 23, 46 integer, 27 interval, 32, 172 modifying with ALTER COLUMN, 137 number types, 26–31 numeric, 6, 28 Estadísticos e-Books & Papers real, 29 returned by math operations, 56 serial, 12, 101 smallint, 27 smallserial, 101 text, 25 time, 32, 172 timestamp, 32, 172 transforming values with CAST(), 35–36 tsquery, 232 tsvector, 231 varchar, 6, 24 date data types date, 5, 32, 172 interval, 32, 172 matching with regular expression, 217 date_part() function, 173, 207 dates input format, 5, 8, 33, 173 setting default style, 320 daylight saving time, 178 deciles, 67 decimal data types, 28 decimal, 29 double precision, 29 numeric, 28 real, 29 decimal degrees, 46 DELETE statement, 50 removing rows matching criteria, 147 with subquery, 194 Estadísticos e-Books & Papers DELETE CASCADE statement with foreign key constraint, 104 delimited text files, 39, 40–41 delimiter character, 40 DELIMITER keyword with COPY statement, 43 dense_rank() function, 164 derived table, 194 joining, 195–197 DESC keyword, 15 direct relationship in correlation, 158 dirty data, 11, 129 cleaning, 129 foreign keys help to avoid, 103 when to discard, 137 distance operator (<->), 232, 236 DISTINCT keyword, 14, 118 division, 57 finding the remainder, 58 integer vs. decimal, 57, 58 documenting code, 23 double at sign match operator (@@), 232 double-colon CAST operator (::), 36 double-dollar quoting ($$), 280 double-pipe concatenation operator (||), 143, 225 double quote (\"), 41, 94 DROP statement COLUMN, 148 INDEX, 111 TABLE, 148 Estadísticos e-Books & Papers duplicate data created by spelling variations, 132 guarding against with constraints, 76 E Eastern Standard Time (EST), 33 ELSE", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 263 + }, + { + "text": "CAST operator (::), 36 double-dollar quoting ($$), 280 double-pipe concatenation operator (||), 143, 225 double quote (\"), 41, 94 DROP statement COLUMN, 148 INDEX, 111 TABLE, 148 Estadísticos e-Books & Papers duplicate data created by spelling variations, 132 guarding against with constraints, 76 E Eastern Standard Time (EST), 33 ELSE clause, 208, 227 entity, 2 environment variable, 292 epoch, 174, 189 equals comparison operator (=), 18 error messages, 9 CSV import failure, 47, 49 foreign key violation, 103 out of range value, 27 primary key violation, 99, 101 relation already exists, 95 UNIQUE constraint violation, 106 when using CAST(), 36 escaping characters, 219 EST (Eastern Standard Time), 33 exclamation point (!) as factorial operator, 56, 59 as negation, 228, 232, 236 EXISTS operator in WHERE clause, 139 with subquery, 199 EXPLAIN statement, 109 exponentiation operator (^), 56, 58 exporting data Estadísticos e-Books & Papers all data in table, 51–52 from query results, 52 including header row, 43 limiting columns, 52 to BINARY file format, 42 to CSV file format, 42, 183–184 to TEXT file format, 42 using command line, 307 using COPY statement, 51–52 using pgAdmin wizard, 52–53 expressions, 34, 192 conditional, 207 subquery, 198 extract() function, 174 F factorials, 58 false (Boolean value), 74 Federal Information Processing Standards (FIPS), 259, 269 field, 5 file paths import and export file locations, 42 naming conventions for operating systems, 25, 42 filtering rows HAVING clause, 127 WHERE clause, 17, 192 with subquery, 192 findstr Windows command, 134 FIPS (Federal Information Processing Standards), 259, 269 fixed-point numbers, 28 Estadísticos e-Books & Papers floating-point numbers, 29 inexact math calculations, 30 foreign key creating with REFERENCES keyword, 102 definition, 76, 102 formatting SQL for readability, 10 forward slash (/) as division operator, 56, 57 in macOS file paths, 42 FROM keyword, 12 with COPY, 42 FULL OUTER JOIN keywords, 82 full text search, 231 adjacent words, locating, 236–237 data types, 231–233 functions to rank results, 237–239 highlighting terms, 235 lexemes, 231–232 multiple terms in query, 236 querying, 234 setting default language, 320 table and column setup, 233 to_tsquery() function, 232 to_tsvector() function, 231 ts_headline() function, 235 ts_rank_cd() function, 237 ts_rank() function, 237 using GIN index, 234 functions, 267 creating, 275, 276–277 full text search, 231 Estadísticos e-Books & Papers IMMUTABLE keyword, 277 RAISE NOTICE keywords, 280 RETURNS keyword, 277 specifying language, 276 string, 212 structure of, 276 updating data with, 278–280 G generate_series() function, 176, 207, 315 geography data type, 247 GeoJSON, 243 geometry data type, 247 GIN (Generalized Inverted Index), 108 with full text search, 234 GIS (Geographic Information System), 241 decimal degrees, 46 GiST (Generalized Search Tree) index, 108, 252 greater than comparison operator (>), 18 greater than or equals comparison operator (>=), 18 grep Linux command, 134 GROUP BY clause eliminating duplicate values, 120 on multiple columns, 121 with aggregate functions, 120 GUI (graphical user interface), 257, 291 list of tools, 333 H Estadísticos e-Books & Papers HAVING clause, 127 with aggregate functions, 127, 132 HEADER keyword with COPY statement, 43 header", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 264 + }, + { + "text": "18 grep Linux command, 134 GROUP BY clause eliminating duplicate values, 120 on multiple columns, 121 with aggregate functions, 120 GUI (graphical user interface), 257, 291 list of tools, 333 H Estadísticos e-Books & Papers HAVING clause, 127 with aggregate functions, 127, 132 HEADER keyword with COPY statement, 43 header row found in CSV file, 41 ignoring during import, 41 hyphen subtraction operator (-), 56, 57 I identifiers avoiding reserved keywords, 95 enabling mixed case, 94–95 naming, 10, 94, 96 quoting, 95 identifying and telling stories in data, 325 asking why, 331 assessing the data’s origins, 328 building your own database, 327 communicating your findings, 331 consulting the data’s owner, 328 documenting your process, 326 gathering your data, 326 identifying trends over time, 329 interviewing the data with queries, 328 starting with a question, 326 ILIKE comparison operator, 18, 19–20 importing data, 39, 42–43 adding default column value, 50 choosing a subset of columns, 49 Estadísticos e-Books & Papers from non-text sources, 40 from TEXT file format, 42 from CSV file format, 42 ignoring header row in text files, 41, 43 using command line, 307 using COPY statement, 39 using pgAdmin import wizard, 52–53 IN comparison operator, 18, 144, 198 with subquery, 198 indexes, 108 B-Tree, 108 considerations before adding, 111 creating on columns, 110 dropping, 111 GIN, 108 GiST, 108, 252 measuring effect on performance, 109 not included with table backups, 140 syntax for creating, 108 initcap() function, 212 INSERT statement, 8–9 inserting rows into a table, 9–10 Institute of Museum and Library Services (IMLS), 114 integer data types, 27 auto-incrementing, 27 basic math operations, 57 bigint, 27 bigserial, 27 difference in integer type capacities, 27 integer, 27 serial, 27 Estadísticos e-Books & Papers smallint, 27 smallserial, 27 International Date Line, 46 International Organization for Standardization (ISO), xxiv, 33, 243 interval data type, 32, 172 calculations with, 34, 187 cumulative, 188 value options, 34 interviewing data, 11, 131–132 across joined tables, 124 artificial values as indicators, 120, 124 checking for missing values, 13, 132–134 correlations, 157–159 counting rows and values, 117–119 determining correct format, 13 finding inconsistent values, 134 malformed values, 135–136 maximum and minimum values, 119–120 rankings, 164–167 rates calculations, 167–169 statistics, 155 summing grouped values, 124 unique combinations of values, 15 inverse relationship, 158 ISO (International Organization for Standardization), xxiv, 33, 243 time format, 172 J JOIN keyword, 74 example of using, 80 Estadísticos e-Books & Papers in FROM clause, 74 joining tables, 73 derived tables, 195–197 inequality condition, 90 multiple-table joins, 87 naming tables in column list, 85, 125 performing calculations across tables, 88 spatial joins, 262, 263 specifying columns to link tables, 77 specifying columns to query, 85 using JOIN keyword, 74, 77 join types CROSS JOIN, 82–83 FULL OUTER JOIN, 82 JOIN (INNER JOIN), 80, 125 LEFT JOIN, 80–81 list of, 78 RIGHT JOIN, 80–81 JSON, 35 K key columns foreign key, 76 primary key, 75 relating tables with, 74 L latitude in U.S. Census data, 46 Estadísticos e-Books & Papers in well-known text,", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 265 + }, + { + "text": "CROSS JOIN, 82–83 FULL OUTER JOIN, 82 JOIN (INNER JOIN), 80, 125 LEFT JOIN, 80–81 list of, 78 RIGHT JOIN, 80–81 JSON, 35 K key columns foreign key, 76 primary key, 75 relating tables with, 74 L latitude in U.S. Census data, 46 Estadísticos e-Books & Papers in well-known text, 245 least squares regression line, 161 LEFT JOIN keyword, 80–81 left() string function, 213 length() string function, 135, 213 less than comparison operator (<), 18 less than or equals comparison operator (<=), 18 lexemes, 231 LIKE comparison operator, 18 case-sensitive search, 19 in UPDATE statement, 143 LIMIT clause, 48 limiting number of rows query returns, 48 linear regression, 161 linear relationship, 158 Linux file path declaration, 26, 42 Terminal setup, 299 literals, 8 locale setting, 16 localhost, xxxii, 4 localtime function, 176 localtimestamp function, 176 longitude in U.S. Census data, 46 in well-known text, 245 positive and negative values, 49 lower() function, 212 Estadísticos e-Books & Papers M macOS file path declaration, 25, 42 Terminal, 296 .bash_profile, 296 bash shell, 296 entering instructions, 297 setup, 296, 297 useful commands, 298 make_date() function, 175 make_time() function, 175 make_timestamptz() function, 175 many-to-many table relationship, 85 map projected coordinate system, 245 projection, 245 math across joined table columns, 88 across table columns, 60–64 median, 65–70 mode, 70 order of operations, 59 with aggregate functions, 64–65 math operators, 56–59 addition (+), 57 cube root (||/), 58 division (/), 57 exponentiation (^), 58 factorial (!), 58 modulo (%), 57 Estadísticos e-Books & Papers multiplication (*), 57 square root (|/), 58 subtraction (-), 57 max() function, 119 median, 65 definition, 65 vs. average, 65, 194 with percentile_cont() function, 66 median() user function creation, 69 performance concerns, 70 vs. percentile_cont(), 70 Microsoft Access, xxiv Microsoft Excel, xxiv Microsoft SQL Server, xxviii, 94, 203 Microsoft Windows Command Prompt entering instructions, 295 setup, 292, 294 useful commands, 295 file path declaration, 25, 42 folder permissions, xxvii min() function, 119 mode, 70 mode() function, 70 modifying data, 136–137 for consistency, 142 updating column values, 141 modulo operator (%), 56, 57–58 multiplying numbers, 57 MySQL, xxviii Estadísticos e-Books & Papers N naming conventions camel case, 94 Pascal case, 94 snake case, 94, 96 National Center for Education Statistics, 327 National Center for Health Statistics, 330 natural primary key, 97, 131 New York City taxi data, 180 calculating busiest hour of day, 182 creating table for, 180 exporting results, 183–184 importing, 181 longest trips, 184–185 normal distribution of data, 194 NOT comparison operator, 18 with EXISTS, 200 not equal comparison operator != syntax, 18 <> syntax, 18 NOT NULL keywords adding to column, 137 definition, 106 removing from column, 107, 138 now() function, 33, 176 NULL keyword definition, 83 ordering with FIRST and LAST, 133 using in table joins, 83 number data types, 26 Estadísticos e-Books & Papers decimal types, 28 double precision, 29 fixed-point type, 28 floating-point types, 29 numeric data type, 6, 28 real, 29 integer types, 27 bigint, 27 integer, 27 serial types, 27 smallint, 27 usage considerations, 31", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 266 + }, + { + "text": "and LAST, 133 using in table joins, 83 number data types, 26 Estadísticos e-Books & Papers decimal types, 28 double precision, 29 fixed-point type, 28 floating-point types, 29 numeric data type, 6, 28 real, 29 integer types, 27 bigint, 27 integer, 27 serial types, 27 smallint, 27 usage considerations, 31 O OGC (Open Geospatial Consortium), 243 ON keyword used with DELETE CASCADE, 104 used with JOIN, 74 one-to-many table relationship, 84 one-to-one table relationship, 84 operators addition (+), 56, 57 comparisons with, 17 cube root (||/), 56, 58 division (/), 56, 57 exponentiation (^), 56, 58 factorial (!), 56, 58 modulo (%), 56, 57 multiplication (*), 56, 57 precedence, 59 Estadísticos e-Books & Papers prefix, 58 square root (|/), 56, 58 subtraction (-), 56, 57 suffix, 59 OR operator, 20 Oracle, xxiv ORDER BY clause, 15 ASC, DESC options, 15 on multiple columns, 16 specifying columns to sort, 15 specifying NULLS FIRST or LAST, 133 OVER clause, 164 P Pacific time zone, 33 padding character columns with spaces, 24, 26 parentheses (), 6, 8 to designate order of operations, 20 to specify columns for importing, 50 Pascal case, 94 pattern matching using LIKE and ILIKE, 19 with regular expressions, 214 with wildcards, 19 Pearson correlation coefficient (r), 157 percent sign (%) as modulo operator, 56, 57 wildcard for pattern matching, 19 percentage of the whole, 62 Estadísticos e-Books & Papers percent change, 63 formula, 63, 89, 276 function, 276 percent_change() user function, 276 using with Census data, 277 percentile, 66, 192 continuous vs. discrete values, 66 percentile_cont() function, 66 finding median with, 185 in subquery, 193 using array to enter multiple values, 68 percentile_disc() function, 66 pgAdmin, xxxi connecting to database, 4, 5, 242 connecting to server, xxxii, 4 executing SQL, 3 importing and exporting data, 52–53 installation Linux, xxxi macOS, xxxi, xxxii Windows, xxix, xxxi keyword highlighting, 95 localhost, xxxii, 4 object browser, xxxii, 5, 7 Query Tool, xxxiii, 4, 243 text display in results grid, 218 viewing data, 9, 75, 118 viewing tables, 45 views, 269 pg_ctl utility, 321 pg_dump utility, 321 Estadísticos e-Books & Papers pg_restore utility, 322 pg_size_pretty() function, 315 pg_total_relation_size() function, 315 pipe character (|) as delimiter, 26, 43 to redirect output, 311 pivot table. See cross tabulations PL/pgSQL, 276, 279 BEGIN ... END block, 280, 284 IF ... THEN statement, 284 PL/Python, 281 point, 46 position() string function, 213 PostGIS, xxviii, 242 creating spatial database, 242–243 creating spatial objects, 247 data types, 247 geography, 247 geometry, 247 displaying version, 243 functions ST_AsText(), 260 ST_DFullyWithin(), 254 ST_Distance(), 254 ST_DWithin(), 253 ST_GeogFromText(), 248, 254 ST_GeometryType(), 262 ST_GeomFromText(), 247 ST_Intersection(), 264 ST_Intersects(), 263 ST_LineFromText(), 250 Estadísticos e-Books & Papers ST_MakeLine(), 250 ST_MakePoint(), 249 ST_MakePolygon(), 250 ST_MPolyFromText(), 250 ST_PointFromText(), 249 ST_PolygonFromText(), 250 installation, 242–243 Linux, xxxi macOS, xxx troubleshooting, xxx Windows, xxix–xxx loading extension, 243 shapefile loading, 257, 258, 311 querying, 259 spatial joins, 262, 263 Postgres.app, xxx–xxxi, 4 PostgreSQL advantages of using, xxviii backup and restore, 321 pg_dump, 321 pg_restore, 322 collation setting, 16 command line usage,", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 267 + }, + { + "text": "ST_PointFromText(), 249 ST_PolygonFromText(), 250 installation, 242–243 Linux, xxxi macOS, xxx troubleshooting, xxx Windows, xxix–xxx loading extension, 243 shapefile loading, 257, 258, 311 querying, 259 spatial joins, 262, 263 Postgres.app, xxx–xxxi, 4 PostgreSQL advantages of using, xxviii backup and restore, 321 pg_dump, 321 pg_restore, 322 collation setting, 16 command line usage, 291 comparison operators, 18 configuration, 313 creating functions, 275 default postgres database, 3 description of, 3 documentation, 335 functions, 267 Estadísticos e-Books & Papers GUI tools, 333 importing from other database managers, 40 installation, xxviii Linux, xxxi macOS, xxx–xxxi troubleshooting, xxx Windows, xxix–xxx locale setting, xxix, 16 maintenance, 313 news websites, 335 postgresql.conf settings file, 319 recovering unused space, 314 settings, 318 spatial data analysis, 241, 253, 254 starting and stopping, 321 statistics collector, 317 table size, 314 triggers, 267, 282 utilities, tools, and extensions, 334 views, 267 postgresql.conf settings file, 178, 319 editing, 319 reloading settings, 321 precision argument with numeric and decimal types, 28 primary key, 2, 12 composite, 100–101 definition of, 75, 97 natural, 97, 131 surrogate, 97, 98 auto-incrementing, 101–102 Estadísticos e-Books & Papers creating, 102 data types for, 101 syntax, 98–100 uniqueness, 76 using auto-incrementing serial type, 28 using Universally Unique Identifier, 98 violation, 99, 101 Prime Meridian, 46, 246 procedural language, 276 projection (map), 245 Albers, 246 Mercator, 245 psql command line application, 3, 292 connecting to database, 299, 300 displaying table info, 306 editing queries, 303 executing queries from a file, 309 formatting results, 303, 304 help commands, 300 importing and exporting files, 307 meta-commands, 306 multiline queries, 302 paging results, 303 parentheses in queries, 302 running queries, 301 saving query output, 308 setup Linux, 299 macOS, 296–298 Microsoft Windows, 293–295 superuser prompt, 300 Estadísticos e-Books & Papers Public Libraries Survey, 114 Python programming language, xxv, 335 creating PL/Python extension, 281 in PostgreSQL function, 277, 281 Q quantiles, 66 quartiles, 67 query choosing order of columns, 13 definition, 1 eliminating duplicate values, 14 execution time, 109–110 exporting results of, 52 limiting number of rows returned, 48 measuring performance with EXPLAIN, 109 order of clauses, 21 retrieving a subset of columns, 13 selecting all rows and columns, 12 quintiles, 67 quotes, single vs. double, 8 R rank() function, 164 ranking data, 164 by subgroup, 165–167 rank() and dense_rank() functions, 164–165 rates calculations, 167, 196 record_if_grade_changed() user function, 284 Estadísticos e-Books & Papers REFERENCES keyword, 103 referential integrity, 97 cascading deletes, 104 foreign keys, 102 primary key, 99 regexp_match() function, 219 extracting text from result, 224 regexp_matches() function, 220 regexp_replace() function, 230 regexp_split_to_array() function, 230 regexp_split_to_table() function, 230 regr_intercept() function, 162 regr_r2() function, 163 regr_slope() function, 162 regular expressions, 214 capture group, 215, 221 escaping characters, 219 examples, 216 in WHERE clause, 228–229 notation, 214–216 parsing unstructured data, 216, 222 regexp_match() function, 219 regexp_matches() function, 220 regexp_replace() function, 230 regexp_split_to_array() function, 230 regexp_split_to_table() function, 230 with substring() function, 216 relational databases, 2, 73 join types CROSS JOIN, 82–83 FULL OUTER JOIN, 82 Estadísticos e-Books & Papers JOIN (INNER JOIN), 80, 125 LEFT JOIN, 80–81 list", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 268 + }, + { + "text": "parsing unstructured data, 216, 222 regexp_match() function, 219 regexp_matches() function, 220 regexp_replace() function, 230 regexp_split_to_array() function, 230 regexp_split_to_table() function, 230 with substring() function, 216 relational databases, 2, 73 join types CROSS JOIN, 82–83 FULL OUTER JOIN, 82 Estadísticos e-Books & Papers JOIN (INNER JOIN), 80, 125 LEFT JOIN, 80–81 list of, 78 RIGHT JOIN, 80–81 querying, 77 relating tables, 74–77 relational model, 73, 84 reducing redundant data, 77 table relationships many-to-many, 85 one-to-many, 84 one-to-one, 84 replace() string function, 214 reserved keywords, 95 RIGHT JOIN keywords, 80–81 right() string function, 213 ROLLBACK statement, 149 roots, square and cube, 58 round() function, 64, 160 row counting, 117 definition, 73 deleting, 147–148 in a CSV file, 40 inserting, 8 recovering unused, 314 updating specific, 141 r (Pearson correlation coefficient), 157 r-squared, 163 R programming language, xxv Estadísticos e-Books & Papers S scalar subquery, 192 scale argument with numeric and decimal types, 29 scatterplot, 158, 159 search. See full text search SELECT statement definition, 11 order of clauses, 21 syntax, 12 with DISTINCT keyword, 14–15 with GROUP BY clause, 120 with ORDER BY clause, 15–17 with WHERE clause, 17–20 selecting all rows and columns, 12 semicolon (;), 3 serial, 27, 101 server connecting, 4 localhost, 4 postgresql.conf file, 178 setting time zone, 178 SET keyword clause in UPDATE, 138, 192 timezone, 178 shapefile, 256 contents of, 256–257 loading into database, 257 shp2pgsql command line utility, 311 U.S. Census TIGER/Line, 258, 262 Estadísticos e-Books & Papers SHOW command config_file, 319 data_directory, 321 timezone, 177 shp2pgsql command line utility, 311 significance testing, 163 simple feature standard, 243 single quote ('), 8, 42 slope-intercept formula, 161 smallint data type, 27 smallserial data type, 27, 101 snake case, 10, 94, 96 sorting data, 15 by multiple columns, 16 dependent on locale setting, 16 on aggregate results, 123 spatial data, 241 area analysis, 260 building blocks, 243 distance analysis, 253, 254 finding location, 261 geographic coordinate system, 243, 245, 246 geometries, 243 constructing, 245, 247 LineString, 243, 249–250 MultiLineString, 244 MultiPoint, 244 MultiPolygon, 244 Point, 243, 249 Polygon, 243, 250 intersection analysis, 264 Estadísticos e-Books & Papers joins, 262, 263 projected coordinate system, 245 projection, 245 shapefile, 256 simple feature standard, 243 Spatial Reference System Identifier (SRID), 244, 246 well-known text (WKT), 244 WGS 84 coordinate system, 246 Spatial Reference System Identifier (SRID), 244, 246 setting with ST_SetSRID(), 252 SQL comments in code, xxvii history of, xxiv indenting code, 10 math operators, 56 relational model, 73 reserved keywords, 95 standards, xxiv statistical functions, 155 style conventions, 6, 10, 36, 94 using with external programming languages, xxv value of using, xxiv square root operator (|/), 56, 58 SRID (Spatial Reference System Identifier), 244, 246 setting with ST_SetSRID(), 252 statistical functions, 155 correlation with corr(), 157–159 dependent and independent variables, 158 linear regression, 160 regr_intercept() function, 162 regr_r2() function, 163 Estadísticos e-Books & Papers regr_slope() function, 162 rates calculations, 167 string functions, 135, 212 case formatting, 212 character information, 212 char_length(), 212 extracting and replacing characters, 213 initcap(), 212 left(),", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 269 + }, + { + "text": "functions, 155 correlation with corr(), 157–159 dependent and independent variables, 158 linear regression, 160 regr_intercept() function, 162 regr_r2() function, 163 Estadísticos e-Books & Papers regr_slope() function, 162 rates calculations, 167 string functions, 135, 212 case formatting, 212 character information, 212 char_length(), 212 extracting and replacing characters, 213 initcap(), 212 left(), 213 length(), 135, 213 lower(), 212 position(), 213 removing characters, 213 replace(), 214 right(), 213 to_char(), 187 trim(), 213 upper(), 212 subquery correlated, 192, 199 definition, 192 expressions, 198 generating column with, 197–198 in DELETE statement, 194 in FROM clause, 194 IN operator expression, 198–199 in UPDATE statement, 139, 192 in WHERE clause, 192–194 scalar, 192 uncorrelated, 192 with crosstab() function, 205 Estadísticos e-Books & Papers substring() function, 216 subtracting numbers, 57 across columns, 60 sum() function, 64 example on joined tables, 124 grouping by column value, 125 summarizing data, 113 surrogate primary key, 98 creating, 102 T tab character as delimiter, 42–43 as regular expression, 215 table add column, 137, 140 aliases, 86, 195 alter column, 137 autovacuum, 316 backup, 94 constraints, 6 creation, 5–7 definition of, 1 deleting columns, 137, 148 deleting data, 147–149 deleting from database, 148–149 derived table, 194 design best practices, 93 dropping, 148 holds data on one entity, 73 Estadísticos e-Books & Papers indexes, 108 inserting rows, 8–9 key columns, 74 modifying with ALTER statement, 137–138 naming, 94, 96 querying multiple tables using joins, 77 relationships, 1 size, 314 temporary tables, 50 viewing data, 9 tablefunc module, 203 table relationships many-to-many, 85 one-to-many, 84 one-to-one, 84 temporary table declaring, 50 removing with DROP TABLE, 51 text data types, 24–26 char, 24 text, 25 varchar, 6, 24 text operations case formatting, 212 concatenation, 143 escaping characters, 219 extracting and replacing characters, 213–214 formatting as timestamp, 173 formatting with functions, 212–214 matching patterns with regular expressions, 214 removing characters, 213 Estadísticos e-Books & Papers sorting, 16 text files, delimited. See delimited text files text qualifier ignoring delimiters with, 41 specifying with QUOTE option in COPY, 43 tilde-asterisk case-insensitive matching operator (~*), 228 tilde case-sensitive matching operator (~), 228 time data types interval, 32, 172 matching with regular expression, 215 time, 32, 172 timestamp, 32, 172 timestamp, 32, 172 calculations with, 180 creating from components, 174–175, 225 extracting components from, 173–174 finding current date and time, 175–176 formatting display, 187 subtracting to find interval, 187 timestamptz shorthand, 172 with time zone, 32, 172 within transactions, 176 time zones AT TIME ZONE keywords, 179 automatic conversion of, 173, 175 finding server setting, 177–178 including in timestamp, 32, 173, 226 setting, 178–180 setting server default, 320 standard name database, 33 viewing names of, 177 Estadísticos e-Books & Papers working with, 177 to_char() function, 187 to_tsquery() function, 232 to_tsvector() function, 231 transaction blocks, 149–151 COMMIT, 149 definition, 149 ROLLBACK, 149 START TRANSACTION, 149 visibility to other users, 151 transactions, 149 with time functions, 176 triggers, 267, 282 BEFORE INSERT statement, 288 CREATE TRIGGER statement, 285 FOR EACH ROW statement, 285 FOR EACH STATEMENT statement, 285 NEW and OLD variables, 284", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 270 + }, + { + "text": "231 transaction blocks, 149–151 COMMIT, 149 definition, 149 ROLLBACK, 149 START TRANSACTION, 149 visibility to other users, 151 transactions, 149 with time functions, 176 triggers, 267, 282 BEFORE INSERT statement, 288 CREATE TRIGGER statement, 285 FOR EACH ROW statement, 285 FOR EACH STATEMENT statement, 285 NEW and OLD variables, 284 RETURN statement, 285 testing, 285, 288 trim_county() user function, 281 trim() function, 213 true (Boolean value), 74 ts_headline() function, 235 tsquery data type, 232 ts_rank_cd() function, 237 ts_rank() function, 237 tsvector data type, 231 U Estadísticos e-Books & Papers uncorrelated subquery, 192 underscore wildcard for pattern matching (_), 19 UNIQUE constraint, 76, 105–106 Universally Unique Identifier (UUID), 35, 98 unnest() function, 68 unstructured data, 211 parsing with regular expressions, 216, 222 UPDATE statement definition, 138 PostgreSQL syntax, 139 SET clause, 138 using across tables, 138, 145, 192 with CASE statement, 226 update_personal_days() user function, 279 upper() function, 212 USA TODAY, xxiii U.S. Census 2010 Decennial Census data, 43 calculating population change, 89 county shapefile analysis, 259 description of columns, 45–47 finding total population, 64 importing data, 43–44 racial categories, 60 short form, 60 2011–2015 American Community Survey description of columns, 156 estimates and margin of error, 157 importing data, 156 apportionment of U.S. House of Representatives, 44 methodologies compared, 157, 328 Estadísticos e-Books & Papers U.S. Department of Agriculture, 130 farmers’ market data, 250 U.S. Federal Bureau of Investigation (FBI) crime report data, 167 UTC (Coordinated Universal Time), 33, 174 UTC offset, 33, 179, 187 UTF-8, 16 UUID (Universally Unique Identifier), 35, 98 V VACUUM command, 314 ANALYZE option, 317 autovacuum process, 316 editing server setting, 319 FULL option, 318 monitoring table size, 314 pg_stat_all_tables view, 317 running manually, 318 time of last vacuum, 317 VERBOSE option, 318 VALUES clause with INSERT, 8 varchar data type, 6, 24 views, 267 advantage of using, 268 creating, 269–271 deleting data with, 275 dropping, 269 inserting data with, 273–274 inserting, updating, deleting data, 271 LOCAL CHECK OPTION, 272, 273 materialized, 268 Estadísticos e-Books & Papers pg_stat_all_tables, 317 queries in, 269 retrieving specific columns, 271 updating data with, 274 W well-known text (WKT), 244 extended, 248 order of coordinates, 245 WHEN clause, 208 in CASE statement, 227 WHERE clause, 17 in UPDATE statement, 138 filtering rows with, 17–19 with DELETE FROM statement, 147 with EXISTS clause, 139, 192 with ILIKE operator, 19–20 with IS NULL keywords, 133 with LIKE operator, 19–20, 143 with regular expressions, 228 whole numbers, 27 wildcard asterisk (*) in SELECT statement, 12 percent sign (%), 19 underscore (_), 19 window functions definition of, 164 OVER clause, 164 PARTITION BY clause, 165 WITH Estadísticos e-Books & Papers as Common Table Expression, 200 options with COPY, 42 WKT (well-known text), 244 extended, 248 order of coordinates, 245 working tables, 148 X XML, 35 Z ZIP Codes, 135 loss of leading zeros, 135 repairing botched, 143 Estadísticos e-Books & Papers Practical SQL is set in New Baskerville, Futura, Dogma, and​- TheSansMono Condensed. Estadísticos e-Books & Papers RESOURCES Visit https://www.nostarch.com/practicalSQL/ for resources, errata, and", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 271 + }, + { + "text": "248 order of coordinates, 245 working tables, 148 X XML, 35 Z ZIP Codes, 135 loss of leading zeros, 135 repairing botched, 143 Estadísticos e-Books & Papers Practical SQL is set in New Baskerville, Futura, Dogma, and​- TheSansMono Condensed. Estadísticos e-Books & Papers RESOURCES Visit https://www.nostarch.com/practicalSQL/ for resources, errata, and more information. More no-nonsense books from NO STARCH PRESS THE BOOK OF R A First Course in Programming and Statistics by TILMAN M. DAVIES JULY 2016, 832 pp., $49.95 ISBN 978-1-59327-651-5 color insert Estadísticos e-Books & Papers DATA VISU ALIZATION WITH JAVASCRIPT by STEPHEN A. THOMAS MARCH 2015, 384 pp., $39.95 ISBN 978-1-59327-605-8 full color PYTHON CRASH COURSE A Hands-On, Project-Based Introduction to Programming by ERIC MATTHES NOVEMBER 2015, 560 pp., $39.95 ISBN 978-1-59327-603-4 Estadísticos e-Books & Papers STATISTICS DONE WRONG The Woefully Complete Guide by ALEX REINHART MARCH 2015, 176 pp., $24.95 ISBN 978-1-59327-620-1 THE MANGA GUIDE TO DATABASES by MANA TAKAHASHI, SHOKO AZUMA, and TREND-PRO CO., LTD JANUARY 2009, 224 pp., $19.95 ISBN 978-1-59327-190-9 Estadísticos e-Books & Papers DOING MATH WITH PYTHON Use Programming to Explore Algebra, Statistics, Calculus, and More! by AMIT SAHA AUGUST 2015, 264 pp., $29.95 ISBN 978-1-59327-640-9 PHONE: 1.800.420.7240 or 1.415.863.9900 EMAIL: SALES@NOSTARCH.COM WEB: WWW.NOSTARCH.COM Estadísticos e-Books & Papers Estadísticos e-Books & Papers FIND THE STORY IN YOUR DATA This book uses PostgreSQL but is applicable to MySQL, Microsoft SQL Server, and other database systems. SQL (Structured Query Language) is a popular programming language used to create, manage, and query databases. Whether you’re a marketing analyst, a journalist, or a researcher mapping neurons in the brain of a fruit fly, you’ll benefit from using SQL to tell the story hidden in your data. Practical SQL is a fast-paced, plain-English introduction to programming with SQL. Following a primer on SQL language basics and database fundamentals, you’ll learn how to use the pgAdmin interface and PostgreSQL database system to define, organize, and analyze real-world data sets, such as crime statistics and U.S. Census demographics. Next, you’ll learn how to create databases using your own data, write queries to perform calculations, and handle common roadblocks when dealing with public data. With the help of easy-to-follow exercises in each Estadísticos e-Books & Papers chapter, you’ll discover how to build powerful databases and find meaning in your data sets. You’ll also learn how to: • Define the right data types for your information • Aggregate, sort, and filter data to find patterns • Identify and clean up any errors in your data • Search text for meaningful data • Create advanced queries and automate tedious tasks Organizing and analyzing data doesn’t have to be dry and complicated. Find the story in your data with Practical SQL. ABOUT THE AUTHOR Anthony DeBarros is an award-winning data journalist whose career spans 30 years at news organizations including USA TODAY and Gannett’s Poughkeepsie Journal. He holds a master’s degree in information systems from Marist College. THE FINEST IN GEEK ENTERTAINMENT™ www.nostarch.com Estadísticos e-Books & Papers", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 272 + }, + { + "text": "Anthony DeBarros is an award-winning data journalist whose career spans 30 years at news organizations including USA TODAY and Gannett’s Poughkeepsie Journal. He holds a master’s degree in information systems from Marist College. THE FINEST IN GEEK ENTERTAINMENT™ www.nostarch.com Estadísticos e-Books & Papers", + "source": "Practical SQL A Beginner’s Guide to Storytelling with Data.pdf", + "chunk_id": 273 + }, + { + "text": "SQL for Data Scientists SQL for Data Scientists A Beginner’s Guide for Building Datasets for Analysis Renée M. P. Teate Copyright © 2021 by John Wiley & Sons, Inc. All rights reserved. Published by John Wiley & Sons, Inc., Hoboken, New Jersey. Published simultaneously in Canada. ISBN: 978-­1-­119-­66936-­4 ISBN: 978-­1-­119-­66937-­1 (ebk) ISBN: 978-­1-­119-­66939-­5 (ebk) No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-­copy fee to the Copyright Clear- ance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-­8400, fax (978) 750-­4470, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-­6011, fax (201) 748-­6008, or online at http://www.wiley.com/go/permission. Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materi- als. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. For general information on our other products and services or for technical support, please contact our Cus- tomer Care Department within the United States at (800) 762-­2974, outside the United States at (317) 572-­3993 or fax (317) 572-­4002. Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic formats. For more information about Wiley products, visit our web site at www.wiley.com. Library of Congress Control Number: 2021941400 Trademarks: WILEY and the Wiley logo are trademarks or registered trademarks of John Wiley & Sons, Inc. and/or its affiliates, in the United States and other countries, and may not be used without written permis- sion. All other trademarks are the property of their respective owners. John Wiley & Sons, Inc. is not associ- ated with any product or vendor mentioned in this book. Cover image: © filo/Getty Images Cover design: Wiley In my data science career talks, I warn about tech industry gatekeepers. This book is dedicated to the gate-­openers. vii About the Author Renée M. P. Teate is the Director of Data Science at HelioCampus, leading a team that builds predictive models for colleges and universities. She has worked with data professionally since 2004, in roles including relational", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 0 + }, + { + "text": "about tech industry gatekeepers. This book is dedicated to the gate-­openers. vii About the Author Renée M. P. Teate is the Director of Data Science at HelioCampus, leading a team that builds predictive models for colleges and universities. She has worked with data professionally since 2004, in roles including relational database design, data-­driven website development, data analysis and reporting, and data sci- ence. With degrees in Integrated Science and Technology from James Madison University and Systems Engineering from the University of Virginia, along with a varied career working with data at every stage in a number of systems, she considers herself to be a “data generalist.” Renée regularly speaks at technology and higher ed conferences and meet- ups, and writes in industry publications about her data science work and about navigating data science career paths. She also created the “Becoming a Data Sci- entist” podcast and @BecomingDataSci Twitter account, where she’s known to her over 60k followers as “Data Science Renee.” She always tells aspiring data scientists to learn SQL, since it has been one of the most valuable and enduring skills needed throughout her career. ix About the Technical Editor Vicki Boykis is a machine learning engineer, currently working with recommen- dation systems. She has over a decade of experience in analytics and databases across numerous industries including social media, telecom, and healthcare, and has worked with Postgres, SQL Server, Oracle, and MySQL. She has pre- viously taught courses in object-­oriented programming (OOP) for Python and MySQL for massive open online courses (MOOCs). She has a BS in Economics with Honors from Penn State University and an MBA from Temple University in Philadelphia. xi Acknowledgments When I first started this book in Fall 2019, I was new to the book authoring and publication process, and I couldn’t have anticipated how everything around us would change due to a deadly pandemic and political upheaval. I want to first acknowledge the healthcare and other essential workers who risked their lives during this era of COVID-­19. Nothing any of us have accomplished throughout this time would have been possible without your selfless efforts saving lives and allowing some of us to work safely from home. I also want to thank those who continue fighting for equality in the face of injustice. You inspire me and give me hope. As a first-­time book author, the process of transferring my knowledge and experience to the page, and bringing this book to completion, has been a major learning experience. I would like to thank the team at Wiley for taking the chance on me and for all of your work, especially project editor Kelly Talbot for guiding me through this process, improving my content, and eventually getting me across the finish line! I was so excited when I found out that Vicki Boykis, whose writing about our industry is fascinating and insightful, would be my technical editor. Her thoughtful feedback was invaluable. I truly appreciate her sticking with me throughout this extended process. I would", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 1 + }, + { + "text": "eventually getting me across the finish line! I was so excited when I found out that Vicki Boykis, whose writing about our industry is fascinating and insightful, would be my technical editor. Her thoughtful feedback was invaluable. I truly appreciate her sticking with me throughout this extended process. I would also like to thank my family and teachers, who encouraged my interest in computers and technology from a young age and fostered my love of reading, and my friends and mentors who have helped me continue to progress in my education and my career since. Those who have had an impact on me are too numerous to list, but know that I acknowledge your role in helping me get to where I am today. My parents and sister, my husband and step-­children, my teachers and managers, my colleagues and friends, your time and energy and patience is so appreciated. xii Acknowledgments And I want to give heartfelt thanks to my husband and my step-­son, Tony and Anthony Teate, for always believing in me, giving invaluable feedback, and bearing with me during this extended project. Tony has been a vital part of my “data science journey” from the very beginning, and I’m fittingly wrapping up this long phase of it on his birthday (Happy Birthday, Sweetheart!). The love and support the two of you have shown me is beyond measure. I love you. Before I close, I want to give shout-­outs to two special communities. First, one that might be a bit unexpected: the vegetable gardening community on Instagram. Growing a garden in my backyard while enjoying your company online honestly helped me get through some of the difficult aspects of writing this book, especially during a pandemic. The fictional Farmer’s Market database used in every example was inspired by you. And last but not least, to the data science communities in my local area (Harrisonburg and Charlottesville, Virginia—­data science isn’t only done in big cities!) and online, plus those of you reading this book. I feel blessed to be a part of such a vibrant professional community, and honored that you value my experience and advice. Thank you to everyone who has been a part of my data career to date, and who has let me know I have been a part of yours. — ­Renée M. P. Teate xiii Contents at a Glance Introduction xix Chapter 1 Data Sources 1 Chapter 2 The SELECT Statement 15 Chapter 3 The WHERE Clause 31 Chapter 4 CASE Statements 49 Chapter 5 SQL JOINs 61 Chapter 6 Aggregating Results for Analysis 79 Chapter 7 Window Functions and Subqueries 97 Chapter 8 Date and Time Functions 113 Chapter 9 Exploratory Data Analysis with SQL 127 Chapter 10 Building SQL Datasets for Analytical Reporting 143 Chapter 11 More Advanced Query Structures 159 Chapter 12 Creating Machine Learning Datasets Using SQL 173 Chapter 13 Analytical Dataset Development Examples 191 Chapter 14 Storing and Modifying Data 229 Appendix Answers to Exercises 239 Index 255 xv Contents", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 2 + }, + { + "text": "SQL 127 Chapter 10 Building SQL Datasets for Analytical Reporting 143 Chapter 11 More Advanced Query Structures 159 Chapter 12 Creating Machine Learning Datasets Using SQL 173 Chapter 13 Analytical Dataset Development Examples 191 Chapter 14 Storing and Modifying Data 229 Appendix Answers to Exercises 239 Index 255 xv Contents Introduction xix Chapter 1 Data Sources 1 Data Sources 1 Tools for Connecting to Data Sources and Editing SQL 2 Relational Databases 3 Dimensional Data Warehouses 7 Asking Questions About the Data Source 9 Introduction to the Farmer’s Market Database 11 A Note on Machine Learning Dataset Terminology 12 Exercises 13 Chapter 2 The SELECT Statement 15 The SELECT Statement 15 The Fundamental Syntax Structure of a SELECT Query 16 Selecting Columns and Limiting the Number of Rows Returned 16 The ORDER BY Clause: Sorting Results 18 Introduction to Simple Inline Calculations 20 More Inline Calculation Examples: Rounding 22 More Inline Calculation Examples: Concatenating Strings 24 Evaluating Query Output 26 SELECT Statement Summary 29 Exercises Using the Included Database 30 Chapter 3 The WHERE Clause 31 The WHERE Clause 31 Filtering SELECT Statement Results 32 xvi Contents Filtering on Multiple Conditions 34 Multi-Column Conditional Filtering 40 More Ways to Filter 41 BETWEEN 41 IN 42 LIKE 43 IS NULL 44 A Warning About Null Comparisons 44 Filtering Using Subqueries 46 Exercises Using the Included Database 47 Chapter 4 CASE Statements 49 CASE Statement Syntax 50 Creating Binary Flags Using CASE 52 Grouping or Binning Continuous Values Using CASE 53 Categorical Encoding Using CASE 56 CASE Statement Summary 59 Exercises Using the Included Database 60 Chapter 5 SQL JOINs 61 Database Relationships and SQL JOINs 61 A Common Pitfall when Filtering Joined Data 71 JOINs with More than Two Tables 74 Exercises Using the Included Database 76 Chapter 6 Aggregating Results for Analysis 79 GROUP BY Syntax 79 Displaying Group Summaries 80 Performing Calculations Inside Aggregate Functions 84 MIN and MAX 88 COUNT and COUNT DISTINCT 90 Average 91 Filtering with HAVING 93 CASE Statements Inside Aggregate Functions 94 Exercises Using the Included Database 96 Chapter 7 Window Functions and Subqueries 97 ROW NUMBER 98 RANK and DENSE RANK 101 NTILE 102 Aggregate Window Functions 103 LAG and LEAD 108 Exercises Using the Included Database 111 Chapter 8 Date and Time Functions 113 Setting datetime Field Values 114 EXTRACT and DATE_PART 115 Contents xvii DATE_ADD and DATE_SUB 116 DATEDIFF 118 TIMESTAMPDIFF 119 Date Functions in Aggregate Summaries and Window Functions 119 Exercises 126 Chapter 9 Exploratory Data Analysis with SQL 127 Demonstrating Exploratory Data Analysis with SQL 128 Exploring the Products Table 128 Exploring Possible Column Values 131 Exploring Changes Over Time 134 Exploring Multiple Tables Simultaneously 135 Exploring Inventory vs. Sales 138 Exercises 142 Chapter 10 Building SQL Datasets for Analytical Reporting 143 Thinking Through Analytical Dataset Requirements 144 Using Custom Analytical Datasets in SQL: CTEs and Views 149 Taking SQL Reporting Further 153 Exercises 157 Chapter 11 More Advanced Query Structures 159 UNIONs 159 Self-­Join to Determine To-­Date Maximum", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 3 + }, + { + "text": "Sales 138 Exercises 142 Chapter 10 Building SQL Datasets for Analytical Reporting 143 Thinking Through Analytical Dataset Requirements 144 Using Custom Analytical Datasets in SQL: CTEs and Views 149 Taking SQL Reporting Further 153 Exercises 157 Chapter 11 More Advanced Query Structures 159 UNIONs 159 Self-­Join to Determine To-­Date Maximum 163 Counting New vs. Returning Customers by Week 167 Summary 171 Exercises 171 Chapter 12 Creating Machine Learning Datasets Using SQL 173 Datasets for Time Series Models 174 Datasets for Binary Classification 176 Creating the Dataset 178 Expanding the Feature Set 181 Feature Engineering 185 Taking Things to the Next Level 189 Exercises 189 Chapter 13 Analytical Dataset Development Examples 191 What Factors Correlate with Fresh Produce Sales? 191 How Do Sales Vary by Customer Zip Code, Market Distance, and Demographic Data? 211 How Does Product Price Distribution Affect Market Sales? 217 Chapter 14 Storing and Modifying Data 229 Storing SQL Datasets as Tables and Views 229 Adding a Timestamp Column 232 xviii Contents Inserting Rows and Updating Values in Database Tables 233 Using SQL Inside Scripts 236 In Closing 237 Exercises 238 Appendix Answers to Exercises 239 Index 255 xix Introduction Who I Am and Why I’m Writing About This Topic When I was first brainstorming topics for this book, I used two questions to narrow down my list: “Who is my audience?” and “What topic do I know well enough to write a book that would be worth publishing for that audience?” The first question had an easy initial answer: I already have an audience of data-­science-­learning Twitter followers with whom I share resources and advice on “Becoming a Data Scientist” that I could keep in mind while narrowing down the topics. So then I was left to figure out what I know that I could teach to people who want to become data scientists. I have been designing and querying relational databases professionally for about 17 years: first as a database and web developer, then as a data analyst, and for the last 5 years, as a data scientist. SQL (Structured Query Language) has been a key tool for me throughout—­whether I was working with MS Access, MS SQL Server, MySQL, Oracle, or Redshift databases, and whether I was summarizing data into reporting views in a data mart, extracting data to use in a data visualization tool like Tableau, or preparing a dataset for a machine learning project. Since SQL is a tool I have used throughout my career, and because creating and retrieving datasets for analysis has been such an integral part of my job as a data scientist, I was surprised to learn that some data scientists don’t know SQL or don’t regularly write SQL code. But in an informal Twitter poll I conducted, which received responses from 979 data scientists, 19% of them reported wanting to learn, or learn more, SQL (74% reported already using SQL professionally). Additionally, 55% of 713 respondents who were working toward becoming data xx Introduction scientists said they wanted", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 4 + }, + { + "text": "code. But in an informal Twitter poll I conducted, which received responses from 979 data scientists, 19% of them reported wanting to learn, or learn more, SQL (74% reported already using SQL professionally). Additionally, 55% of 713 respondents who were working toward becoming data xx Introduction scientists said they wanted to learn, or learn more, SQL. So, my target audience had an interest in this topic. According to an analysis of online job postings conducted by Jeff Hale of Towards Data Science, SQL is in the top three technology skills that data scien- tist jobs require. (See towardsdatascience.com/the-­most-­in-­demand-­skills-­ for-­data-­scientists-­4a4a8db896db.) In an Indeed BeSeen article, Joy Garza lists SQL as one of the top-­five in-­demand tech skills for data scientists. (See https://web.archive.org/web/20200624031802/https://www.beseen.com/ blog/talent/data-scientist-skills/.) After learning how many working and prospective data scientists wanted to learn SQL, and how much of a need there is in the industry for people who know how to use it, SQL dataset development started to move to the top of the list of topics I could share my knowledge of with others. There are many SQL books on the market that can be used to learn query syntax and advanced SQL functions—­after all, the language has been around for 45 years and has been standardized since the late 1980s—­but I hadn’t found any definitive resources to refer people to when they asked me if I knew of any books that taught how to use SQL to construct datasets for machine learning, so I decided to write this book to cover SQL from a data scientist’s point of view. So, my goal in writing this book is not only to teach you how to write SQL code but to teach you how to think about summarizing data into analytical datasets that can be used for reports and machine learning: to use SQL like a data scientist does. Like I do. Who This Book Is For SQL for Data Scientists is designed to be a learning resource for anyone who wants to become (or who already is) a data analyst or data scientist, and wants to be able to pull data from databases to build their own datasets without having to rely on others in the organization to query the source system and transform it into flat files (or spreadsheets) for them. There are plenty of SQL books out there, but many are either written as syntax references or written for people in other roles that create, query from, and maintain databases. However, this book is written from the perspective of a data scientist and is aimed at those who will primarily be extracting data from existing databases in order to generate datasets for analysis. I won’t assume that you’ve ever written SQL queries before, and we’ll start with the basics, but I do assume that you have some basic understanding of what databases are and a general idea of how data might be used in reports, analyses, and machine learning algorithms. This book is meant to fill", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 5 + }, + { + "text": "that you’ve ever written SQL queries before, and we’ll start with the basics, but I do assume that you have some basic understanding of what databases are and a general idea of how data might be used in reports, analyses, and machine learning algorithms. This book is meant to fill in the steps between finding a database that contains the data you need and starting the analysis. I aim to teach you how to think about structuring datasets for analysis and how to use SQL to extract the data from the database and get it into that form. Introduction xxi Why You Should Learn SQL if You Want to Be a Data Scientist If you can use SQL to pull your own datasets, you don’t have to rely on others in your organization to pull it for you, enabling you to work more efficiently. Requesting datasets usually involves a process of filling out a form or ticket describing in detail what data you need, waiting for your request to be ful- filled, then often clarifying your request after seeing the initial results, and then waiting again for modifications. If you can edit your own queries, you can not only design and retrieve your own datasets but then also adjust calculations or add fields as needed. Additionally, running a SQL query that writes to a database table or exports to a file—­effectively snapshotting the data in the form you need it in for your analysis—­means you don’t have to retrieve and reprocess the data in your machine learning script every time you run your code, speeding up the usually iterative model development process. Some summaries and calculations can be done more efficiently in SQL than in other types of code, as well, so even if you are running the queries “live” each time you run your script, you may be able to lower the computational cost of your code by doing some of the transformations in SQL. Finally, because it is a high-­demand tech skill in data scientist job postings, learning SQL will increase your marketability and value to employers. What I Hope You Gain from This Book My goal is that by the time you finish reading this book and practicing the queries within (ideally both on the provided example database and on another database of your choosing, so you have to modify the example queries and apply them in another context), you will be able to think through the process of creating an analytical dataset and develop the SQL code necessary to generate your intended output. I hope that even if you end up needing to use a SQL function that’s not covered in this book, you will have gained enough baseline knowledge from the book to go look it up online and determine how to best use it in the query you are developing. I also hope that this book will help you feel confident that you can pull your own data at work and get it into", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 6 + }, + { + "text": "gained enough baseline knowledge from the book to go look it up online and determine how to best use it in the query you are developing. I also hope that this book will help you feel confident that you can pull your own data at work and get it into the form you need it in for your report or model without having to wait on others to do it for you. xxii Introduction Conventions This book uses MySQL version 8.0–style SQL. No matter what type of database system you use (MS SQL Server, Redshift, PostgreSQL, Oracle, etc.), the query design concepts and syntax are very similar, when not identical across plat- forms. So, if you work with a database system other than MySQL, you might have to search for the equivalent code syntax for a few functions in the book, but the overall dataset design concepts are platform-­independent, and the SQL keywords are cross-­platform standards. When you see code displayed in the following style: SELECT * FROM Product that means it is a complete SQL query that you can use to select data from the Farmer’s Market database described in Chapter 1, “Data Sources.” If you’re reading the printed version of this book, you can go to the book’s website to get digital versions of the queries that you can copy and paste to try them out yourself. Reserved SQL keywords like SELECT will appear in all-­uppercase throughout the book, and column names will appear in all-­lowercase. This isn’t a requirement of SQL syntax (neither are line breaks), but is a convention used for readability. Be aware that the Farmer’s Market database will continue to evolve, and I will likely continue adding rows to its tables after this book goes to print, so the data values you see in the output when you run the queries yourself may not exactly match the screenshots included in the printed book. Reader Support for This Book Companion Download Files As you work through the examples in this book, you may choose either to type in all the code manually or to use the source code files that accompany the book. All the source code used in this book, along with the Farmer’s Market database, is available for download from both sqlfordatascientists.com and www.wiley.com/go/sqlfordatascientists. How to Contact the Publisher If you believe you’ve found a mistake in this book, please bring it to our attention. At John Wiley & Sons, we understand how important it is to provide our cus- tomers with accurate content, but even with our best efforts an error may occur. In order to submit your possible errata, please email it to our Customer Ser- vice Team at wileysupport@wiley.com with the subject line “Possible Book Errata Submission”. Introduction xxiii How to Contact the Author I’m known as “Data Science Renee” on Twitter, and my username is @becomingdatasci. I’m happy to interact with readers via social media, so feel free to tweet me your questions and suggestions. Thank you for giving", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 7 + }, + { + "text": "the subject line “Possible Book Errata Submission”. Introduction xxiii How to Contact the Author I’m known as “Data Science Renee” on Twitter, and my username is @becomingdatasci. I’m happy to interact with readers via social media, so feel free to tweet me your questions and suggestions. Thank you for giving me the chance to help guide you through the topic of SQL for Data Scientists. Let’s dive in! C H A P T E R 1 1 As a data analyst or data scientist, you will encounter data from many sources— from databases to spreadsheets to Application Programming Interfaces (APIs)— which you are expected to use for predictive modeling. Understanding the source system your data comes from, how it was initially gathered and stored, and how frequently it is updated, will take you a long way toward an effective analysis. In my experience, issues with a predictive model can often be traced back all the way to the source data or the query that first pulls the data from the source. Exploring the data available for your analysis starts with exploring the structure of the source database. Data Sources Data can be stored in many forms and structures. Examples of unstructured data include text documents or images stored as individual files in a computer’s file system. In this book, we’ll be focusing on structured data, which is typically organized into a tabular format, like a spreadsheet or database table containing limited-length text or numeric values. Many software applications enable the organization of data into structured forms. One example you are likely familiar with is Microsoft Excel, for creating and maintaining spreadsheets. Excel also includes some analysis capabilities, such as pivot tables for summarizing spreadsheets and data visualization tools Data Sources 2 Chapter 1 ■ Data Sources for plotting data points from a spreadsheet. Some functions in Excel allow you to connect data in one spreadsheet to another, but in order to create a true relational database model and define rules for how the data tables are interconnected, Microsoft offers a relational database application called Access. My first experiences with relational database design were in MS Access, and the basic Structured Query Language (SQL) concepts I learned in order to query data from an Access database are the same concepts I have used throughout my career—in increasingly complex ways. I have since extracted data from other Relational Database Management Systems (RDBMSs) such as MS SQL Server, Oracle Database, MySQL, and Amazon Redshift. Though the syntax for each can differ slightly, the general concepts, many of which you will learn in this book, are consistent across products. SQL-style RDBMSs were first developed in the 1970s, and the basic database design concepts have stood the test of time; many of the database systems that originated then are still in use today. The longevity of these tools is another reason that SQL is so ubiquitous and so valuable to learn. As a professional who works with data, you will likely encounter several of the", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 8 + }, + { + "text": "stood the test of time; many of the database systems that originated then are still in use today. The longevity of these tools is another reason that SQL is so ubiquitous and so valuable to learn. As a professional who works with data, you will likely encounter several of the following popular Relational Database Management Systems: ■ ■Oracle ■ ■MySQL ■ ■MS SQL Server ■ ■PostgreSQL ■ ■Amazon Redshift ■ ■IBM DB2 ■ ■MS Access ■ ■SQLite ■ ■Snowflake You will also likely work with data retrieved from other types of files at some point, such as CSV text files, JSON retrieved via API, XML in a NoSQL database, Graph databases with special query languages, key-value stores, and so on. However, relational SQL databases still dominate the industry for structured data storage and are the most likely database systems you will encounter on the job. Tools for Connecting to Data Sources and Editing SQL When you start an analysis project, the first step is often connecting to a data- base on a server. This is generally done through a SQL Integrated Development Environment (IDE) or with code that connects to the database without a graphical Chapter 1 ■ Data Sources 3 user interface (GUI) to run queries that extract the data and store it in a struc- ture that you can work with downstream in your analysis, such as a dataframe. The IDE referenced for demonstration purposes throughout this book is MySQL Workbench Community Edition, which was chosen because we’ll be querying a MySQL database in the examples. MySQL is open source under the GPL license, and MySQL Workbench CE is free to download. Many other IDEs will allow you to connect to databases and will perform syntax-highlighting of SQL (highlighting keywords to make it easier to read and to spot errors). All major database systems support Open Database Con- nectivity (ODBC), which uses drivers to standardize the interfaces between software applications and databases. Whoever has granted you permission to access a database should give you documentation on how to securely connect to it via your selected IDE. You can also connect to a database directly from code such as Python or R. Search for your preferred language and the type of database (for example, “R SQL Server” or “Python Redshift”) and you will find packages or add-ons that enable you to embed SQL queries in your code and return results in the form of a dataframe or other data structure. The database system’s official documen- tation will also provide information about connecting to it from other software and from within your code. Searching “MySQL connector” brings up a list of drivers for use with different languages, for example. If you are writing code in a language like Python and will be passing a SQL statement to a function as a string, where it won’t be syntax highlighted, you can write SQL in a free text tool that performs SQL syntax highlighting, such as Notepad++, or in a SQL IDE,", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 9 + }, + { + "text": "you are writing code in a language like Python and will be passing a SQL statement to a function as a string, where it won’t be syntax highlighted, you can write SQL in a free text tool that performs SQL syntax highlighting, such as Notepad++, or in a SQL IDE, and then paste the final result into your code. Relational Databases If you have never explored a database, you can think of a database table like a well-defined spreadsheet, with row identifiers and named column headers. Each table may store different subsets and types of data at different levels of detail. An entity is the “thing” (object or concept) that the table represents and cap- tures data for. If there is a table that contains data about books, the entity is “Books,” and the “Book table” is the data structure that contains information about the Book entity. Some people use the terms entity and table interchangeably. You may see me using the terms row and record interchangeably in this book: a record in a database is like a row in a table and displayed the same way. Some people call a database row a tuple. You may also see me using the terms column, field, and attribute as synonyms. A column header in a spreadsheet is the equivalent of an attribute name in a table. Each column in a database table stores data about an attribute of the entity. 4 Chapter 1 ■ Data Sources For example, as illustrated in Figure 1.1, in a table of Books there would be a row for each book, with an ISBN number column to identify each book. The ISBN is an attribute of the book entity. The Author column in the row in the Books table representing this book would have my name in it, so you could say that “the value in the Author field in the SQL for Data Scientists record in the Books table is ‘Renée M. P. Teate’.” Or, “In the Books table, the row repre- senting the book SQL for Data Scientists contains the value ‘Renée M. P. Teate’ in the Author column.” A database is a collection of related tables, and a database schema stores information about the tables (and other database objects), as well as the rela- tionships between them, defining the structure of the database. To illustrate an example of a relationship between database tables, imagine that one table in a database contains a record (row) for every patient that’s ever scheduled an appointment at a doctor’s office, with each patient’s name, birthdate, and phone number, like a directory. Another table contains a record of every appointment, with the patient’s name, appointment time, reason for the visit, and the name of the doctor the patient has an appointment with. The connection between these two tables could be the patient’s name. (In reality, a unique identifier would be assigned to each patient, since two people can have the same name, but for this illustration, the name", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 10 + }, + { + "text": "the visit, and the name of the doctor the patient has an appointment with. The connection between these two tables could be the patient’s name. (In reality, a unique identifier would be assigned to each patient, since two people can have the same name, but for this illustration, the name will suffice.) In order to create a report of every patient who has an appointment scheduled in the next week along with their contact information, there would have to be an established connection between the patient directory table and the appointment table, enabling someone to pull data from both tables simultaneously. See Figure 1.2. Figure 1.1 Figure 1.2 Chapter 1 ■ Data Sources 5 The relationship between the entities just described is called a one-to-many relationship. Each patient only appears in the patient directory table one time but can have many appointments in the related appointment-tracking table. Each appointment only has one patient’s name associated with it. Database relationships like this one are depicted in what’s called an entity- relationship diagram (ERD). The ERD for these two tables is shown in Figure 1.3. NOTE In an ERD, an infinity symbol, “N”, or “crow’s feet” on the end of a line connecting two tables indicates that it is the “many” side of a one-to-many relationship. You can see the infinity symbol next to the Appointments table in Figure 1.3. The primary key in a table is a column or combination of columns that uniquely identifies a row. The combination of values in the primary key columns must be unique per record, and cannot all be NULL (empty). The primary key can be made of values that occur in the data that are unique per record—such as a Student ID Card number in a table of students at a university—or it can be gen- erated by the database and not carry meaning elsewhere “in real life,” like an integer value that increments automatically every time a new record is created. The primary key in a table can be used to identify the records in other tables that relate to each of its records. When a table’s primary key is referenced in another table, it is called a foreign key. NOTE Notice that the NULL value is described in this section as “empty” and not as “blank.” In database terms, NULL and “blank” aren’t necessarily the same thing. For example, a single space “ ” can be considered a “blank” value in a string field, but is not NULL, because there is a space character stored there. A NULL is the absence of any value, a totally empty field. NULLs are treated differently than blanks in SQL. As mentioned, using the Patient Name in the previous example is a poor selection of primary key, because two patients can have the same name, so your primary key won’t necessarily end up uniquely identifying patients. One option that is common practice in the industry is to create a field that generates an auto-incrementing integer to", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 11 + }, + { + "text": "in the previous example is a poor selection of primary key, because two patients can have the same name, so your primary key won’t necessarily end up uniquely identifying patients. One option that is common practice in the industry is to create a field that generates an auto-incrementing integer to serve as a unique identifier for each new row, so as not to rely on other values unique to a record that may be a privacy concern or unavailable at the time the record is created, such as Social Security numbers. Figure 1.3 6 Chapter 1 ■ Data Sources So, let’s say that instead, the doctor’s office database assigned an auto-incre- menting integer value to serve as the primary key for each patient record in the Patients table and for each appointment record in the Appointments table. Then, the appointment-tracking table can use that generated Patient ID value to link each appointment to each patient, and the patient’s name doesn’t even need to be stored in the Appointments table. In Figure 1.4, you can see a data- base design where the Patient ID is serving as a primary key in the Patients table, and as a foreign key in the Appointments table. Another type of relationship found in RDBMSs is called many-to-many. As you might guess, it’s a connection between entities where the records on each side of the relationship can connect to multiple records on the other side. Using our Books example, if we had a table of Authors, there would be a many-to- many relationship between books and authors, because each author can write multiple books, and each book can have multiple authors. In order to create this relationship in the database, a junction or associative table will be needed to capture the pairs of related rows. See Figure 1.5. Figure 1.4 Figure 1.5 Chapter 1 ■ Data Sources 7 In the ERD shown in Figure 1.5 you can see that the ISBN, which is the pri- mary key in the Books table, and the Author ID, which is the primary key in the Authors table (denoted by asterisks) are both foreign keys in the Books-Authors Junction table (denoted by double asterisks). Each pairing of ISBN and Author ID in the junction table would be unique, so the pair of fields can be considered a multi-column primary key in the Books-Authors Junction table. By setting up this database relationship so that we don’t end up with mul- tiple rows per book in the Books table or multiple authors listed per book in the Authors column of the Books table and have a junction table that only contains the identifiers matching up the related tables, we are reducing the amount of redundant data stored in the database and clarifying how the entities are related in real life. The idea of not storing redundant data in a database unnecessarily is known as database normalization. In the book database example, we only have to store each author’s full name once, no", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 12 + }, + { + "text": "amount of redundant data stored in the database and clarifying how the entities are related in real life. The idea of not storing redundant data in a database unnecessarily is known as database normalization. In the book database example, we only have to store each author’s full name once, no matter how many books they have written. In the doctor’s office example, there’s no need to store a patient’s phone number repeatedly in the Appointments table, because it’s already stored in the related “patient directory” table, and can be found by connecting the two tables via the Patient ID (we will cover SQL JOINs, which are used to merge data from mul- tiple tables, in Chapter 5, “SQL JOINs”). Normalization can reduce the amount of storage space a database requires and also reduce the complexity of updating data, since each value is stored a minimal number of times. We won’t go into all of the details of normalization here, but if you are interested in learning more about it, research “relational database design.” Dimensional Data Warehouses Data warehouses often contain data from multiple underlying data sources. They can be designed in a normalized relational database form, as described in the previous section, or using other design standards. They may contain both “raw data” extracted from other databases directly, and “summary” tables that are combined or transformed versions of that raw data (for example, the analytical datasets you will learn to build in this book could be permanently stored as tables in a data warehouse, to be referenced by multiple reports). Data ware- houses can contain historical data logs with past and current records, tables that are updated in real time as the source systems are updated, or snapshots of data to preserve it as it existed at a past moment in time. Often, data warehouses are designed using dimensional modeling techniques. We won’t go in-depth into the details of dimensional modeling here, but one concept you are likely to come across when querying tables in data warehouses is a “star schema” design that divides the data into facts and dimensions. 8 Chapter 1 ■ Data Sources The way I think of facts and dimensions is that a record in a fact table contains the “metadata” of an entity, as well as any measures (which are usually numeric values) you want to track and later summarize. A dimension is property of that entity you can group or “slice and dice” the fact records by, and a dimension table will contain further information of that property. So, for example, a transactional record of an item purchased at a retail store is a fact, containing the timestamp of the purchase, the store number, order number, customer number, and the amount paid. The store the purchase was made at is a dimension of the item purchase fact, and the associated store dimension table would contain additional information about the store, such as its name. You could then query both the fact and the dimension", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 13 + }, + { + "text": "order number, customer number, and the amount paid. The store the purchase was made at is a dimension of the item purchase fact, and the associated store dimension table would contain additional information about the store, such as its name. You could then query both the fact and the dimension tables to get a summary of purchases by store. If we transformed our doctor’s office database into a star schema, we might have an appointments fact table capturing the occurrence of every appointment, which patient it was for, when it was booked, the reason for the appointment, which doctor it was with, and when it is scheduled to occur. We might also have a date dimension and a time dimension, storing the various properties of each appointment date and time (such as year or day of week) and appointment- booking date and time. This would allow us to easily count up how many appointments occurred per time period or determine when the highest volume of appointment-booking calls take place, by grouping the “transactional” fact information by different dimensions. Figure 1.6 depicts an example dimensional data warehouse design. Can you see why this design is called a star schema? There might also be an appointment history log in this data warehouse, with a record of each time the appointment was changed. That way, not only could we tell when the appointment is supposed to take place, but how many times it was modified, whether it was initially assigned to another doctor, etc. Dim Patient Dim Date Dim Doctor Dim Time Fact Appointment Dim Appointment Reason Figure 1.6 Chapter 1 ■ Data Sources 9 Note that when compared to a normalized relational database, a dimensional model stores a lot more information. Appointment records will appear multiple times in an appointment log table. There may be a record for every calendar date in the date dimension table, even if no appointments are scheduled for that date yet, and the list of dates might extend for decades into the future! If you’re designing a database or data warehouse, you need to understand these concepts in much more detail than we’ll cover here. But in order to query the database to build an analytical dataset, you primarily need to under- stand the data warehouse table grain (level of detail; what set of columns makes a row unique) and how the tables are related to one another. Once you have that information, querying a dimensional data warehouse with SQL is much like querying a relational database with SQL. Asking Questions About the Data Source Once you find out what type of data source you’re working with and learn about the schema design and the relationships between the database tables, there is still a lot of information you should gather about the tables you’ll be querying before you dive into writing any SQL. If you are lucky enough to have access to subject matter experts (SMEs) who know the details of why the database was designed the", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 14 + }, + { + "text": "the database tables, there is still a lot of information you should gather about the tables you’ll be querying before you dive into writing any SQL. If you are lucky enough to have access to subject matter experts (SMEs) who know the details of why the database was designed the way it is, how the data is collected and updated, and what to expect in terms of the frequency and types of data that may be updating as you work with the database, stay in commu- nication with them throughout the data exploration and query development process. These might be database administrators (DBAs), ETL engineers (the people who extract, transform, and load data from a source system into a data warehouse), or the people who actually generate or enter the data into the source system in the first place. If you spot some values that don’t seem to make sense, you can sometimes look in a data dictionary to learn more (if one exists and is correct), but often going directly to the SMEs to get the details is the best approach. If your questions are easily answered by existing documentation, they will point you to it! Here are some example questions you might want to ask the SMEs as you’re first learning about the data source: ■ ■“Here are the questions I’m being asked to answer in my analysis. Which tables in this database should I look in first for the relevant data? And is there an entity-relationship diagram documenting the relationships bet- ween them that I can reference?” These questions are especially helpful for large data warehouses with a lot of tables, where being pointed in the right direction from the start can save a lot of time searching for the data you need. 10 Chapter 1 ■ Data Sources ■ ■“What set of fields make up the primary key for this table?” Or, “What is the grain of this fact table?” Understanding the level of detail of each table is important in order to know how to filter, group, and summarize the data in the table, and join it to other tables. ■ ■“Are these records imported directly from the source system, or have they been transformed or merged in some way before being stored in this table?” This is helpful to know when debugging data that doesn’t look like you expected. If the database is “raw” data from the source system, you might talk to those entering the data to learn more about it. If it has gone through a transformation or includes data from several different tables in the system of origin, then the first stop to understand a value would likely be the ETL engineers who programmed the code that modified it. ■ ■“Is this a static snapshot table, or does it update regularly? At what fre- quency does it update? And are older records expired and kept as new data is added, or is the existing record overwritten when changes occur?” If a", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 15 + }, + { + "text": "engineers who programmed the code that modified it. ■ ■“Is this a static snapshot table, or does it update regularly? At what fre- quency does it update? And are older records expired and kept as new data is added, or is the existing record overwritten when changes occur?” If a table you’re querying contains “live” data that is being updated as you work, and you are using it to perform calculations or as an input to a machine learning algorithm, you may want to make a copy of it to use while working. That way, you know that changes in the calculations or model output are due to changes in your code, and not due to data that is changing as you debug. For datasets updated on a nightly basis, you might want to know what time they refresh, so you can schedule other things that depend on it, like an extract refresh, to occur after the table gets the latest data. If old records are maintained in the table as a log, you can use the expira- tion date to filter out old records if you only want the latest ones, or keep past records if you’re reporting on historical trends. ■ ■“Is this data collected automatically as events occur, or are the values entered by people? Do we have documentation on the interface that they can see with the data entry form field labels?” Data entered by people may be more prone to error because of manual entry, but the people doing the data entry are often extremely valuable to talk to if you want to understand the business processes that generated the data. You can ask them why certain values were selected, what might trigger an update of a record, or what automated processes are kicked off when they make a change or process a batch. It’s a good idea to check to see how the values in each field are distributed: What is the range of possible values? If a column contains categorical Chapter 1 ■ Data Sources 11 values, how many rows fall into each category? If the column contains continuous or discrete numeric values, what is the shape of the statistical distribution? I find that it helps to visualize the data at this exploratory stage, called Exploratory Data Analysis (EDA). Histograms are especially useful for this purpose. Additionally, you might explore the data broken down by time period (such as by fiscal year) to see if those distributions change over time. If you find that they do, you may find out by talking to the SMEs that there is a point at which old records stop being updated, a business process changed, or past values get zeroed out in certain cases, for example. Knowing how their data entry forms look can also help with communi- cation with SMEs about the data, because they may not know the field names in the underlying database but will be able to describe the data using the labels", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 16 + }, + { + "text": "values get zeroed out in certain cases, for example. Knowing how their data entry forms look can also help with communi- cation with SMEs about the data, because they may not know the field names in the underlying database but will be able to describe the data using the labels they can see on the front-end interface. Knowing the type of database is also important for writing more efficient queries, but that is something you are likely to know from the start, as you will probably need that information in order to connect to it. In some database sys- tems, limiting the number of rows returned will make a query run faster. How- ever, in “columnar” database systems like Redshift, even when limiting results to a single row, returning data from all columns may take longer to complete than summarizing all of the values in a single column across thousands of rows because of how the data is physically stored and compiled behind the scenes before being returned to you. Additionally, you will need to know the type of database in order to look up SQL syntax details in the official documentation, since syntax can differ slightly between database systems. Introduction to the Farmer’s Market Database The MySQL database we’ll be using for example queries throughout much of this book serves as a tracking system for vendors, products, customers, and sales at a fictional farmer’s market. This relational database contains information about each day the farmer’s market is open, such as the date, the hours, the day of the week, and the weather. There is data about each vendor, including their booth assignments, products, and prices. We’re going to pretend that (unlike at many real farmer’s markets) vendors use networked cash registers to ring up individual items, and customers scan farmer’s market loyalty cards with every transaction, so we have detailed logs of their purchases (we know who purchased which items and exactly when). The Farmer’s Market database was designed to allow for demonstration of a variety of queries, including those a data analyst might write to answer business 12 Chapter 1 ■ Data Sources questions about the market, such as “How many people visit the market each week throughout the year, and when do our sales peak?” “How much does inclement weather impact attendance?” “When is each type of fresh fruit or vegetable in season, locally?” and “Is this proposed new vendor likely to take business away from existing vendors, or is there enough demand to support another vendor selling these goods?” We will also prepare datasets based on this database for use with predictive modeling techniques like classification and time-series forecasting, transform- ing the data into a format that can be used as an input for statistical models and machine learning algorithms. These can help answer questions such as “How many shoppers should we expect to have next month? Next year?” and “Based on what we know about their purchase history, is this customer likely to return in", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 17 + }, + { + "text": "that can be used as an input for statistical models and machine learning algorithms. These can help answer questions such as “How many shoppers should we expect to have next month? Next year?” and “Based on what we know about their purchase history, is this customer likely to return in the next month?” Figure 1.7 shows the ERD for the entire Farmer’s Market database. Throughout this book, we will be diving into details of the different tables and relationships depicted in Figure 1.7, as we learn how to write SQL statements that can actu- ally be used to pull data from this database, giving realistic examples of queries that data analysts and data scientists develop. A Note on Machine Learning Dataset Terminology So far, we have defined rows (or records) and columns (or attributes or fields) the way a database developer might. However, if the table is a transformed dataset designed for use in training a predictive model (which is what you will be learning to create throughout this book), a data scientist might use different terminology to describe the rows and columns in that dataset. Figure 1.7 Chapter 1 ■ Data Sources 13 In this special use case, the set of values in each row can be used as inputs to train a model, and that row is often called a “training example” (or “instance” or “data point”). And each input column is a “feature” (or “input variable”). A machine learning algorithm might rank important features, letting you know which attributes are most useful to the model for making its prediction. The column that contains the output that the model is trying to predict is called the “target variable.” By the end of this book, you will have learned how to convert “rows and columns” in database tables into “training examples” with “features” for a pre- dictive model to learn from. Exercises 1. What do you think will happen in the described Books and Authors data- base depicted in Figure 1.5 if an author changes their name? Which records might be added or updated, and what might be the effect on the results of future queries based on this data? 2. Think of something in your life that you could track using a database. What entities in this database might have one-to-many relationships with one another? Many-to-many? C H A P T E R 15 2 Before we can discuss designing analytical datasets, you first need to understand the syntax of Structured Query Language (SQL), so the next several chapters will cover SQL basics. Throughout the book, you will see many variations on the themes introduced here, combining these basic concepts with increasing complexity, because even the most complex SQL queries boil down to the same basic concepts. The SELECT Statement The majority of the queries in this book will be SELECT statements. A SELECT statement is SQL code that retrieves data from the database. When used in combination with other SQL keywords, SELECT can be used to view", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 18 + }, + { + "text": "SQL queries boil down to the same basic concepts. The SELECT Statement The majority of the queries in this book will be SELECT statements. A SELECT statement is SQL code that retrieves data from the database. When used in combination with other SQL keywords, SELECT can be used to view data from a set of columns in a database table, combine data from multiple tables, filter the results, perform calculations, and more. NOTE You’ll often see the word SELECT capitalized in SQL code, and I chose to follow that formatting standard in this book, because SELECT is a reserved SQL key- word, meaning it is a special instructional word that the code interpreter uses to exe- cute your query. Capitalizing it visually differentiates the keyword from other text in your query, such as field names. The SELECT Statement 16 Chapter 2 ■ The SELECT Statement The Fundamental Syntax Structure of a SELECT Query SQL SELECT queries follow this basic syntax, though most of the clauses are optional: SELECT [columns to return] FROM [schema.table] WHERE [conditional filter statements] GROUP BY [columns to group on] HAVING [conditional filter statements that are run after grouping] ORDER BY [columns to sort on] The SELECT and FROM clauses are generally required, because those indicate which columns to select and from what table. The words in brackets are place- holders, and as you go through the next several chapters of this book, you will learn what to put in each section. Selecting Columns and Limiting the Number of Rows Returned The simplest SELECT statement is SELECT * FROM [schema.table] where [schema.table] is the name of the database schema and table you want to retrieve data from. NOTE In Chapter 5, “SQL JOINs,” you’ll learn the syntax for pulling data from mul- tiple tables at once. For example, SELECT * FROM farmers_market.product can be read as “Select everything from the product table in the farmers_market schema.” The asterisk really represents “all columns,” so technically it’s “Select all columns from the product table in the farmers_market schema,” but since there are no filters in this query (no WHERE clause, which you’ll learn about in Chapter 3, “The WHERE Clause”), it will also return all rows, hence “everything.” There is another optional clause I didn’t include in the basic SELECT syntax in the previous section: the LIMIT clause. (The syntax is different in some database systems. See the note about the TOP keyword and WHERE clause.) I frequently use LIMIT while developing queries. LIMIT sets the maximum number of rows Chapter 2 ■ The SELECT Statement 17 that are returned, in cases when you don’t want to see all of the results you would otherwise get. It is especially useful when you’re developing queries that pull from a large database, when queries take a long time to run, since the query will stop executing and show the output once it’s generated the set number of rows. By using LIMIT, you can get a preview of your current query’s results", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 19 + }, + { + "text": "useful when you’re developing queries that pull from a large database, when queries take a long time to run, since the query will stop executing and show the output once it’s generated the set number of rows. By using LIMIT, you can get a preview of your current query’s results without having to wait for all of the results to be compiled. For example, SELECT * FROM farmers_market.product LIMIT 5 returns all columns for the first 5 rows of the product table, as shown in Figure 2.1. NOTE When querying MS SQL Server databases, there is a keyword similar to MySQL’s LIMIT called TOP (which goes before the SELECT statement). For Oracle databases, this is accomplished in the WHERE clause using “WHERE rownum <= [number]”. This is one example of the slight differences in SQL syntax across database systems. However, the basic concepts are the same. So, if you learn SQL for one type of database, you will know enough to know what to look up to find the equivalents for other databases. In MySQL syntax, the keyword is LIMIT, which goes at the end of the query. You might have noticed that I put the FROM clause on the second line in the SELECT query in the preceding example, while the first SELECT query was written as a single line. Line breaks and tabs don’t matter to SQL code execution and are treated like spaces, so you can indent your SQL and break it into multiple lines for readability without affecting the output. To specify which columns you want returned, list the column names imme- diately after SELECT, separated by commas, instead of using the asterisk. The following query lists five product IDs and their associated product names from the product table, as displayed in Figure 2.2. SELECT product_id, product_name FROM farmers_market.product LIMIT 5 Figure 2.1 18 Chapter 2 ■ The SELECT Statement TIP Even if you want all columns returned, it’s good practice to list the names of the columns instead of using the asterisk, especially if the query will be used as part of a data pipeline (if the query is automatically run nightly and the results are used as an input into the next step of a series of code functions without human review, for example). This is because returning “all” columns may result in a different output if the underlying table is modified, such as when a new column is added, or columns appear in a different order, which could break your automated data pipeline. The following query lists five rows of farmer’s market vendor booth assign- ments, displaying the market date, vendor ID, and booth number from the vendor_booth_assignments table, depicted in Figure 2.3: SELECT market_date, vendor_id, booth_number FROM farmers_market.vendor_booth_assignments LIMIT 5 But this output would make a lot more sense if it were sorted by the market date, right? In the next section, we’ll learn how to set the sort order of query results. The ORDER BY Clause: Sorting Results The ORDER BY", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 20 + }, + { + "text": "SELECT market_date, vendor_id, booth_number FROM farmers_market.vendor_booth_assignments LIMIT 5 But this output would make a lot more sense if it were sorted by the market date, right? In the next section, we’ll learn how to set the sort order of query results. The ORDER BY Clause: Sorting Results The ORDER BY clause is used to sort the output rows. In it, you list the columns you want to sort the results by, in order, separated by commas. You can also specify whether you want the sorting to be in ascending (ASC) or descending (DESC) order. ASC sorts text alphabetically and numeric values from low to high, and DESC sorts them in the reverse order. In MySQL, NULL values appear first when sorting in default ascending order. Figure 2.2 Figure 2.3 Chapter 2 ■ The SELECT Statement 19 NOTE The sort order is ascending by default, so if you want your values sorted in ascending order, the ASC keyword is optional. The following query sorts the results by product name, even though the product ID is listed first in the output, shown in Figure 2.4: SELECT product_id, product_name FROM farmers_market.product ORDER BY product_name LIMIT 5 And the following modification to the ORDER BY clause changes the query to now sort the results by product ID, highest to lowest, as shown in Figure 2.5: SELECT product_id, product_name FROM farmers_market.product ORDER BY product_id DESC LIMIT 5 Note that the rows returned display a different set of products than we saw in the previous query. That’s because the ORDER BY clause is executed before the LIMIT is imposed. So in addition to limiting the number of rows returned for quicker development (or to conserve space on the page of a book!), a LIMIT clause can also be combined with an ORDER BY statement to sort the results and return the top x number of results, for reporting or validation purposes. To sort the output of the last query in the previous section, we can add an ORDER BY line, specifying that we want it to sort the output first by market date, then by vendor ID. In Figure 2.6, we only see rows with the earliest market date available in the database, since we sorted by market_date first, there are more than 5 records Figure 2.4 Figure 2.5 20 Chapter 2 ■ The SELECT Statement in the table for that same date, and we limited this query to only return the first 5 rows. After sorting by market date, the records are then sorted by vendor ID in ascending order: SELECT market_date, vendor_id, booth_number FROM farmers_market.vendor_booth_assignments ORDER BY market_date, vendor_id LIMIT 5 Introduction to Simple Inline Calculations In this section, you will see examples of calculations being performed on the data in some columns. We’ll dive deeper into calculations throughout later chapters, but this will give you a sense of how calculations look in the basic SELECT query syntax. Let’s say we wanted to do a calculation using the data in two different columns in", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 21 + }, + { + "text": "calculations being performed on the data in some columns. We’ll dive deeper into calculations throughout later chapters, but this will give you a sense of how calculations look in the basic SELECT query syntax. Let’s say we wanted to do a calculation using the data in two different columns in each row. In the customer_purchases table, we have a quantity column and a cost_to_customer_per_qty column, so we can multiply those to get a price. In Figure 2.7, you can see how the raw data in the selected columns of the customer_purchases table looks prior to adding any calculations to the fol- lowing query: SELECT market_date, customer_id, vendor_id, quantity, cost_to_customer_per_qty FROM farmers_market.customer_purchases LIMIT 10 Figure 2.6 Figure 2.7 Chapter 2 ■ The SELECT Statement 21 The following query demonstrates how the values in two different columns can be multiplied by one another by putting an asterisk between them. When used this way, the asterisk represents a multiplication sign. The results of the calculation are shown in the last column in Figure 2.8. SELECT market_date, customer_id, vendor_id, quantity, cost_to_customer_per_qty, quantity * cost_to_customer_per_qty FROM farmers_market.customer_purchases LIMIT 10 To give the calculated column a meaningful name, we can create an alias by adding the keyword AS after the calculation and then specifying the new name. If your alias includes spaces, it should be surrounded by single quotes. I prefer not to use spaces in my aliases, to avoid having to remember to surround them with quotes if I need to reference them later. Here, we’ll give the alias “price” to the result of the “quantity times cost_ to_customer_per_qty” calculation. There is no need for the columns used for the calculation to also be included individually like they were in the previous query, so we can remove them in this version. SELECT market_date, customer_id, vendor_id, quantity * cost_to_customer_per_qty AS price FROM farmers_market.customer_purchases LIMIT 10 The column alias is shown in the header of the last column in Figure 2.9. Figure 2.8 22 Chapter 2 ■ The SELECT Statement In MySQL syntax, the AS keyword is actually optional, so the following query will return the same results as the previous query. For clarity, we will use the AS convention for assigning column aliases in this book. SELECT market_date, customer_id, vendor_id, quantity * cost_to_customer_per_qty price FROM farmers_market.customer_purchases LIMIT 10 A sensible next step would be to calculate transaction totals (how much the customer paid for all of the products they purchased on that day from each vendor). This could be accomplished by adding up the price per customer per market vendor per market date. In Chapter 6, “Aggregating Results for Anal- ysis,” you will learn about aggregate calculations, which calculate summaries across multiple rows. Because we aren’t aggregating our results yet, the calcu- lation in the preceding query is applied to the values in each row, as displayed in Figure 2.9, and does not calculate across rows or summarize multiple rows. More Inline Calculation Examples: Rounding A SQL function is a piece of code that takes inputs", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 22 + }, + { + "text": "aggregating our results yet, the calcu- lation in the preceding query is applied to the values in each row, as displayed in Figure 2.9, and does not calculate across rows or summarize multiple rows. More Inline Calculation Examples: Rounding A SQL function is a piece of code that takes inputs that you give it (which are called parameters), performs some operation on those inputs, and returns a value. You can use functions inline in your query to modify the raw values from the database tables before displaying them in the output. A SQL function call uses the following syntax: FUNCTION_NAME([parameter 1],[parameter 2], . . . .[parameter n]) Each bracketed item shown is a placeholder for an input parameter. Param- eters go inside the parentheses following the function name and are separated by commas. The input parameter might be a field name or a value that gives the Figure 2.9 Chapter 2 ■ The SELECT Statement 23 function further instructions. To determine what input parameters a particular function requires, you can look it up in the database documentation, which you can find at dev.mysql.com/doc for MySQL. To give an example of how functions are used in SELECT statements, we’ll use the ROUND() function to round a number. In the query from the previous section, the “price” field was displayed with four digits after the decimal point. Let’s say we wanted to display the number rounded to the nearest penny (in US dollars), which is two digits after the decimal. That can be accomplished using the ROUND() function. The syntax of the ROUND() function is for the first parameter (the first item inside the parentheses) to represent the value to be rounded, followed by a comma, then the second parameter indicating the desired number of digits after the decimal. So ROUND([column name], 3) will round the values in the specified column to 3 digits after the decimal. We can update our query from the previous section to put the price calcula- tion inside the ROUND() function: SELECT market_date, customer_id, vendor_id, ROUND(quantity * cost_to_customer_per_qty, 2) AS price FROM farmers_market.customer_purchases LIMIT 10 The result of this rounding is shown in the final column of the output, dis- played in Figure 2.10. TIP The ROUND() function can also accept negative numbers for the second parameter, to round digits that are to the left of the decimal point. For example, SELECT ROUND(1245, -2) will return a value of 1200. Figure 2.10 24 Chapter 2 ■ The SELECT Statement More Inline Calculation Examples: Concatenating Strings In addition to performing numeric operations, there are also inline functions that can be used to modify string values in SQL, as well. In our customer table, there are separate columns for each customer’s first and last names, as shown in Figure 2.11. SELECT * FROM farmers_market.customer LIMIT 5 Let’s say we wanted to merge each customer’s name into a single column that contains the first name, then a space, and then the last name. We can accom- plish that by using", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 23 + }, + { + "text": "each customer’s first and last names, as shown in Figure 2.11. SELECT * FROM farmers_market.customer LIMIT 5 Let’s say we wanted to merge each customer’s name into a single column that contains the first name, then a space, and then the last name. We can accom- plish that by using the CONCAT() function. The list of string values you want to merge together are entered into the CONCAT() function as parameters. A space can be included by surrounding it with quotes. You can see the result of this concatenation in Figure 2.12. SELECT customer_id, CONCAT(customer_first_name, \" \", customer_last_name) AS customer_ name FROM farmers_market.customer LIMIT 5 Note that we can still add an ORDER BY clause and sort by last name first, even though the columns are merged together in the output, as shown in Figure 2.13. Figure 2.11 Figure 2.12 Chapter 2 ■ The SELECT Statement 25 NOTE As we discussed earlier in the “Selecting Columns and Limiting the Number of Rows Returned” section, the LIMIT clause determines how many results we will display. Because more than five names are stored in our table, and we are limiting our results to 5, you see the names change from Figure 2.12 to Figure 2.13 as the sort order changes. SELECT customer_id, CONCAT(customer_first_name, \" \", customer_last_name) AS customer_ name FROM farmers_market.customer ORDER BY customer_last_name, customer_first_name LIMIT 5 It’s also possible to nest functions inside other functions, which are executed by the SQL interpreter from the “inside” to the “outside.” UPPER() is a function that capitalizes string values. We can enclose the CONCAT() function inside it to uppercase the full name. Let’s also change the order of the concatenation parameters to put the last name first, and add a comma after the last name (note the comma before the space inside the double quotes). The result is shown in Figure 2.14. SELECT customer_id, UPPER(CONCAT(customer_last_name, \", \", customer_first_name)) AS customer_name FROM farmers_market.customer ORDER BY customer_last_name, customer_first_name LIMIT 5 Because the CONCAT() function is contained inside the parentheses of the UPPER() function, the concatenation is performed first, and then the combined string is uppercased. Figure 2.13 Figure 2.14 26 Chapter 2 ■ The SELECT Statement NOTE Note that we did not sort on the new derived column alias customer_ name here, but on columns that exist in the customer table. In some cases (depend- ing on what database system you’re using, which functions are used, and the execution order of your query) you can’t reuse aliases in other parts of the query. It is possible to put some functions or calculations in the ORDER BY clause, to sort by the result- ing value. Some other options for referencing derived values will be covered in later chapters. Evaluating Query Output When you are developing a SQL SELECT statement, how do you know if the result will include the rows and columns you expect, in the form you expect? As previously demonstrated, one method is to run the query with a LIMIT each time you make a", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 24 + }, + { + "text": "chapters. Evaluating Query Output When you are developing a SQL SELECT statement, how do you know if the result will include the rows and columns you expect, in the form you expect? As previously demonstrated, one method is to run the query with a LIMIT each time you make a modification. This gives a quick preview of the first x number of rows to ensure the changes you expect to see are returned, and you can inspect the column names and format of a few output values to verify that they look the way you intended. However, you still might want to confirm how many rows would have been returned if you hadn’t placed the LIMIT on the results. Similarly, there’s the concern that your function might not perform as expected on some values that didn’t appear in your limited results preview. I therefore use the query editor to help me review the results of my query a bit further. This method doesn’t provide a full quality control of the output (which should be done before putting any query into production), but it can give me a sanity check that the output looks correct enough for me to continue on with my work. To demonstrate, we’ll use the “rounded price” query from the earlier “More Inline Calculation Examples: Rounding” section. First, I remove the LIMIT. Note that your query editor might have a built-in limit (such as 2000 rows) to prevent you from generating a gigantic dataset by accident, so you might need to go into the settings and turn off any pre-set row limits to actually return the full dataset. Figure 2.15 shows the “Don’t Limit” option available in MySQL Workbench, under the Query menu. Then, I’ll run the query to generate the output for inspection: SELECT market_date, customer_id, vendor_id, ROUND(quantity * cost_to_customer_per_qty, 2) AS price FROM farmers_market.customer_purchases Chapter 2 ■ The SELECT Statement 27 The first thing I’ll look at in the output is the total count of rows returned, to see if it matches my expectations. This is often displayed at the bottom of the results window or in the output window. In this prototype version of the Farmer’s Market database, there are only 21 rows in the customer_purchases table, which is indicated in the Message column of the Output section of MySQL Workbench, shown in the lower right of Figure 2.16. (Note that there will be more rows when you run this query yourself, after a more realistic volume of data has been added to the database). Next, I’ll look at the resulting dataset that was generated (which is called the “Result Grid” in MySQL Workbench). I check the column headers, to see if I need to change any aliases. Then, I scroll through and spot-check a few of the values in the output, looking for anything that stands out as incorrect or surprising. If I included an ORDER BY clause (which I did not in this case), I’ll also ensure the results are sorted the", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 25 + }, + { + "text": "to change any aliases. Then, I scroll through and spot-check a few of the values in the output, looking for anything that stands out as incorrect or surprising. If I included an ORDER BY clause (which I did not in this case), I’ll also ensure the results are sorted the way I intended. Then, I can use the editor to manually sort each column first in one direction and then the other. For example, Figures 2.17 and 2.18 show the query results manually sorted in the Result Grid by market_date and vendor_id, respectively. This allows me to look at the minimum and maximum values in each, because that’s often where “edge cases” exist, such as unexpected NULLs, strings that Figure 2.15 28 Chapter 2 ■ The SELECT Statement start with spaces or numbers, or data entry or calculation errors that increase numeric values by a large factor. I can explore anything that looks strange, and I might also spot if there is an unusual value present, because series of frequent values will appear side by side in the sorted results, making both frequent and unique values noticeable as you scroll down the sorted column. In Chapter 6, you will learn about aggregate queries, which will provide more options for inspecting your results, but these are a few simple steps you Figure 2.17 Figure 2.16 Chapter 2 ■ The SELECT Statement 29 can take, without writing a more complicated query, to make sure your output looks sensible. SELECT Statement Summary In this chapter, you learned basic SQL SELECT statement syntax and how to pull your desired columns from a single table and sort the output. You also learned how simple inline calculations look. Note that every query in this chapter, even the ones that started to look more complex by including calculations, all followed this basic syntax: SELECT [columns to return] FROM [schema.table] ORDER BY [columns to sort on] You should now be able to describe what the following two queries do. The results for the following query are shown in Figure 2.19: SELECT * FROM farmers_market.vendor The results for the following query are shown in Figure 2.20: SELECT vendor_name, vendor_id, vendor_type FROM farmers_market.vendor ORDER BY vendor_name Figure 2.18 30 Chapter 2 ■ The SELECT Statement Exercises Using the Included Database The following exercises refer to the customer table. The columns contained in the customer table, and some example rows with data values, are shown in Figure 2.11. 1. Write a query that returns everything in the customer table. 2. Write a query that displays all of the columns and 10 rows from the cus- tomer table, sorted by customer_last_name, then customer_first_ name. 3. Write a query that lists all customer IDs and first names in the customer table, sorted by first_name. Figure 2.19 Figure 2.20 C H A P T E R 31 3 Now that you have the basic idea of how SQL queries look and have learned how to return the columns of data that you want from", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 26 + }, + { + "text": "IDs and first names in the customer table, sorted by first_name. Figure 2.19 Figure 2.20 C H A P T E R 31 3 Now that you have the basic idea of how SQL queries look and have learned how to return the columns of data that you want from a single database table, we can talk about how to filter that result to include only the rows that you want returned. The WHERE Clause The WHERE clause is the part of the SELECT statement in which you list con- ditions that are used to determine which rows in the table should be included in the results set. In other words, the WHERE clause is used for filtering. If you have programmed in other languages, you have likely encountered other conditional statements such as “IF” statements, which use boolean logic (think “AND” or “OR”) to determine what action to take, based on whether certain conditions are met. SQL uses boolean logic to check the available data against conditions in your WHERE clause to determine whether to include each row in the output. I use the WHERE clause in almost every query I write as a data scientist to accomplish things like narrowing down categories of records to be displayed in a report, or filtering a dataset to a particular date range from the past that will be used to train a predictive model. The WHERE Clause 32 Chapter 3 ■ The WHERE Clause Filtering SELECT Statement Results The WHERE clause goes after the FROM statement and before any GROUP BY, ORDER BY, or LIMIT statements in the SELECT query: SELECT [columns to return] FROM [table] WHERE [conditional filter statements] ORDER BY [columns to sort on] For example, to get a list of product IDs and product names that are in product category 1, you could use a conditional statement in the WHERE clause to select only rows from the product table in which the product_category_id is 1, as demonstrated by the following query and the output in Figure 3.1: SELECT product_id, product_name, product_category_id FROM farmers_market.product WHERE product_category_id = 1 LIMIT 5 One of the queries in Chapter 2 , “The SELECT Statement,” returned a list of prices for items purchased by customers at the farmer’s market, displaying the market date, customer ID, vendor ID, and calculated price. Let’s say we wanted to print a report of everything a particular customer has ever purchased at the farmer’s market, sorted by market date, vendor ID, and product ID. We could use a WHERE clause to specify that we want to filter the results of that query to a specified customer ID, and an ORDER BY clause to customize the field sort order: SELECT market_date, customer_id, vendor_id, Figure 3.1 Chapter 3 ■ The WHERE Clause 33 product_id, quantity, quantity * cost_to_customer_per_qty AS price FROM farmers_market.customer_purchases WHERE customer_id = 4 ORDER BY market_date, vendor_id, product_id LIMIT 5 Figure 3.2 shows the result of this query, which is essentially a line-item receipt of all purchases", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 27 + }, + { + "text": "SELECT market_date, customer_id, vendor_id, Figure 3.1 Chapter 3 ■ The WHERE Clause 33 product_id, quantity, quantity * cost_to_customer_per_qty AS price FROM farmers_market.customer_purchases WHERE customer_id = 4 ORDER BY market_date, vendor_id, product_id LIMIT 5 Figure 3.2 shows the result of this query, which is essentially a line-item receipt of all purchases made at the farmer’s market by the customer with an ID of 4. Changing the “4” in the WHERE clause to another customer’s ID and re-running the query would result in the same columns, but different rows, listing another customer’s purchases. Note that in order to simplify the query, I did not format the price as currency in the output, but you can refer to the section in Chapter 2 that covers the ROUND() function if you want to format your numeric price output to 2 digits after the decimal. What’s actually happening behind the scenes is that each of the conditional statements (like “customer_id = 4”) listed in the WHERE clause will evaluate to TRUE or FALSE for each row, and only the rows for which the combination of conditions evaluates to TRUE will be returned. For example, if a database table contains the transactional data shown in the first six columns of Table 3.1, and the WHERE clause condition is “customer_id = 4” as shown in the previous query, the condition will evaluate to TRUE only for the rows in the table where the customer_id is exactly 4 (as shown in the last column of Table 3.1), and the results will be filtered to only include those six rows, resulting in the output previously shown in Figure 3.2. The customer_id values in the database table are integers, not string characters. If the customer_id values were strings, the comparison value in the WHERE clause would also need to be a string, meaning the ‘4’ would need to be enclosed in single quotes. Figure 3.2 34 Chapter 3 ■ The WHERE Clause Filtering on Multiple Conditions You can combine multiple conditions with boolean operators, such as “AND,” “OR,” or “AND NOT” between them in order to filter using multiple criteria in the WHERE clause. Clauses with OR between them will jointly evaluate to TRUE, meaning the row will be returned, if any of the clauses are TRUE. Clauses with AND between them will only evaluate to TRUE in combination if all of the clauses evaluate to TRUE. Otherwise, the row will not be returned. Remember that NOT flips the following boolean value to its opposite (TRUE becomes FALSE, and vice versa). See Table 3.2. Table 3.2 CONDITION 1 EVALUATES TO BOOLEAN OPERATOR CONDITION 2 EVALUATES TO ROW RETURNED? TRUE OR FALSE TRUE TRUE OR TRUE TRUE FALSE OR FALSE FALSE TRUE AND FALSE FALSE TRUE AND TRUE TRUE TRUE AND NOT FALSE TRUE Table 3.1 MARKET_ DATE CUSTOMER_ ID VENDOR_ID PRODUCT_ ID QUANTITY PRICE CONDITION: CUSTOMER_ ID = 4 2019-03-02 3 4 4 8.4 16.80 FALSE 2019-03-02 1 1 11 1.7 20.40 FALSE 2019-03-02 4 4 9 1.4 2.80 TRUE", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 28 + }, + { + "text": "TRUE AND FALSE FALSE TRUE AND TRUE TRUE TRUE AND NOT FALSE TRUE Table 3.1 MARKET_ DATE CUSTOMER_ ID VENDOR_ID PRODUCT_ ID QUANTITY PRICE CONDITION: CUSTOMER_ ID = 4 2019-03-02 3 4 4 8.4 16.80 FALSE 2019-03-02 1 1 11 1.7 20.40 FALSE 2019-03-02 4 4 9 1.4 2.80 TRUE 2019-03-02 4 8 4 2.0 8.00 TRUE 2019-03-09 5 9 7 1.0 16.00 FALSE 2019-03-09 4 1 10 1.0 5.50 TRUE 2019-03-09 4 4 9 9.9 19.80 TRUE 2019-03-09 4 7 12 2.0 6.00 TRUE 2019-03-09 4 7 13 0.3 1.72 TRUE 2019-03-16 3 4 9 5.5 11.00 FALSE 2019-03-16 3 9 8 1.0 18.00 FALSE Chapter 3 ■ The WHERE Clause 35 So if the WHERE clause lists two conditions with OR between them, like “WHERE customer_id = 3 OR customer_id = 4,” then each condition will be evaluated for each row, and rows where the customer_id is either 3 or 4 (either condition is met) will be returned, as shown in Table 3.3 (some columns have been removed for readability). Because there is an OR between the two conditions, only one of the conditions has to evaluate to TRUE in order for a row to be returned. In fact, if there is a long list of conditions, with OR between all of them, only one condition in the entire list has to evaluate to TRUE per row in order for the row to be returned, because it can be read as “Either [Condition 1] is TRUE OR [Condition 2] is TRUE OR [Condition 3] is TRUE,” etc. So, only if all items in the list of “OR conditions” evaluate to FALSE is a row not returned. Table 3.3 MARKET_ DATE CUSTOMER _ID VENDOR _ID PRICE CONDITION: CUSTOMER_ ID = 3 OR CONDITION: CUSTOMER_ ID = 4 ROW RETURNED? 2019-03-02 3 4 16.80 TRUE OR FALSE TRUE 2019-03-02 1 1 20.40 FALSE OR FALSE FALSE 2019-03-02 4 4 2.80 FALSE OR TRUE TRUE 2019-03-02 4 8 8.00 FALSE OR TRUE TRUE 2019-03-09 5 9 16.00 FALSE OR FALSE FALSE 2019-03-09 4 1 5.50 FALSE OR TRUE TRUE 2019-03-09 4 4 19.80 FALSE OR TRUE TRUE 2019-03-09 4 7 6.00 FALSE OR TRUE TRUE 2019-03-09 4 7 1.72 FALSE OR TRUE TRUE 2019-03-16 3 4 11.00 TRUE OR FALSE TRUE 2019-03-16 3 9 18.00 TRUE OR FALSE TRUE CONDITION 1 EVALUATES TO BOOLEAN OPERATOR CONDITION 2 EVALUATES TO ROW RETURNED? FALSE AND NOT TRUE FALSE FALSE AND NOT FALSE FALSE FALSE OR NOT FALSE TRUE 36 Chapter 3 ■ The WHERE Clause Here is a query containing the conditions illustrated in Table 3.3 in the WHERE clause, and in Figure 3.3, you can see the actual output: SELECT market_date, customer_id, vendor_id, product_id, quantity, quantity * cost_to_customer_per_qty AS price FROM farmers_market.customer_purchases WHERE customer_id = 3 OR customer_id = 4 ORDER BY market_date, customer_id, vendor_id, product_id What would happen if the WHERE clause condition were “customer_id = 3 AND customer_id = 4”? Let’s use the same table setup to illustrate what each", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 29 + }, + { + "text": "vendor_id, product_id, quantity, quantity * cost_to_customer_per_qty AS price FROM farmers_market.customer_purchases WHERE customer_id = 3 OR customer_id = 4 ORDER BY market_date, customer_id, vendor_id, product_id What would happen if the WHERE clause condition were “customer_id = 3 AND customer_id = 4”? Let’s use the same table setup to illustrate what each condition evaluates to for each row. See Table 3.4. Figure 3.3 Table 3.4 MARKET_ DATE CUSTOMER _ID VENDOR _ID PRICE CONDITION: CUSTOMER_ ID = 3 AND CONDITION: CUSTOMER_ ID = 4 ROW RETURNED? 2019-03-02 3 4 16.80 TRUE AND FALSE FALSE 2019-03-02 1 1 20.40 FALSE AND FALSE FALSE 2019-03-02 4 4 2.80 FALSE AND TRUE FALSE 2019-03-02 4 8 8.00 FALSE AND TRUE FALSE 2019-03-09 5 9 16.00 FALSE AND FALSE FALSE Chapter 3 ■ The WHERE Clause 37 The correct way to read a query with the conditional statement “WHERE cus- tomer_id = 3 AND customer_id = 4” is “Return each row where the customer ID is 3 and the customer ID is 4.” But there is only a single customer_id value per row, so it’s impossible for the customer_id to be both 3 and 4 at the same time, therefore no rows are returned! Some people make the mistake of reading the logical AND operator the way we might request in English, “Give me all of the rows with customer IDs 3 and 4,” when what we really mean by that phrase is “Give me all of the rows where the customer ID is either 3 or 4,” which would require an OR operator in SQL. When the AND operator is used, all of the conditions with AND between them must evaluate to TRUE for a row in order for that row to be returned in the query results. One example where you could use AND in a WHERE clause referring to only a single column is when you want to return rows with a range of values. If someone requests “Give me all of the rows with a customer ID greater than 3 and less than or equal to 5,” the conditions would be written as “WHERE customer_id > 3 AND customer_id <= 5,” and would evaluate as shown in Table 3.5. Because of the AND, both conditions must evaluate to TRUE in order for a row to be returned. Let’s try it in SQL, and see the output in Figure 3.4: SELECT market_date, customer_id, vendor_id, product_id, MARKET_ DATE CUSTOMER _ID VENDOR _ID PRICE CONDITION: CUSTOMER_ ID = 3 AND CONDITION: CUSTOMER_ ID = 4 ROW RETURNED? 2019-03-09 4 1 5.50 FALSE AND TRUE FALSE 2019-03-09 4 4 19.80 FALSE AND TRUE FALSE 2019-03-09 4 7 6.00 FALSE AND TRUE FALSE 2019-03-09 4 7 1.72 FALSE AND TRUE FALSE 2019-03-16 3 4 11.00 TRUE AND FALSE FALSE 2019-03-16 3 9 18.00 TRUE AND FALSE FALSE 38 Chapter 3 ■ The WHERE Clause quantity, quantity * cost_to_customer_per_qty AS price FROM farmers_market.customer_purchases WHERE customer_id > 3 AND customer_id <= 5 ORDER BY market_date, customer_id, vendor_id, product_id You can", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 30 + }, + { + "text": "AND TRUE FALSE 2019-03-16 3 4 11.00 TRUE AND FALSE FALSE 2019-03-16 3 9 18.00 TRUE AND FALSE FALSE 38 Chapter 3 ■ The WHERE Clause quantity, quantity * cost_to_customer_per_qty AS price FROM farmers_market.customer_purchases WHERE customer_id > 3 AND customer_id <= 5 ORDER BY market_date, customer_id, vendor_id, product_id You can combine multiple AND, OR, and NOT conditions, and control in which order they get evaluated, by using parentheses the same way you would in an algebraic expression to specify the order of operations. The conditions inside the parentheses get evaluated first. Figure 3.4 Table 3.5 MARKET_ DATE CUSTOMER _ID VENDOR _ID PRICE CONDITION: CUSTOMER_ ID > 3 AND CONDITION: CUSTOMER_ ID <= 5 ROW RETURNED? 2019-03-02 3 4 16.80 FALSE AND TRUE FALSE 2019-03-02 1 1 20.40 FALSE AND TRUE FALSE 2019-03-02 4 4 2.80 TRUE AND TRUE TRUE 2019-03-02 4 8 8.00 TRUE AND TRUE TRUE 2019-03-09 5 9 16.00 TRUE AND TRUE TRUE 2019-03-09 4 1 5.50 TRUE AND TRUE TRUE 2019-03-09 4 4 19.80 TRUE AND TRUE TRUE 2019-03-09 4 7 6.00 TRUE AND TRUE TRUE 2019-03-09 4 7 1.72 TRUE AND TRUE TRUE 2019-03-16 3 4 11.00 FALSE AND TRUE FALSE 2019-03-16 3 9 18.00 FALSE AND TRUE FALSE Chapter 3 ■ The WHERE Clause 39 Returning to the product table, let’s examine a couple of queries and compare their output. First, see this query and its output in Figure 3.5: SELECT product_id, product_name FROM farmers_market.product WHERE product_id = 10 OR (product_id > 3 AND product_id < 8) Now look at this query and its output in Figure 3.6: SELECT product_id, product_name FROM farmers_market.product WHERE (product_id = 10 OR product_id > 3) AND product_id < 8 When the product ID is 10, the WHERE clause in the first query is evaluated as: TRUE OR (TRUE AND FALSE) = TRUE OR (FALSE) = TRUE and the WHERE clause in the second query is evaluated as: (TRUE OR TRUE) AND FALSE = (TRUE) AND FALSE = FALSE Figure 3.5 Figure 3.6 40 Chapter 3 ■ The WHERE Clause Since the OR statement evaluates to TRUE if any of the conditions are TRUE, but the AND statement only evaluates to TRUE if all of the conditions are true, the row with a product_id value of 10 is only returned by the first query. Multi-Column Conditional Filtering So far, all of the examples in this chapter have shown combinations of condi- tions that reference only one field at a time. WHERE clauses can also impose conditions using values in multiple columns. For example, if we wanted to know the details of purchases made by cus- tomer 4 at vendor 7, we could use the following query: SELECT market_date, customer_id, vendor_id, quantity * cost_to_customer_per_qty AS price FROM farmers_market.customer_purchases WHERE customer_id = 4 AND vendor_id = 7 The results of this query are shown in Figure 3.7. Let’s try a WHERE clause that uses an OR condition to apply comparisons across multiple fields. This query will return anyone in the customer table", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 31 + }, + { + "text": "quantity * cost_to_customer_per_qty AS price FROM farmers_market.customer_purchases WHERE customer_id = 4 AND vendor_id = 7 The results of this query are shown in Figure 3.7. Let’s try a WHERE clause that uses an OR condition to apply comparisons across multiple fields. This query will return anyone in the customer table with the first name of “Carlos” or the last name of “Diaz,” and its results are shown in Figure 3.8: SELECT customer_id, customer_first_name, customer_last_name FROM farmers_market.customer WHERE customer_first_name = 'Carlos' OR customer_last_name = 'Diaz' Figure 3.7 Chapter 3 ■ The WHERE Clause 41 And of course the conditions don’t both have to be “exact match” filters using equals signs. If you wanted to find out what booth(s) vendor 2 was assigned to on or before (less than or equal to) March 9, 2019, you could use this query (see Figure 3.9 for the output): SELECT * FROM farmers_market.vendor_booth_assignments WHERE vendor_id = 9 AND market_date <= '2019-03-09' ORDER BY market_date More Ways to Filter The filters you have seen so far in this chapter include numeric, string, and date comparisons to determine if a value in a field is greater than, less than, or equal to a given comparison value. Other ways to filter rows based on the values in that row include checking if a field is NULL, comparing a string against another partial string value using a wildcard comparison, determining if a field value is found within a list of values, and determining if a field value lies between two other values, among others. BETWEEN In the previous query, we checked if a date was less than or equal to another date. We can also use the BETWEEN keyword to see if a value, such as a date, is within a specified range of values. This query will find the booth assignments for vendor 7 for any market date that occurred between March 2, 2019, and March 16, 2019, including either of those two dates. The output is shown in Figure 3.10. Figure 3.8 Figure 3.9 42 Chapter 3 ■ The WHERE Clause SELECT * FROM farmers_market.vendor_booth_assignments WHERE vendor_id = 7 AND market_date BETWEEN '2019-03-02' and '2019-03-16' ORDER BY market_date IN To return a list of customers with selected last names, we could use a long list of OR comparisons, as shown in the first query in the following example. An alternative way to do the same thing, which may come in handy if you had a long list of names, is to use the IN keyword and provide a comma-separated list of values to compare against. This will return TRUE for any row with a customer_last_name that is in the provided list. Both queries in the following example return the same results, shown in Figure 3.11: SELECT customer_id, customer_first_name, customer_last_name FROM farmers_market.customer WHERE customer_last_name = 'Diaz' OR customer_last_name = 'Edwards' OR customer_last_name = 'Wilson' ORDER BY customer_last_name, customer_first_name SELECT customer_id, customer_first_name, customer_last_name FROM farmers_market.customer WHERE customer_last_name IN ('Diaz' , 'Edwards', 'Wilson') ORDER BY customer_last_name, customer_first_name Figure 3.10 Figure", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 32 + }, + { + "text": "return the same results, shown in Figure 3.11: SELECT customer_id, customer_first_name, customer_last_name FROM farmers_market.customer WHERE customer_last_name = 'Diaz' OR customer_last_name = 'Edwards' OR customer_last_name = 'Wilson' ORDER BY customer_last_name, customer_first_name SELECT customer_id, customer_first_name, customer_last_name FROM farmers_market.customer WHERE customer_last_name IN ('Diaz' , 'Edwards', 'Wilson') ORDER BY customer_last_name, customer_first_name Figure 3.10 Figure 3.11 Chapter 3 ■ The WHERE Clause 43 Another use of the IN list comparison is if you’re searching for a person in the customer table, but don’t know the spelling of their name. For example, if someone asked you to look up my customer ID, but you didn’t know how to spell my name or whether it had any accents (or if it was entered incorrectly), you might try searching against a list with multiple spellings, like this: SELECT customer_id, customer_first_name, customer_last_name FROM farmers_market.customer WHERE customer_first_name IN ('Renee', 'Rene', 'Renée', 'René', 'Renne') There are not currently any records in the database with any of these names, so the query won’t return any results. But there is another way to accomplish the same type of search without typing all of the various spellings. LIKE Let’s say that there was a farmer’s market customer you knew as “Jerry,” but you weren’t sure if he was listed in the database as “Jerry” or “Jeremy” or “Jeremiah.” All you knew for sure was that the first three letters were “Jer.” In SQL, instead of listing every variation you can think of, you can search for partially matched strings using a comparison operator called LIKE, and wildcard characters, which serve as a placeholder for unknown characters in a string. In MS SQL Server–style SQL, the wildcard character % (percent sign) can serve as a stand-in for any number of characters (including none). So the comparison LIKE ‘Jer%’ will search for strings that start with “Jer” and have any (or no) additional characters after the “r”: SELECT customer_id, customer_first_name, customer_last_name FROM farmers_market.customer WHERE customer_first_name LIKE 'Jer%' In Figure 3.12, you can see that this query returned two customers whose first names matched this pattern, “Jeremy” and “Jeri”. Figure 3.12 44 Chapter 3 ■ The WHERE Clause IS NULL It’s often useful to find rows in the database where a field is blank or NULL. In the product table, the product_size field is not required, so it’s possible to add a record for a product with no size. If you wanted to find all of the products without sizes, maybe in order to fill in that missing data, you could use the IS NULL condition to filter to just those rows (with results shown in Figure 3.13): SELECT * FROM farmers_market.product WHERE product_size IS NULL Keep in mind that “blank” and NULL are not the same thing in database terms. If someone asked you to find all products that didn’t have product sizes, you might also want to check for blank strings, which would equal ‘’ (two single-quotes with nothing between), or rows where someone entered a space or any number of spaces into that field.", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 33 + }, + { + "text": "thing in database terms. If someone asked you to find all products that didn’t have product sizes, you might also want to check for blank strings, which would equal ‘’ (two single-quotes with nothing between), or rows where someone entered a space or any number of spaces into that field. The TRIM() function removes excess spaces from the beginning or end of a string value, so if you use a combination of the TRIM() function and blank string comparison, you can find any row that is blank or contains only spaces. In this case, the “Red Potatoes - Small” row, shown in Figure 3.14, has a product_size with one space in it, ' ', so could be found using the following query: SELECT * FROM farmers_market.product WHERE product_size IS NULL OR TRIM(product_size) = '' A Warning About Null Comparisons You might wonder why the comparison operator IS NULL is used instead of equals NULL in the previous section. NULL is not actually a value, it’s the absence of a value, so it can’t be compared to any existing value. If your query were filtered to WHERE product_size = NULL, no rows would be returned, even though there is a record with a NULL product_size, because nothing “equals” NULL, even NULL. Figure 3.14 Figure 3.13 Chapter 3 ■ The WHERE Clause 45 This is important for other types of comparisons as well. Look at the follow- ing two queries and their output in Figures 3.15 and 3.16: SELECT market_date, transaction_time, customer_id, vendor_id, quantity FROM farmers_market.customer_purchases WHERE customer_id = 1 AND vendor_id = 7 AND quantity > 1 SELECT market_date, transaction_time, customer_id, vendor_id, quantity FROM farmers_market.customer_purchases WHERE customer_id = 1 AND vendor_id = 7 AND quantity <= 1 You might think that if you ran both of the queries, you would get all records in the database, since in one case you’re looking for quantities over 1, and in the other you’re looking for quantities less than or equal to 1, the combination of which appears to contain all possible values. But since NULL values aren’t comparable to numbers in that way, there is a record that is never returned when there’s a numeric comparison used, because it has a NULL value in the quantity field. You can see that if you run this query, which results in Figure 3.17: Figure 3.15 Figure 3.16 46 Chapter 3 ■ The WHERE Clause SELECT market_date, transaction_time, customer_id, vendor_id, quantity FROM farmers_market.customer_purchases WHERE customer_id = 1 AND vendor_id = 7 Ideally, the database should be designed so that the quantity value for a pur- chase record isn’t allowed to be NULL because you can’t buy a NULL number of items, but since NULL values weren’t prevented, one was entered. If you wanted to return all records that don’t have NULL values in a field, you could use the condition “[field name] IS NOT NULL” in the WHERE clause. Filtering Using Subqueries When the IN list comparison was demonstrated earlier, it used a hard-coded list of", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 34 + }, + { + "text": "prevented, one was entered. If you wanted to return all records that don’t have NULL values in a field, you could use the condition “[field name] IS NOT NULL” in the WHERE clause. Filtering Using Subqueries When the IN list comparison was demonstrated earlier, it used a hard-coded list of values. What if you wanted to filter to a list of values that was returned by another query? In other words, you wanted a dynamic list. There is a way to do that in SQL, using a subquery (a query inside a query). Let’s say we wanted to analyze purchases that were made at the farmer’s market on days when it rained. There is a value in the market_date_info table called market_rain_flag that has a value of 0 if it didn’t rain while the market was open and a value of 1 if it did. First, let’s write a query that gets a list of market dates when it rained, using this query: SELECT market_date, market_rain_flag FROM farmers_market.market_date_info WHERE market_rain_flag = 1 The results of this query are shown in Figure 3.18. Figure 3.18 Figure 3.17 Chapter 3 ■ The WHERE Clause 47 Now let’s use the list of dates generated by that query to return purchases made on those dates. Note that when using a query in an IN comparison, you can only return the field you’re comparing to, so we will not include the market_ rain_flag field in the following subquery. Therefore, the query inside the parentheses just returns the dates shown in Figure 3.18, and the “outer” query looks for customer_purchases records with a market_date value in that list of dates. You can see in the results in Figure 3.19 that all of the purchase records returned occurred on the days it rained. SELECT market_date, customer_id, vendor_id, quantity * cost_to_customer_per_qty price FROM farmers_market.customer_purchases WHERE market_date IN ( SELECT market_date FROM farmers_market.market_date_info WHERE market_rain_flag = 1 ) LIMIT 5 Creating results that depend on data in more than one table can also be accomplished using something called a JOIN, which you will learn about in Chapter 5, “SQL JOINs.” Exercises Using the Included Database 1. Refer to the data in Table 3.1. Write a query that returns all customer purchases of product IDs 4 and 9. 2. Refer to the data in Table 3.1. Write two queries, one using two conditions with an AND operator, and one using the BETWEEN operator, that will return all customer purchases made from vendors with vendor IDs between 8 and 10 (inclusive). 3. Can you think of two different ways to change the final query in the chapter so it would return purchases from days when it wasn’t raining? Figure 3.19 C H A P T E R 49 4 In Chapters 2, “The SELECT Statement,” and 3, “The WHERE Clause,” you learned how to specify which columns and rows you want to pull from a data- base table into your dataset. We used the WHERE clause to filter rows using conditional", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 35 + }, + { + "text": "H A P T E R 49 4 In Chapters 2, “The SELECT Statement,” and 3, “The WHERE Clause,” you learned how to specify which columns and rows you want to pull from a data- base table into your dataset. We used the WHERE clause to filter rows using conditional statements that must evaluate to TRUE in order for a row to be returned. But what if, instead of using conditional statements to filter rows, you want a column or value in your dataset to be based on a conditional statement? For example, instead of filtering your results to purchases over $50, say you just want to return all rows and create a new column that flags each purchase as being above or below $50? Or, maybe the machine learning algorithm you want to use can’t accept a categorical string column as an input feature, so you want to encode those categories into numeric values. These are a version of what SQL developers call “derived columns” or “calculated fields,” and creating new columns that present the values differently is what data scientists call “feature engineering.” This is where CASE statements come in. NOTE If you’re familiar with other scripting languages like Python that use “if” statements, you’ll find that SQL handles conditional logic somewhat similarly, just with different syntax. CASE Statements 50 Chapter 4 ■ CASE Statements CASE Statement Syntax You use conditional reasoning in your daily life any time you think “If [one condition] is true, then [take this action]. Otherwise, [take this other action].” “If the weather forecast predicts it will rain today, then I’ll take an umbrella with me. Otherwise, I’ll leave the umbrella at home.” In SQL, the code to delineate this type of logic is called a CASE statement, which uses the following syntax: CASE WHEN [first conditional statement] THEN [value or calculation] WHEN [second conditional statement] THEN [value or calculation] ELSE [value or calculation] END This statement indicates that you want a column to contain different values under different conditions. If we put the umbrella example into this form: CASE WHEN weather_forecast = 'rain' THEN 'take umbrella' ELSE 'leave umbrella at home' END the WHENs are evaluated in order, from top to bottom, and the first time a condition evaluates to TRUE, the corresponding THEN part of the statement is executed, and no other WHEN conditions are evaluated. To illustrate, consider this nonsense query: SELECT CASE WHEN 1=1 THEN 'Yes' WHEN 2=2 THEN 'No' END This query will always evaluate to “Yes,” because 1=1 is always TRUE, and therefore the 2=2 conditional statement is never evaluated, even though it is also true. The ELSE part of the statement is optional, and that value or calculation result is returned if none of the conditional statements above it evaluate to TRUE. If the ELSE is not included and none of the WHEN conditionals evaluate to TRUE, the resulting value will be NULL. You should always alias columns that contain CASE statements so the resulting column headers", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 36 + }, + { + "text": "calculation result is returned if none of the conditional statements above it evaluate to TRUE. If the ELSE is not included and none of the WHEN conditionals evaluate to TRUE, the resulting value will be NULL. You should always alias columns that contain CASE statements so the resulting column headers are readable, as demonstrated in the queries in this chapter. Chapter 4 ■ CASE Statements 51 Let’s say that we want to know which vendors primarily sell fresh produce and which don’t. Figure 4.1 shows the vendor types currently in our Farmer’s Market database. The vendors we want to label as “Fresh Produce” have the word “Fresh” in the vendor_type column. We can use a CASE statement and the LIKE oper- ator that was covered in Chapter 3 to create a new column, which we’ll alias vendor_type_condensed, that condenses the vendor types to just “Fresh Pro- duce” or “Other”: SELECT vendor_id, vendor_name, vendor_type, CASE WHEN LOWER(vendor_type) LIKE '%fresh%' THEN 'Fresh Produce' ELSE 'Other' END AS vendor_type_condensed FROM farmers_market.vendor In the last two columns of Figure 4.2, you can see how the vendor types were converted to condensed vendor types. We’re using the LOWER() function (which does the opposite of the UPPER() function demonstrated in Chapter 2) to lowercase the vendor type string, because we don’t want the comparison to fail because of capitalization. UPPER() would have also worked, if we then made the comparison string all caps: '%FRESH%'. Figure 4.1 Figure 4.2 52 Chapter 4 ■ CASE Statements If a new vendor type is added to the database that includes the word “fresh,” this query using the LIKE comparison would automatically categorize it as “Fresh Produce” in the vendor_type_condensed column. If we only wanted existing vendor types to be labeled using this logic, we could instead use the IN keyword and explicitly list the existing vendor types we want to label with the “Fresh Produce” category. As a data analyst or data scientist building a dataset that may be refreshed as new data is added to the database, you should always consider what might happen to your transformed columns if the underlying data changes. Creating Binary Flags Using CASE A CASE statement can be used to create a “binary flag field,” which is a type of field that’s often found in machine learning datasets. A binary flag field contains only 1s or 0s, usually indicating a “Yes” or “No” or “exists” or “doesn’t exist” type of value. For example, the Farmer’s Markets in our database all occur on Wednesday evenings or Saturday mornings. Many machine learning algo- rithms won’t know what to do with the words “Wednesday” and “Saturday” that appear in our database, as shown in the market_day column of Figure 4.3: SELECT market_date, market_day FROM farmers_market.market_date_info LIMIT 5 But, the algorithm could use a numeric value as an input. So, how might we turn this string column into a number? One approach we can take to including the market day in our dataset is to generate a binary", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 37 + }, + { + "text": "of Figure 4.3: SELECT market_date, market_day FROM farmers_market.market_date_info LIMIT 5 But, the algorithm could use a numeric value as an input. So, how might we turn this string column into a number? One approach we can take to including the market day in our dataset is to generate a binary flag field that indicates whether it’s a weekday or weekend market. We can do this with a CASE state- ment, making a new column that contains a 1 if the market occurs on a Saturday or Sunday, and a 0 if it doesn’t, calling the field “weekend_flag,” as shown in Figure 4.4. SELECT market_date, Figure 4.3 Chapter 4 ■ CASE Statements 53 CASE WHEN market_day = 'Saturday' OR market_day = 'Sunday' THEN 1 ELSE 0 END AS weekend_flag FROM farmers_market.market_date_info LIMIT 5 You may have noticed that I included “Sunday” in the OR statement, even though we said earlier that our farmer’s markets currently occur on Wednesday evenings and Saturday mornings. I had decided to call the field “weekend_flag” instead of “saturday_flag” because when creating this example, I imagined an analytical question that could be asked: “Do farmers sell more produce at our weekend market or at our weekday market?” If the farmer’s market ever changes or expands its schedule to hold a market on a Sunday, this CASE statement will still correctly flag it as a weekend market for the analysis. There is not much downside to making the field aliased “weekend_flag” actually mean what it’s called (except for the tiny additional computation done to check the second OR condition when necessary, which is unlikely to make any noticeable difference for data on the scale most farmer’s markets could collect) and planning for the future possibility of other market days when designing a dataset to answer this question. Grouping or Binning Continuous Values Using CASE In Chapter 3, we had a query that filtered to only customer purchases where an item or quantity of an item cost over $50, by putting a conditional statement in the WHERE clause. But let’s say we wanted to return all rows, and instead of using that value as a filter, only indicate whether the cost was over $50 or not. We could write the query like this: SELECT market_date, customer_id, vendor_id, ROUND(quantity * cost_to_customer_per_qty, 2) AS price, Figure 4.4 Continues 54 Chapter 4 ■ CASE Statements CASE WHEN quantity * cost_to_customer_per_qty > 50 THEN 1 ELSE 0 END AS price_over_50 FROM farmers_market.customer_purchases LIMIT 10 The final column in Figure 4.5 now contains a flag indicating which item purchases were over $50. The price is displayed here for explanatory purposes, but could be left out if only the flag indicator were important. CASE statements can also be used to “bin” a continuous variable, such as price. Let’s say we wanted to put the line-item customer purchases into bins of under $5.00, $5.00–$9.99, $10.00–$19.99, or $20.00 and over. We could accomplish that with a CASE statement in which we surround the values after the THENs", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 38 + }, + { + "text": "can also be used to “bin” a continuous variable, such as price. Let’s say we wanted to put the line-item customer purchases into bins of under $5.00, $5.00–$9.99, $10.00–$19.99, or $20.00 and over. We could accomplish that with a CASE statement in which we surround the values after the THENs in single quotes to generate a column that contains a string label, as shown in Figure 4.6: SELECT market_date, customer_id, vendor_id, ROUND(quantity * cost_to_customer_per_qty, 2) AS price, CASE WHEN quantity * cost_to_customer_per_qty < 5.00 THEN 'Under $5' WHEN quantity * cost_to_customer_per_qty < 10.00 THEN '$5-$9.99' WHEN quantity * cost_to_customer_per_qty < 20.00 THEN '$10-$19.99' WHEN quantity * cost_to_customer_per_qty >= 20.00 THEN '$20 and Up' END AS price_bin FROM farmers_market.customer_purchases LIMIT 10 Figure 4.5 (continued) Chapter 4 ■ CASE Statements 55 Or, if the result needs to be numeric, a different approach is to output the bottom end of the numeric range, as shown in Figure 4.7: SELECT market_date, customer_id, vendor_id, ROUND(quantity * cost_to_customer_per_qty, 2) AS price, CASE WHEN quantity * cost_to_customer_per_qty < 5.00 THEN 0 WHEN quantity * cost_to_customer_per_qty < 10.00 THEN 5 WHEN quantity * cost_to_customer_per_qty < 20.00 THEN 10 WHEN quantity * cost_to_customer_per_qty >= 20.00 THEN 20 END AS price_bin_lower_end FROM farmers_market.customer_purchases LIMIT 10 One of these queries generates a new column of strings, and one generates a new column of numbers. You might actually want to include both columns in Figure 4.6 Figure 4.7 56 Chapter 4 ■ CASE Statements your query if you were building it to be used in a report, because the price_bin column is a more explanatory label for the bin, but will sort alphabetically instead of in bin value order. With both available to use in your report, you could use the numeric version of the column to sort the bins correctly, and the string version to label the bins. Remember that because neither of the preceding queries included an ELSE inside the CASE statement, the output will be NULL if the quantity field is blank or the calculation can’t be completed with the available values for whatever reason. If there is a mis-entered price, or perhaps a record of a refund, and the value in the price column turns out to be negative in one row, what do you think will happen? In the preceding queries, the first condition is “less than 5,” so negative values will end up in the “Under $5,” or 0, bin. Therefore, the name price_bin_lower_end is a misnomer, since 0 might not actually represent the lowest value possible in the first bin. It’s important when writing CASE state- ments for analytical purposes to determine what the result will be if there end up being unexpected values in any of the referenced database fields. Categorical Encoding Using CASE When developing datasets for machine learning, you will often need to “encode” categorical string variables as numeric variables, in order for a mathematical algorithm to be able to use them as input. If the categories represent something that can be", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 39 + }, + { + "text": "any of the referenced database fields. Categorical Encoding Using CASE When developing datasets for machine learning, you will often need to “encode” categorical string variables as numeric variables, in order for a mathematical algorithm to be able to use them as input. If the categories represent something that can be sorted in a rank order, it might make sense to convert the string variables into numeric values that rep- resent that rank order. For example, the vendor booths at the farmer’s market are rented out at different costs, depending on their size and proximity to the entrance. These booth price levels are labeled with the letters “A,” “B,” and “C,” in order by increasing price, which could be converted into either numeric values 1, 2, 3 or the actual booth prices. The following CASE statement converts the booth price levels into numeric values, and the results are shown in Figure 4.8: SELECT booth_number, booth_price_level, CASE WHEN booth_price_level = 'A' THEN 1 WHEN booth_price_level = 'B' THEN 2 WHEN booth_price_level = 'C' THEN 3 END AS booth_price_level_numeric FROM farmers_market.booth LIMIT 5 Chapter 4 ■ CASE Statements 57 If the categories aren’t necessarily in any kind of rank order, like our vendor type categories, we might use a method called “one-hot encoding.” This helps us avoid inadvertently indicating a sort order when none exists. One-hot encod- ing means that we create a new column representing each category, assigning it a binary value of 1 if a row falls into that category, and a 0 otherwise. These columns are sometimes called “dummy variables.” The following CASE state- ment one-hot encodes our vendor type categories, and the results are demon- strated in Figure 4.9: SELECT vendor_id, vendor_name, vendor_type, CASE WHEN vendor_type = 'Arts & Jewelry' THEN 1 ELSE 0 END AS vendor_type_arts_jewelry, CASE WHEN vendor_type = 'Eggs & Meats' THEN 1 ELSE 0 END AS vendor_type_eggs_meats, CASE WHEN vendor_type = 'Fresh Focused' THEN 1 ELSE 0 END AS vendor_type_fresh_focused, CASE WHEN vendor_type = 'Fresh Variety: Veggies & More' THEN 1 ELSE 0 END AS vendor_type_fresh_variety, CASE WHEN vendor_type = 'Prepared Foods' THEN 1 ELSE 0 END AS vendor_type_prepared FROM farmers_market.vendor Figure 4.8 58 Chapter 4 ■ CASE Statements Figure 4.9 Chapter 4 ■ CASE Statements 59 A situation to be aware of when manually encoding one-hot categorical var- iables this way is that if a new category is added (a new type of vendor in this case), there will be no column in your dataset for the new vendor type until you add another CASE statement. CASE Statement Summary In this chapter, you learned SQL CASE statement syntax for creating new col- umns with values based on conditions. You also learned how to consolidate categorical values into fewer categories, create binary flags, bin continuous values, and encode categorical values. You should now be able to describe what the following two queries do. The results for the first query are displayed in Figure 4.10: SELECT customer_id, CASE WHEN customer_zip = '22801' THEN 'Local' ELSE 'Not", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 40 + }, + { + "text": "categorical values into fewer categories, create binary flags, bin continuous values, and encode categorical values. You should now be able to describe what the following two queries do. The results for the first query are displayed in Figure 4.10: SELECT customer_id, CASE WHEN customer_zip = '22801' THEN 'Local' ELSE 'Not Local' END customer_location_type FROM farmers_market.customer LIMIT 10 The results for the following query are displayed in Figure 4.11: SELECT booth_number, CASE WHEN booth_price_level = 'A' THEN 1 ELSE 0 END booth_price_level_A, CASE WHEN booth_price_level = 'B' THEN 1 ELSE 0 Figure 4.10 60 Chapter 4 ■ CASE Statements END booth_price_level_B, CASE WHEN booth_price_level = 'C' THEN 1 ELSE 0 END booth_price_level_C FROM farmers_market.booth LIMIT 5 Exercises Using the Included Database Look back at Figure 2.1 in Chapter 2 for sample data and column names for the product table referenced in these exercises. 1. Products can be sold by the individual unit or by bulk measures like lbs. or oz. Write a query that outputs the product_id and product_name col- umns from the product table, and add a column called prod_qty_type_ condensed that displays the word “unit” if the product_qty_type is “unit,” and otherwise displays the word “bulk.” 2. We want to flag all of the different types of pepper products that are sold at the market. Add a column to the previous query called pepper_flag that outputs a 1 if the product_name contains the word “pepper” (regard- less of capitalization), and otherwise outputs 0. 3. Can you think of a situation when a pepper product might not get flagged as a pepper product using the code from the previous exercise? Figure 4.11 C H A P T E R 61 5 Now that you have learned how to select the data you want from a database table and filter to the rows you want, you might wonder what to do if the data you need exists across multiple related tables in the database. For example, one analytical question mentioned in Chapter 1, “When is each type of fresh fruit or vegetable in season, locally?” requires data from the product_category table (to filter to the categories with fresh fruit and vegetables), the product table (to get details about each specific item, including product names and quantity types), and the vendor_inventory table (to find out when vendors were selling these products). This is where SQL JOINs come in. Database Relationships and SQL JOINs In Chapter 1, “Data Sources,” we introduced different types of database rela- tionships and the entity-relationship diagram (ERD). The type of relationship between database tables, and the key fields that connect them, give us information we need to combine them using a JOIN statement in SQL. Let’s say we wanted to list each product name along with its product cate- gory name. Since only the ID of the product category exists in the product table, and the product category’s name is in the product_category table, we have to combine the data in the product and product_category tables together in", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 41 + }, + { + "text": "wanted to list each product name along with its product cate- gory name. Since only the ID of the product category exists in the product table, and the product category’s name is in the product_category table, we have to combine the data in the product and product_category tables together in order to generate this list. SQL JOINs 62 Chapter 5 ■ SQL JOINs Figure 5.1 shows the one-to-many relationship between these two tables: each product can only belong to one category, but each category can contain many products. The primary key in the product_category table is the product_cate- gory_id. There is also a product_category_id in each row of the product table that serves as a foreign key, identifying which category each product belongs to. NOTE Remember that in an ERD, an infinity symbol, “N,” or “crow’s feet” on the end of a line connecting two tables indicates that it is the “many” side of a one-to-many relationship. You can see the infinity symbol next to the product table in Figure 5.1. In order to combine these tables, we need to figure out which type of JOIN to use. To illustrate the different types of SQL JOINs, we’ll use the two tables from the Farmer’s Market database found in Figure 5.1, but remove some columns to simplify the illustration, as shown in Figure 5.2. Figure 5.1 Figure 5.2 Chapter 5 ■ SQL JOINs 63 Figure 5.2 shows the one-to-many relationship between these tables, as well as some sample data. Their primary keys are each identified with an asterisk, and the foreign key with a double asterisk. Each row in the product_category table can be associated with many rows in the product table, but each row in the product table is associated with only one row in the product_category table. The fields that connect the two tables are product_category.product_­ category_id and product.product_category_id. The first type of JOIN we’ll cover is the one I’ve seen used most frequently when building analytical datasets: the LEFT JOIN. This tells the query to pull all records from the table on the “left side” of the JOIN, and only the matching records (based on the criteria specified in the JOIN clause) from the table on the “right side” of the JOIN. See Figure 5.3. Note that the product table will be on the left side of the join we’re setting up, even though it was on the right side of the relationship diagram! In our demonstration tables shown in Figure 5.2, you can see that there are two records in the product table that have a product_category_id of 1, and only one record in the product_category table with the product_category_id of 1 (because it is the primary key in this table). When we LEFT JOIN the product_category table to the product table, the records with a value of 1 in the product_category_id fields of both tables will be joined in the results, as shown in the first two rows of the output in Figure 5.4. Note that", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 42 + }, + { + "text": "key in this table). When we LEFT JOIN the product_category table to the product table, the records with a value of 1 in the product_category_id fields of both tables will be joined in the results, as shown in the first two rows of the output in Figure 5.4. Note that the data for product_category_id 1 from the product_category table is repeated in two rows in the resulting output, even though it was only in one row in the product_category table, because it is matched up with two records in the joined product table. Figure 5.3 64 Chapter 5 ■ SQL JOINs What makes this JOIN a LEFT JOIN is that all of the records from the product table on the left side of the join are included in the output, even if they don’t have a match in the product_category table. But records from the product_­ category table on the right side of the join are only included if they match up with product records on the left side. In Figure 5.4, you can see that the row with a product_id of 99 is included, but the values in the resulting output columns from the product_category are NULL (empty) This is because there was no product_category_id value on which to join this product to the product_category table. However, the row from the product_category table with a product_category_id of 6 is not included in the output at all, because it was in the “right” table and does not have a matching record on the left side, and this is a LEFT JOIN. The syntax for creating this output is: SELECT [columns to return] FROM [left table] [JOIN TYPE] [right table] ON [left table].[field in left table to match] = [right table].[field in right table to match] Let’s look at the actual output of a query that joins our product and ­product_ category tables from the Farmer’s Market database using a LEFT JOIN. In order to pull a list of all products with each product’s category name listed, we make the product table the “left” table in this query by listing it first after FROM, and the product_category table the “right” side by listing it after the LEFT JOIN, and we’ll match up the records using the product_category_id fields, as seen after the ON keyword. This is how a query pulling all fields from both tables looks: SELECT * FROM product LEFT JOIN product_category Figure 5.4 Chapter 5 ■ SQL JOINs 65 ON product.product_category_id = product_category. product_category_id This query can be read as “Select everything from the product table, left joined with the product_category table, matched on the product_category_id that’s common to both tables,” or more specifically: “Select all columns and rows from the product table, and all columns from the product_category table for rows where the product_category’s product_category_id matches a prod- uct’s product_category_id.” The first 10 rows of the output from this query are shown in Figure 5.5. NOTE You may have noticed that there are two columns called product_category_id in", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 43 + }, + { + "text": "from the product table, and all columns from the product_category table for rows where the product_category’s product_category_id matches a prod- uct’s product_category_id.” The first 10 rows of the output from this query are shown in Figure 5.5. NOTE You may have noticed that there are two columns called product_category_id in Figure 5.5. That is because we selected all fields, using the asterisk, and there are fields in both tables with the same name. To remedy this, we could either specify the list of fields to be returned, and only include the product_category_id from one of the tables, or we could alias the column names to indicate which table each came from, as shown in the next query and pictured in Figure 5.6. The LEFT JOIN indicates that we want all rows from the product table (which is listed on the left side of the JOIN keyword) and only the associated rows from the product_category table. So, if for some reason there is a category that is not associated with any products, it will not be included in the results. If there were a product without a category (which can’t happen in the actual database, because that is a required field), it would be included in the results, with the fields on the product_category side being NULL, as was illustrated in the last row of Figure 5.4. The ON part of the JOIN clause tells the query to match up the rows in the two tables using the values in the product_category_id field in each table. Note in Figure 5.5 that the product_category_id fields from both tables match on every row. (It would be helpful for us to alias those field names so they don’t appear with identical column headers in the output, which we will do in the next query.) Now, if we want to retrieve specific columns from the merged dataset, we have to specify which table each column is from, since it’s possible to have iden- tically named columns in different tables. And we can alias identically named Figure 5.5 66 Chapter 5 ■ SQL JOINs columns to differentiate them. For example, the following code accomplishes the changes shown in Figure 5.6. SELECT product.product_id, product.product_name, product.product_category_id AS product_prod_cat_id, product_category.product_category_id AS category_prod_cat_id, product_category.product_category_name FROM product LEFT JOIN product_category ON product.product_category_id = product_category. product_category_id There is another type of aliasing in a SQL query that is for developer convenience, because it isn’t visible in the output: table aliasing. If you don’t want to write out the entire table name every time you reference it, you can assign it a short alias in the FROM clause that can then be used to reference it throughout the query, as demonstrated in the following query with the table aliases “p” and “pc.” Also, we will also only show a single product_category_id now that we are convinced from the output that they are matching up between tables. SELECT p.product_id, p.product_name, pc.product_category_id, pc.product_category_name FROM product AS p LEFT JOIN product_category AS pc ON p.product_category_id =", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 44 + }, + { + "text": "following query with the table aliases “p” and “pc.” Also, we will also only show a single product_category_id now that we are convinced from the output that they are matching up between tables. SELECT p.product_id, p.product_name, pc.product_category_id, pc.product_category_name FROM product AS p LEFT JOIN product_category AS pc ON p.product_category_id = pc.product_category_id ORDER BY pc.product_category_name, p.product_name Again, the AS keyword between the table name and the alias is optional, but I will stick with that standard in this book for consistency. The output from this query is shown in Figure 5.7. Figure 5.6 Chapter 5 ■ SQL JOINs 67 The next type of SQL JOIN we’ll discuss is called a RIGHT JOIN, which is illustrated in Figure 5.8. In a RIGHT JOIN, all of the rows from the “right table” are returned, along with only the matching rows from the “left table,” using the fields specified in the ON part of the query. If we use a RIGHT JOIN to merge the data in the tables shown in Figure 5.2, the result will look like the table in Figure 5.9. All of the records from the right table, product_category, are returned, but only matching records from product. There are no products in this example with a product_category_id value of 6, so the first three columns of the last row are NULL. Note that the record in the product table with a product_id of 99 that was in the last row of Figure 5.4 is now missing, because we performed a RIGHT JOIN this time, and that record does not have a match in the product_category table, since its product_category_id was NULL. Figure 5.7 Figure 5.8 68 Chapter 5 ■ SQL JOINs You would use a RIGHT JOIN if you wanted to list all product categories and the products in each. (And you didn’t care about products that were not put into a category, but you did care about categories that didn’t contain any products.) An INNER JOIN only returns records that have matches in both tables. Can you tell which rows from each table in Figure 5.2 will not be returned if we INNER JOIN the product and product_category tables on product_­category_id? Figure 5.10 illustrates an INNER JOIN. In Figure 5.4, the fields on the right side of the join (from product_category) were NULL in the row where the left side had a NULL product_category_id. In Figure 5.9, the fields on the left side of the table (from product) were NULL in the row where the product_category_id on the right side hadn’t been assigned to any products. If we INNER JOIN the tables, neither of those rows will be included in the output, as shown in Figure 5.11. Figure 5.9 Figure 5.10 Chapter 5 ■ SQL JOINs 69 In the INNER JOINed output in Figure 5.11, there is a row for every matched pair of product_category_ids, and no rows without a matching product_category_id on the other side. To practice all of these types of JOINs, let’s now look at the customer", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 45 + }, + { + "text": "5.10 Chapter 5 ■ SQL JOINs 69 In the INNER JOINed output in Figure 5.11, there is a row for every matched pair of product_category_ids, and no rows without a matching product_category_id on the other side. To practice all of these types of JOINs, let’s now look at the customer and customer_purchase tables in the Farmer’s Market database. Again, this is a one-to-many type relationship. A customer can have multiple purchases, but each purchase is made by only one customer. The two tables are related via the customer_id field, which is the primary key in the customer table, and a foreign key in the customer_purchases table. If we do a LEFT JOIN, using the following query, we can see in the output in Figure 5.12 that for some rows, the fields that come from the customer_­purchases table are all NULL. What do you think this means? SELECT * FROM customer AS c LEFT JOIN customer_purchases AS cp ON c.customer_id = cp.customer_id Unlike for the product-product_category relationship, there can be customers without any purchases. Such customers were added to the customer table when they signed up for the farmer’s market loyalty card, so we have their customer data, but they have not yet purchased any products. Since we did a LEFT JOIN, we’re getting a list of all customers, and their associated ­purchases, if there are Figure 5.11 Figure 5.12 70 Chapter 5 ■ SQL JOINs any. Customers with multiple purchases will show up in the output multiple times—once for each item purchased. Customers without any purchases will have NULL values in all fields displayed that are from the customer_purchases table. We can use the WHERE clause you learned about in Chapter 3, “The WHERE Clause,” to filter the list to only customers with no purchases, if we’d like: SELECT c.* FROM customer AS c LEFT JOIN customer_purchases AS cp ON c.customer_id = cp.customer_id WHERE cp.customer_id IS NULL In this case, we only selected columns from the customer table, using c.*, because all of the columns on the customer_purchases side of the relationship will be NULL (since we’re filtering to NULL customer_id, and there are no purchases in the customer_purchases table without a customer_id, since it is a required field; remember that in this imaginary farmer’s market, every purchase is logged at checkout, and every customer uses their loyalty card). Figure 5.13 shows what the output of this query looks like, listing all cus- tomers who don’t have any purchases. What if we wanted to list all purchases and the customers associated with them? In that case, we could do a RIGHT JOIN, pulling all records from the customer_purchases table, and only customers from the customer table with a purchase (a record in the customer_purchases table with their ID on it): SELECT * FROM customer AS c RIGHT JOIN customer_purchases AS cp ON c.customer_id = cp.customer_id The output in Figure 5.14 is truncated to save space and doesn’t show all results, but there are no rows returned with NULL values in", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 46 + }, + { + "text": "record in the customer_purchases table with their ID on it): SELECT * FROM customer AS c RIGHT JOIN customer_purchases AS cp ON c.customer_id = cp.customer_id The output in Figure 5.14 is truncated to save space and doesn’t show all results, but there are no rows returned with NULL values in the customer table columns, because there is no such thing as a purchase without a customer_id. And because we did a RIGHT JOIN, we will no longer get customers without purchases in the results. Figure 5.13 Chapter 5 ■ SQL JOINs 71 If you only want records from each table that have matches in both tables, use an INNER JOIN. Using these customer and customer_purchases tables, an INNER JOIN happens to return the same results as the RIGHT JOIN, because there aren’t any records on the “right side” of the join without matches on the “left side”—every purchase is associated with a customer. A Common Pitfall when Filtering Joined Data Going back to the LEFT JOIN example between the customer and customer_ purchases tables whose output is depicted in Figure 5.12, how do you think the output of the following query will differ from the original LEFT JOIN query without the added WHERE clause? SELECT * FROM customer AS c LEFT JOIN customer_purchases AS cp ON c.customer_id = cp.customer_id WHERE cp.customer_id > 0 All customer_id values are integers above 0, so it might initially appear like the addition of this WHERE clause will make no difference in the output. How- ever, notice that it is filtering on the customer_id on the “right side” table, ­customer_purchases (note the alias cp in the WHERE clause). That means that the customers without purchases will be filtered out, because they wouldn’t have matching records in the customer_purchases table. The addition of this filter makes the query return results like an INNER JOIN instead of a LEFT JOIN, by filtering out records that return NULL values on the “right side” table columns in the output. So, instead of the output shown in Figure 5.12, this query’s output would look like the one shown in Figure 5.14. If you are using a LEFT JOIN because you want to return all rows from the “left” table, even those that don’t have a match on the “right” side of the join, be sure not to filter on any fields from the “right” table without also allowing NULL results on the right side, or you will filter out results you intended to keep. Figure 5.14 72 Chapter 5 ■ SQL JOINs Let’s say we want to write a query that returns a list of all customers who did not make a purchase at the March 2, 2019, farmer’s market. We will use a LEFT JOIN, since we want to include the customers who have never made a purchase at any farmer’s market, so wouldn’t have any records in the customer_purchases table: SELECT c.*, cp.market_date FROM customer AS c LEFT JOIN customer_purchases AS cp ON c.customer_id = cp.customer_id WHERE cp.market_date", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 47 + }, + { + "text": "We will use a LEFT JOIN, since we want to include the customers who have never made a purchase at any farmer’s market, so wouldn’t have any records in the customer_purchases table: SELECT c.*, cp.market_date FROM customer AS c LEFT JOIN customer_purchases AS cp ON c.customer_id = cp.customer_id WHERE cp.market_date <> '2019-03-02' Figure 5.15 displays the output we get with this query. There are multiple problems with this output. The first problem is that we’re missing customers who have never made a purchase, like Betty Bullard shown in Figure 5.12, since we filtered to the market_date field in the customer_purchases table, which is on the “right side” of the JOIN, and because (as shown in Chapter 3) SQL doesn’t evaluate value comparisons to TRUE when one of the values being compared is NULL. But we need that filter in order to remove customers who made a purchase that day. One solution that will allow us to filter the results returned using a field in the table on the right side of the join while still returning records that only exist in the left side table is to write the WHERE clause to allow NULL values in the field: SELECT c.*, cp.market_date FROM customer AS c LEFT JOIN customer_purchases AS cp ON c.customer_id = cp.customer_id WHERE (cp.market_date <> '2019-03-02' OR cp.market_date IS NULL) Figure 5.15 Chapter 5 ■ SQL JOINs 73 Now we see customers without purchases in Figure 5.16, like Betty Bullard, in addition to customers who have made purchases on other dates. The second problem with this output is that it contains one row per customer per item purchased, because the customer_purchases table has a record for each item purchased, when we just wanted a list of customers. We can resolve this problem by removing the market_date field from the customer_purchases “side” of the relationship, so the purchase dates aren’t displayed, then using the DISTINCT keyword, which removes duplicate records in the output, only displaying distinct (unique) results. Figure 5.17 shows the output associated with the following query: SELECT DISTINCT c.* FROM customer AS c LEFT JOIN customer_purchases AS cp ON c.customer_id = cp.customer_id WHERE (cp.market_date <> '2019-03-02' OR cp.market_date IS NULL) Figure 5.16 Figure 5.17 74 Chapter 5 ■ SQL JOINs With this approach, we were able to filter out records we didn’t want, using values on the customer_purchases side of the relationship, without excluding records we did want from the customer side of the relationship. And, we only displayed data from one of the tables, even though we were using fields from both in the query. JOINs with More than Two Tables Let’s say we want details about all farmer’s market booths, as well as every vendor booth assignment for every market date. Perhaps we’re building an interactive report that lets us filter to a booth, a vendor, or a date, to see the resulting list of booth assignments with additional both and vendor details, so we need a merged dataset that contains all of their records.", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 48 + }, + { + "text": "booth assignment for every market date. Perhaps we’re building an interactive report that lets us filter to a booth, a vendor, or a date, to see the resulting list of booth assignments with additional both and vendor details, so we need a merged dataset that contains all of their records. This requires joining the three tables shown in Figure 5.18 together. What kind of JOINs do you think we could use to ensure that all booths are included, even if they aren’t assigned to a vendor yet, and all vendors assigned to booths are included? We can LEFT JOIN the vendor_booth_assignments to booth, therefore including all of the booths, and LEFT JOIN vendor to vendor_booth_assignments in the results. The query to accomplish these joins looks like the following and results in Figure 5.19: SELECT b.booth_number, b.booth_type, Figure 5.18 Chapter 5 ■ SQL JOINs 75 vba.market_date, v.vendor_id, v.vendor_name, v.vendor_type FROM booth AS b LEFT JOIN vendor_booth_assignments AS vba ON b.booth_number = vba. booth_number LEFT JOIN vendor AS v ON v.vendor_id = vba.vendor_id ORDER BY b.booth_number, vba.market_date You can think of the second JOIN as being merged into the result of the first JOIN. Because in this case the vendor_id field in the third table, vendor, is joined to the vendor_id field in the second table, vendor_booth_assignments, only vendors that exist in the vendor_booth_assignments table will be included, resembling the diagram shown in Figure 5.20. Figure 5.19 Figure 5.20 76 Chapter 5 ■ SQL JOINs If the third table was instead joined to the first table, using a field common to both of them (which is common, but isn’t actually possible using the tables in this Farmer’s Market database, because there aren’t any other tables joined to the booth table), the arrangement would look like the diagram in Figure 5.21. This method of joining multiple tables together is a common one that you will see in machine learning applications, where you have one “primary” table that contains one row per entity record you want data summarized for, with many other tables LEFT JOINed into it. This allows you to pull additional data about the entity in from other tables. Data in these other tables is frequently summarized, so the resulting dataset remains at one row per input example, and data from other tables is represented by counts or sums, for example. You will learn more about this in Chapter 6, “Aggregating Results for Analysis,” when we cover aggregation. We will continue to use JOINs throughout the rest of the book, so look in later chapters for more examples of JOINed tables and the resulting output. Exercises Using the Included Database 1. Write a query that INNER JOINs the vendor table to the vendor_booth_ assignments table on the vendor_id field they both have in common, and sorts the result by vendor_name, then market_date. Figure 5.21 Chapter 5 ■ SQL JOINs 77 2. Is it possible to write a query that produces an output identical to the output of the following query, but using", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 49 + }, + { + "text": "vendor_booth_ assignments table on the vendor_id field they both have in common, and sorts the result by vendor_name, then market_date. Figure 5.21 Chapter 5 ■ SQL JOINs 77 2. Is it possible to write a query that produces an output identical to the output of the following query, but using a LEFT JOIN instead of a RIGHT JOIN? SELECT * FROM customer AS c RIGHT JOIN customer_purchases AS cp ON c.customer_id = cp.customer_id 3. At the beginning of this chapter, the analytical question “When is each type of fresh fruit or vegetable in season, locally?” was asked, and it was explained that the answer requires data from the product_category table, the product table, and the vendor_inventory table. What type of JOINs do you expect would be needed to combine these three tables in order to be able to answer this question? C H A P T E R 79 6 SQL starts becoming especially powerful for analysis when you use it to aggregate data. By using the GROUP BY statement, you can specify the level of summari- zation and then use aggregate functions to summarize values for the records in each group. Data analysts can use SQL to build dynamic summary reports that can be automatically updated as the database is updated with new data, by simply triggering a refresh that reruns the query. Dashboards and reports built using software like Tableau and Cognos often rely on SQL queries to get the data they need from the underlying database in an aggregated form that can be used for reporting, which we’ll cover in Chapter 10, “Building Analytical Reports with SQL.” Data scientists can use SQL to summarize data at the level of granularity needed for training a classification model, which we’ll get into in more depth in Chapter 12, “SQL for Machine Learning.” But it all starts with basic SQL aggregation. GROUP BY Syntax You saw this basic SQL SELECT query syntax in Chapter 2, “The SELECT Statement.” Two sections of this query that we haven’t yet covered, which are both related to aggregation, are the GROUP BY and HAVING clauses: SELECT [columns to return] Aggregating Results for Analysis 80 Chapter 6 ■ Aggregating Results for Analysis FROM [table] WHERE [conditional filter statements] GROUP BY [columns to group on] HAVING [conditional filter statements that are run after grouping] ORDER BY [columns to sort on] The GROUP BY keywords are followed by a comma-­separated list of column names that indicate how you want to summarize the query results. Using what you’ve learned so far without grouping, you might write a query like the following to get a list of the customer IDs of customers who made pur- chases on each market date: SELECT market_date, customer_id FROM farmers_market.customer_purchases ORDER BY market_date, customer_id However, this approach would result in one row per item each customer purchased, displaying duplicates in the output, because you’re querying the customer_purchases table with no grouping specified. To instead get one row per customer per market date, you can", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 50 + }, + { + "text": "date: SELECT market_date, customer_id FROM farmers_market.customer_purchases ORDER BY market_date, customer_id However, this approach would result in one row per item each customer purchased, displaying duplicates in the output, because you’re querying the customer_purchases table with no grouping specified. To instead get one row per customer per market date, you can group the results by adding a GROUP BY clause that specifies that you want to summarize the results by the customer_id and market_date fields: SELECT market_date, customer_id FROM farmers_market.customer_purchases GROUP BY market_date, customer_id ORDER BY market_date, customer_id You can also accomplish the same result by using SELECT DISTINCT to remove duplicates, but here we are using GROUP BY with the intention of adding sum- mary columns to the output. Displaying Group Summaries Now that you have grouped the data at the desired level, you can add aggregate functions like SUM and COUNT to return summaries of the customer_purchases data per group. This query uses the COUNT() function to count the rows in the customer_purchases table per market date per customer. The output of this query is shown in Figure 6.1. Chapter 6 ■ Aggregating Results for Analysis 81 SELECT market_date, customer_id, COUNT(*) AS items_purchased FROM farmers_market.customer_purchases GROUP BY market_date, customer_id ORDER BY market_date, customer_id LIMIT 10 Now, remember that the granularity of the customer_purchases table is such that if a customer were to buy three identical items, such as tomatoes, at once from a vendor, that would show up as 1 in the items_purchased column of this query’s output, since the item purchase is recorded in one row in the table, with a quantity value of 3. (See Figures 1.7 and 2.7 in Chapters 1, “Data Sources,” and 2 for reference.) If the customer were to buy three tomatoes, walk away from the stand, then go back and purchase another three tomatoes, that would be counted as two by the preceding query, since the new separate purchase would generate a new line in the database. If instead of counting up line items, we wanted to add up all quantities pur- chased, to count all six tomatoes, we can sum up the quantity column using the following query. The output of this query is shown in Figure 6.2. SELECT market_date, customer_id, SUM(quantity) AS items_purchased FROM farmers_market.customer_purchases GROUP BY market_date, customer_id ORDER BY market_date, customer_id LIMIT 10 Figure 6.1 82 Chapter 6 ■ Aggregating Results for Analysis The items_purchased column is no longer an integer, because some of the quantities we’re adding up are bulk product weights. After seeing these results and realizing bulk weight quantities are included, you may decide that it doesn’t make sense to report the purchases this way, and you instead want to know how many different kinds of items were purchased by each customer. So now you only want to count “1” if they bought tomatoes, no matter how many individual tomatoes they purchased, or how many times they checked out, and only add to that count if they bought other items, such as lettuce. NOTE Note", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 51 + }, + { + "text": "of items were purchased by each customer. So now you only want to count “1” if they bought tomatoes, no matter how many individual tomatoes they purchased, or how many times they checked out, and only add to that count if they bought other items, such as lettuce. NOTE Note that this type of modification occurs frequently while designing reports—­either by the data analyst or by the requester/customer—­so it’s important to understand the granularity and structure of the underlying table to ensure that your result means what you think it does. This is why I recommend writing the query without aggregation first to see the values you will be summarizing before grouping the results. What you want now is a DISTINCT count of product IDs, shown in the fol- lowing query and in Figure 6.3. So instead of counting how many rows there were in the customer_purchases table per customer per market date, like we did with COUNT(*), or adding up the quantities, like we did with SUM(quantity), we’re identifying how many unique product_id values exist across those rows in the group—­how many different kinds of products were purchased by each customer on each market date: SELECT market_date, customer_id, COUNT(DISTINCT product_id) AS different_products_purchased FROM farmers_market.customer_purchases c GROUP BY market_date, customer_id ORDER BY market_date, customer_id LIMIT 10 Figure 6.2 Chapter 6 ■ Aggregating Results for Analysis 83 We can also combine these summaries into a single query, as shown here and in Figure 6.4: SELECT market_date, customer_id, SUM(quantity) AS items_purchased, COUNT(DISTINCT product_id) AS different_products_purchased FROM farmers_market.customer_purchases GROUP BY market_date, customer_id ORDER BY market_date, customer_id LIMIT 10 You can include as many different aggregate functions as you want in a single query, and they will all be applied at the same level of grouping—­in this case, summarizing per market date per customer ID. Note how we are using column name aliases to describe the different summary values. If you do want to sum up the quantities but don’t like how the “items pur- chased” column includes both discrete items and bulk weights (some of which may be in pounds and some in ounces, so shouldn’t be added together, anyway), we will demonstrate one possible solution later in the chapter in the “CASE Statements Inside Aggregate Functions” section. Figure 6.3 Figure 6.4 84 Chapter 6 ■ Aggregating Results for Analysis Performing Calculations Inside Aggregate Functions You can also include mathematical operations, which are calculated at the row level prior to summarization, inside the aggregate functions. In Chapter 3, “The WHERE Clause,” you learned how to display a list of customer purchases at the farmer’s market, using a WHERE clause to filter it to a specific customer. The customer with ID 3 has purchased the items in Figure 6.5, which can be retrieved using the following query that calculates the price per line item: SELECT market_date, customer_id, vendor_id, quantity * cost_to_customer_per_qty AS price FROM farmers_market.customer_purchases WHERE customer_id = 3 ORDER BY market_date, vendor_id Let’s say we wanted to know how much money this customer", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 52 + }, + { + "text": "the items in Figure 6.5, which can be retrieved using the following query that calculates the price per line item: SELECT market_date, customer_id, vendor_id, quantity * cost_to_customer_per_qty AS price FROM farmers_market.customer_purchases WHERE customer_id = 3 ORDER BY market_date, vendor_id Let’s say we wanted to know how much money this customer spent total on each market_date, regardless of item or vendor. We can GROUP BY market_date, and use the SUM aggregate function on the price calculation to add up the prices of the items purchased, as follows: SELECT customer_id, market_date, SUM(quantity * cost_to_customer_per_qty) AS total_spent FROM farmers_market.customer_purchases WHERE customer_id = 3 GROUP BY market_date ORDER BY market_date The SUM() function surrounds the “price” calculation, which means that the price will be calculated per row of the table, as we saw in the first query, and then the results will be summed up per group—­in this case, per market date. The summarized results are shown in Figure 6.6, where you can see that the prices of the two line items from March 16, 2019 have been added up, and the alias has been updated to total_spent to better reflect the meaning of the Figure 6.5 Chapter 6 ■ Aggregating Results for Analysis 85 summarized value. We grouped by customer_id and market_date, so there is now just one summarized row per customer_id per market_date. Notice that vendor_id has been removed from the list of columns to be dis- played and from the ORDER BY clause. That’s because if we want the aggregation level of one row per customer per date, we can’t also include vendor_id in the output, because the customer can purchase from multiple vendors on a single date, so the results wouldn’t be aggregated at the level we wanted. Even though it’s not required in order to get these results, we should also add customer_id to the GROUP BY list, so the query will work without error even when it’s not filtered to a single customer. We’ll make this change in the next query. What if we wanted to find out how much this customer had spent at each vendor, regardless of date? Then we can group by customer_id and vendor_id: SELECT customer_id, vendor_id, SUM(quantity * cost_to_customer_per_qty) AS total_spent FROM farmers_market.customer_purchases WHERE customer_id = 3 GROUP BY customer_id, vendor_id ORDER BY customer_id, vendor_id The results of this query are shown in Figure 6.7. We can also remove the customer_id filter—­in this case by removing the entire WHERE clause since there are no other filter values—­and GROUP BY customer_id only, to get a list of every customer and how much they have ever spent at the farmer’s market. The results of the following query are shown in Figure 6.8: SELECT customer_id, SUM(quantity * cost_to_customer_per_qty) AS total_spent FROM farmers_market.customer_purchases Figure 6.6 Figure 6.7 Continues 86 Chapter 6 ■ Aggregating Results for Analysis GROUP BY customer_id ORDER BY customer_id So far, we have been doing all of this aggregation on a single table, but it can be done on joined tables, as well. It’s", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 53 + }, + { + "text": "* cost_to_customer_per_qty) AS total_spent FROM farmers_market.customer_purchases Figure 6.6 Figure 6.7 Continues 86 Chapter 6 ■ Aggregating Results for Analysis GROUP BY customer_id ORDER BY customer_id So far, we have been doing all of this aggregation on a single table, but it can be done on joined tables, as well. It’s a good idea to join the tables without the aggregate functions first, to make sure the data is at the level of granularity you expect (and not generating duplicates) before adding the GROUP BY. Let’s say that for the query that was grouped by customer_id and vendor_id, we want to bring in some customer details, such as first and last name, and the vendor name. We can first join the three tables together, select columns from all of the tables, and inspect the output before grouping, as shown in Figure 6.9: SELECT c.customer_first_name, c.customer_last_name, cp.customer_id, v.vendor_name, cp.vendor_id, cp.quantity * cp.cost_to_customer_per_qty AS price FROM farmers_market.customer c LEFT JOIN farmers_market.customer_purchases cp ON c.customer_id = cp.customer_id LEFT JOIN farmers_market.vendor v ON cp.vendor_id = v.vendor_id WHERE cp.customer_id = 3 ORDER BY cp.customer_id, cp.vendor_id To summarize at the level of one row per customer per vendor, we will have to group by a lot more fields, including all of the customer table fields and all of Figure 6.8 Figure 6.9 (continued) Chapter 6 ■ Aggregating Results for Analysis 87 the vendor table fields. Basically, we want to group by all of the displayed fields that don’t include aggregate functions. The following query shows the list of fields used for grouping, and the output of this query is shown in Figure 6.10. The ROUND() function was added here to format the total_spent calculation nicely in dollar form: SELECT c.customer_first_name, c.customer_last_name, cp.customer_id, v.vendor_name, cp.vendor_id, ROUND(SUM(quantity * cost_to_customer_per_qty), 2) AS total_spent FROM farmers_market.customer c LEFT JOIN farmers_market.customer_purchases cp ON c.customer_id = cp.customer_id LEFT JOIN farmers_market.vendor v ON cp.vendor_id = v.vendor_id WHERE cp.customer_id = 3 GROUP BY c.customer_first_name, c.customer_last_name, cp.customer_id, v.vendor_name, cp.vendor_id ORDER BY cp.customer_id, cp.vendor_id We can also keep the same level of aggregation and filter to a single vendor instead of a single customer, to get a list of customers per vendor instead of vendors per customer, as shown in the following code. Note that the only line of code that is changed is the WHERE clause condition, because even though we’re changing the filter, we want the grouping level and the output fields to stay the same. You can see in Figure 6.11 that the customer_id column now has values other than 3, and the vendor_id column now is limited to vendor 9: SELECT c.customer_first_name, c.customer_last_name, cp.customer_id, v.vendor_name, cp.vendor_id, ROUND(SUM(quantity * cost_to_customer_per_qty), 2) AS total_spent FROM farmers_market.customer c Figure 6.10 Continues 88 Chapter 6 ■ Aggregating Results for Analysis LEFT JOIN farmers_market.customer_purchases cp ON c.customer_id = cp.customer_id LEFT JOIN farmers_market.vendor v ON cp.vendor_id = v.vendor_id WHERE cp.vendor_id = 9 GROUP BY c.customer_first_name, c.customer_last_name, cp.customer_id, v.vendor_name, cp.vendor_id ORDER BY cp.customer_id, cp.vendor_id Or, we could remove the WHERE clause altogether and get one row for every customer-­vendor pair", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 54 + }, + { + "text": "Results for Analysis LEFT JOIN farmers_market.customer_purchases cp ON c.customer_id = cp.customer_id LEFT JOIN farmers_market.vendor v ON cp.vendor_id = v.vendor_id WHERE cp.vendor_id = 9 GROUP BY c.customer_first_name, c.customer_last_name, cp.customer_id, v.vendor_name, cp.vendor_id ORDER BY cp.customer_id, cp.vendor_id Or, we could remove the WHERE clause altogether and get one row for every customer-­vendor pair in the database. This would be useful as a query to support a reporting system that allows for front-­end filtering, such as Tableau. The query can provide a list of any customer that has shopped at any vendor and the sum of how much they have spent, and the reporting tool can then allow the user to choose any customer or vendor, to narrow down the results dynamically. You can now see how all of the basic SQL components you have learned in previous chapters are coming together to build analytical reports! MIN and MAX If we wanted to get the most and least expensive items per product category, con- sidering the fact that each vendor sets their own prices and can adjust prices per customer (which is why the customer_purchases table has a cost_to_customer_ per_qty field, so the original price can be overridden at the time of purchase, if needed), we will use the vendor_inventory table, which has a field for the original price the vendors set for each item they bring to market on each market date. First, let’s look at all of the available fields in the vendor_inventory table by using the following SELECT * query. The output is shown in Figure 6.12. SELECT * FROM farmers_market.vendor_inventory ORDER BY original_price LIMIT 10 Figure 6.11 (continued) Chapter 6 ■ Aggregating Results for Analysis 89 We can get the least and most expensive item prices in the entire table by using the MIN() and MAX() functions without grouping in MySQL, as shown here and in Figure 6.13: SELECT MIN(original_price) AS minimum_price, MAX(original_price) AS maximum_price FROM farmers_market.vendor_inventory ORDER BY original_price But if we want to get the lowest and highest prices within each product category, we have to group by the product_category_id (and product_ category_name, if we want to display it), then the summary values will be cal- culated per group, as shown in the following query and in Figure 6.14. Table aliases were added here since we’re referencing multiple tables and need to distinguish which table each field is from: SELECT pc.product_category_name, p.product_category_id, MIN(vi.original_price) AS minimum_price, MAX(vi.original_price) AS maximum_price FROM farmers_market.vendor_inventory AS vi INNER JOIN farmers_market.product AS p ON vi.product_id = p.product_id INNER JOIN farmers_market.product_category AS pc ON p.product_category_id = pc.product_category_id GROUP BY pc.product_category_name, p.product_category_id Figure 6.12 Figure 6.13 90 Chapter 6 ■ Aggregating Results for Analysis If we were to also add columns for MIN(product_name) and MAX(product_name), we would not get the product names associated with the lowest and highest prices; instead, we would get the first and last product names, sorted alpha- betically. If we wanted to get the products associated with these min and max prices per category, we would use window functions, which will be", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 55 + }, + { + "text": "would not get the product names associated with the lowest and highest prices; instead, we would get the first and last product names, sorted alpha- betically. If we wanted to get the products associated with these min and max prices per category, we would use window functions, which will be covered in the next chapter. COUNT and COUNT DISTINCT Suppose we wanted to count how many products were for sale on each market date, or how many different products each vendor offered. We can determine these values using COUNT and COUNT DISTINCT. COUNT will count up the rows within a group when used with GROUP BY, and COUNT DISTINCT will count up the unique values present in the specified field within the group. To determine how many products are offered for sale each market date, we can count up the rows in the vendor_inventory table, grouped by date. This doesn’t tell us what quantity of each product was offered or sold (because we’re not adding up the quantity column, or counting customer purchases), but counts the number of products available, because there is a row in this table for each product for each vendor for each market date. Of course, the values shown in the screenshots are too small to be realistic numbers, because the database has only been populated with a small number of sample rows per table, but you can see in Figure 6.15 that the result is a count for each market_date. SELECT market_date, COUNT(product_id) AS product_count FROM farmers_market.vendor_inventory GROUP BY market_date ORDER BY market_date If we wanted to know how many different products—­with unique product IDs—­each vendor brought to market during a date range, we could use COUNT DISTINCT on the product_id field, like so: Figure 6.14 Chapter 6 ■ Aggregating Results for Analysis 91 SELECT vendor_id, COUNT(DISTINCT product_id) AS different_products_offered FROM farmers_market.vendor_inventory WHERE market_date BETWEEN '2019-­03-­02' AND '2019-­03-­16' GROUP BY vendor_id ORDER BY vendor_id Note that the DISTINCT goes inside the parentheses for the COUNT() aggregate function. The results of the query are shown in Figure 6.16. Average What if, in addition to the count of different products per vendor, we also want the average original price of a product per vendor? We can add a line to the preceding query, and use the AVG() function, like we do in the following query, with results shown in Figure 6.17: SELECT vendor_id, COUNT(DISTINCT product_id) AS different_products_offered, AVG(original_price) AS average_product_price FROM farmers_market.vendor_inventory WHERE market_date BETWEEN '2019-­03-­02' AND '2019-­03-­16' GROUP BY vendor_id ORDER BY vendor_id Figure 6.15 Figure 6.16 92 Chapter 6 ■ Aggregating Results for Analysis However, we have to think about what we’re actually averaging here. Is it fair to call it “average product price” when the underlying table has one row per type of product? If the vendor brought 100 tomatoes to market, those would all be in one line of the underlying vendor inventory table, so the price of a tomato would only be included in the average once. Then if that same vendor", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 56 + }, + { + "text": "the underlying table has one row per type of product? If the vendor brought 100 tomatoes to market, those would all be in one line of the underlying vendor inventory table, so the price of a tomato would only be included in the average once. Then if that same vendor also sold bouquets of flowers for $20, no matter how many bouquets they brought, that would only be included in the average once. If you calculated the “average product price” for the vendor this way, you would just get the average of the price of one tomato and one bouquet. To get an actual average price of items in each vendor’s inventory between the specified dates, it might make more sense to multiply the quantity of each type of item times the price of that item, which is a calculation that would occur per row, then sum that up and divide by the total quantity of items, which is a calculation that would occur per vendor. Let’s try a calculation that includes these two summary values. We also surrounded the calculation with a ROUND() function to format the output in dollars, as shown in Figure 6.18. SELECT vendor_id, COUNT(DISTINCT product_id) AS different_products_offered, SUM(quantity * original_price) AS value_of_inventory, SUM(quantity) AS inventory_item_count, ROUND(SUM(quantity * original_price) / SUM(quantity), 2) AS average_item_price FROM farmers_market.vendor_inventory WHERE market_date BETWEEN '2019-­03-­02' AND '2019-­03-­16' GROUP BY vendor_id ORDER BY vendor_id Figure 6.17 Figure 6.18 Chapter 6 ■ Aggregating Results for Analysis 93 The multiplication of quantity * original_price inside the aggregate function is performed per row, then the aggregate SUMs are calculated, then the division of one SUM into the other to determine the “average item price” is calculated. So we’re performing mathematical operations both before and after the GROUP BY summarization occurs. Filtering with HAVING Filtering is another thing that can be done in the query after summarization occurs. In previous chapters and in the following query, we filtered rows using the WHERE clause. Here, we’re filtering to a date range in the WHERE clause prior to grouping. If you want to filter values after the aggregate functions are applied, you can add a HAVING clause to the query. This filters the groups based on the summary values. So, modifying the previous query, let’s filter to vendors who brought at least 100 items to the farmer’s market over the specified time period. You can see the HAVING clause usage in the following code, and the results in Figure 6.19: SELECT vendor_id, COUNT(DISTINCT product_id) AS different_products_offered, SUM(quantity * original_price) AS value_of_inventory, SUM(quantity) AS inventory_item_count, SUM(quantity * original_price) / SUM(quantity) AS average_item_price FROM farmers_market.vendor_inventory WHERE market_date BETWEEN '2019-­03-­02' AND '2019-­03-­16' GROUP BY vendor_id HAVING inventory_item_count >= 100 ORDER BY vendor_id TIP If you GROUP BY all of the fields that are supposed to be distinct in your resulting dataset, then add a HAVING clause that filters to aggregated rows with a COUNT(*) > 1, any results returned indicate that there is more than one row with your “unique” combination", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 57 + }, + { + "text": "vendor_id TIP If you GROUP BY all of the fields that are supposed to be distinct in your resulting dataset, then add a HAVING clause that filters to aggregated rows with a COUNT(*) > 1, any results returned indicate that there is more than one row with your “unique” combination of values, highlighting the existence of unwanted dupli- cates in your database or query results! Figure 6.19 94 Chapter 6 ■ Aggregating Results for Analysis CASE Statements Inside Aggregate Functions Earlier in this chapter, in the query that generated the output in Figure 6.4, we added up the quantity value in the customer_purchases table, which included discrete items sold individually as well as bulk items sold by ounce or pound, and it was awkward to add those quantities together. In Chapter 4, “Conditionals / CASE Statements,” you learned about conditional CASE statements. Here, we’ll use a CASE statement to specify which type of item quantities to add together using each SUM aggregate function. First, we’ll need to JOIN the customer_purchases table to the product table to pull in the product_qty_type column, which currently only contains the values “unit” and “lbs,” as shown in Figure 6.20. SELECT cp.market_date, cp.vendor_id, cp.customer_id, cp.product_id, cp.quantity, p.product_name, p.product_size, p.product_qty_type FROM farmers_market.customer_purchases AS cp INNER JOIN farmers_market.product AS p ON cp.product_id = p.product_id To create one column that only adds up quantities of products that are sold by unit, another column that adds up quantities of products sold by the pound, and a third for any products that may be entered in the future that are sold by other units (like bulk ounces), we’ll put CASE statements inside the SUM functions to indicate which values to add up in each summary column. First, we’ll review the results with the CASE statements included before grouping or using aggregate functions. Notice in Figure 6.21 that the CASE statements have Figure 6.20 Chapter 6 ■ Aggregating Results for Analysis 95 separated the quantity values into three different columns, by product_qty_type. These are the values we’ll be adding up per group in the next step: SELECT cp.market_date, cp.vendor_id, cp.customer_id, cp.product_id, CASE WHEN product_qty_type = \"unit\" THEN quantity ELSE 0 END AS quantity_units, CASE WHEN product_qty_type = \"lbs\" THEN quantity ELSE 0 END AS quantity_lbs, CASE WHEN product_qty_type NOT IN (\"unit\",\"lbs\") THEN quantity ELSE 0 END AS quantity_other, p.product_qty_type FROM farmers_market.customer_purchases cp INNER JOIN farmers_market.product p ON cp.product_id = p.product_id Now we can add the SUM functions around each CASE statement to add up these values per market date per customer, as defined in the GROUP BY clause. The results are shown in Figure 6.22. (The prior screenshot was just a subset of the full results, so there may be values added into the rows in Figure 6.22 that are not visible in Figure 6.21.) SELECT cp.market_date, cp.customer_id, SUM(CASE WHEN product_qty_type = \"unit\" THEN quantity ELSE 0 END) AS qty_units_purchased, SUM(CASE WHEN product_qty_type = \"lbs\" THEN quantity ELSE 0 END) AS qty_lbs_purchased, SUM(CASE WHEN product_qty_type NOT IN (\"unit\",\"lbs\") THEN quantity ELSE", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 58 + }, + { + "text": "added into the rows in Figure 6.22 that are not visible in Figure 6.21.) SELECT cp.market_date, cp.customer_id, SUM(CASE WHEN product_qty_type = \"unit\" THEN quantity ELSE 0 END) AS qty_units_purchased, SUM(CASE WHEN product_qty_type = \"lbs\" THEN quantity ELSE 0 END) AS qty_lbs_purchased, SUM(CASE WHEN product_qty_type NOT IN (\"unit\",\"lbs\") THEN quantity ELSE 0 END) AS qty_other_purchased Figure 6.21 Continues 96 Chapter 6 ■ Aggregating Results for Analysis FROM farmers_market.customer_purchases cp INNER JOIN farmers_market.product p ON cp.product_id = p.product_id GROUP BY market_date, customer_id ORDER BY market_date, customer_id So now you have seen examples of how to use COUNT, COUNT DISTINCT, SUM, AVG, MIN, and MAX aggregate SQL functions, as well as CASE statements and cal- culations inside the functions, and calculations performed with the summarized values. I hope that by now you are starting to dream up how to apply these skills to your own work! Exercises Using the Included Database 1. Write a query that determines how many times each vendor has rented a booth at the farmer’s market. In other words, count the vendor booth assignments per vendor_id. 2. In Chapter 5, “SQL Joins,” Exercise 3, we asked “When is each type of fresh fruit or vegetable in season, locally?” Write a query that displays the product category name, product name, earliest date available, and latest date avail- able for every product in the “Fresh Fruits & Vegetables” product category. 3. The Farmer’s Market Customer Appreciation Committee wants to give a bumper sticker to everyone who has ever spent more than $50 at the market. Write a query that generates a list of customers for them to give stickers to, sorted by last name, then first name. (HINT: This query requires you to join two tables, use an aggregate function, and use the HAVING keyword.) Figure 6.22 (continued) C H A P T E R 97 7 All of the functions that have been covered in this book so far, like ROUND(), return one value in each row of the results dataset. When GROUP BY is used, the functions operate on multiple values in an aggregated group of records, sum- marizing across multiple rows in the underlying dataset, like AVG(), but each value returned is associated with a single row in the results. Window functions operate across multiple records, as well, but those records don’t have to be grouped in the output. This gives the ability to put the values from one row of data into context compared to a group of rows, or partition, enabling an analyst to write queries that answer questions like: If the dataset were sorted, where would this row land in the results? How does a value in this row compare to a value in the prior row? How does a value in the current row compare to the average value for its group? So, window functions return group aggregate calculations alongside individual row-­level information for items in that group, or partition. They can also be used to rank or sort values within each partition. One", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 59 + }, + { + "text": "prior row? How does a value in the current row compare to the average value for its group? So, window functions return group aggregate calculations alongside individual row-­level information for items in that group, or partition. They can also be used to rank or sort values within each partition. One use for window functions in data science is to include some information from a past record alongside the most recent detail record related to an entity. For example, we could use window functions to get the date of the first purchase a person made at the farmer’s market, to be returned alongside their detailed purchase records, which could then be used to determine how long they had been a customer at the time each purchase was made. Window Functions and Subqueries 98 Chapter 7 ■ Window Functions and Subqueries ROW NUMBER Based on what you’ve learned in previous chapters, if you wanted to determine how much the most expensive product sold by each vendor costs, you could group the records in the vendor_inventory table by vendor_id, and return the maximum original_price value using the following query: SELECT vendor_id, MAX(original_price) AS highest_price FROM farmers_market.vendor_inventory GROUP BY vendor_id ORDER BY vendor_id But this just gives you the price of the most expensive item per vendor. If you wanted to know which item was the most expensive, how would you determine which product_id was associated with that MAX(original_price) per vendor? There is a window function that enables you to rank rows by a value—­in this case, ranking products per vendor by price—­called ROW_NUMBER(). This approach will allow you to maintain the detail-­level information that you would otherwise lose by aggregating like we did in the preceding query: SELECT vendor_id, market_date, product_id, original_price, ROW_NUMBER() OVER (PARTITION BY vendor_id ORDER BY original_price DESC) AS price_rank FROM farmers_market.vendor_inventoryORDER BY vendor_id, original_price DESC Let’s break that syntax down a bit. I would interpret the ROW_NUMBER() line as “number the rows of inventory per vendor, sorted by original price, in descend- ing order.” The part inside the parentheses says how to apply the ROW_NUMBER() function. We’re going to PARTITION BY vendor_id (you can think of this like a GROUP BY without actually combining the rows, so we’re telling it how to split the rows into groups, without aggregating). Then within the partition, the ORDER BY indicates how to sort the rows. So, we’ll sort the rows by price, high to low, within each vendor_id partition, and number each row. That means the highest-­ priced item per vendor will be first, and assigned row number 1. You can see in Figure 7.1 that for each vendor, the products are sorted by original_price, high to low, and the row numbering column is called price_ rank. The row numbering starts over when you get to the next vendor_id, so the most expensive item per vendor has a price_rank of 1. Chapter 7 ■ Window Functions and Subqueries 99 To return only the record of the highest-­priced item per vendor, you can", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 60 + }, + { + "text": "column is called price_ rank. The row numbering starts over when you get to the next vendor_id, so the most expensive item per vendor has a price_rank of 1. Chapter 7 ■ Window Functions and Subqueries 99 To return only the record of the highest-­priced item per vendor, you can query the results of the previous query (which is called a subquery), and limit the output to the #1 ranked item per vendor_id. With this approach, you’re not using a GROUP BY to aggregate the records. You’re sorting the records within each partition (a set of records that share a value or combination of values—­vendor_id in this case), then filtering to a value (the row number called price_rank here) that was evaluated over that partition. Figure 7.2 shows the highest-­priced product per vendor using the following query: SELECT * FROM ( SELECT vendor_id, market_date, product_id, original_price, ROW_NUMBER() OVER (PARTITION BY vendor_id ORDER BY original_price DESC) AS price_rank FROM farmers_market.vendor_inventory ORDER BY vendor_id) x WHERE x.price_rank = 1 Figure 7.1 Figure 7.2 100 Chapter 7 ■ Window Functions and Subqueries This will only return one row per vendor, even if there are multiple products with the same price. To return all products with the highest price per vendor when there is more than one with the same price, use the RANK function found in the next section. If you want to determine which one of the multiple items gets returned by this ROW_NUMBER function, you can add additional sorting columns in the ORDER BY section of the ROW_NUMBER function. For example, you can sort by both original_price (descending) and market_date (ascending) to get the product brought to market by each vendor the earliest that had this top price. You’ll notice that the preceding query has a different structure than the queries we have written so far. There is one query embedded inside the other! Sometimes this is called “querying from a derived table,” but is more commonly called a “subquery.” What we’re doing is treating the results of the “inner” SELECT statement like a table, here given the table alias x, selecting all columns from it, and filtering to only the rows with a particular ROW_NUMBER. Our ROW_NUMBER column is aliased price_rank, and we’re filtering to price_rank = 1, because we numbered the rows by original_price in descending order, so the most expensive item will have the lowest row number. The reason we have to structure this as a subquery is that the entire dataset has to be processed in order for the window function to find the highest price per vendor. So we can’t filter the results using a WHERE clause (which you’ll remember evaluates the conditional statements row by row) because when that filtering is applied, the ROW_NUMBER has not yet been calculated for every row. Figure 7.3 illustrates which parts of the SQL statement are considered the “inner” and “outer” queries. The “outer” part of a subquery is processed after the “inner” query is complete, so the row", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 61 + }, + { + "text": "row) because when that filtering is applied, the ROW_NUMBER has not yet been calculated for every row. Figure 7.3 illustrates which parts of the SQL statement are considered the “inner” and “outer” queries. The “outer” part of a subquery is processed after the “inner” query is complete, so the row numbers have been determined, and we can then filter by the values in the price_rank column. TIP Many SQL editors allow you to run the inner query by itself, by highlighting it and executing the selected SQL only. This allows you to preview the results of the inner query that will then be used by the outer query. Figure 7.3 Chapter 7 ■ Window Functions and Subqueries 101 If we didn’t use a subquery, and had attempted to filter based on the values in the price_rank field by adding a WHERE clause to the first query with the ROW_ NUMBER function, we would get an error. The price_rank value is unknown at the time the WHERE clause conditions are evaluated per row, because the window functions have not yet had a chance to check the entire dataset to determine the ranking. If we tried to put the ROW_NUMBER function in the WHERE clause, instead of referencing the price_rank alias, we would get a different error, but for the same reason. You will see the subquery format throughout this chapter, because if you want to do anything with the results of most window functions, you have to allow them to calculate across the entire dataset first. Then, by treating the results like a table, you can query from and filter by the results returned by the window functions. Note that you can also use ROW_NUMBER without a PARTITION BY clause, to number every record across the whole result (instead of numbering per parti- tion). If you were to use the same ORDER BY clause we did earlier, and eliminate the PARTITION BY clause, then only one item with the highest price in the entire results set would get the price_rank of 1, instead of one item per vendor. RANK and DENSE RANK Two other window functions are very similar to ROW_NUMBER and have the same syntax, but provide slightly different results. The RANK function numbers the results just like ROW_NUMBER does, but gives rows with the same value the same ranking. If we run the same query as before, but replace ROW_NUMBER with RANK, we get the output shown in Figure 7.4. SELECT vendor_id, market_date, product_id, original_price, RANK() OVER (PARTITION BY vendor_id ORDER BY original_price DESC) AS price_rank FROM farmers_market.vendor_inventory ORDER BY vendor_id, original_price DESC If we used subquery structure and embedded this query inside another SELECT statement like we did previously, and filtered to price_rank = 1, multiple rows per vendor would be returned. Notice in Figure 7.4 that the ranking for vendor_id 1 goes from 1 to 2 to 4, skipping 3. That’s because there’s a tie for second place, so there’s no third place. If you don’t want", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 62 + }, + { + "text": "previously, and filtered to price_rank = 1, multiple rows per vendor would be returned. Notice in Figure 7.4 that the ranking for vendor_id 1 goes from 1 to 2 to 4, skipping 3. That’s because there’s a tie for second place, so there’s no third place. If you don’t want to skip numbers like this in your ranking when there is a tie 102 Chapter 7 ■ Window Functions and Subqueries (so the items for vendor_id in the example would be numbered 1 and 2 instead of 1 and 5), use the DENSE_RANK function. If you don’t want any ties in your num- bering at all, and want each row to have its own number, use the ROW_NUMBER function (compare the output in Figure 7.4 to the output in Figure 7.1). NTILE The ROW_NUMBER() and RANK() functions can help answer a question that asks something like “What are the top 10 items sold at the farmer’s market, by price?” (by filtering the results to rows numbered less than or equal to 10). But what if you were asked to return the “top tenth” of the inventory, when sorted by price? You could start by running a query that used the COUNT() function, dividing the number returned by 10, then writing another query that numbers the rows, and filtering to those with a row number less than or equal to the number you just determined. But that isn’t a dynamic solution, and you’d have to modify it as the number of rows in the database changed. The dynamic solution is to use the NTILE function. With NTILE, you specify a number inside the parentheses, NTILE(n), to indicate that you want the results broken up into n blocks. So, to get the top tenth, you could put 10 in the paren- theses, with no partition (segmenting the entire results set), then filter to the rows in NTILE 1, like so: SELECT vendor_id, market_date, Figure 7.4 Chapter 7 ■ Window Functions and Subqueries 103 product_id, original_price, NTILE(10) OVER (ORDER BY original_price DESC) AS price_ntile FROM farmers_market.vendor_inventory ORDER BY original_price DESC If the number of rows in the results set can be divided evenly, the results will be broken up into n equally sized groups, labeled 1 to n. If they can’t be divided up evenly, some groups will end up with one more row than others. Note that the NTILE is only using the count of rows to split the groups (or to split the partition into groups, if you specify a partition), and is not using a field value to determine where to make the splits. Therefore, it’s possible that two rows with the same value specified in the ORDER BY clause (two products with the same original_price, in this case) will end up in two different NTILE groups. You can sort on additional fields if you want a little more control over how the rows are split into NTILE groups. But if you want to ensure that all items with the", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 63 + }, + { + "text": "(two products with the same original_price, in this case) will end up in two different NTILE groups. You can sort on additional fields if you want a little more control over how the rows are split into NTILE groups. But if you want to ensure that all items with the same price are grouped together, for example, then it would make more sense to use RANK than NTILE, because in that case, you aren’t looking for evenly sized groupings. Aggregate Window Functions You learned about aggregate SQL functions like SUM() in Chapter 6, “Aggregating Results for Analysis,” and in this chapter you have learned about window functions that partition the results set. Can you imagine how they might be used together? It turns out that you can use most aggregate functions across partitions like the window functions, returning an aggregate calculation for a partition on every row in that partition (or, for the whole results set, if you don’t use the PARTITION BY clause). One way this approach can be used is to compare each row’s value to the aggregate value for that grouped category. For example, what if you are a farmer selling products at the market, and you want to know which of your products were above the average price per product on each market date? (Remember that because of the way our database is designed, this isn’t a true average for the full inventory, because we’re not multiplying by a quantity, but you can think of it as the average display price in a product catalog.) We can use the AVG() function as a window function, partitioned by market_date, and compare each product’s price to that value. First, let’s try using AVG() as a window function. The output of the following query is shown in Figure 7.5: SELECT vendor_id, market_date, Continues 104 Chapter 7 ■ Window Functions and Subqueries product_id, original_price, AVG(original_price) OVER (PARTITION BY market_date ORDER BY market_date) AS average_cost_product_by_market_date FROM farmers_market.vendor_inventory The AVG() function in this query is structured as a window function, meaning it has “OVER (PARTITION BY __ ORDER BY __)” syntax, so instead of returning a single row per group with the average for that group, like you would get with GROUP BY, this function displays the average for the partition on every row within the partition. You can see in Figure 7.5 that when you get to a new market_date value in the results dataset, the average_cost_product_by_market_date value changes. Now, let’s wrap that query inside another query (use it as a subquery) so we can compare the original price per item to the average cost of products on each market date that has been calculated by the window function. In this example, we are comparing the values in the last two columns of Figure 7.5. Remember that we can’t compare the two values in the original query, because the window function is calculated over multiple rows and won’t have a value for the parti- tion yet when the WHERE clause filters", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 64 + }, + { + "text": "example, we are comparing the values in the last two columns of Figure 7.5. Remember that we can’t compare the two values in the original query, because the window function is calculated over multiple rows and won’t have a value for the parti- tion yet when the WHERE clause filters are being applied row by row. Using a subquery, we can filter the results to a single vendor, with vendor_id 1, and only display products that have prices above the market date’s average product cost. Here we will also format the average_cost_product_by_market_ date to two digits after the decimal point using the ROUND() function: SELECT * FROM ( SELECT vendor_id, market_date, Figure 7.5 (continued) Chapter 7 ■ Window Functions and Subqueries 105 product_id, original_price, ROUND(AVG(original_price) OVER (PARTITION BY market_date ORDER BY market_date), 2) AS average_cost_product_by_market_date FROM farmers_market.vendor_inventory ) x WHERE x.vendor_id = 1 AND x.original_price > x.average_cost_product_by_market_date ORDER BY x.market_date, x.original_price DESC Note that we will get different (and incorrect) results if we put the WHERE clause filtering by vendor_id inside the parentheses with the original query in this case. That’s because the results set of the inner SELECT statement would be filtered to vendor_id 1 before the window function was calculated, we would only be calculating the average price of vendor 1’s products! Since we want to compare vendor 1’s prices on each market date to the average price of all vendors’ products on each market date, we don’t want to filter to vendor_id 1 until after the averages have been calculated, so we put the WHERE clause on the “outer” query outside the parentheses. The results of the preceding query are shown in Figure 7.6. So vendor_id 1 had a single product, with product_id 11, that was above the average product cost on each of the market dates listed. Another use of an aggregate window function is to count how many items are in each partition. The following is a query that counts how many different products each vendor brought to market on each date, and displays that count on each row. This way, even if the results weren’t sorted in a way that let you quickly determine how many inventory rows there are for each vendor, you would know that the row you’re looking at represents just one of the products in a counted set: SELECT vendor_id, market_date, product_id, original_price, COUNT(product_id) OVER (PARTITION BY market_date, vendor_id) vendor_product_count_per_market_date FROM farmers_market.vendor_inventory ORDER BY vendor_id, market_date, original_price DESC Figure 7.6 106 Chapter 7 ■ Window Functions and Subqueries The output for this query is shown in Figure 7.7. You can see that even if I’m only looking at one row for vendor 9 on March 9, 2019, I would know that it is one of three products that vendor had in their inventory on that market date. You can also use aggregate window functions to calculate running totals. In the first query shown next, we’re not using a PARTITION BY clause, so the running total of the price is calculated", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 65 + }, + { + "text": "that it is one of three products that vendor had in their inventory on that market date. You can also use aggregate window functions to calculate running totals. In the first query shown next, we’re not using a PARTITION BY clause, so the running total of the price is calculated across the entire results set, in the sort order specified in the ORDER BY clause of the SUM() window function. The results are displayed in Figure 7.8. SELECT customer_id, market_date, vendor_id, product_id, quantity * cost_to_customer_per_qty AS price, SUM(quantity * cost_to_customer_per_qty) OVER (ORDER BY market_date, transaction_time, customer_id, product_id) AS running_total_purchases FROM farmers_market.customer_purchases Figure 7.7 Figure 7.8 Chapter 7 ■ Window Functions and Subqueries 107 In this next query, we are calculating the same running total, but it is parti- tioned by customer_id. That means that each time we get to a new customer_id, the running total resets. So we’re getting a running total of the cost of items purchased by each customer, sorted by the date and time, and the product ID (in case any two items have identical purchase times). The result is shown in Figure 7.9. SELECT customer_id, market_date, vendor_id, product_id, quantity * cost_to_customer_per_qty AS price, SUM(quantity * cost_to_customer_per_qty) OVER (PARTITION BY customer_id ORDER BY market_date, transaction_time, product_id) AS customer_spend_running_total FROM farmers_market.customer_purchases This SUM functions as a running total because of the combination of the PARTITION BY and ORDER BY clauses in the window function. We showed what happens when there is only an ORDER BY clause, and when both clauses are pre- sent. What do you expect to happen when there is only a PARTITION BY clause (and no ORDER BY clause)? SELECT customer_id, market_date, vendor_id, product_id, ROUND(quantity * cost_to_customer_per_qty, 2) AS price, ROUND(SUM(quantity * cost_to_customer_per_qty) OVER (PARTITION BY customer_id), 2) AS customer_spend_total FROM farmers_market.customer_purchases As hinted at by the field name alias, this version with no in-­partition sorting calculates the total spent by the customer and displays that summary total on Figure 7.9 108 Chapter 7 ■ Window Functions and Subqueries every row. So, without the ORDER BY, the SUM is calculated across the entire par- tition, instead of as a per-­row running total, as shown in Figure 7.10. We also added the ROUND() function so this final output displays the prices with two numbers after the decimal point. LAG and LEAD With the running total example in the previous section, you can start to see how SQL can be used to calculate changes in a value over time. Using the vendor_booth_assignments table in the Farmer’s Market database, we can display each vendor’s booth assignment for each market_date alongside their previous booth assignments using the LAG() function. LAG retrieves data from a row that is a selected number of rows back in the dataset. You can set the number of rows (offset) to any integer value x to count x rows backwards, following the sort order specified in the ORDER BY section of the window function: SELECT market_date, vendor_id, booth_number, LAG(booth_number,1) OVER (PARTITION BY vendor_id ORDER BY", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 66 + }, + { + "text": "number of rows back in the dataset. You can set the number of rows (offset) to any integer value x to count x rows backwards, following the sort order specified in the ORDER BY section of the window function: SELECT market_date, vendor_id, booth_number, LAG(booth_number,1) OVER (PARTITION BY vendor_id ORDER BY market_date, vendor_id) AS previous_booth_number FROM farmers_market.vendor_booth_assignments ORDER BY market_date, vendor_id, booth_number In this case, for each vendor_id for each market_date, we’re pulling the booth_number the vendor had 1 market date in the past. As you can see in Figure 7.11, the values are all NULL for the first market date, because there is no prior market date to pull values from. Figure 7.10 Chapter 7 ■ Window Functions and Subqueries 109 The recipient of a report like this, such as the manager of the farmer’s market, may want to filter these query results to a specific market date to determine which vendors are new or changing booths that day, so we can contact them and ensure setup goes smoothly. We will create this report by wrapping the query with the LAG function in another query, which we can use to filter the results to a market_date and vendors whose current booth_number is different from their previous_booth_number: SELECT * FROM ( SELECT market_date, vendor_id, booth_number, LAG(booth_number,1) OVER (PARTITION BY vendor_id ORDER BY market_ date, vendor_id) AS previous_booth_number FROM farmers_market.vendor_booth_assignments ORDER BY market_date, vendor_id, booth_number ) x WHERE x.market_date = '2019-­04-­10' AND (x.booth_number <> x.previous_booth_number OR x.previous_ booth_number IS NULL) If you look closely at Figure 7.11, you can see that for the April 10, 2019 market, vendor 1 and vendor 4 have swapped booths compared to the previous market date. This would be hard to spot from a printout of this output, but using the preceding query, we can return just the rows with booth changes on the speci- fied date, as shown in Figure 7.12. Figure 7.11 110 Chapter 7 ■ Window Functions and Subqueries To show another example use case, let’s say we want to find out if the total sales on each market date are higher or lower than they were on the previous market date. In this example, we are going to use the customer_purchases table from the Farmer’s Market database, and also add in a GROUP BY function, which the previous examples did not include. The window functions are calculated after the grouping and aggregation occurs. First, we need to get the total sales per market date, using a GROUP BY and reg- ular aggregate SUM. The results of the following query are shown in Figure 7.13: SELECT market_date, SUM(quantity * cost_to_customer_per_qty) AS market_date_total_sales FROM farmers_market.customer_purchases GROUP BY market_date ORDER BY market_date Then, we can add the LAG() window function to output the previous market_ date’s calculated sum on each row. We ORDER BY market_date in the window function to ensure it’s the previous market date we’re comparing to and not another date. You can see in Figure 7.14 that each row has a new total", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 67 + }, + { + "text": "the LAG() window function to output the previous market_ date’s calculated sum on each row. We ORDER BY market_date in the window function to ensure it’s the previous market date we’re comparing to and not another date. You can see in Figure 7.14 that each row has a new total value (for that market date), as well as the previous market date’s total: SELECT market_date, SUM(quantity * cost_to_customer_per_qty) AS market_date_total_sales, LAG(SUM(quantity * cost_to_customer_per_qty), 1) OVER (ORDER BY market_date) AS previous_market_date_total_sales FROM farmers_market.customer_purchases GROUP BY market_date ORDER BY market_date Figure 7.12 Figure 7.13 Chapter 7 ■ Window Functions and Subqueries 111 LEAD works the same way as LAG, but it gets the value from the next row instead of the previous row (assuming the offset integer is 1). You can set the offset integer to any value x to count x rows forward, following the sort order specified in the ORDER BY section of the window function. If the rows are sorted by a time value, LAG would be retrieving data from the past, and LEAD would be retrieving data from the future (relative to the current row). These values can also now be used in calculations; for example, to determine the change in sales week to week. This chapter just covers the tip of the iceberg when it comes to window functions! Look in the documentation for the type of database you’re working with to see what other functions are available, and what caveats to be aware of for each. Some database systems offer additional capabilities. For example, PostgreSQL supports something called “window naming,” Oracle has additional useful aggregate functions like LISTAGG (which operates on string values), and some database systems allow for additional clauses like RANGE. Once you understand the concept of a window function and how to use it in your query, you have the knowledge you need to research and apply the many variations. Exercises Using the Included Database 1. Do the following two steps: a. Write a query that selects from the customer_purchases table and numbers each customer’s visits to the farmer’s market (labeling each market date with a different number). Each customer’s first visit is labeled 1, second visit is labeled 2, etc. (We are of course not counting visits where no purchases are made, because we have no record of those.) You can either display all rows in the customer_purchases table, with the counter changing on each new market date for each customer, or select only the unique market dates per customer (without purchase details) and number those visits. HINT: One of these approaches uses ROW_NUMBER() and one uses DENSE_RANK(). Figure 7.14 112 Chapter 7 ■ Window Functions and Subqueries b. Reverse the numbering of the query from a part so each customer’s most recent visit is labeled 1, then write another query that uses this one as a subquery and filters the results to only the customer’s most recent visit. 2. Using a COUNT() window function, include a value along with each row", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 68 + }, + { + "text": "of the query from a part so each customer’s most recent visit is labeled 1, then write another query that uses this one as a subquery and filters the results to only the customer’s most recent visit. 2. Using a COUNT() window function, include a value along with each row of the customer_purchases table that indicates how many different times that customer has purchased that product_id. 3. In the last query associated with Figure 7.14 from the chapter, we used LAG and sorted by market_date. Can you think of a way to use LEAD in place of LAG, but get the exact same output? C H A P T E R 113 8 Data scientists use date and time functions many different ways in our queries. We may use two dates to calculate a duration, for example. Many machine learning algorithms are “trained” to identify patterns in data from the past and use those patterns to predict future outcomes. In order to build a dataset for that purpose, we have to be able to filter queries by time range. Often, datasets that are built for predictive models include summaries of activities within dynamic date ranges—­for example, a count of some activity occurrence during each of the past three months. Or, in the case of time-­series analysis, an input dataset might include one row per time period (hour, day, week, month) with a count of something associated with each time period; for example, the number of patients a doctor sees per week. Many predictive models are time-­bound. For example, the question “Will this first-­time customer become a repeat customer?” will be further refined as “What is the likelihood that each first-­time customer at today’s farmer’s market will return and make a second purchase within the next month?” To answer this question, we could create a dataset with a row for every customer, columns containing data values as of the time of their first purchase, and a binary “target variable” that indicates whether that customer made another purchase within a month of their first purchase date. Let’s look at some different ways to work with date and time values in our Farmer’s Market database. Date and Time Functions 114 Chapter 8 ■ Date and Time Functions Setting datetime Field Values The Farmer’s Market market_date_info table doesn’t include any fields stored as datetime values, so in order to demonstrate date and time functions without having to combine fields in every query, I’m going to first create a demonstra- tion table with datetimes created by combining the market_date and market_ start_time fields in the market_date_info table using the following query: CREATE TABLE farmers_market.datetime_demo AS ( SELECT market_date, market_start_time, market_end_time, STR_TO_DATE(CONCAT(market_date, ' ', market_start_time), '%Y-­%m-­%d %h:%i %p') AS market_start_datetime, STR_TO_DATE(CONCAT(market_date, ' ', market_end_time), '%Y-­%m-­%d %h:%i %p') AS market_end_datetime FROM farmers_market.market_date_info ) We will go over table creation in Chapter 14, “Storing Machine Learning Results,” but I want to explain what the functions here are doing. Refer to Figure 8.1 to see data in the table", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 69 + }, + { + "text": "%h:%i %p') AS market_start_datetime, STR_TO_DATE(CONCAT(market_date, ' ', market_end_time), '%Y-­%m-­%d %h:%i %p') AS market_end_datetime FROM farmers_market.market_date_info ) We will go over table creation in Chapter 14, “Storing Machine Learning Results,” but I want to explain what the functions here are doing. Refer to Figure 8.1 to see data in the table generated by this query. The innermost part of the nested functions in the line of the query used to create the market_start_datetime field concatenates the market_date and market_start_time into a single string value, using CONCAT(). The surround- ing STR_TO_DATE() function, as you might guess, converts string values to date values. The string of percent signs and letters in single quotes at the end is an input parameter that tells the function how the date and time are formatted. %Y is a 4-­digit year, %m is a 2-­digit month, %d is a 2-­digit day, %h is the hour, %i represents the minutes, and %p indicates there is an AM/PM indicator in the time string. Every database system has some codes for the date and time format- ting, which can be found in the documentation, but these values are common, originating from the C programming language. Figure 8.1 Chapter 8 ■ Date and Time Functions 115 NOTE You can find the SQL date and time function documentation for any data- base system by searching the internet for “[database system] date and time functions.” For MySQL 8.0, you can find this documentation at dev.mysql.com/doc/refman/ 8.0/en/date-­and-­time-­functions.html. The combination of functions in the query associated with Figure 8.1 is taking each date and time string, concatenating them into a combined datetime string, and converting that to a datetime data type. So, the final market_start_datetime and market_end_datetime fields are actually stored as datetime values, which we can then use to perform calculations, like finding the difference between two datetimes. The STR_TO_DATE() function does the type conversion to a date, time, or datetime, depending on the input. It will return a NULL value if the input string isn’t formatted in a way it can interpret. You’ll notice that the dates in the final two columns in Figure 8.1 are formatted as YYYY-­MM-­DD, and the times are in 24-­hour time (HH:MM:SS), which is one indication that the fields are datetimes (though I should note it would also be possible to format a string to look like a datetime using the DATE_FORMAT() function and a particular formatting string). EXTRACT and DATE_PART You will encounter datetime data types, such as timestamps, in the databases you work with and might only need a portion of the stored date and time value. For example, you might only want the month and day from a full date, in one field, with the year stripped out into a second field, to create a year-­over-­year comparison (to align and visualize daily totals from different years by month and day). Depending on the database system you are using, the function that retrieves different portions of a datetime value may be called EXTRACT (MySQL), DATE_ PART (Redshift),", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 70 + }, + { + "text": "stripped out into a second field, to create a year-­over-­year comparison (to align and visualize daily totals from different years by month and day). Depending on the database system you are using, the function that retrieves different portions of a datetime value may be called EXTRACT (MySQL), DATE_ PART (Redshift), or DATEPART (Oracle and SQL Server). The example Farmer’s Market database is in MySQL, so these examples use EXTRACT(), but the con- cepts are the same for the other functions, even though the syntax will vary. The market_start_datetime field in Figure 8.2 is an example of a MySQL datetime type field. In addition to EXTRACT(), MySQL offers the functions DATE() and TIME() to extract the date and time parts of a datetime field, respectively (you put the date- time value inside the parentheses, and just the date or time portion is returned). Using datetime values established in the datetime_demo table created in the previous section, we can EXTRACT date and time parts from the fields. The following query demonstrates five different “date parts” that can be extracted from the datetime and results in the output shown in Figure 8.2. Using 116 Chapter 8 ■ Date and Time Functions the time intervals allowed by the database system (see the documentation for others), you can extract portions of a datetime field as needed: SELECT market_start_datetime, EXTRACT(DAY FROM market_start_datetime) AS mktsrt_day, EXTRACT(MONTH FROM market_start_datetime) AS mktsrt_month, EXTRACT(YEAR FROM market_start_datetime) AS mktsrt_year, EXTRACT(HOUR FROM market_start_datetime) AS mktsrt_hour, EXTRACT(MINUTE FROM market_start_datetime) AS mktsrt_minute FROM farmers_market.datetime_demo WHERE market_start_datetime = '2019-­03-­02 08:00:00' There are also shortcuts for extracting the entire date and entire time from the datetime field, so you don’t have to extract each part and re-­concatenate it together. The following query and the output in Figure 8.3 demonstrate the DATE() and TIME() functions: SELECT market_start_datetime, DATE(market_start_datetime) AS mktsrt_date, TIME(market_start_datetime) AS mktsrt_time FROM farmers_market.datetime_demo WHERE market_start_datetime = '2019-­03-­02 08:00:00' DATE_ADD and DATE_SUB The powerful thing about storing string dates as datetime values (or converting them using SQL) is that you can do date calculations, which is not possible when they are stored as numbers and punctuation and letters in a string field. Date math can get complex when dealing with multiple time zones, so in this case we’re assuming that all datetimes we’re working with are from the same time zone. Here, we’ll use the market_start_datetime and market_end_ datetime fields to demonstrate. If you wanted to determine how many sales occurred within the first 30 min- utes after the farmer’s market opened, how would you dynamically determine what cutoff time to use (automatically calculate it for every market date in your database)? This is where the DATE_ADD function comes in. We can use SQL to Figure 8.2 Figure 8.3 Chapter 8 ■ Date and Time Functions 117 add 30 minutes to the start time by passing the datetime, the interval (minutes, in this case), and the number of minutes we want to add into the DATE_ADD function, as shown in the second line of the following", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 71 + }, + { + "text": "8.2 Figure 8.3 Chapter 8 ■ Date and Time Functions 117 add 30 minutes to the start time by passing the datetime, the interval (minutes, in this case), and the number of minutes we want to add into the DATE_ADD function, as shown in the second line of the following query: SELECT market_start_datetime, DATE_ADD(market_start_datetime, INTERVAL 30 MINUTE) AS mktstrt_date_ plus_30min FROM farmers_market.datetime_demo WHERE market_start_datetime = '2019-­03-­02 08:00:00' I filtered the results to a single market date for clarity. You can see in Figure 8.4 that the calculated mktstrt_date_plus_30min is 30 minutes after the displayed market_start_datetime. If we instead wanted to do a calculation that required looking 30 days past a date (like the example analysis mentioned in the introduction, which would require calculating 30 days past a customer’s first purchase to determine if they made a second purchase within that time frame), we could change the interval parameter from MINUTE to DAY, and add 30 days instead: SELECT market_start_datetime, DATE_ADD(market_start_datetime, INTERVAL 30 DAY) AS mktstrt_date_ plus_30days FROM farmers_market.datetime_demo WHERE market_start_datetime = '2019-­03-­02 08:00:00' You can see in Figure 8.5 that the calculated mktstrt_date_plus_30min is 30 days after market_start_datetime. There is also a related function called DATE_SUB() that subtracts intervals from datetimes. However, instead of switching to DATE_SUB(), you could also just add a negative number to the datetime if you prefer. The following query demonstrates that using DATE_ADD() to add –30 days to a date has the same effect as using DATE_SUB() to subtract 30 days from a date, and the results are shown in Figure 8.6: Figure 8.4 Figure 8.5 118 Chapter 8 ■ Date and Time Functions SELECT market_start_datetime, DATE_ADD(market_start_datetime, INTERVAL -­30 DAY) AS mktstrt_date_ plus_neg30days, DATE_SUB(market_start_datetime, INTERVAL 30 DAY) AS mktstrt_date_ minus_30days FROM farmers_market.datetime_demo WHERE market_start_datetime = '2019-­03-­02 08:00:00' DATEDIFF In the previous section we added 30 days to a date using DATE_ADD(), and I mentioned that the result could be used to determine if an action occurs within 30 days of the first purchase date. However, there is another way to determine whether two dates are within 30 days of one another: DATEDIFF()! DATEDIFF is a SQL function available in most database systems that accepts two dates or datetime values, and returns the difference between them in days. Here, the inner query (by which I mean the query inside parentheses, aliased “x”) returns the first and last market dates from the datetime_demo table, and the outer query (which is selecting from “x”) calculates the difference between those two dates using DATEDIFF. The output of this query is shown in Figure 8.7: SELECT x.first_market, x.last_market, DATEDIFF(x.last_market, x.first_market) days_first_to_last FROM ( SELECT min(market_start_datetime) first_market, max(market_start_datetime) last_market FROM farmers_market.datetime_demo ) x There are additional examples using DATEDIFF later in this chapter. Figure 8.6 Figure 8.7 Chapter 8 ■ Date and Time Functions 119 TIMESTAMPDIFF The DATEDIFF function returns the difference in days, but there is also a function in MySQL called TIMESTAMPDIFF that returns the difference between two date- times in any chosen interval. Here, we calculate the hours", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 72 + }, + { + "text": "this chapter. Figure 8.6 Figure 8.7 Chapter 8 ■ Date and Time Functions 119 TIMESTAMPDIFF The DATEDIFF function returns the difference in days, but there is also a function in MySQL called TIMESTAMPDIFF that returns the difference between two date- times in any chosen interval. Here, we calculate the hours and minutes between the market start and end times on each market date. The results are shown in Figure 8.8: SELECT market_start_datetime, market_end_datetime, TIMESTAMPDIFF(HOUR, market_start_datetime, market_end_datetime) AS market_duration_hours, TIMESTAMPDIFF(MINUTE, market_start_datetime, market_end_datetime) AS market_duration_mins FROM farmers_market.datetime_demo In Oracle SQL, you can simply subtract two datetimes from one another and use the EXTRACT function to specify which interval you want the result returned in. In Redshift and MS SQL Server, the TIMESTAMPDIFF doesn’t exist and isn’t necessary, because the DATEDIFF function allows for specification of a datepart interval as a parameter. NOTE An interesting note about the timestamp values in many database (and other) systems is that they are stored as 32-­bit integers “under the hood” that represent the number of seconds since January 1, 1970. Because of this, the latest timestamp that can be stored that fits within 32 bits is 2038-­01-­19 03:14:07. Timestamps above this value will cause an integer overflow (similar to the “Y2K” issue) until database systems are updated to use a new timestamp standard. Date Functions in Aggregate Summaries and Window Functions In this section, we’ll explore a few ways that you can use date functions when summarizing data. Figure 8.8 120 Chapter 8 ■ Date and Time Functions Let’s say we wanted to get a profile of each farmer’s market customer’s habits over time. So, we’ll want to group the results at the customer level and include some date-­related summary information in the output. Our database isn’t very heavily populated with example purchases over a long time period yet, but we can use the sample data to demonstrate these concepts. First, let’s get each customer’s purchase detail records, particularly the dates on which each customer made purchases. We’ll start by querying the database for the records for customer_id 1: SELECT customer_id, market_date FROM farmers_market.customer_purchases WHERE customer_id = 1 Figure 8.9 shows all of the purchases made by customer 1 over time. Let’s summarize this data and get their earliest purchase date, latest purchase date, and number of different days on which they made a purchase. We’ll GROUP BY customer_id, use MIN and MAX to get the lowest (earliest) and highest (latest) purchase dates, and COUNT DISTINCT to determine on how many different dates they made purchases: SELECT customer_id, MIN(market_date) AS first_purchase, MAX(market_date) AS last_purchase, COUNT(DISTINCT market_date) AS count_of_purchase_dates FROM farmers_market.customer_purchases WHERE customer_id = 1 GROUP BY customer_id Figure 8.10 shows the output of this query. Figure 8.9 Figure 8.10 Chapter 8 ■ Date and Time Functions 121 If we wanted to determine for how long this person has been a customer of the farmer’s market, we can get the difference between the first and last pur- chase. Note that in this query, we’re using a DATEDIFF", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 73 + }, + { + "text": "8.9 Figure 8.10 Chapter 8 ■ Date and Time Functions 121 If we wanted to determine for how long this person has been a customer of the farmer’s market, we can get the difference between the first and last pur- chase. Note that in this query, we’re using a DATEDIFF on the aggregate MIN and MAX dates. Those are still date values, so are therefore valid parameters to pass to the DATEDIFF function. I’ll also remove the customer filter here, so we can see the results for all customers in Figure 8.11: SELECT customer_id, MIN(market_date) AS first_purchase, MAX(market_date) AS last_purchase, COUNT(DISTINCT market_date) AS count_of_purchase_dates, DATEDIFF(MAX(market_date), MIN(market_date)) AS days_between_first_ last_purchase FROM farmers_market.customer_purchases GROUP BY customer_id If we wanted to also know how long it’s been since the customer last made a purchase, we can use the CURDATE() function (which may be called CURRENT_DATE, TODAY(), SYSDATE, or GETDATE() in your particular database system’s SQL syntax; check the documentation). The following query demonstrates its usage. CUR- DATE() can be used to represent the current system date in any calculation that requires a date or datetime parameter. Keep in mind that the server’s current time might differ from your local time, depending on what time zone it is set to: SELECT customer_id, MIN(market_date) AS first_purchase, MAX(market_date) AS last_purchase, COUNT(DISTINCT market_date) AS count_of_purchase_dates, DATEDIFF(MAX(market_date), MIN(market_date)) AS days_between_first_ last_purchase, DATEDIFF(CURDATE(), MAX(market_date)) AS days_since_last_purchase FROM farmers_market.customer_purchases GROUP BY customer_id Going back to the window functions covered in Chapter 7, “Window Functions Frequently Used by Data Scientists,” we can also write a query that gives us the days between each purchase a customer makes. Let’s go back to customer 1’s detailed purchases (previously shown in Figure 8.9) and use both the RANK Figure 8.11 122 Chapter 8 ■ Date and Time Functions and LAG window functions to retrieve each purchase date, along with the next purchase date, so we can have both values per row to enable us to display both and calculate the time between each: SELECT customer_id, market_date, RANK() OVER (PARTITION BY customer_id ORDER BY market_date) AS purchase_number, LEAD(market_date,1) OVER (PARTITION BY customer_id ORDER BY market_ date) AS next_purchase FROM farmers_market.customer_purchases WHERE customer_id = 1 The results of this query are shown in Figure 8.12. You can see that we didn’t quite accomplish the goal of retrieving each purchase date and the previous purchase date in order to show the time between them, because there are multiple rows with the same date in cases where the customer purchased multiple items on the same date. We can resolve this a few ways. One approach is to remove the duplicates by using the DISTINCT keyword, and then use a WHERE clause filter to remove rows where the two dates (current and next purchase) are the same (because multiple purchases were made on the same date). Another is to remove duplicates in the initial dataset and use a subquery (a query inside a query) to get the date differences. Doing this and moving the window functions to the outer", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 74 + }, + { + "text": "dates (current and next purchase) are the same (because multiple purchases were made on the same date). Another is to remove duplicates in the initial dataset and use a subquery (a query inside a query) to get the date differences. Doing this and moving the window functions to the outer query will also fix the issue of the RANK counting each purchase, when we really want to count each purchase date. This is what that second approach looks like: SELECT x.customer_id, x.market_date, RANK() OVER (PARTITION BY x.customer_id ORDER BY x.market_date) AS purchase_number, LEAD(x.market_date,1) OVER (PARTITION BY x.customer_id ORDER BY x.market_date) AS next_purchase Figure 8.12 Chapter 8 ■ Date and Time Functions 123 FROM ( SELECT DISTINCT customer_id, market_date FROM farmers_market.customer_purchases WHERE customer_id = 1 ) x and we can now add a line to the query to use that next_purchase date in a DATEDIFF calculation: SELECT x.customer_id, x.market_date, RANK() OVER (PARTITION BY x.customer_id ORDER BY x.market_date) AS purchase_number, LEAD(x.market_date,1) OVER (PARTITION BY x.customer_id ORDER BY x.market_date) AS next_purchase, DATEDIFF( LEAD(x.market_date,1) OVER (PARTITION BY x.customer_id ORDER BY x.market_date), x.market_date ) AS days_between_purchases FROM ( SELECT DISTINCT customer_id, market_date FROM farmers_market.customer_purchases WHERE customer_id = 1 ) x This may look confusing, but we used the same exact LEAD function inside the DATEDIFF as we used in the next_purchase field above it, and the second DATEDIFF parameter is just market_date, so we are calculating the days between the current row’s market_date and next_purchase columns. We can’t just insert the next_purchase column name into the query there; we have to calculate it for the days_between_purchases field as well, because the calculations don’t happen sequentially and are at the same level (the outer query). The results of the preceding query are shown in Figure 8.13. You might notice that the final days_between_purchases value is NULL. That’s because that row’s next_purchase date is NULL, since there are no more purchases for customer 1 after March 20, 2019. Figure 8.13 124 Chapter 8 ■ Date and Time Functions If we wanted to use the next_purchase field name inside the DATEDIFF() function to avoid inserting that LEAD() calculation twice, we could use another query layer and have a query of a query of a query, as shown in the following code. Here, we’ll remove the customer_id filter to return all customers, then filter to each customer’s first purchase by adding a filter on the calculated pur- chase_number. This query answers the question “How many days pass between each customer’s first and second purchase?” The results of this query are shown in Figure 8.14. SELECT a.customer_id, a.market_date AS first_purchase, a.next_purchase AS second_purchase, DATEDIFF(a.next_purchase, a.market_date) AS time_between_1st_2nd_ purchase FROM ( SELECT x.customer_id, x.market_date, RANK() OVER (PARTITION BY x.customer_id ORDER BY x.market_date) AS purchase_number, LEAD(x.market_date,1) OVER (PARTITION BY x.customer_id ORDER BY x.market_date) AS next_purchase FROM ( SELECT DISTINCT customer_id, market_date FROM farmers_market.customer_purchases ) x ) a WHERE a.purchase_number = 1 In Chapter 10, “Building Analytical Reports with SQL,” we will cover a con- cept called Common Table Expression, also known as", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 75 + }, + { + "text": "x.market_date) AS purchase_number, LEAD(x.market_date,1) OVER (PARTITION BY x.customer_id ORDER BY x.market_date) AS next_purchase FROM ( SELECT DISTINCT customer_id, market_date FROM farmers_market.customer_purchases ) x ) a WHERE a.purchase_number = 1 In Chapter 10, “Building Analytical Reports with SQL,” we will cover a con- cept called Common Table Expression, also known as a CTE or “WITH clause,” which offers another way to select from precalculated values instead of nesting Figure 8.14 Chapter 8 ■ Date and Time Functions 125 multiple queries inside one another as we did earlier, which you can imagine will get increasingly complex and difficult to read as we attempt to answer more complex questions. To get back to simpler aggregate functions that use dates, we will again return to customer 1’s purchase history (originally shown in Figure 8.9). Let’s say that today’s date is March 31, 2019, and the marketing director of the farmer’s market wants to give infrequent customers an incentive to return to the market in April. The director asks you for a list of everyone who only made a purchase at one market event during the previous month, because they want to send an email to all of those customers with a coupon to receive a discount on a purchase made in April. How would you pull up that list? Well, first we have to find everyone who made a purchase in the 31 days prior to March 31, 2019. Then, we need to filter that list to those who only made a purchase on a single market date during that time. This query would retrieve a list of one row per market date per customer within that date range: SELECT DISTINCT customer_id, market_date FROM farmers_market.customer_purchases WHERE DATEDIFF('2019-­03-­31', market_date) <= 31 Then, we could query the results of that query, count the distinct market_date values per customer during that time, and filter to those with exactly one market date, using the HAVING clause (which remember is like the WHERE clause, but calculated after the GROUP BY aggregation): SELECT x.customer_id, COUNT(DISTINCT x.market_date) AS market_count FROM ( SELECT DISTINCT customer_id, market_date FROM farmers_market.customer_purchases WHERE DATEDIFF('2019-­03-­31', market_date) <= 31 ) x GROUP BY x.customer_id HAVING COUNT(DISTINCT market_date) = 1 The results of this query are shown in Figure 8.15 If we were actually fulfilling a report request, we would want to next join these results to the customer table to get the customer name and contact information, but here we have shown how to use date calculations to filter a list of customers by the actions they took. Figure 8.15 126 Chapter 8 ■ Date and Time Functions Exercises 1. Get the customer_id, month, and year (in separate columns) of every purchase in the farmers_market.customer_purchases table. 2. Write a query that filters to purchases made in the past two weeks, returns the earliest market_date in that range as a field called sales_since_date, and a sum of the sales (quantity * cost_to_customer_per_qty) during that date range. Your final answer should use the CURDATE() function, but if you want to", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 76 + }, + { + "text": "a query that filters to purchases made in the past two weeks, returns the earliest market_date in that range as a field called sales_since_date, and a sum of the sales (quantity * cost_to_customer_per_qty) during that date range. Your final answer should use the CURDATE() function, but if you want to test it out on the Farmer’s Market database, you can replace your CUR- DATE() with the value ‘2019-­03-­31’ to get the report for the two weeks prior to March 31, 2019 (otherwise your query will not return any data, because none of the dates in the database will have occurred within two weeks of you writing the query). 3. In MySQL, there is a DAYNAME() function that returns the full name of the day of the week on which a date occurs. Query the Farmer’s Market database market_date_info table, return the market_date, the market_day, and your calculated day of the week name that each market_date occurred on. Create a calculated column using a CASE statement that indicates whether the recorded day in the database differs from your calculated day of the week. This is an example of a quality control query that could be used to check manually entered data for correctness. C H A P T E R 127 9 Exploratory Data Analysis (EDA) is often discussed in a data science context as a first step in the predictive modeling process, when a data scientist explores what the data in a provided dataset looks like prior to using it to build a predic- tive model. The SQL we’ll be using in this chapter could be used at that point in the process, to explore an already-­prepared dataset. But what if you don’t have a dataset to work with yet? Here we’ll show examples that could occur even earlier in the data pipeline, as we explore raw data straight from the database tables (as opposed to an already-­aggregated dataset in which the raw data has been combined and transformed using SQL that is ready to be ingested into a model). If you are given access to a database for the first time, these are the types of queries you can run to familiarize yourself with the tables and data in it. There are of course many ways to conduct EDA, including in a Jupyter note- book with Python code, in a Tableau workbook, or using SQL. (I regularly do all three in my job as a data scientist.) In the later EDA, once a dataset has been prepared, the focus is often on distributions of values, relationships between columns, and identifying correlations between input features and the target variable (column with values to be predicted by the model). Here, we will use the types of queries we’ve covered so far in this book to explore some tables in the Farmer’s Market database, as a demonstration of a real EDA focusing on familiarizing ourselves with the data in the database for the first time. Exploratory Data Analysis with SQL 128 Chapter", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 77 + }, + { + "text": "will use the types of queries we’ve covered so far in this book to explore some tables in the Farmer’s Market database, as a demonstration of a real EDA focusing on familiarizing ourselves with the data in the database for the first time. Exploratory Data Analysis with SQL 128 Chapter 9 ■ Exploratory Data Analysis with SQL Demonstrating Exploratory Data Analysis with SQL Let’s start with a real-­world scenario for this example Exploratory Data Analysis: Let’s say the Director of the Farmer’s Market asks us to help them build some reports to use throughout the year, and gives us access to the database referenced in this book. They haven’t yet given us any specific report requirements, but they have told us that they’ll be asking questions related to general product availability and purchase trends, and have given us the E-­R diagram found in Chapter 1, “Data Sources,” so we know the relationships between the tables. Based on the little information we have, we might guess that we should famil- iarize ourselves with the product, vendor_inventory, and customer_­purchases tables, because we’ve been told we’ll be building reports on “product avail- ability” and “purchase trends.” Some sensible questions to ask via query are: ■ ■How large are the tables, and how far back in time does the data go? ■ ■What kind of information is available about each product and each purchase? ■ ■What is the granularity of each of these tables; what makes a row unique? ■ ■Since we’ll be looking at trends over time, what kind of date and time dimensions are available, and how do the different values look when summarized over time? ■ ■How is the data in each table related to the other tables? How might we join them together to summarize the details for reporting? Exploring the Products Table Some databases (like MySQL) offer a function called DESCRIBE [table name] or DESC [table name], or have a special schema to select from to list the columns, data types, and other settings for fields in tables, but this function isn’t available in every database system and doesn’t show a preview of the data, so we’ll take a more universal approach here to preview data in a table. Let’s start with the product table first. We’ll select everything in the table, to see what kind of data is in each column, but limit it to 10 rows in case it is a large table: SELECT * FROM farmers_market.product LIMIT 10 The output from this query is shown in Figure 9.1. What do we notice in this output? We can see that there is a product_id, product_name, product_size, Chapter 9 ■ Exploratory Data Analysis with SQL 129 product_category_id, and product_qty_type on each row, and at least in this small subset, it appears that most of the fields are populated (there aren’t a lot of NULL values). This table looks like a catalog of products, with product metadata like name and category. It doesn’t list individual items for sale by", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 78 + }, + { + "text": "product_qty_type on each row, and at least in this small subset, it appears that most of the fields are populated (there aren’t a lot of NULL values). This table looks like a catalog of products, with product metadata like name and category. It doesn’t list individual items for sale by vendors or purchased by customers, like a transactional table would. The product_cat- egory_id is an integer, and we know there is a product_category table, so we might assume that is a foreign key and check out that relationship later in the EDA. There are many product_name and product_size values, but in this preview, only two product_qty_type values, “lbs” and “unit.” The product_id appears to be the primary key. What if we didn’t know whether it was a unique identifier? To check to see if any two rows have the same product_id, we can write a query that groups by the product_id and returns any groups with more than one record. This isn’t a guarantee that it is the primary key, but can tell you whether, at least currently, the product_id is unique per record: SELECT product_id, count(*) FROM farmers_market.product GROUP BY product_id HAVING count(*) > 1 There are no results returned, so no product_id groups have more than one row, meaning each product_id is unique, and we can say that this table has a granularity of one row per product What about the product categories we see IDs for in the product table? How many different categories are there, and what do those look like? Let’s see what’s in the product_category table: SELECT * FROM farmers_market.product_category The results of this query are shown in Figure 9.2, and the listing of categories gives a sense of how the Farmer’s Market groups its different types of products. This might be useful when we need to write reports, because we could report on inventory and trends by product category. Figure 9.1 130 Chapter 9 ■ Exploratory Data Analysis with SQL How many different products are there in the catalog-­like product metadata table? SELECT count(*) FROM farmers_market.product As you can see in Figure 9.3, only 23 products have been entered into this database so far. If this were a report, we would want to give this column a header besides count(*), but during an Exploratory Data Analysis, you are often moving quickly and writing queries for your own information and not for display purposes, so there is not a need to alias the columns in every query. We might next ask, “How many products are there per product category?” We’ll quickly join the product table and the product_category table to pull in the category names that we think go with the IDs here, and count up the prod- ucts in each category: SELECT pc.product_category_id, pc.product_category_name, count(product_id) AS count_of_products FROM farmers_market.product_category AS pc LEFT JOIN farmers_market.product AS p ON pc.product_category_id = p.product_category_id GROUP BY pc.product_category_id In Figure 9.4, you can see that the IDs did all match up with categories, and the most common product", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 79 + }, + { + "text": "count up the prod- ucts in each category: SELECT pc.product_category_id, pc.product_category_name, count(product_id) AS count_of_products FROM farmers_market.product_category AS pc LEFT JOIN farmers_market.product AS p ON pc.product_category_id = p.product_category_id GROUP BY pc.product_category_id In Figure 9.4, you can see that the IDs did all match up with categories, and the most common product category is “Fresh Fruits & Vegetables.” The “Freshly Prepared Food” category does not yet have any products in it. Figure 9.2 Figure 9.3 Chapter 9 ■ Exploratory Data Analysis with SQL 131 Exploring Possible Column Values To further familiarize ourselves with the range of values in a column that appears to contain string values that represent categories, we might ask, “What is in the product_qty_type field we saw in our first preview of the product table? And how many different quantity types are there?” which could be answered with this query using the DISTINCT keyword: SELECT DISTINCT product_qty_type FROM farmers_market.product As you can see in Figure 9.5, the two quantity types we saw in the 10-­row preview turned out to be the only two values in the column, though some prod- ucts have a NULL product_qty_type, so we’ll have to remember that when doing any sort of filtering or calculation based on this column. Let’s take a look at some of the data in the vendor_inventory table next: SELECT * FROM farmers_market.vendor_inventory LIMIT 10 In the output shown in Figure 9.6, we can see that it looks like there is one row per market_date, vendor_id, and product_id, with the quantity of that product that the vendor brought to each market, and the original_price. We might need to ask the Director what the “original price” of an item is, but with that name, we might guess that it is the item price before any sales or special deals. And it is tracked per market date, so changes over time would be recorded. We can also see that the quantity is a decimal value, at least for the product displayed. Figure 9.4 Figure 9.5 132 Chapter 9 ■ Exploratory Data Analysis with SQL We should confirm our assumption about the primary key. If your database system offers the DESCRIBE function or a schema to query from to see table properties, that could give definite confirmation that a field is set to be a unique primary key. (See your database’s documentation.) Otherwise, we can group by the fields we expect are unique and use HAVING to check whether there is more than one record with each combination, like we did previously with the product_id in the product table: SELECT market_date, vendor_id, product_id, count(*) FROM farmers_market.vendor_inventory GROUP BY market_date, vendor_id, product_id HAVING count(*) > 1 There are no combinations of these three values that occur on more than one row, so at least with the currently entered data, this combination of fields is indeed unique, and the vendor_inventory table has a granularity of one record per market date, vendor, and product. It is not visible in the version of the E-­R diagram displayed in", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 80 + }, + { + "text": "occur on more than one row, so at least with the currently entered data, this combination of fields is indeed unique, and the vendor_inventory table has a granularity of one record per market date, vendor, and product. It is not visible in the version of the E-­R diagram displayed in Chapter 1, but it’s possible to highlight the primary key of a table in MySQL Workbench, as shown in Figure 9.7. Here, we confirm that it is a composite primary key, made up of the three fields we guessed. Figure 9.6 Figure 9.7 Chapter 9 ■ Exploratory Data Analysis with SQL 133 We also saw in Figure 9.6 that there are dates in this table, in the market_date field. So we might ask, “How far back does the data go: When was the first market that was tracked in this database, and how recent is the latest data?” To answer this, we can get the minimum (earliest) and maximum (latest) values from that field: SELECT min(market_date), max(market_date) FROM farmers_market.vendor_inventory As you can see in Figure 9.8, it looks like we have about one and a half years’ worth of records. So, if we’re asked to build any kind of forecast involving an annual seasonality, we will have to explain that we have limited training data for that, since we don’t have multiple complete years of seasonal trends yet. It would be good to check to see if purchases were tracked during that entire period, as well. Another question we might ask to better understand the Farmer’s Market and the data it generates is: How many different vendors are there, and when did they each start selling at the market? And which are still selling at the most recent market_date? We can do that by grouping the previous query by vendor_id to get the earliest and latest dates for which each vendor had inventory. We will also sort by these dates, to see which vendors were selling the longest ago, and the most recently: SELECT vendor_id, min(market_date), max(market_date) FROM farmers_market.vendor_inventory GROUP BY vendor_id ORDER BY min(market_date), max(market_date) The output in Figure 9.9 shows that the inventories of only three vendors have been added to the database so far. Vendors 7 and 8 have been selling since April 3, 2019, and most recently brought inventory to the October 10, 2020 market. Vendor 4 started later and was last at the market on September 30, 2020. Figure 9.8 Figure 9.9 134 Chapter 9 ■ Exploratory Data Analysis with SQL Exploring Changes Over Time One thought that comes to mind looking at the dates during this exploration is that we should check to find out if there is an indicator in the database showing whether the market changed format, perhaps allowing online orders during the COVID-­19 pandemic in 2020, since that might have impacted sales, and that indicator could be valuable for predictive modeling. Another question we might ask the database after seeing the output in Figure 9.9 is: Do most vendors sell at", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 81 + }, + { + "text": "whether the market changed format, perhaps allowing online orders during the COVID-­19 pandemic in 2020, since that might have impacted sales, and that indicator could be valuable for predictive modeling. Another question we might ask the database after seeing the output in Figure 9.9 is: Do most vendors sell at the market year-­round, or is there a certain time of year when there are different numbers of vendors at the farmer’s market? We’ll extract the month and year from each market date and look at the counts of vendors that appear in the data each month: SELECT EXTRACT(YEAR FROM market_date) AS market_year, EXTRACT(MONTH FROM market_date) AS market_month, COUNT(DISTINCT vendor_id) AS vendors_with_inventory FROM farmers_market5.vendor_inventory GROUP BY EXTRACT(YEAR FROM market_date), EXTRACT(MONTH FROM market_date) ORDER BY EXTRACT(YEAR FROM market_date), EXTRACT(MONTH FROM market_date) Since only three vendors have inventory entered into this example database, there isn’t much variation seen in this output, but you can see in Figure 9.10 that there are three vendors in June through September, and two vendors per month the rest of the year. So one of the vendors (likely vendor 4, from the date ranges we saw in Figure 9.9) may be a seasonal vendor. Figure 9.10 Chapter 9 ■ Exploratory Data Analysis with SQL 135 Perhaps surprisingly, we can also see in Figure 9.10 that there are no months 1 and 2 listed in this output, only months 3–12, so the farmer’s market must be closed in January and February. (We’ll have to remember to check with the Director to see whether that is true, because if not, there might be an issue with the data.) These are the aspects of the data that we want to discover during EDA instead of later when we’re doing analysis, so we know what to expect in the report output. Next, let’s look at the details of what a particular vendor’s inventory looks like. We saw a vendor with an ID of 7 in a previous query, so we’ll explore their inventory first: SELECT * FROM farmers_market.vendor_inventory WHERE vendor_id = 7 ORDER BY market_date, product_id Part of the output of this query is shown in Figure 9.11. In the limited amount of data visible in the screenshot, we can see that this vendor was only selling one product through most of May and June, then there are some other prod- ucts (with IDs 1–3) that appear in July. We can also see that product 4 has not changed price at all and is $4.00 throughout the period visible in Figure 9.11. Exploring Multiple Tables Simultaneously Some of the products have round quantities, and some appear to be continuous numbers, possibly products sold by weight. This vendor always brings either 30 or 40 of product 4 to each market. It would be interesting to see how many of those items are sold at each market, in comparison. Figure 9.11 136 Chapter 9 ■ Exploratory Data Analysis with SQL Let’s jump over to the customer_purchases table to get a sense of what the", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 82 + }, + { + "text": "40 of product 4 to each market. It would be interesting to see how many of those items are sold at each market, in comparison. Figure 9.11 136 Chapter 9 ■ Exploratory Data Analysis with SQL Let’s jump over to the customer_purchases table to get a sense of what the purchases of that product look like, compared to the vendor’s inventory. First, we need to see what data is available in that table, so we’ll select all fields and 10 rows to get a preview: SELECT * FROM farmers_market.customer_purchases LIMIT 10 As shown in Figure 9.12, each row has a product_id, vendor_id, market_date, customer_id, quantity, cost_to_customer_per_qty, and transaction_time. We can see that each product purchased by a customer is in its own row, but multiple units of the same product can be recorded in one row, as indicated by the quantity field. We might also guess from the cost_to_customer_per_qty field in this output that the reason the original_price is recorded in the ­vendor_inventory table is because different customers might pay different prices for the same items. We also notice that the quantity and cost_to_­customer_per_qty values are numeric values with two digits after the decimal point. It’s also interesting that each individual purchase’s transaction_time is recorded, in addition to the date. This could allow us to build reports on the flow of customers to the market throughout the day, or to estimate how long each customer spends at the market, by looking for their earliest and latest purchase times. Since we see vendor_id and product_id are both included here, we can look closer at purchases of vendor 7’s product #4: SELECT * FROM farmers_market.customer_purchases WHERE vendor_id = 7 AND product_id = 4 ORDER BY market_date, transaction_time Glancing at this data in Figure 9.13, we can see that for the two market dates that are visible, most customers are buying between 1 and 5 of these items at a time and spending $4 per item. Figure 9.12 Chapter 9 ■ Exploratory Data Analysis with SQL 137 I see customer_id 12 in Figure 9.13 several times, and out of curiosity, we might run the same query but filtered to, or sorted by, the customer_id to explore one customer’s purchase history of a product in more detail: SELECT * FROM farmers_market.customer_purchases WHERE vendor_id = 7 AND product_id = 4 AND customer_id = 12 ORDER BY customer_id, market_date, transaction_time This is simulated (generated) data, and I know this result shown in Figure 9.14 occurred because there are a limited number of customer IDs in the database to associate with purchases so far, but it looks like customer 12 really likes this product. They first purchased it on April 3, then came back and bought five more three days later on April 6, only to come back 30 minutes later on the same day to buy five more! Sorting and filtering the data different ways—­by market date and time, by vendor_id, by customer_id—­you can get a sense of the processes that the data represents and see", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 83 + }, + { + "text": "more three days later on April 6, only to come back 30 minutes later on the same day to buy five more! Sorting and filtering the data different ways—­by market date and time, by vendor_id, by customer_id—­you can get a sense of the processes that the data represents and see if anything stands out that might be worth following up on with the subject matter experts. We can see by looking at this detailed view of the customer_purchases data that there are multiple sales recorded per vendor per product per day, but we can’t get a good sense of what that looks like in summary. And since we wanted to compare the sales per day to the inventory that the vendor brought to each market, we’ll want to aggregate these sales by market date. So, we can group by Figure 9.13 Figure 9.14 138 Chapter 9 ■ Exploratory Data Analysis with SQL market_date, vendor_id, and product_id, and add up the quantities sold and each row’s quantity (?): multiplied by the cost to get the total sales for each group: SELECT market_date, vendor_id, product_id, SUM(quantity) quantity_sold, SUM(quantity * cost_to_customer_per_qty) total_sales FROM farmers_market.customer_purchases WHERE vendor_id = 7 and product_id = 4 GROUP BY market_date, vendor_id, product_id ORDER BY market_date, vendor_id, product_id You can see the results of this in Figure 9.15. Again, if we were building a report, we would probably want to round those dollars in the last column to two numbers after the decimal, but since this is an exploration for our own information, I won’t spend the extra time on formatting. Exploring Inventory vs. Sales Now we have all of the information we need to answer our question about the sales of this product compared to the inventory except the inventory counts! Throughout the EDA so far, we have gotten a sense of what the data in each of these related tables looks like, and now we can start joining them together to get a better sense of the relationship between entities. For example, now that we have aggregated the customer_purchases to the same granularity of the vendor_inventory table—­one row per market_date, vendor_id, and product_id—­ we can join the two tables together to view inventory side by side with sales. Figure 9.15 Chapter 9 ■ Exploratory Data Analysis with SQL 139 First, we’ll join the two tables (the details of one and the summary of the other) and display all columns to check to make sure it’s combining the way we expect it to. We’ll give the customer_purchases summary table the alias “sales,” and limit the output to 10 rows since we haven’t filtered to a vendor or product yet, so this query will return a lot of rows: SELECT * FROM farmers_market.vendor_inventory AS vi LEFT JOIN ( SELECT market_date, vendor_id, product_id, SUM(quantity) AS quantity_sold, SUM(quantity * cost_to_customer_per_qty) AS total_sales FROM farmers_market.customer_purchases GROUP BY market_date, vendor_id, product_id ) AS sales ON vi.market_date = sales.market_date AND vi.vendor_id = sales.vendor_id AND vi.product_id = sales.product_id ORDER BY vi.market_date, vi.vendor_id, vi.product_id LIMIT 10", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 84 + }, + { + "text": "SELECT * FROM farmers_market.vendor_inventory AS vi LEFT JOIN ( SELECT market_date, vendor_id, product_id, SUM(quantity) AS quantity_sold, SUM(quantity * cost_to_customer_per_qty) AS total_sales FROM farmers_market.customer_purchases GROUP BY market_date, vendor_id, product_id ) AS sales ON vi.market_date = sales.market_date AND vi.vendor_id = sales.vendor_id AND vi.product_id = sales.product_id ORDER BY vi.market_date, vi.vendor_id, vi.product_id LIMIT 10 We can see in Figure 9.16 that the vendor_id, product_id, and market_date do match on every row, and the summary values for vendor_id 8 and product_id 4 match what we saw when looking at the customer_purchases table alone. This confirms that our join looks right, and we can remove these redundant columns by specifying which columns we want to display from each table. We’ll make sure to pull these columns in from the vendor_inventory table, which we made the “left” side of the JOIN relationship because a customer can’t buy inventory that doesn’t exist (Theoretically! We should check the data to make sure that never happens, which might indicate a vendor made a mistake in entering their available inventory.) But there could be inventory that doesn’t get purchased, which we would want to see in the output, but would be missing if we chose a RIGHT JOIN instead. Figure 9.16 140 Chapter 9 ■ Exploratory Data Analysis with SQL We can also join in additional “lookup” tables to convert the various IDs to human-­readable values, pulling in the vendor name and product names. Then, we can filter to vendor 7 and product 4 to get the information we were looking for earlier, comparing this vendor’s inventory of this product to the sales made at each market: SELECT vi.market_date, vi.vendor_id, v.vendor_name, vi.product_id, p.product_name, vi.quantity AS quantity_available, sales.quantity_sold, vi.original_price, sales.total_sales FROM farmers_market.vendor_inventory AS vi LEFT JOIN ( SELECT market_date, vendor_id, product_id, SUM(quantity) AS quantity_sold, SUM(quantity * cost_to_customer_per_qty) AS total_sales FROM farmers_market.customer_purchases GROUP BY market_date, vendor_id, product_id ) AS sales ON vi.market_date = sales.market_date AND vi.vendor_id = sales.vendor_id AND vi.product_id = sales.product_id LEFT JOIN farmers_market.vendor v ON vi.vendor_id = v.vendor_id LEFT JOIN farmers_market.product p ON vi.product_id = p.product_id WHERE vi.vendor_id = 7 AND vi.product_id = 4 ORDER BY vi.market_date, vi.vendor_id, vi.product_id Now we can see in a sample of this query’s output in Figure 9.17 that this vendor is called Marco’s Peppers, and the product we were looking at is jars of Banana Peppers. He brings 30–40 jars each time and sells between 1 and 40 jars per market (which we quickly determined by sorting the output in the SQL editor ascending and descending by the quantity_sold, but we could’ve also done by adding quantity_sold to the ORDER BY clause of the query and scrolling to the top and bottom of the output or surrounding this query with another query that calculated the MIN and MAX quantity_sold). Chapter 9 ■ Exploratory Data Analysis with SQL 141 To take a closer look at the distribution of sales and continue our EDA visu- ally, or to easily filter to different combinations of vendors and products, we can remove the WHERE clause filter and pull the", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 85 + }, + { + "text": "MIN and MAX quantity_sold). Chapter 9 ■ Exploratory Data Analysis with SQL 141 To take a closer look at the distribution of sales and continue our EDA visu- ally, or to easily filter to different combinations of vendors and products, we can remove the WHERE clause filter and pull the dataset generated by this query into reporting software such as Tableau, building dashboards and reports. Figure 9.18 shows a dashboard with the inventory and sales of the selected products during the selected date range. Figure 9.19 shows a histogram with how many different market dates each count of jars of Banana Peppers were sold. We could also do a lot more calculations in SQL using this dataset, such as calculating the percent of each vendor’s inventory sold at each market, but the purpose of this demonstration was to show how much you can learn about the data in a database using the types of SQL queries you have learned in this book so far. In the next chapter, we’ll explore more analytical queries for building reports using SQL that go beyond simple data previews and basic summaries. Figure 9.17 Figure 9.18 142 Chapter 9 ■ Exploratory Data Analysis with SQL Exercises 1. In the chapter, it was suggested that we should see if the customer_­purchases data was collected for the same time frame as the vendor_inventory table. Write a query that gets the earliest and latest dates in the customer_­purchases table. 2. There is a MySQL function DAYNAME() that returns the name of the day of the week for a date. Using the DAYNAME and EXTRACT functions on the customer_purchases table, select and group by the weekday and hour of the day, and count the distinct number of customers during each hour of the Wednesday and Saturday markets. See Chapters 6, “Aggregating Results for Analysis,” and 8, “Date and Time Functions,” for information on the COUNT DISTINCT and EXTRACT functions. 3. What other questions haven’t we yet asked about the data in these tables that you would be curious about? Write two more queries further explor- ing or summarizing the data in the product, vendor_inventory, or customer_purchases tables. Figure 9.19 C H A P T E R 143 10 In previous chapters, we covered basic SQL SELECT syntax and started to use SQL to construct datasets to answer specific questions. In the data analysis world, being asked questions, exploring a database, writing SQL statements to find and pull the data needed to determine the answers, and conducting the analysis of that data to calculate the answers to the questions, is called ad-­hoc reporting. I often say in my data science conference presentations that the process depicted in Figure 10.1 is what is expected of any data analyst or data scientist: to be able to listen to a question from a business stakeholder, determine how it might be answered using data from the database, retrieve the data needed to answer it, calculate the answers, and present that result in a form", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 86 + }, + { + "text": "what is expected of any data analyst or data scientist: to be able to listen to a question from a business stakeholder, determine how it might be answered using data from the database, retrieve the data needed to answer it, calculate the answers, and present that result in a form that the business stake- holder can understand and use to make decisions. You now know enough SQL to answer some basic ad-­hoc questions about what is occurring at the fictional farmer’s market using the demonstration database by filtering, joining, and summarizing the data. In the remaining chapters, we’ll take those skills to the next level and dem- onstrate how to think through multiple analysis questions, simulating what it might be like to write queries to answer a question posed by a business stake- holder. We’ll design and develop analytical datasets that can be used repeat- edly to facilitate ad-­hoc reporting, build dashboards, and serve as inputs into predictive models. Building SQL Datasets for Analytical Reporting 144 Chapter 10 ■ Building SQL Datasets for Analytical Reporting Thinking Through Analytical Dataset Requirements In this chapter, we’ll walk through some examples of designing reusable datasets that can be queried to build many report variations. An experienced analyst who goes through the steps in Figure 10.1 won’t only think about writing a query to answer the immediate question at hand but will think more generally about “Building Datasets for Analysis” (we’re finally getting to this book’s subtitle!) and designing SQL queries that combine and summarize data in a way that can then be used to answer many similar questions that might arise as offshoots of the original question. You can think of a dataset as being like a table, already summarized at the desired level of detail, with multiple measures and dimensions that you can break down those metrics by. An analytical dataset is designed for use in reports and predictive models, and usually combines data from several tables summa- rized at a granularity (row level of detail) that lends itself to multiple analyses. If the dataset is meant to be used in a visual report or dashboard, it’s a good idea to join in all fields that could be used for human-­readable report labels (like vendor names instead of just IDs), measures (the numeric values you’ll want to summarize and report on, such as sales), and dimensions (for grouping and slicing and dicing the measures, like product category). Because I know that the first question I’m asked to answer with an ad-­hoc query is almost never the only question, I will use any remaining project time to try to anticipate follow-­up questions and design a dataset that may be useful for answering them. Adding additional relevant columns or calculations to a query also makes the resulting dataset reusable for future reporting purposes. Anticipating potential follow-­up questions is a skill that analysts develop over time, through experience. For example, if the manager of the farmer’s market asked me “What were the total sales at", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 87 + }, + { + "text": "Adding additional relevant columns or calculations to a query also makes the resulting dataset reusable for future reporting purposes. Anticipating potential follow-­up questions is a skill that analysts develop over time, through experience. For example, if the manager of the farmer’s market asked me “What were the total sales at the market last week?” I would expect to be asked more questions after delivering the answer to that one, such as, Business Question Data Question Data Answer Business Answer Figure 10.1 Chapter 10 ■ Building SQL Datasets for Analytical Reporting 145 “How many sales were at the Wednesday market versus the Saturday market last week?” or “Can you calculate the total sales over another time period?” or “Let’s track these weekly market sales over time,” or “Now that we have the total sales for last week, can we break this down by vendor?” Given time, I could build a single dataset that could be imported into a reporting system like Tableau and used to answer all of these questions. Since we’re talking about summary sales and time, I would first think about all of the different time periods by which someone might want to “slice and dice” market sales. Someone could ask to summarize sales by minute, hour, day, week, month, year, and so on. Then I would think about dimensions other than time that people might want to filter or summarize sales by, such as vendor or customer zip code. Whatever granularity I choose to make the dataset (at what level of detail I choose to summarize it) dictates the lowest level of detail at which I can then filter or summarize a report based on that dataset. So, for example, if I build a dataset that summarizes the data by week, I will not be able to produce a daily sales report from that dataset, because weeks are less granular than days. Conversely, the granularity I choose means I will always need to write sum- mary queries for any question that’s at a higher level of aggregation than the dataset. For example, if I create a dataset that summarizes sales per minute, I will always have to use GROUP BY in the query, or use a reporting tool to sum- marize sets of rows, to answer any question that needs those minutes combined into a longer time period like hours or days. If you are developing a dataset for use in a reporting tool that makes aggregation simple, such as Tableau, you may want to keep it as granular as possible, so you can drill down to as small of a time period as is available in the data. In Tableau, measures are automatically summarized by default as you build a report, and you have to instead break down the summary measures by add- ing dimensions. Summarizing by date in Tableau is as simple as dragging any datetime value into a report and choosing at what timescale you want to view that date field, regardless of whether the", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 88 + }, + { + "text": "you build a report, and you have to instead break down the summary measures by add- ing dimensions. Summarizing by date in Tableau is as simple as dragging any datetime value into a report and choosing at what timescale you want to view that date field, regardless of whether the underlying dataset has one row per day or one row per second. However, if you are primarily going to be querying the dataset with SQL, you will have to use GROUP BY and structure your queries to summarize at the desired level every time you use it to build a report. So, if you anticipate that you will frequently be summarizing by day, it would make sense to build a dataset that is already summarized to one row per day. You can always go back to the source data and write a custom query for that rare case when you’re expected to report on sales per hour, but reuse the pre-­summarized daily sales dataset as a shortcut for the more common report requests in this case. Let’s say that for this example, I can safely assume that most reports will be summarized at the daily or weekly level, so I will choose to build a dataset that has one row per day. In each row, I can summarize not only the total daily 146 Chapter 10 ■ Building SQL Datasets for Analytical Reporting sales but also include other summary metrics and attributes that sales might be reported by. Once I think it through, I realize that the reports I’m asked to provide are often broken down by vendor, so I will revise my design to be one row per day per vendor. This will allow us to filter any report built on this dataset to any date range and to a specific vendor, or set of vendors. In the Farmer’s Market database, the sales are tracked in the customer_ purchases table, which has one row per item purchased (with multiples of the same item potentially included in a single row, since there is a quantity column). Each row in customer_purchases contains the ID of the product purchased, the ID of the vendor the product was purchased from, the ID of the customer making the purchase, the transaction date and time, and the quantity and cost per quantity. Because I am designing my dataset to have one row per date and vendor, I do not need to include detailed information about the customers or products. And because my most granular time dimension is market_date, I do not need to consider the field that tracks the time of purchase. I will start by writing a SELECT statement that pulls only the fields I need, leav- ing out unnecessary information, and allowing me to summarize at the selected level of detail. I don’t need to include the quantity of an item purchased in this dataset, only the final sale amount, so I’ll multiply the quantity and the cost_ to_customer_per_qty fields, like we did", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 89 + }, + { + "text": "I need, leav- ing out unnecessary information, and allowing me to summarize at the selected level of detail. I don’t need to include the quantity of an item purchased in this dataset, only the final sale amount, so I’ll multiply the quantity and the cost_ to_customer_per_qty fields, like we did in Chapter 3, “The WHERE Clause”: SELECT market_date, vendor_id, quantity * cost_to_customer_per_qty FROM farmers_market.customer_purchases After reviewing the output without aggregation, to ensure that these are the fields and values I expected to see, I’ll group and sort by vendor_id and market_date, SUM the calculated cost column, round it to two decimal places, and give it an alias of sales. The resulting output is shown in Figure 10.2. Figure 10.2 Chapter 10 ■ Building SQL Datasets for Analytical Reporting 147 SELECT market_date, vendor_id, ROUND(SUM(quantity * cost_to_customer_per_qty),2) AS sales FROM farmers_market.customer_purchases GROUP BY market_date, vendor_id ORDER BY market_date, vendor_id This is a good time to review whether my results will allow me to answer the other questions I assumed I might be asked in the future and to determine what other information might be valuable to add to the dataset for reporting purposes. Let’s go through each one: ■ ■What were the total sales at the market last week? There are multiple ways to accomplish this. A simple option is to filter the results by last week’s date range and sum the sales. If we wanted the report to update dynamically as data is added, always adding up sales “over the last week,” we could have our query subtract 7 days from the current date and filter to only sales in rows with a market_date value that occurred after that date. We have the fields necessary to accomplish this. ■ ■How many of last week’s sales were at the Wednesday market versus the Saturday market? In addition to the approach to answering the previous question, we can use the DAYNAME() function in MySQL to return the name of the day of the week of each market date. It might be a good idea to add the day of the week to the dataset so it’s easily accessible for report labeling and grouping. If we explore the market_date_info table, we’ll find that there is a field called market_day that already includes the weekday label of each market date, so there is actually no need to calculate it; we can join it in. ■ ■Can we calculate the total sales over another time period? Yes, we can filter the results of the existing query to any date range and sum up the sales. ■ ■Can we track the weekly market sales over time? We can use the MySQL functions WEEK() and YEAR() on the market_date field to pull those values into the dataset and group rows by week number (which could be espe- cially useful for year over year comparisons) or by year and week (for a weekly summary time series). However, these values are also already in the market_date_info table, so we can", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 90 + }, + { + "text": "market_date field to pull those values into the dataset and group rows by week number (which could be espe- cially useful for year over year comparisons) or by year and week (for a weekly summary time series). However, these values are also already in the market_date_info table, so we can join those in to allow reporting by that information. 148 Chapter 10 ■ Building SQL Datasets for Analytical Reporting ■ ■Can we break down the weekly sales by vendor? Because we included vendor_id in the output, we can group or filter any of the preceding approaches by vendor. It might be a good idea to join in the vendor name and type in this dataset in case we end up grouping by vendor in a report, so we can label those rows with more than just the numeric vendor ID. While evaluating whether the dataset includes the data that we need to answer the expected questions, we identified some additional information that might be valuable to have on hand for reporting from other tables, including market_day, market_week, and market_year from the market_date_info table (which also contains fields indicating properties of that date such as whether it was raining, which might be a dimension someone wants to summarize by in the future), and vendor_name and vendor_type from the vendor table. Let’s LEFT JOIN those into our dataset, so we keep all of the existing rows, and add in more columns where available. A sample of the dataset generated by this query is shown in Figure 10.3: SELECT cp.market_date, md.market_day, md.market_week, md.market_year, cp.vendor_id, v.vendor_name, v.vendor_type, ROUND(SUM(cp.quantity * cp.cost_to_customer_per_qty),2) AS sales FROM farmers_market.customer_purchases AS cp LEFT JOIN farmers_market.market_date_info AS md ON cp.market_date = md.market_date LEFT JOIN farmers_market.vendor AS v ON cp.vendor_id = v.vendor_id GROUP BY cp.market_date, cp.vendor_id ORDER BY cp.market_date, cp.vendor_id Now we can use this custom dataset to create reports and conduct further analysis. Figure 10.3 Chapter 10 ■ Building SQL Datasets for Analytical Reporting 149 Using Custom Analytical Datasets in SQL: CTEs and Views There are multiple ways to store queries (and the results of queries) for reuse in reports and other analyses. Some techniques, such as creating new database tables, will be covered in later chapters. Here, we will cover two approaches for more easily querying from the results of custom dataset queries you build: Common Table Expressions and views. Most database systems, including MySQL since version 8.0, support Common Table Expressions (CTEs), also known as “WITH clauses.” CTEs allow you to cre- ate an alias for an entire query, which allows you to reference it in other queries like you would any database table. The syntax for CTEs is: WITH [query_alias] AS ( [query] ), [query_2_alias] AS ( [query_2] ) SELECT [column list] FROM [query_alias] ... [remainder of query that references aliases created above] where “[query_alias]” is a placeholder for the name you want to use to refer to a query later, and “[query]” is a placeholder for the query you want to reuse. If you want to alias", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 91 + }, + { + "text": ") SELECT [column list] FROM [query_alias] ... [remainder of query that references aliases created above] where “[query_alias]” is a placeholder for the name you want to use to refer to a query later, and “[query]” is a placeholder for the query you want to reuse. If you want to alias multiple queries in the WITH clause, you put each query inside its own set of parentheses, separated by commas. You only use the WITH keyword once at the top, and enter “[alias_name] AS” before each new query you want to later reference. (The AS is not optional in this case.) Each query in the WITH clause can reference any query that preceded it, by using its alias. Then below the WITH clause, you start your SELECT statement like you nor- mally would, and use the query aliases to refer to the results of each of them. They are run before the rest of your queries that rely on their results, the same way they would be if you put them inside the SELECT statement as subqueries, which were covered in Chapter 7, “Window Functions Frequently Used by Data Scientists.” For example, if we wanted to reuse the previous query we wrote to generate the dataset of sales summarized by date and vendor for a report that summarizes 150 Chapter 10 ■ Building SQL Datasets for Analytical Reporting sales by market week, we could put that query inside a WITH clause, then query from it using another SELECT statement, like so: WITH sales_by_day_vendor AS ( SELECT cp.market_date, md.market_day, md.market_week, md.market_year, cp.vendor_id, v.vendor_name, v.vendor_type, ROUND(SUM(cp.quantity * cp.cost_to_customer_per_qty),2) AS sales FROM farmers_market.customer_purchases AS cp LEFT JOIN farmers_market.market_date_info AS md ON cp.market_date = md.market_date LEFT JOIN farmers_market.vendor AS v ON cp.vendor_id = v.vendor_id GROUP BY cp.market_date, cp.vendor_id ORDER BY cp.market_date, cp.vendor_id ) SELECT s.market_year, s.market_week, SUM(s.sales) AS weekly_sales FROM sales_by_day_vendor AS s GROUP BY s.market_year, s.market_week A subset of results of this query is shown in Figure 10.4. Figure 10.4 Chapter 10 ■ Building SQL Datasets for Analytical Reporting 151 Notice how the SELECT statement at the bottom references the sales_by_ day_vendor Common Table Expression using its alias, treating it just like a table, and even giving it an even shorter alias, s. You can filter it, perform calculations, and do anything with its fields that you would do with a normal database table. By using a WITH statement instead of a subquery, it keeps this query at the bot- tom cleaner and easier to understand. In most SQL editors, you can highlight the query within each set of paren- theses to run just that code inside the WITH statement and view its results, so you know what data is available as you develop your SELECT query below the WITH statement. You can’t highlight only the SELECT statement at the bottom to run it alone, however, because it references sales_by_day_vendor, which is dynamically created by running the WITH statement above it along with it. Another approach allows you to develop SELECT statements that depend", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 92 + }, + { + "text": "your SELECT query below the WITH statement. You can’t highlight only the SELECT statement at the bottom to run it alone, however, because it references sales_by_day_vendor, which is dynamically created by running the WITH statement above it along with it. Another approach allows you to develop SELECT statements that depend on a custom dataset in their own SQL editor window, or inside other code such as a Python script, without first including the entire CTE. This involves storing the query as a database view. A view is treated just like a table in SQL, the only difference being that it has run when it’s referenced to dynamically generate a result set (where a table stores the data instead of storing the query), so queries that reference views can take longer to run than queries that reference tables. However, the view is retrieving the latest data from the underlying tables each time it is run, so you are working with the freshest data available when you query from a view. If you want to store your dataset as a view (assuming you have been granted database permissions to create a view in a schema), you simply precede your SELECT statement with “CREATE VIEW [schema_name].[view_name] AS”, replac- ing the bracketed statements with the actual schema name, and the name you are giving the view. This is one query in this book that you will not be able to test using an online SQL editor and the sample database, because you won’t have permissions to create a new database object there. The vw_ prefix in the following view name serves as an indicator when writing a query that the object you’re referencing is a view (stored query) and not a table (stored data): CREATE VIEW farmers_market.vw_sales_by_day_vendor AS SELECT cp.market_date, md.market_day, md.market_week, md.market_year, cp.vendor_id, v.vendor_name, v.vendor_type, ROUND(SUM(cp.quantity * cp.cost_to_customer_per_qty),2) AS sales Continues 152 Chapter 10 ■ Building SQL Datasets for Analytical Reporting FROM farmers_market.customer_purchases AS cp LEFT JOIN farmers_market.market_date_info AS md ON cp.market_date = md.market_date LEFT JOIN farmers_market.vendor AS v ON cp.vendor_id = v.vendor_id GROUP BY cp.market_date, cp.vendor_id ORDER BY cp.market_date, cp.vendor_id No results will be displayed when you run this query, other than a confirma- tion message indicating that the view was created. But if you were to select the data from this view, it would be identical to the data in Figure 10.3. Since this dataset we created has one row per market date per vendor, we can filter this view by vendor_id and a range of market_date values, just like we could if it were a table. Our stored view query is then run to retrieve and summarize data from the underlying tables, then those results are modified by the SELECT statement, as shown in Figure 10.5. SELECT * FROM farmers_market.vw_sales_by_day_vendor AS s WHERE s.market_date BETWEEN '2020-­04-­01' AND '2020-­04-­30' AND s.vendor_id = 7 ORDER BY market_date Because the results of CTEs and views are not stored, they pull the data dynamically each time they are referenced. So, if you use the", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 93 + }, + { + "text": "statement, as shown in Figure 10.5. SELECT * FROM farmers_market.vw_sales_by_day_vendor AS s WHERE s.market_date BETWEEN '2020-­04-­01' AND '2020-­04-­30' AND s.vendor_id = 7 ORDER BY market_date Because the results of CTEs and views are not stored, they pull the data dynamically each time they are referenced. So, if you use the preceding SQL to report on weekly sales using the vw_sales_by_day_vendor view, each time you run the query, it will include the latest week for which data exists in the customer_purchases table, which the view code references. We can also paste the SQL that generates this dataset into Business Intelli- gence software such as Tableau, effectively pulling our summary dataset into a visual drag-­and-­drop reporting and dashboarding interface. The Tableau reports in Figures 10.6 and 10.7 were built using the same dataset query we developed earlier, and are data visualizations that filter and summa- rize data using the same fields and values displayed in Figures 10.4 and 10.5. Figure 10.5 (continued) Chapter 10 ■ Building SQL Datasets for Analytical Reporting 153 Taking SQL Reporting Further Now that you have seen one example of the development of a reusable ana- lytical dataset for reporting, let’s walk through another example, starting with the dataset we built at the end of Chapter 9, “Exploratory Data Analysis with SQL.” The last query in that chapter creates a dataset that has one row per market_date, vendor_id, and product_id, and includes information about the vendor and product, including the total inventory brought to market and the total sales for that day. This is an example of an analytical dataset that can be reused for many report variations. Some examples of questions that could be answered with that dataset include: ■ ■What quantity of each product did each vendor sell per market/week/ month/year? ■ ■When are certain products in season (most available for sale)? Figure 10.6 Figure 10.7 154 Chapter 10 ■ Building SQL Datasets for Analytical Reporting ■ ■What percentage of each vendor’s inventory is selling per time period? ■ ■Did the prices of any products change over time? ■ ■What are the total sales per vendor for the season? ■ ■How frequently do vendors discount their product prices? ■ ■Which vendor sold the most tomatoes last week? We can’t answer questions about any time periods shorter than a day, because the timestamp of the sale isn’t included. We also don’t have any detailed information about customers, but because we have date, vendor, and product dimensions, we can slice and dice different metrics by those values. We can add some calculated fields to the query for reporting purposes without pulling in any additional columns. The following query adds fields calculating the percentage of product quantity sold and the total discount and aliases them percent_of_available_sold and discount_amount, respectively. Let’s store this dataset as a view and use it to answer a business question: CREATE VIEW farmers_market.vw_sales_per_date_vendor_product AS SELECT vi.market_date, vi.vendor_id, v.vendor_name, vi.product_id, p.product_name, vi.quantity AS quantity_available, sales.quantity_sold, ROUND((sales.quantity_sold / vi.quantity) * 100, 2) AS percent_of_ available_sold, vi.original_price, (vi.original_price", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 94 + }, + { + "text": "the total discount and aliases them percent_of_available_sold and discount_amount, respectively. Let’s store this dataset as a view and use it to answer a business question: CREATE VIEW farmers_market.vw_sales_per_date_vendor_product AS SELECT vi.market_date, vi.vendor_id, v.vendor_name, vi.product_id, p.product_name, vi.quantity AS quantity_available, sales.quantity_sold, ROUND((sales.quantity_sold / vi.quantity) * 100, 2) AS percent_of_ available_sold, vi.original_price, (vi.original_price * sales.quantity_sold) -­ sales.total_sales AS discount_amount, sales.total_sales FROM farmers_market.vendor_inventory AS vi LEFT JOIN ( SELECT market_date, vendor_id, product_id, SUM(quantity) quantity_sold, SUM(quantity * cost_to_customer_per_qty) AS total_sales FROM farmers_market.customer_purchases GROUP BY market_date, vendor_id, product_id ) AS sales ON vi.market_date = sales.market_date AND vi.vendor_id = sales.vendor_id AND vi.product_id = sales.product_id LEFT JOIN farmers_market.vendor v ON vi.vendor_id = v.vendor_id Chapter 10 ■ Building SQL Datasets for Analytical Reporting 155 LEFT JOIN farmers_market.product p ON vi.product_id = p.product_id ORDER BY vi.vendor_id, vi.product_id, vi.market_date To clarify what’s happening in the calculation for discount_amount: (vi.original_price * sales.quantity_sold) -­ sales.total_sales we’re taking the original_price of a product and multiplying it by the quantity of that product that was sold, to get the total value of the products sold, and subtracting from it the actual sales for the day, so we get a total for how much potential profit the vendor gave away in the form of discounts for that product on that day. If a vendor asked, “What percent of my sales at each market came from each product I brought?” you could use this dataset to build a report to answer the question, because you have a summary of sales per product per vendor per market date. You will need to use a window function to get the answer, because the total you are dividing is the sum of all of a vendor’s sales per date, which means adding up the total_sales values across multiple rows of the dataset. Once you have the total_sales for a vendor on a market date, then you can divide each row’s total sales (remember there is one row per product per vendor per market date) into the vendor’s total sales of all products for the day. The query needed to generate this report is pictured in Figure 10.8, because the window functions make the calculations quite long, and the syntax high- lighting provided by the IDE helps make the various sections of the SQL state- ment clearer. Let’s walk through these calculations step by step. First, note that we’re querying from the view created by the previous query, vw_sales_per_date_vendor_prod- uct. We give the total sales, which is summarized per market date, vendor, and product in the view, an alias of vendor_product_sales_on_market_date, and round it to have two digits after the decimal point, since this is a report and we want to format everything nicely. Figure 10.8 156 Chapter 10 ■ Building SQL Datasets for Analytical Reporting In the next line of the SQL statement in Figure 10.8, we are summing up each vendor’s sales (of all of their products) on each market date, using a window function that partitions sales by market_date and vendor_id. (See Chapter 7 for more information about", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 95 + }, + { + "text": "SQL Datasets for Analytical Reporting In the next line of the SQL statement in Figure 10.8, we are summing up each vendor’s sales (of all of their products) on each market date, using a window function that partitions sales by market_date and vendor_id. (See Chapter 7 for more information about window functions.) Then we give that sum an alias of vendor_total_sales_on_market_date and round it to two decimal places. We now have the total sales for the vendor for the day on each row, and we already had the total sales of each product the vendor sold that day. The calcu- lation in the next line is that first dollar amount divided by the second dollar amount, which calculates the percentage of the vendor’s sales on that market date represented by each product. In the pictured rows of output at the bottom of Figure 10.8, you can see that Marco’s Peppers is only selling one product on 4/22/2020, so the sales on that row represent 100% of Marco’s sales for the day. Annie’s Pies is selling three different products, and you can see in the final column what portion of Annie’s total sales was contributed by each product. We can write additional queries against this reusable dataset to build other reports in SQL, too. To use SQL to get the same data summary that is shown in Figure 9.18, which was grouped and visualized in Tableau, we can query the view as follows: SELECT market_date, vendor_name, product_name, quantity_available, quantity_sold FROM farmers_market.vw_sales_per_date_vendor_product AS s WHERE market_date BETWEEN '2020-­06-­01' AND '2020-­07-­31' AND vendor_name = 'Marco''s Peppers' AND product_id IN (2, 4) ORDER BY market_date, product_id A partial view of the output of this query is in Figure 10.9, and you can com- pare the numbers to those in the bar chart in Figure 9.18. One benefit of saving queries that generate summary datasets so they are available to reuse as needed, is that any tool you use to pull the data will be referencing the underlying data, table joins, and calculated values. As long as the data isn’t changing between the report generation times, everyone using the defined dataset can get the same results. Chapter 10 ■ Building SQL Datasets for Analytical Reporting 157 Exercises 1. Using the view created in this chapter called farmers_market.vw_ sales_by_day_vendor, referring to Figure 10.3 for a preview of the data in the dataset, write a query to build a report that summarizes the sales per vendor per market week. 2. Rewrite the query associated with Figure 7.11 using a CTE (WITH clause). 3. If you were asked to build a report of total and average market sales by vendor booth type, how might you modify the query associated with Figure 10.3 to include the information needed for your report? Figure 10.9 C H A P T E R 159 11 Most of this book is targeted at beginners, but because beginners can quickly become more advanced in developing SQL, I wanted to give you some ideas of", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 96 + }, + { + "text": "associated with Figure 10.3 to include the information needed for your report? Figure 10.9 C H A P T E R 159 11 Most of this book is targeted at beginners, but because beginners can quickly become more advanced in developing SQL, I wanted to give you some ideas of what is possible when you think a little more creatively and go beyond the simplest SELECT statements. SQL is a powerful way to shape and summarize data into a wide variety forms that can be used for many types of analyses. This chapter includes a few examples of more complex query structures. UNIONs One query structure that I haven’t yet covered in this book, but which certainly deserves a mention, is the UNION query. Using a UNION, you can combine any two queries that result in the same number of columns with the same data types. The columns must be in the same order in both queries. There are many possible use cases for UNION queries, but the syntax is simple: write two queries with the same number and type of fields, and put a UNION keyword between them: SELECT market_year, MIN(market_date) AS first_market_date FROM farmers_market.market_date_info WHERE market_year = '2019' UNION More Advanced Query Structures Continues 160 Chapter 11 ■ More Advanced Query Structures SELECT market_year, MIN(market_date) AS first_market_date FROM farmers_market.market_date_info WHERE market_year = '2020' Of course, this isn’t a sensible use case, because you could just write one query, GROUP BY market_year, and filter to WHERE market_year IN (‘2019’,’2020’) and get the same output. There are always multiple ways to write queries, but sometimes combining two queries with identical columns selected but different criteria or different aggregation is the quickest way to get the results you want. For a more complex example combining CTEs and UNIONs, we’ll build a report that shows the products with the largest quantities available at each market: the bulk product with the largest weight available, and the unit product with the highest count available: WITH product_quantity_by_date AS ( SELECT vi.market_date, vi.product_id, p.product_name, SUM(vi.quantity) AS total_quantity_available, p.product_qty_type FROM farmers_market.vendor_inventory vi LEFT JOIN farmers_market.product p ON vi.product_id = p.product_id GROUP BY market_date, product_id ) SELECT * FROM ( SELECT market_date, product_id, product_name, total_quantity_available, product_qty_type, RANK() OVER (PARTITION BY market_date ORDER BY total_quantity_ available DESC) AS quantity_rank FROM product_quantity_by_date WHERE product_qty_type = 'unit' UNION SELECT market_date, product_id, product_name, total_quantity_available, product_qty_type, (continued) Chapter 11 ■ More Advanced Query Structures 161 RANK() OVER (PARTITION BY market_date ORDER BY total_quantity_ available DESC) AS quantity_rank FROM product_quantity_by_date WHERE product_qty_type = 'lbs' ) x WHERE x.quantity_rank = 1 ORDER BY market_date The WITH statement (CTE) at the top of this query totals up the quantity of each product that is available at each market from the vendor_inventory table, and joins in helpful information from the product table such as product name and the type of quantity, which has product_qty_type values of “lbs” or “unit.” The inner part of the bottom query contains two different queries of the same view created in the", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 97 + }, + { + "text": "each market from the vendor_inventory table, and joins in helpful information from the product table such as product name and the type of quantity, which has product_qty_type values of “lbs” or “unit.” The inner part of the bottom query contains two different queries of the same view created in the CTE, product_quantity_by_date, UNIONed together. Each ranks the information available in the CTE by total_quantity_available (the sum of the quantity field, aggregated in the WITH clause), as well as returning all of the available fields. Note that both queries return the fields in the exact same order. The only difference between the two queries is their WHERE clauses, which separate the results by product_qty_type. In this case, we can’t simply GROUP BY the product_qty_type and remove the UNION as was possible for the initial example query in this section, because the RANK() window function is ranking by quantity available in each query, and we want to see the top item per product_qty_type, so want to return the top ranked item from each set separately. The outer part of the bottom query selects the results of the union, and filters to only the top-­ranked quantities, so we get one row per market date with the highest number of lbs, and one row per market date with the highest number of units. Some results of this query are shown in Figure 11.1. You can see that for the month of August 2019, the bulk product with the highest weight each week was organic jalapeno peppers, and the product sold by unit with the highest count each week was sweet corn. Figure 11.1 162 Chapter 11 ■ More Advanced Query Structures For the sake of instruction, and because I frequently say that there are mul- tiple ways to construct queries in SQL that result in identical outputs, there is at least one other way to get the preceding output that doesn’t require a UNION. In the following query, the second query in the WITH clause queries from the first query in the WITH clause, and the final SELECT statement simply filters the result of the second query: WITH product_quantity_by_date AS ( SELECT vi.market_date, vi.product_id, p.product_name, SUM(vi.quantity) AS total_quantity_available, p.product_qty_type FROM farmers_market.vendor_inventory vi LEFT JOIN farmers_market.product p ON vi.product_id = p.product_id GROUP BY market_date, product_id ), rank_by_qty_type AS ( SELECT market_date, product_id, product_name, total_quantity_available, product_qty_type, RANK() OVER (PARTITION BY market_date, product_qty_type ORDER BY total_quantity_available DESC) AS quantity_rank FROM product_quantity_by_date ) SELECT * FROM rank_by_qty_type WHERE quantity_rank = 1 ORDER BY market_date We were able to accomplish the same result without the UNION by partition- ing by both the market_date and product_qty_type in the RANK() function, resulting in a ranking for each date and quantity type. Because I have shown two examples of UNION queries that don’t actually require UNIONs, I wanted to mention one case when a UNION is definitely required: when you have separate tables with the same columns, representing different time periods. This could happen, for example, when you have event logs (such", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 98 + }, + { + "text": "Because I have shown two examples of UNION queries that don’t actually require UNIONs, I wanted to mention one case when a UNION is definitely required: when you have separate tables with the same columns, representing different time periods. This could happen, for example, when you have event logs (such as website traffic logs) that are stored across multiple files, and each file is loaded into its own table in the database. Or, the tables could be static snapshots of the same dynamic dataset from different points in time. Or, maybe the data was Chapter 11 ■ More Advanced Query Structures 163 migrated from one system into another, and you need to pull data from tables in two different systems and combine them together into one view to see the entire history of records. Self-­Join to Determine To-­Date Maximum A self-­join in SQL is when a table is joined to itself (you can think of it like two copies of the table joined together) in order to compare rows to one another. You write the SQL for a self-­join just like any other join, but reference the same table name twice. To differentiate the two “copies” of the table, give each one its own alias: SELECT t1.id1, t1.field2, t2.field2, t2.field3 FROM mytable AS t1 LEFT JOIN mytable AS t2 ON t1.id1 = t2.id1 This particular example is meant to demonstrate the syntax and not high- light a typical use case, because you would not normally be joining on a pri- mary key and comparing a row to itself. One more realistic use case is using a comparison operator other than an equal sign to accomplish something such as joining every row to every previous row, as shown in the queries associated with Figures 11.3 through 11.5 below. Let’s say we wanted to show an aggregate metric changing over time, com- paring each value to all previous values. One reason you might want to com- pare a value to all previous values is to create a “record high to-­date” indicator. One possible use case might be the need to automatically determine when the count of positive COVID-­19 tests for a region on a particular day set a record as the highest count to-­date. One way you could use this data point is to create a visual indicator on a COVID-­19 tracking dashboard that appears when the count of new cases hits a new record high. Building this indicator into your dataset allows you to look back at the reported positive case counts at any past point in time and know if the number that day had set a new record high at the time. We can demonstrate an example of that type of query using the Farmer’s Market database by creating a report showing whether the total sales on each market date were the highest for any market to-­date. If we were always looking at data filtered to dates before a selected date, we could simply use the SUM() and MAX()", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 99 + }, + { + "text": "type of query using the Farmer’s Market database by creating a report showing whether the total sales on each market date were the highest for any market to-­date. If we were always looking at data filtered to dates before a selected date, we could simply use the SUM() and MAX() functions to determine the highest total sales for the given date range. But, if we’re looking back at a log of sales on all dates, and want to do the cal- culation using data from dates prior to each past date without filtering out the later dates from view, we need a different approach. 164 Chapter 11 ■ More Advanced Query Structures In this case, we want to determine whether there is any previous date that has a higher sales total than the “current” row we’re looking at, and we can use a self-­join to do that comparison. First, we’ll need to summarize the sales by market_date, which we have done previously. We’ll put this query into a CTE (WITH clause), and alias it sales_per_market_date. If we select the first 10 rows of this query, we get the output shown in Figure 11.2: WITH sales_per_market_date AS ( SELECT market_date, ROUND(SUM(quantity * cost_to_customer_per_qty),2) AS sales FROM farmers_market.customer_purchases GROUP BY market_date ORDER BY market_date ) SELECT * FROM sales_per_market_date LIMIT 10 We can select data from this “table” twice. The trick here is to join the table to itself using the market_date field—­but we won’t be using an equal sign in the join. In this case, we want to join every row to all other rows that have a date that occurred prior to the “current” row’s date, so we’ll use a less-­than sign (<) in the join. I’ll use an alias of cm to represent the “current market date” row (the left side of the join), and an alias of pm to represent a “previous market date” row. NOTE It’s easy to make an error when building this kind of join, so be sure to check your results carefully to ensure you created the intended output, especially if other tables are also joined in. Figure 11.2 Chapter 11 ■ More Advanced Query Structures 165 So we’re joining every row to every other row in the database that has a lower market_date value than it does. We’ll filter this to view the row from April 13, 2019, and you can see it is joined to the rows representing earlier market dates in the output in Figure 11.3: WITH sales_per_market_date AS ( SELECT market_date, ROUND(SUM(quantity * cost_to_customer_per_qty),2) AS sales FROM farmers_market.customer_purchases GROUP BY market_date ORDER BY market_date ) SELECT * FROM sales_per_market_date AS cm LEFT JOIN sales_per_market_date AS pm ON pm.market_date < cm.market_date WHERE cm.market_date = '2019-­04-­13' Now we’ll use a MAX() function on the pm.sales field and GROUP BY cm.market_date to get the previous highest sales value. The output of this ­version of the query is shown in Figure 11.4. If you compare this to Figure 11.3, you’ll recognize the sales total", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 100 + }, + { + "text": "WHERE cm.market_date = '2019-­04-­13' Now we’ll use a MAX() function on the pm.sales field and GROUP BY cm.market_date to get the previous highest sales value. The output of this ­version of the query is shown in Figure 11.4. If you compare this to Figure 11.3, you’ll recognize the sales total from April 6, 2019, which had the highest sales of any date prior to April 13, 2019: WITH sales_per_market_date AS ( SELECT market_date, ROUND(SUM(quantity * cost_to_customer_per_qty),2) AS sales FROM farmers_market.customer_purchases GROUP BY market_date ORDER BY market_date ) SELECT cm.market_date, Figure 11.3 Continues 166 Chapter 11 ■ More Advanced Query Structures cm.sales, MAX(pm.sales) AS previous_max_sales FROM sales_per_market_date AS cm LEFT JOIN sales_per_market_date AS pm ON pm.market_date < cm.market_date WHERE cm.market_date = '2019-­04-­13' GROUP BY cm.market_date, cm.sales We can now remove the date filter in the WHERE clause to get the previous_max_sales for each date. Additionally, we can use a CASE statement to create a flag field that indicates whether the current sales are higher than the previous maximum sales, indicating whether each row’s market_date set a sales record as of that date. This query’s output is displayed in Figure 11.5. Note that now that we can see the row for April 6, 2019, we can see that it is labeled as being a record sales day at the time, and its sales record is the value we saw displayed as the previous_max_sales in Figure 11.4. You can see that after a new record is set on May 29, 2019, the value in the previous_max_sales field updates to the new record value: WITH sales_per_market_date AS ( SELECT market_date, ROUND(SUM(quantity * cost_to_customer_per_qty),2) AS sales FROM farmers_market.customer_purchases GROUP BY market_date ORDER BY market_date ) SELECT cm.market_date, cm.sales, MAX(pm.sales) AS previous_max_sales, CASE WHEN cm.sales > MAX(pm.sales) THEN \"YES\" ELSE \"NO\" END sales_record_set FROM sales_per_market_date AS cm LEFT JOIN sales_per_market_date AS pm ON pm.market_date < cm.market_date GROUP BY cm.market_date, cm.sales Figure 11.4 (continued) Chapter 11 ■ More Advanced Query Structures 167 Counting New vs. Returning Customers by Week Another common report has to do with summarizing customers by time period. The manager of the farmer’s market might want to monitor how many customers are visiting the market per week, and how many of those are new, making a purchase for the first time. Remember that in this case, a customer is only counted if they make a pur- chase. So, if the farmer’s market manager asks what percentage of visitors to the market end up making a purchase, we can’t give them that information using the data in this database. We can only report on customers who purchased items, and they are identified using their loyalty card with each purchase (we assume for the sake of simplicity that 100% of customers are identifiable at purchase). We have built queries and used functions to summarize customers per week before, but how do we determine whether a customer is new? One way is to compare each purchase date to the minimum purchase date per customer. If a customer’s", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 101 + }, + { + "text": "simplicity that 100% of customers are identifiable at purchase). We have built queries and used functions to summarize customers per week before, but how do we determine whether a customer is new? One way is to compare each purchase date to the minimum purchase date per customer. If a customer’s minimum purchase date is today, then the customer made their first purchase today, and therefore is new. Let’s get a summary of every market date attended by every customer, and determine their first purchase dates. This query finds the customer’s first purchase date by using MIN() as a window function, partitioned by customer_id: SELECT DISTINCT customer_id, Figure 11.5 Continues 168 Chapter 11 ■ More Advanced Query Structures market_date, MIN(market_date) OVER(PARTITION BY cp.customer_id) AS first_purchase_ date FROM farmers_market.customer_purchases cp The DISTINCT has been added because there is a row in the customer_ purchases table for each item purchased, and we only need one row per market_ date per customer_id. You might have noticed that we didn’t need a GROUP BY here. Because the window function does its own partitioning, and the DIS- TINCT ensures we don’t return duplicate rows, no further grouping is needed. A portion of the results from the preceding query is shown in Figure 11.6. You can see that each row depicts a date that customer shopped at the farmer’s market alongside the customer’s first purchase date. Now we can put that query inside a WITH clause and query its results with some calculations added. We also need to join it to the market_date_info table to get the year and week of each market_date. We’re going to group by week, so if a customer made purchases at both markets within that week, they will have two rows to be grouped into each year-­week combination. So let’s do two types of counts. The field we alias, customer_visit_count, will count each of those rows, so a customer shopping at both markets in a week would be counted twice in that summary. We’ll create a second calculation that counts unique customer_id values using DISTINCT and alias that distinct_customer_count, which will only count unique customers per week without double-­counting anyone. The results of the following counts are shown in Figure 11.7: WITH customer_markets_attended AS ( Figure 11.6 (continued) Chapter 11 ■ More Advanced Query Structures 169 SELECT DISTINCT customer_id, market_date, MIN(market_date) OVER(PARTITION BY cp.customer_id) AS first_purchase_ date FROM farmers_market.customer_purchases cp ) SELECT md.market_year, md.market_week, COUNT(customer_id) AS customer_visit_count, COUNT(DISTINCT customer_id) AS distinct_customer_count FROM customer_markets_attended AS cma LEFT JOIN farmers_market.market_date_info AS md ON cma.market_date = md.market_date GROUP BY md.market_year, md.market_week ORDER BY md.market_year, md.market_week The results are something we could’ve achieved with a much simpler query, so what was the point of the query we turned into a CTE? The data is now in a form that facilitates performing further calculations, and we also have access to each customer’s first purchase date, which we haven’t made use of yet. We also want to get a count of new customers per week, so let’s add", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 102 + }, + { + "text": "we turned into a CTE? The data is now in a form that facilitates performing further calculations, and we also have access to each customer’s first purchase date, which we haven’t made use of yet. We also want to get a count of new customers per week, so let’s add a column display- ing what percent of each week’s customers are new. This requires adding two more fields to the query. The first looks like this: COUNT( DISTINCT CASE WHEN cma.market_date = cma.first_purchase_date THEN customer_id ELSE NULL END ) AS new_customer_count Figure 11.7 170 Chapter 11 ■ More Advanced Query Structures Inside the COUNT() function is a CASE statement. Can you tell what it does? It is looking for rows in the results of the customer_markets_attended CTE where the market_date (the date when the customer made the purchase) is equal to the customer’s first purchase date. If those values match, the CASE statement returns a customer_id to count. If not, the CASE statement returns NULL. So, the result is a distinct count of customers that made their first purchase that week. The second field, which is the last listed in the following full query, then divides that same value by the total distinct count of customer IDs, giving us a percentage. The result of this query is shown in Figure 11.8. WITH customer_markets_attended AS ( SELECT DISTINCT customer_id, market_date, MIN(market_date) OVER(PARTITION BY cp.customer_id) AS first_purchase_ date FROM farmers_market.customer_purchases cp ) SELECT md.market_year, md.market_week, COUNT(customer_id) AS customer_visit_count, COUNT(DISTINCT customer_id) AS distinct_customer_count, COUNT(DISTINCT CASE WHEN cma.market_date = cma.first_purchase_date THEN customer_id ELSE NULL END)AS new_customer_count, COUNT(DISTINCT CASE WHEN cma.market_date = cma.first_purchase_date THEN customer_id ELSE NULL END) / COUNT(DISTINCT customer_id) AS new_customer_percent FROM customer_markets_attended AS cma LEFT JOIN farmers_market.market_date_info AS md ON cma.market_date = md.market_date GROUP BY md.market_year, md.market_week ORDER BY md.market_year, md.market_week Chapter 11 ■ More Advanced Query Structures 171 It makes sense that on the first market date, 100% of the customers are new. With the data that’s currently entered into the Farmer’s Market database as of the time of this writing, there are no new customers added after week 18 of 2019. Summary These are just a few examples of more complicated query structures that can be created with SQL. I hope by demonstrating these approaches that it gives you a sense of the wide variety of analyses and datasets you can create with differ- ent combinations of the SQL you have learned in this book. In the next chapter, we’ll reuse some of these concepts to generate datasets designed specifically to be used as inputs into machine learning and forecasting algorithms. Exercises 1. Starting with the query associated with Figure 11.5, put the larger SELECT statement in a second CTE, and write a query that queries from its results to display the current record sales and associated market date. Can you think of another way to generate the same results? 2. Modify the “New vs. Returning Customers Per Week” report (associated with Figure 11.8) to summarize the counts by vendor", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 103 + }, + { + "text": "and write a query that queries from its results to display the current record sales and associated market date. Can you think of another way to generate the same results? 2. Modify the “New vs. Returning Customers Per Week” report (associated with Figure 11.8) to summarize the counts by vendor by week. 3. Using a UNION, write a query that displays the market dates with the highest and lowest total sales. Figure 11.8 C H A P T E R 173 12 In previous chapters, we introduced SQL concepts and walked through some analytical reporting examples, but we have not yet focused on the specifics of dataset design for predictive modeling applications. In this chapter, we’ll discuss the development of datasets for two types of algorithms: classification and time series models. A binary classification model predicts whether a record belongs to one category or another. For example, a heart disease classification model might analyze data from a patient’s medical history to determine whether they’re likely to develop heart disease, or not. A weather model could use past and current temperature, precipitation, pressure, and wind measurements, as well as those from surround- ing geographic areas, to predict whether or not it will rain in the next 24 hours. In a retail scenario like a farmer’s market, the seller may want to predict whether a customer will return to make another purchase within a certain time frame, or not. In order to make predictions, the model needs to be trained. Binary classifiers are a type of supervised learning model, which means they are trained by passing example rows of data (also called instances, observations, or feature vectors) labeled with each of the possible outcomes into the algorithm, so it can detect patterns and identify characteristics that are more strongly associated with one result or the other. Some example instances are set aside to test the trained model, feeding them into the algorithm to generate predictions we can compare to the known actual outcomes in order to check the model’s performance and deter- mine in what ways the model is incorrect, so we can make adjustments to it. Creating Machine Learning Datasets Using SQL 174 Chapter 12 ■ Creating Machine Learning Datasets Using SQL A time series model performs statistical operations on a series of measurements over time to forecast what the measurement might be at some point in time in the future. The training data for a time series model is a running log of data measurements from past points in time. Someone could use an hourly history of a stock’s prices to attempt to predict the value of an investment at the end of the day. A college might use historical counts of applications, admission offers, enrolled students, and deposits paid per week to generate a weekly forecast of incoming freshman class enrollment. A farmer’s market may use purchases over time to detect seasonal product sales trends and growth in the customer base, or try to predict how many ears of", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 104 + }, + { + "text": "counts of applications, admission offers, enrolled students, and deposits paid per week to generate a weekly forecast of incoming freshman class enrollment. A farmer’s market may use purchases over time to detect seasonal product sales trends and growth in the customer base, or try to predict how many ears of corn will sell per week next month. Each type of model requires a different type of dataset, and we’ll review some approaches for preparing datasets for these two common types of models using SQL. Datasets for Time Series Models The simplest type of time series forecasting uses a single variable measured over specified time intervals to predict the value of that same variable at a future point in time. A dataset for training a simple time series model can consist of just two columns: a column with dates or datetime values that indicate the time of measurement, and a column with the value being measured. For example, a model to predict the high temperature in a location tomorrow could have a dataset with years’ worth of daily high temperatures measured at that location. The dataset would have one row per day, with one column for the date, and another for the daily high temperature measured. A time series algorithm could detect seasonal temperature patterns, long-­term trends, and the most recent daily high temperatures to predict what the high temperature might be tomorrow. Let’s create a dataset that allows us to plot a time series of farmer’s market sales per week. Note that this will be a simplistic view of sales, because it will not take into consideration changes in vendors over time, available inventory at different times of year, or external economic factors. In Chapter 10, “Building SQL Datasets for Analytical Reporting,” we created a dataset that summarized sales per market date. Here, we’ll further summarize that data to a weekly level. Because we have joined the customer_purchases table to the market_date_info table and there is a market_week field available, you may assume that we want to group by that field. However, remember that market_week is a number that represents the week of the year, and every year has week numbers 1 through 52. So, if you group by market_week only, you will be adding sales from the same calendar weeks from different years together! There- fore, we will need to GROUP BY both market_year and market_week. ­However, Chapter 12 ■ Creating Machine Learning Datasets Using SQL 175 we want only a single field indicating the time period in our dataset, so we don’t want to output the market_year and market_week fields in the results. Because so many time series algorithms are designed to use calendar dates to indicate when an event occurred or a measurement was taken, we’re going to use the first market date of each week to label the weekly intervals. So, we’ll find the minimum market date per week and output that, with the column alias first_market_date_of_week: SELECT MIN(cp.market_date) AS first_market_date_of_week, ROUND(SUM(cp.quantity * cp.cost_to_customer_per_qty),2) AS weekly_sales", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 105 + }, + { + "text": "an event occurred or a measurement was taken, we’re going to use the first market date of each week to label the weekly intervals. So, we’ll find the minimum market date per week and output that, with the column alias first_market_date_of_week: SELECT MIN(cp.market_date) AS first_market_date_of_week, ROUND(SUM(cp.quantity * cp.cost_to_customer_per_qty),2) AS weekly_sales FROM farmers_market.customer_purchases AS cp LEFT JOIN farmers_market.market_date_info AS md ON cp.market_date = md.market_date GROUP BY md.market_year, md.market_week ORDER BY md.market_year, md.market_week Figure 12.1 shows the last 13 rows of data generated by this query. Tableau has some built-­in time series forecasting functions, including one that uses a method called exponential smoothing. We’ll import the results of our query into Tableau and have it use the weekly sales data to generate a sales forecast for the eight weeks beyond the last date in our dataset. The line chart in Figure 12.2 shows a visualization of the numbers generated by the preceding query, in the lighter gray color labeled “Actual” in the legend. These are the actual sales per week from our query. The forecasted sales for the next eight weeks are plotted in a darker gray color. These “Estimate” values are surrounded by a shaded area that represents a 90% confidence interval for each forecasted value. Figure 12.1 176 Chapter 12 ■ Creating Machine Learning Datasets Using SQL By summarizing past sales into a dataset that has one row per week, with a date column and a weekly sales total column, we enabled Tableau’s forecast to identify patterns in past weekly sales and use those to forecast future weekly sales. However, we didn’t provide it with enough granularity to forecast daily sales. There is no information in our dataset indicating that each week in our dataset actually represents sales from two market dates (as opposed to, say, seven different days of sales summarized by week). And if we asked Tableau to forecast monthly sales using this dataset, it only has the date of the first market per week, so if a new month started between two market dates we grouped into a single week, the second market’s sales will be categorized into the wrong month, creating erroneous training data for the forecasting algorithm. It’s important to know the intended use of a dataset in order to make the correct choices when designing it. It’s also important to provide documenta- tion to accompany a dataset, so others who might use it later are aware of its original purpose, and any caveats that might be important to know when using it for analysis. Datasets for Binary Classification Classification algorithms detect patterns in training datasets, which contain example records from the past that fall into known categories, and use those detected patterns to then categorize new data. Binary classification algorithms categorize inputs into one of two outcomes, which are determined according to the purpose of the model and the available data. For example: Given the medical Figure 12.2 Chapter 12 ■ Creating Machine Learning Datasets Using SQL 177 history of a patient, is it", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 106 + }, + { + "text": "categorize new data. Binary classification algorithms categorize inputs into one of two outcomes, which are determined according to the purpose of the model and the available data. For example: Given the medical Figure 12.2 Chapter 12 ■ Creating Machine Learning Datasets Using SQL 177 history of a patient, is it likely that they have heart disease, or not? Given the current and summarized past weather data, is it likely to rain within the next 24 hours, or not? Many of these algorithms can also output a probability or likelihood score, so you can also determine how “sure” the algorithm is that the instance should be classified into one category or the other (how well it matches the patterns detected for training examples in either class). These algorithms need training data that is in the same form as the data to be classified. For example, if you want the algorithm to take a patient’s medical history, including current vital measurements, as input, then the training data has to be at the same granularity and level of summary. Based on the dataset it was trained on, your classification model may expect one row of data per patient, with input fields such as a patient’s age, sex, cholesterol as of five years ago, cholesterol as of one year ago, cholesterol measured today (the day of diag- nosis), number of years that the patient has smoked cigarettes (as of the day of diagnosis), resting blood pressure as of five years ago, resting blood pressure as of one year ago, resting blood pressure measured today, resting ECG results, chest pain level indicator, and other summary metrics. In this case, each training “instance” (row of data, or vector) should have the data as of the diagnosis date, as well as the measurements or cholesterol and blood pressure from one and five years prior to the diagnosis date, so the duration between those two data points in the training data is as similar as possible to the duration between those two data points in the data you are passing through the trained model to be classified. The conditions under which the model will be applied need to be considered when designing the dataset. Say you trained your model on data from a study that was structured that way, collecting data over a five-­year period, but it’s unlikely that you will have a five-­year history of those measurements for current patients you want this algorithm to make a prediction for. You might not want to include the data from five years ago in the training dataset, since the model might not be good at classifying records with NULL values in the “resting blood pressure as of five years ago” field and other fields requiring past data, if all of the training instances included those values. Alternatively, training a model on data col- lected five years ago with outcomes as of the current year would be ideal if the thing you’re trying to predict is whether a patient will", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 107 + }, + { + "text": "ago” field and other fields requiring past data, if all of the training instances included those values. Alternatively, training a model on data col- lected five years ago with outcomes as of the current year would be ideal if the thing you’re trying to predict is whether a patient will develop heart disease five years from now. Every classification model requires a target variable, which is the thing you’re trying to predict. In binary classifiers, you can usually input the target as a binary value of 1 or 0, with each number representing each outcome. The target var- iable in the preceding example is a binary flag indicating a diagnosis of heart disease (or no heart disease) as of the date of the latest cholesterol and blood pressure measurements. 178 Chapter 12 ■ Creating Machine Learning Datasets Using SQL We won’t get into the details of how much past data with known outcomes is required for training and testing a model here, as it depends on the type of model, the number of columns, the variation in the data values, and many other factors beyond the scope of this book. We also won’t be training classification models in this book. However, we will discuss how to structure the datasets needed for binary classification model training and prediction, and how to use SQL to pull the data you need to train and run a classifier. When thinking about how to structure a dataset for binary classification, the first thing you’ll need to determine is the target variable, meaning the categories you’re building a model to classify records into. Often, the target variable needs a time bounding, so instead of predicting “Will this patient develop heart dis- ease?” the outcome being predicted could be “Will this patient be diagnosed with heart disease in the next five years?” The time bounding will affect choices you make when designing the training dataset. Creating the Dataset To have a concrete example to discuss, we’ll build a dataset that could be used to train a model that can predict an answer to the question “Will this customer who just made a purchase return to make another purchase within the next month?” The binary target variable is “makes another purchase within 30 days,” with the values 1 for “yes” and 0 for “no.” With this time-­limited target variable, we can create a training dataset that summarizes information about each cus- tomer as of the date of each purchase, and flag that row with a 1 if that customer made another purchase within a month, and 0 if they did not. Now, instead of having one training example per customer, we have one training example per customer per purchase date. The benefit of having multiple records per customer is that a lot more training instances are available for the model to use to detect patterns in the data. Addi- tionally, a person’s behavior can change over time, so having a snapshot record of their summary activity every time", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 108 + }, + { + "text": "purchase date. The benefit of having multiple records per customer is that a lot more training instances are available for the model to use to detect patterns in the data. Addi- tionally, a person’s behavior can change over time, so having a snapshot record of their summary activity every time they make a purchase, and a record of whether they came back within a month of making that purchase, can help the algorithm determine what impact certain activities may have on behavior. One effect of this approach to be aware of is that frequent customers will be over-­ represented in the dataset, which could have different impacts depending on the model, and could lead to overfitting the model to that type of customer. (Because a one-­time customer will only be in the training dataset one time, while a frequent customer will be in the training dataset many times.) Setting up your query to produce a dataset that is at the correct granularity with a target variable that is time-­bound can be the most complicated step of the query design process, and it’s worth taking the time to make sure it is cal- culated correctly before pulling in any other data fields. When I first designed Chapter 12 ■ Creating Machine Learning Datasets Using SQL 179 this example and the dataset to pair with it, I set it up to determine whether each customer makes a purchase in the next calendar month (for example: if the purchase record is from April, will the customer make a purchase in May?). That is one way to approach this model that would be perfectly valid, if that’s the type of prediction you wanted to make. However, I quickly realized that the time duration for the target variable wouldn’t be consistent. Depending on when in April the customer made a purchase, the target variable could be 1 (yes) if the next purchase date was one day after the initial purchase, up to almost two months later if the initial purchase was on April 1 and the second purchase was on May 30. So, I decided to instead put a dynamic one-­month time limit after each purchase for determining the returning customer flag value. So in the following queries, the target variable purchased_again_within_30_days represents whether the customer returned within 30 days of making a purchase, without considering the calendar month. With this approach, we can start by creating a CTE that has a row for every purchase, like the query we developed in Chapter 11, “More Advanced Query Structures,” that was associated with Figure 11.5. We can reuse the customer_ markets_attended CTE in the following query to look up whether a customer made another purchase within 30 days after the date of each purchase. Before we look at the calculated columns in the following query, let’s look at its FROM, WHERE, and GROUP BY clauses, which determine which table(s) we’re selecting from, how they’re filtered, and the granularity of the final result. You can see that", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 109 + }, + { + "text": "30 days after the date of each purchase. Before we look at the calculated columns in the following query, let’s look at its FROM, WHERE, and GROUP BY clauses, which determine which table(s) we’re selecting from, how they’re filtered, and the granularity of the final result. You can see that we’re selecting from the customer_purchases table, which has one row per customer per product purchased. There is no WHERE clause, so we’re returning all rows. The GROUP BY clause includes the customer_id and market_date from the customer_purchases table, so we’ll end up with one row per customer per market date at which they made a purchase: WITH customer_markets_attended AS ( SELECT DISTINCT customer_id, market_date FROM farmers_market.customer_purchases ORDER BY customer_id, market_date ) SELECT cp.market_date, cp.customer_id, SUM(cp.quantity * cp.cost_to_customer_per_qty) AS purchase_total, COUNT(DISTINCT cp.vendor_id) AS vendors_patronized, COUNT(DISTINCT cp.product_id) AS different_products_purchased, (SELECT MIN(cma.market_date) FROM customer_markets_attended AS cma Continues 180 Chapter 12 ■ Creating Machine Learning Datasets Using SQL WHERE cma.customer_id = cp.customer_id AND cma.market_date > cp.market_date GROUP BY cma.customer_id) AS customer_next_market_date, DATEDIFF( (SELECT MIN(cma2.market_date) FROM customer_markets_attended AS cma2 WHERE cma2.customer_id = cp.customer_id AND cma2.market_date > cp.market_date GROUP BY cma2.customer_id), cp.market_date) AS days_until_customer_next_market_date, CASE WHEN DATEDIFF( (SELECT MIN(cma3.market_date) FROM customer_markets_attended AS cma3 WHERE cma3.customer_id = cp.customer_id AND cma3.market_date > cp.market_date GROUP BY cma3.customer_id), cp.market_date) <=30 THEN 1 ELSE 0 END AS purchased_again_within_30_days FROM farmers_market.customer_purchases AS cp GROUP BY cp.customer_id, cp.market_date ORDER BY cp.customer_id, cp.market_date The purchase_total column should be familiar by now, multiplying the quantity and cost of each item purchased and summing that up to get the total spent by each customer at each market date. The vendors_patronized column is a distinct count of how many different vendors the customer made purchases from that day, and the different_products_purchased column is a distinct count of how many different kinds of products the customer purchased. These calculated columns are also called engineered features when you’re talking about datasets for machine learning, and we’re including them so we can explore the relationship between these values and the target variable. Maybe the more vendors the customer makes purchases from, the more likely they are to pur- chase an item they like so much that they’ll return within 30 days to buy more. Including this column in the dataset enables the exploration of the relationship between this feature and the target variable. The next column, customer_next_market_date, is generated by a subquery that references our CTE. Look at all of the code inside parentheses after dif- ferent_products_purchased, and before customer_next_market_date. This subquery selects the minimum market date a customer attended, which occurs after the current row’s market_date value. In other words, we’re finding the date of this customer’s next purchase. In the WHERE clause of this subquery, we’re matching up the subquery’s customer_id with the main query’s cus- tomer_id (to ensure we’re looking at a single customer’s trips to the market). We’re also limiting the subquery rows to those where the market date occurs (continued) Chapter 12 ■ Creating Machine Learning Datasets Using SQL 181 after the main query’s market", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 110 + }, + { + "text": "the subquery’s customer_id with the main query’s cus- tomer_id (to ensure we’re looking at a single customer’s trips to the market). We’re also limiting the subquery rows to those where the market date occurs (continued) Chapter 12 ■ Creating Machine Learning Datasets Using SQL 181 after the main query’s market date, with the cma.market_date > cp.market_ date filter. In effect, we’re pulling a list of all of this customer’s future market dates, and then only returning the minimum date from that list, and aliasing it customer_next_market_date. The next column, aliased days_until_customer_next_market_date, uses the same subquery, but this time calculates the difference between the current row’s date and that calculated next market date. Then the last column, aliased pur- chased_again_within_30_days, uses the same calculation and wraps it inside a CASE statement that returns a binary flag (1 or 0) value indicating whether that next purchase was made within 30 days. If you review these three calcu- lated columns, you will see that the subqueries are all the same, and we’re just performing different calculations on the result. Some example rows generated by this query are shown in Figure 12.3. I aliased the customer_markets_attended CTE reference differently in each of the subqueries, as cma, cma2, and cma3 for clarity, though that isn’t actually necessary. The table alias only applies within the subquery. Also note that each of these subqueries are aggregates that return only one value to be inserted into each row of the dataset. The customer_next_market_date and days_until_customer_next_market are useful for validating the output of the preceding query, but once we have checked that the target variable purchased_again_within_30_days looks correct in context, we can remove those two columns from our machine learning data- set. All of the columns other than the target variable will be available as inputs to our algorithm, so we don’t want to encode that value indicating whether or not they returned within 30 days in any other field, to avoid data leakage. Data leakage occurs when data that would not have been known as of the snapshot in time that the row represents is fed into the algorithm, artificially improving the predictions. Expanding the Feature Set What other features might we include that might contain “signal (detectable patterns)” for the model to detect? Maybe particular vendors have more loyal customers, or sell items that need to be replenished more frequently, so shopping at a particular vendor is an indicator that a shopper might return sooner. Or, the number of days since a customer’s last purchase could be an indicator that they Figure 12.3 182 Chapter 12 ■ Creating Machine Learning Datasets Using SQL are a frequent shopper and have a higher likelihood to come back again soon. Let’s add some columns that indicate which vendors each customer shopped at on each market day and flip the days_until_customer_next_market_date cal- culation to instead indicate how long it’s been since the customer last shopped before the visit represented by the row: WITH customer_markets_attended AS ( SELECT DISTINCT customer_id, market_date FROM", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 111 + }, + { + "text": "Let’s add some columns that indicate which vendors each customer shopped at on each market day and flip the days_until_customer_next_market_date cal- culation to instead indicate how long it’s been since the customer last shopped before the visit represented by the row: WITH customer_markets_attended AS ( SELECT DISTINCT customer_id, market_date FROM farmers_market.customer_purchases ORDER BY customer_id, market_date ) SELECT cp.market_date, cp.customer_id, SUM(cp.quantity * cp.cost_to_customer_per_qty) AS purchase_total, COUNT(DISTINCT cp.vendor_id) AS vendors_patronized, MAX(CASE WHEN cp.vendor_id = 7 THEN 1 ELSE 0 END) AS purchased_from_vendor_7, MAX(CASE WHEN cp.vendor_id = 8 THEN 1 ELSE 0 END) AS purchased_from_vendor_8, COUNT(DISTINCT cp.product_id) AS different_products_purchased, DATEDIFF(cp.market_date, (SELECT MAX(cma.market_date) FROM customer_markets_attended AS cma WHERE cma.customer_id = cp.customer_id AND cma.market_date < cp.market_date GROUP BY cma.customer_id)) AS days_since_last_customer_market_date, CASE WHEN DATEDIFF( (SELECT MIN(cma.market_date) FROM customer_markets_attended AS cma WHERE cma.customer_id = cp.customer_id AND cma.market_date > cp.market_date GROUP BY cma.customer_id), cp.market_date) <=30 THEN 1 ELSE 0 END AS purchased_again_within_30_days FROM farmers_market.customer_purchases AS cp GROUP BY cp.customer_id, cp.market_date ORDER BY cp.customer_id, cp.market_date You can see in the preceding query that we added a couple demonstration columns indicating whether each customer purchased from vendors 7 or 8. This technique could be repeated to create a column for every vendor. We also flipped the greater-­than sign in the date comparison to a less-­than sign, to find Chapter 12 ■ Creating Machine Learning Datasets Using SQL 183 how many days it had been since the customer last made a purchase, in the feature aliased days_since_last_customer_market_date. Another type of aggregate value that might be useful to input into a pre- dictive model is some type of representation of the customer’s entire history of farmer’s market shopping, up to the date the row represents. For example, how many times has the customer shopped at the farmer’s market before? A long-­time shopper might be more likely to return than a brand-­new shopper. The ROW_NUMBER window function is one way to calculate this value, counting how many prior rows exist for each customer, but you have to be careful where you put it in the query, because ROW_NUMBER only counts the rows returned by the query. So, if we wanted to count how many times the customer has shopped at the market before as of the market date in the current row, but our main query is filtered to only return data from the year 2019, then our ROW_NUMBER window function will only be able to count previous purchases in 2019 and not the customer’s entire shopping history. One solution for our use case is to put the ROW_NUMBER function in the cus- tomer_markets_attended CTE, which doesn’t need to be filtered by date, even if you wanted to filter the final output, so we can calculate the number of past markets attended using a similar approach we used to determine the previous purchase date, referencing that CTE. This time, instead of returning the maximum market date that’s less than the market_date in the row, we’ll return the row number that’s associated with the current market_date from that same query. In the following", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 112 + }, + { + "text": "using a similar approach we used to determine the previous purchase date, referencing that CTE. This time, instead of returning the maximum market date that’s less than the market_date in the row, we’ll return the row number that’s associated with the current market_date from that same query. In the following query, this value has the alias market_count in the CTE and is summarized as customer_markets_attended_count in the main query. One important note is that when we add the ROW_NUMBER to the customer_ markets_attended query in the WITH clause, we have to modify it to use a GROUP BY instead of a COUNT DISTINCT to summarize per customer_id and market_date. The first eight rows of the output of the COUNT DISTINCT approach are shown in Figure 12.4, and the first eight rows of the output of the GROUP BY approach are shown in Figure 12.5. The reason the ROW_NUMBER returns much higher counts when the query is summarized using COUNT DISTINCT is that the window function is calculated on the dataset before the DISTINCT, so since the results aren’t grouped, it’s returning one row per customer per prod- uct purchased, not one row per customer per market_date, then numbering every row in the customer_purchases table for that customer. Grouping by market_date solves that issue because the window function is calculated after the data is aggregated by the GROUP BY. Sometimes it takes trial and error to get the order of operations correct, and this is why it’s important to view the details of the underlying data prior to aggregating, so you know whether the resulting summary data is correct. 184 Chapter 12 ■ Creating Machine Learning Datasets Using SQL Like with most SQL queries, there are actually several ways to accomplish the same result. The following approach, which moves the ROW_NUMBER() into the WHERE clause, will return the same count for customer_markets_attended_count as a version that has it in the main query, even if the main query is filtered to a date range that doesn’t include a customer’s entire purchase history: WITH customer_markets_attended AS ( SELECT customer_id, market_date, ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY market_date) AS market_count FROM farmers_market.customer_purchases GROUP BY customer_id, market_date ORDER BY customer_id, market_date ) select cp.customer_id, cp.market_date, (SELECT MAX(market_count) FROM customer_markets_attended AS cma WHERE cma.customer_id = cp.customer_id AND cma.market_date <= cp.market_date) AS customer_markets_ attended_count Figure 12.4 Figure 12.5 Chapter 12 ■ Creating Machine Learning Datasets Using SQL 185 FROM farmers_market.customer_purchases AS cp GROUP BY cp.customer_id, cp.market_date ORDER BY cp.customer_id, cp.market_date One feature we could add that is likely predictive of whether a customer returns in the next 30 days is how many times they shopped in the previous 30 days. So, we can create another column that puts a time range on the calculation for past markets attended and counts the market dates in the last 30 days. This is demonstrated in the following query by the calculation aliased customer_markets_attended_30days_count. You might extrapolate from this calculation one of the aforementioned alternative ways of determining the total", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 113 + }, + { + "text": "that puts a time range on the calculation for past markets attended and counts the market dates in the last 30 days. This is demonstrated in the following query by the calculation aliased customer_markets_attended_30days_count. You might extrapolate from this calculation one of the aforementioned alternative ways of determining the total count of markets attended: WITH customer_markets_attended AS ( SELECT customer_id, market_date, ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY market_date) AS market_count FROM farmers_market.customer_purchases GROUP BY customer_id, market_date ORDER BY customer_id, market_date ) select cp.customer_id, cp.market_date, (SELECT COUNT(market_date) FROM customer_markets_attended AS cma WHERE cma.customer_id = cp.customer_id AND cma.market_date < cp.market_date AND DATEDIFF(cp.market_date, cma.market_date) <= 30) AS customer_markets_attended_30days_count FROM farmers_market.customer_purchases AS cp GROUP BY cp.customer_id, cp.market_date ORDER BY cp.customer_id, cp.market_date Feature Engineering This process of creating different input values that might be helpful to the predic- tion algorithm is called feature engineering. Most binary classification algorithms require numeric inputs, so sometimes features are engineered to convert another data type into a numeric representation. Other types of features you might create include sets of one-­hot encoded flag columns (as mentioned in Chapter 4, “CASE Statements”), converting categorical text columns to numeric values, aggregate 186 Chapter 12 ■ Creating Machine Learning Datasets Using SQL high or low metrics (such as the maximum ever spent by the customer at a market to-­date), other incrementing totals (such as the length of time the person has been a customer of the market), and other summaries for different time periods. One important factor when engineering features is that each of these fea- ture values is only what would be knowable as of the date represented by the row—­the market_date in this case. We want to train the model on examples of customers with a variety of traits as of specific points in time that can be cor- related with a specific outcome or target variable relative to that time. In fact, I’ve been outputting the market_date in each of the previous queries for ver- ification purposes, but I wouldn’t input the full date into a predictive model, because then the training data would all be tied to past dates, when only the relative dates for the events of interest (the time between purchases) are impor- tant. If we ran a current customer’s record through the model to try to make a prediction based on data collected this week, but the model had been trained on full date values from the past, the model wouldn’t know what to do with the market_date, because there will have been no training examples with sim- ilar dates. So, when I train my classification model using this dataset, I will not input the customer_id or market_date into the algorithm. I will only use them as unique identifiers, or index values, so the predictions the model outputs can be tied back to their respective rows. However, the month of the market date is likely predictive, because the market closes for the months of January and February. So, customers shopping in December would have a", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 114 + }, + { + "text": "them as unique identifiers, or index values, so the predictions the model outputs can be tied back to their respective rows. However, the month of the market date is likely predictive, because the market closes for the months of January and February. So, customers shopping in December would have a lower likelihood of returning in the next 30 days than customers in other months. The final version of this example classification dataset query will include a column representing the month, which can be seen in Figures 12.6 and 12.7. The number of columns is too wide to fit into one figure, so the output is split into two figures, with the customer_id and market_date index visible in both sections so you can find the continuation of each row in the second block: WITH customer_markets_attended AS ( SELECT customer_id, market_date, ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY market_date) AS market_count FROM farmers_market.customer_purchases GROUP BY customer_id, market_date ORDER BY customer_id, market_date ) Chapter 12 ■ Creating Machine Learning Datasets Using SQL 187 SELECT cp.customer_id, cp.market_date, EXTRACT(MONTH FROM cp.market_date) as market_month, SUM(cp.quantity * cp.cost_to_customer_per_qty) AS purchase_total, COUNT(DISTINCT cp.vendor_id) AS vendors_patronized, MAX(CASE WHEN cp.vendor_id = 7 THEN 1 ELSE 0 END) purchased_from_ vendor_7, MAX(CASE WHEN cp.vendor_id = 8 THEN 1 ELSE 0 END) purchased_from_ vendor_8, COUNT(DISTINCT cp.product_id) AS different_products_purchased, DATEDIFF(cp.market_date, (SELECT MAX(cma.market_date) FROM customer_markets_attended AS cma WHERE cma.customer_id = cp.customer_id AND cma.market_date < cp.market_date GROUP BY cma.customer_id) ) days_since_last_customer_market_date, (SELECT MAX(market_count) FROM customer_markets_attended AS cma WHERE cma.customer_id = cp.customer_id AND cma.market_date <= cp.market_date) AS customer_markets_ attended_count, (SELECT COUNT(market_date) FROM customer_markets_attended AS cma WHERE cma.customer_id = cp.customer_id AND cma.market_date < cp.market_date AND DATEDIFF(cp.market_date, cma.market_date) <= 30) AS customer_markets_attended_30days_count, CASE WHEN DATEDIFF( (SELECT MIN(cma.market_date) FROM customer_markets_attended AS cma WHERE cma.customer_id = cp.customer_id AND cma.market_date > cp.market_date GROUP BY cma.customer_id), cp.market_date) <=30 THEN 1 ELSE 0 END AS purchased_again_within_30_days FROM farmers_market.customer_purchases AS cp GROUP BY cp.customer_id, cp.market_date ORDER BY cp.customer_id, cp.market_date 188 Chapter 12 ■ Creating Machine Learning Datasets Using SQL Figure 12.7 Figure 12.6 Chapter 12 ■ Creating Machine Learning Datasets Using SQL 189 Taking Things to the Next Level In this chapter, we have built datasets that could be used as inputs to train time series models and binary classification models. Sometimes you will be join- ing in data from additional tables to add columns to your dataset, and you’ll have to be careful not to change the granularity as you do so. In this case, we engineered features using only data in the customer_purchases table in the Farmer’s Market database, aggregating data from the same table in many ways. You can use the SQL you have learned in this book to engineer a wide variety of features, improving your model by providing it with many different signals to correlate with the target variable. Some people do feature engineering in their model-­building script or other software. Tools like the pandas package in Python do make certain types of feature engineering straightforward to include in your machine learning script. Benefits of conducting some feature engineering in", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 115 + }, + { + "text": "many different signals to correlate with the target variable. Some people do feature engineering in their model-­building script or other software. Tools like the pandas package in Python do make certain types of feature engineering straightforward to include in your machine learning script. Benefits of conducting some feature engineering in the SQL code as a separate step in your data pipeline include the ability to easily store your results in a database table to use repeatedly during training (without having to regenerate the calculated columns each time your script is run) or to share with others. Additionally, some types of summarization are more efficient to do in SQL at the point of data extraction from the database than in other coding environments. If you need it to run more quickly, you could ask an experienced data engineer to help make your SQL more computationally efficient, once you have it returning the results you want. Now that you know how to build your own dataset, you can provide them with a query that generates the results you need instead of having to explain the granularity and define every column. And you don’t have to rely on a data engineer to simply add a column to an existing dataset, since you can now read SQL that someone else has developed and modify it yourself. The next step after building the dataset will be conducting Exploratory Data Analysis (EDA) on it to better understand the relationship between your input features and the target variable. Then you will go through a training and test- ing process with the portion of the dataset that contains known outcomes from the past. Once your model is trained, you can feed in current data summarized using the same query, but without values in the target variable column, and have it predict what those values will be. Then, after evaluating your model’s performance, you’ll likely be right back here engineering more features or join- ing in more data in order to improve your model’s predictions! Exercises 1. Add a column to the final query in the chapter that counts how many markets were attended by each customer in the past 14 days. 190 Chapter 12 ■ Creating Machine Learning Datasets Using SQL 2. Add a column to the final query in the chapter that contains a 1 if the customer purchased an item that cost over $10, and a 0 if not. HINT: The calculation will follow the same form as the purchased_from_vendor_x flags. 3. Let’s say that the farmer’s market started a customer reward program that gave customers a market goods gift basket and branded reusable market bag when they had spent at least $200 total. Create a flag field (with a 1 or 0) that indicates whether the customer has reached this loyal customer status. HINT: One way to accomplish this involves modifying the CTE (WITH clause) to include purchase totals, and adding a column to the main query with a similar structure to the one that", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 116 + }, + { + "text": "flag field (with a 1 or 0) that indicates whether the customer has reached this loyal customer status. HINT: One way to accomplish this involves modifying the CTE (WITH clause) to include purchase totals, and adding a column to the main query with a similar structure to the one that calculates customer_markets_attended_count, to calculate a running total spent. C H A P T E R 191 13 In this chapter, I will walk through the development of datasets for answering different types of analytical questions. This involves combining multiple con- cepts from previous chapters into more complex queries and therefore is more advanced. Note that the example database doesn’t currently contain enough data with correlations for actually doing the analyses that would follow the dataset development, so we won’t be looking for trends in the output screenshots. The focus here is on how I would go about designing and building a dataset from our Farmer’s Market database using SQL to answer each of the following ana- lytical questions: ■ ■What factors correlate with fresh produce sales? ■ ■How do sales vary by customer zip code, market distance, and demo- graphic data? ■ ■How does product price distribution affect market sales? What Factors Correlate with Fresh Produce Sales? Let’s say we’re asked the analytical question “What factors are correlated with sales of fresh produce at the farmer’s market?” So what we’re being asked is to determine the relationships between a selection of different variables and a Analytical Dataset Development Examples 192 Chapter 13 ■ Analytical Dataset Development Examples subset of market product sales. That means from a data perspective that we’ll need to summarize different variables over periods of time and explore how sales during those same time periods change as each variable changes. For example, “As the number of different available products at the market increases, do sales of fresh produce go up or down?” is a question exploring the relationship between two variables: product variety and sales. If sales go up when the product variety goes up, then the two variables are positively cor- related. If sales go down when product variety goes up, then the two variables are negatively correlated. I could choose to summarize each value per week and then create a scatterplot of the weekly pairs of numbers to visualize the relationship between them, for example. To do that for a variety of variables, I’ll need to write a query that gen- erates a dataset with one row per market week containing weekly summaries of each value to be explored. I’ll first need to determine what products are considered “fresh produce,” then calculate sales of those products per week, and pull in other variables (factors) summarized per week to explore in relation to those sales. Some ideas for values to compare to sales include: product availability (number of vendors carrying the products, volume of inventory available for purchase, or special high-­demand product seasonal availability, to give a few examples), product cost, time of year/season, sales trends", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 117 + }, + { + "text": "per week to explore in relation to those sales. Some ideas for values to compare to sales include: product availability (number of vendors carrying the products, volume of inventory available for purchase, or special high-­demand product seasonal availability, to give a few examples), product cost, time of year/season, sales trends over time, and things that affect all sales at the market such as weather and the number of customers shopping at the market. First, I will look at all of the different product categories to determine which make the most sense to use to answer this question. The result of the following query is shown in Figure 13.1: SELECT * FROM farmers_market.product_category From the list in Figure 13.1, I can see that product category 1 is “Fresh Fruits & Vegetables,” which sounds like “fresh produce” to me. The “Plants & Flowers” and “Eggs & Meat” categories may contain products that the requester considers fresh produce, so I can generate a list of all products in categories 1, 5, and 6 and take this list to the requester to double-­check that category 1 contains the types of products they would like me to analyze sales for. If they request analysis of a list of products instead of an entire category, I might mention that if this analysis Figure 13.1 Chapter 13 ■ Analytical Dataset Development Examples 193 is meant to be repeated over time, we will need to check for product additions and changes to assess the included product list every time we run the report (where if we simply filter to a category, any product added to that category in the future will be automatically included). The output from this query is shown in Figure 13.2: SELECT * FROM farmers_market.product WHERE product_category_id IN (1, 5, 6) ORDER BY product_category_id We’ll assume that going with product category 1 was the correct guess. Now that I have the basic filter requirements, I can design the structure of my query. I know that I want some summary information about sales, which will need to come from the customer_purchases table. I’ll need data about the availability, which will come from the vendor_inventory table. And I’ll want some time-­ related information. Both the customer_purchases and vendor_inventory ­tables have market dates in them. Even though neither is directly related to the market_date_info table in the E-­R diagram, I can join it to them by market_date to pull in additional information about that date. Because this is a question about something related to sales over time, I’m going to start with the product sales part of the question, then join other information to the results of that query. I need to select the details needed to summarize sales per week for products in the “Fresh Fruits & Vegetables” category 1 by inner joining sales (customer_­ purchases) and products (product) by product_id. If we were looking at products in multiple categories, it might also be worth joining in the product_category table to get the category names, but for", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 118 + }, + { + "text": "sales per week for products in the “Fresh Fruits & Vegetables” category 1 by inner joining sales (customer_­ purchases) and products (product) by product_id. If we were looking at products in multiple categories, it might also be worth joining in the product_category table to get the category names, but for now, I’ll simplify and leave that out. Let’s look at the details, shown in Figure 13.3, first: Figure 13.2 194 Chapter 13 ■ Analytical Dataset Development Examples SELECT * FROM customer_purchases cp INNER JOIN product p ON cp.product_id = p.product_id WHERE p.product_category_id = 1 I used an INNER JOIN instead of a LEFT JOIN, because for this sales calcula- tion, I’m not interested in products that don’t have purchases. At this stage, it’s good to check the count of rows returned, to ensure it makes sense given your join selection. I also look at the details, to make sure I’m pulling the right data from the tables I meant to pull from, that I’m joining on the correct fields, and that my results are filtered the way I expect. Since I’ll be summarizing by week, I don’t need the transaction_time field, and I don’t currently have a need to know about the size of each product. We might eventually need the product quantity type and vendor information for when we start adding up how much product is available for purchase. However, do we need those fields from these tables? Total sales is the dependent variable—­ the value we’re trying to correlate different values with. If we wanted to get a count of vendors who had the product for sale, we don’t want to get that from the customer_purchases table, because there could be products available for sale that no one purchased, which means their existence wouldn’t be recorded in the customer_purchases table. So, we’ll want to get those pieces of data from the vendor_inventory table at a later step, and not from the customer_purchases table. I can join in the market_date_info table to get the week number, to make sum- marization easier, as well as other date-­related information such as the season and the weather. I decided to RIGHT JOIN it to the other tables, because I want to know whether there are market dates with no fresh produce sales at all, and the RIGHT JOIN will still pull in market dates with no corresponding records in the customer_purchases table, as shown in Figure 13.4: SELECT cp.market_date, cp.customer_id, cp.quantity, cp.cost_to_customer_per_qty, p.product_category_id, mdi.market_date, mdi.market_week, mdi.market_year, mdi.market_rain_flag, mdi.market_snow_flag FROM customer_purchases cp INNER JOIN product p ON cp.product_id = p.product_id RIGHT JOIN market_date_info mdi ON mdi.market_date = cp.market_date WHERE p.product_category_id = 1 Figure 13.3 Chapter 13 ■ Analytical Dataset Development Examples 195 Figure 13.4 196 Chapter 13 ■ Analytical Dataset Development Examples Chapter 13 ■ Analytical Dataset Development Examples 197 But look what happened. Even though I’m right joining the market_date_info table to the customer_purchases table, I’m not seeing any market dates that don’t have sales in this category, even though I know", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 119 + }, + { + "text": "13.4 196 Chapter 13 ■ Analytical Dataset Development Examples Chapter 13 ■ Analytical Dataset Development Examples 197 But look what happened. Even though I’m right joining the market_date_info table to the customer_purchases table, I’m not seeing any market dates that don’t have sales in this category, even though I know they exist in the data! This is a common SQL design error. You might think that the solution is to rearrange all of the joins, but if that’s the only change you make, you will still have the same issue. What’s happening is that our WHERE clause is filtering the results to only rows with a customer purchase from product category 1. So if there are no sales on a market date, there are no product categories associated with that date, so we are filtering it out, defeating the purpose of the RIGHT JOIN. (This is also a good reason to look at the data in each table before joining and write some quality control queries such as distinct counts of market dates, so you are aware if some expected values are missing after you join the tables together.) The solution to this filter issue is to put the product category filter in the JOIN ON clause instead of in the WHERE clause, which is something we haven’t covered previously. I can join to the product table on the product_id and product_category_id fields, and filter the product_category_id in the ON clause. This makes the filter only apply to the data from the product table (and now the customer_purchases table, since they’re inner joined), and not to the results set, the way the WHERE clause does. So now all of our market dates will be returned. I moved the market_date_info fields to appear first, and modified the join to include the filter. You can now see that we’re joining on the product_id and filtering the product_category_id in the ON section of the JOIN. Note that now there is no WHERE clause, but we are still filtering the results of one of the tables being joined into the dataset! The output of this query is displayed in Figure 13.5: SELECT mdi.market_date, mdi.market_week, mdi.market_year, mdi.market_rain_flag, mdi.market_snow_flag, cp.market_date, cp.customer_id, cp.quantity, cp.cost_to_customer_per_qty, p.product_category_id FROM customer_purchases cp INNER JOIN product p ON cp.product_id = p.product_id AND p.product_category_id = 1 RIGHT JOIN market_date_info mdi ON mdi.market_date = cp.market_date Figure 13.5 198 Chapter 13 ■ Analytical Dataset Development Examples Chapter 13 ■ Analytical Dataset Development Examples 199 Now we can see rows for market dates with no fresh produce purchases. I can summarize the customer purchases to one row per week to get sales per week by grouping on market_year and market_week, and we don’t need most of the other columns with additional details about the purchases, so we can remove them from our query. The sales calculation is the same one we used in Chapter 12, “Creating Machine Learning Datasets Using SQL,” with a slight addition. COALESCE is a function that returns the first non-­NULL value in", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 120 + }, + { + "text": "the other columns with additional details about the purchases, so we can remove them from our query. The sales calculation is the same one we used in Chapter 12, “Creating Machine Learning Datasets Using SQL,” with a slight addition. COALESCE is a function that returns the first non-­NULL value in a list of values. In this case, when the query returns a market date with no sales in product category 1, the weekly_category1_sales value would be NULL. If we want it to be 0 instead, we can use the syntax COALESCE([value 1], 0) which will return a 0 if “value 1” is NULL, and will otherwise return the cal- culated value. We wrap our SUM function in this COALESCE function, then ROUND the result of the COALESCE function to two digits after the decimal point. To summarize, in that final line before the FROM clause, we’re adding up the sales, converting the result to 0 if there are no sales, then rounding the numeric result to two digits. I’m also returning the MAX of the snow and rain flags, because if there was precipitation at either of the markets during the week, I want to return a 1 in this field. And I returned the minimum market_season value, so only one value is returned if a week happens to be split across two seasons. The updated result using the following query is displayed in Figure 13.6: SELECT mdi.market_year, mdi.market_week, MAX(mdi.market_rain_flag) AS market_week_rain_flag, MAX(mdi.market_snow_flag) AS market_week_snow_flag, MIN(mdi.market_min_temp) AS minimum_temperature, MAX(mdi.market_max_temp) AS maximum_temperature, MIN(mdi.market_season) AS market_season, ROUND(COALESCE(SUM(cp.quantity * cp.cost_to_customer_per_qty), 0), 2) AS weekly_category1_sales FROM customer_purchases cp INNER JOIN product p ON cp.product_id = p.product_id AND p.product_category_id = 1 RIGHT JOIN market_date_info mdi ON mdi.market_date = cp.market_date GROUP BY mdi.market_year, mdi.market_week Figure 13.6 200 Chapter 13 ■ Analytical Dataset Development Examples Chapter 13 ■ Analytical Dataset Development Examples 201 So now we have total sales by week. Some of the other aggregate values that could be added to this dataset include the number of vendors carrying products in the category, the volume of inventory available for purchase, and special high-­demand product seasonal availability. These are values that come from the vendor_inventory table, because I want to know what the vendors brought to market, regardless of whether people purchased the items. We can set up the query for vendor_inventory just like we did for customer_purchases, joining it to the product and market_date_info tables the same way, and filtering to product_category_id = 1 in the JOIN statement as we did previously. The results of this query are shown in Figure 13.7: SELECT mdi.market_date, mdi.market_year, mdi.market_week, vi.*, p.* FROM vendor_inventory vi INNER JOIN product p ON vi.product_id = p.product_id AND p.product_category_id = 1 RIGHT JOIN market_date_info mdi ON mdi.market_date = vi.market_date Removing the fields we don’t need for our weekly summary, we can narrow down the vendor_inventory table to keep the features we need for calculating the number of vendors (vendor_id), the number of products (product_id), and volume of products (quantity). We can also use the", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 121 + }, + { + "text": "mdi ON mdi.market_date = vi.market_date Removing the fields we don’t need for our weekly summary, we can narrow down the vendor_inventory table to keep the features we need for calculating the number of vendors (vendor_id), the number of products (product_id), and volume of products (quantity). We can also use the product_id to flag the existence of certain products. Let’s say that we suspect that when the sweet corn vendors are at the market, some customers come that don’t come at any other time of year, just to get the locally famous corn on the cob. We want to know if overall fresh produce sales go up during the weeks when corn is available, so we’ll create a product availability flag for product 16, sweet corn, called corn_available_flag, as shown in the following query and in Figure 13.8: SELECT mdi.market_year, mdi.market_week, COUNT(DISTINCT vi.vendor_id) AS vendor_count, COUNT(DISTINCT vi.product_id) AS unique_product_count, SUM(CASE WHEN p.product_qty_type = 'unit' THEN vi.quantity ELSE 0 END) AS unit_products_qty, SUM(CASE WHEN p.product_qty_type = 'lbs' THEN vi.quantity ELSE 0 END) AS bulk_products_lbs, ROUND(COALESCE(SUM(vi.quantity * vi.original_price), 0), 2) AS total_product_value, Continues 202 Chapter 13 ■ Analytical Dataset Development Examples MAX(CASE WHEN p.product_id = 16 THEN 1 ELSE 0 END) AS corn_available_flag FROM vendor_inventory vi INNER JOIN product p ON vi.product_id = p.product_id RIGHT JOIN market_date_info mdi ON mdi.market_date = vi.market_date GROUP BY mdi.market_year, mdi.market_week Now that I see these results, I realize that I would like to have a count of vendors selling and products available at the entire market, in addition to the product availability for product category 1. To avoid developing another query that will need to be joined in, I will remove the product_category_id filter and use CASE statements to create a set of fields that provides the same metrics, but only for products in the category. Then, the existing fields will turn into a count for all vendors and products at the market: SELECT mdi.market_year, mdi.market_week, COUNT(DISTINCT vi.vendor_id) AS vendor_count, COUNT(DISTINCT CASE WHEN p.product_category_id = 1 THEN vi.vendor_id ELSE NULL END) AS vendor_count_product_category1, COUNT(DISTINCT vi.product_id) AS unique_product_count, COUNT(DISTINCT CASE WHEN p.product_category_id = 1 THEN vi.product_id ELSE NULL END) AS unique_product_count_product_category1, SUM(CASE WHEN p.product_qty_type = 'unit' THEN vi.quantity ELSE 0 END) AS unit_products_qty, SUM(CASE WHEN p.product_category_id = 1 AND p.product_qty_type = 'unit' THEN vi.quantity ELSE 0 END) AS unit_products_qty_product_category1, SUM(CASE WHEN p.product_qty_type = 'lbs' THEN vi.quantity ELSE 0 END) AS bulk_products_lbs, SUM(CASE WHEN p.product_category_id = 1 AND p.product_qty_type = 'lbs' THEN vi.quantity ELSE 0 END) AS bulk_products_lbs_product_category1, ROUND(COALESCE(SUM(vi.quantity * vi.original_price), 0), 2) AS total_ product_value, ROUND(COALESCE(SUM(CASE WHEN p.product_category_id = 1 THEN vi.quantity * vi.original_price ELSE 0 END), 0), 2) AS total_product_value_product_ category1, MAX(CASE WHEN p.product_id = 16 THEN 1 ELSE 0 END) AS corn_available_ flag FROM vendor_inventory vi INNER JOIN product p ON vi.product_id = p.product_id RIGHT JOIN market_date_info mdi ON mdi.market_date = vi.market_date GROUP BY mdi.market_year, mdi.market_week (continued) Figure 13.7 Chapter 13 ■ Analytical Dataset Development Examples 203 204 Chapter 13 ■ Analytical Dataset Development Examples Figures 13.9, 13.10, and 13.11 display the columns resulting from this", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 122 + }, + { + "text": "INNER JOIN product p ON vi.product_id = p.product_id RIGHT JOIN market_date_info mdi ON mdi.market_date = vi.market_date GROUP BY mdi.market_year, mdi.market_week (continued) Figure 13.7 Chapter 13 ■ Analytical Dataset Development Examples 203 204 Chapter 13 ■ Analytical Dataset Development Examples Figures 13.9, 13.10, and 13.11 display the columns resulting from this query, with each figure compressing the width of the previously displayed columns so examples of all columns in the output can be shown. One easy quality check to do on the output here is to make sure that every field with the _category1 suffix has an equal or lower value than the corresponding field without the suffix, since the totals include product category 1, so the cate- gory value should never come out to be higher than the overall total. Now I can combine the results of these two queries. I’ll alias each of them in the WITH clause (CTE), then join the views: WITH my_customer_purchases AS ( SELECT mdi.market_year, mdi.market_week, MAX(mdi.market_rain_flag) AS market_week_rain_flag, MAX(mdi.market_snow_flag) AS market_week_snow_flag, MIN(mdi.market_min_temp) AS minimum_temperature, MAX(mdi.market_max_temp) AS maximum_temperature, MIN(mdi.market_season) AS market_season, ROUND(COALESCE(SUM(cp.quantity * cp.cost_to_customer_per_qty), 0), 2) AS weekly_category1_sales FROM customer_purchases cp INNER JOIN product p Figure 13.8 Chapter 13 ■ Analytical Dataset Development Examples 205 ON cp.product_id = p.product_id AND p.product_category_id = 1 RIGHT JOIN market_date_info mdi ON mdi.market_date = cp.market_date GROUP BY mdi.market_year, mdi.market_week ), my_vendor_inventory AS ( SELECT mdi.market_year, mdi.market_week, COUNT(DISTINCT vi.vendor_id) AS vendor_count, COUNT(DISTINCT CASE WHEN p.product_category_id = 1 THEN vi.vendor_id ELSE NULL END) AS vendor_count_product_category1, COUNT(DISTINCT vi.product_id) unique_product_count, COUNT(DISTINCT CASE WHEN p.product_category_id = 1 THEN vi.product_id ELSE NULL END) AS unique_product_count_product_category1, SUM(CASE WHEN p.product_qty_type = 'unit' THEN vi.quantity ELSE 0 END) AS unit_products_qty, SUM(CASE WHEN p.product_category_id = 1 AND p.product_qty_type = 'unit' THEN vi.quantity ELSE 0 END) AS unit_products_qty_product_category1, SUM(CASE WHEN p.product_qty_type <> 'unit' THEN vi.quantity ELSE 0 END) AS bulk_products_qty, SUM(CASE WHEN p.product_category_id = 1 AND p.product_qty_type <> 'unit' THEN vi.quantity ELSE 0 END) AS bulk_products_qty_product_category1, ROUND(COALESCE(SUM(vi.quantity * vi.original_price), 0), 2) AS total_product_value, ROUND(COALESCE(SUM(CASE WHEN p.product_category_id = 1 THEN vi.quantity * vi.original_price ELSE 0 END), 0), 2) AS total_product_value_product_category1, MAX(CASE WHEN p.product_id = 16 THEN 1 ELSE 0 END) AS corn_available_flag FROM vendor_inventory vi INNER JOIN product p ON vi.product_id = p.product_id RIGHT JOIN market_date_info mdi ON mdi.market_date = vi.market_date GROUP BY mdi.market_year, mdi.market_week ) SELECT * FROM my_vendor_inventory LEFT JOIN my_customer_purchases ON my_vendor_inventory.market_year = my_customer_purchases.market_year AND my_vendor_inventory.market_week = my_customer_purchases.market_week ORDER BY my_vendor_inventory.market_year, my_vendor_inventory.market_week Figure 13.9 206 Chapter 13 ■ Analytical Dataset Development Examples Figure 13.10 Chapter 13 ■ Analytical Dataset Development Examples 207 Figure 13.11 208 Chapter 13 ■ Analytical Dataset Development Examples Chapter 13 ■ Analytical Dataset Development Examples 209 I can alter the final SELECT statement to include the prior week’s product category 1 sales, too, because the prior week’s sales might be a good indicator of what to expect this week. I can use the LAG window function that was intro- duced in Chapter 7, ”Window Functions and Subqueries.” And I’ll go ahead and list all of the column names to avoid showing the duplicate market_year", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 123 + }, + { + "text": "the prior week’s sales might be a good indicator of what to expect this week. I can use the LAG window function that was intro- duced in Chapter 7, ”Window Functions and Subqueries.” And I’ll go ahead and list all of the column names to avoid showing the duplicate market_year and market_week columns that are in both CTEs which therefore show in the output twice when we use *. (This query should be preceded by the same CTE/WITH clause as the previous query, but removed here to save space.) SELECT mvi.market_year, mvi.market_week, mcp.market_week_rain_flag, mcp.market_week_snow_flag, mcp.minimum_temperature, mcp.maximum_temperature, mcp.market_season, mvi.vendor_count, mvi.vendor_count_product_category1, mvi.unique_product_count, mvi.unique_product_count_product_category1, mvi.unit_products_qty, mvi.unit_products_qty_product_category1, mvi.bulk_products_qty, mvi.bulk_products_qty_product_category1, mvi.total_product_value, mvi.total_product_value_product_category1, LAG(mcp.weekly_category1_sales, 1) OVER (ORDER BY mvi.market_year, mvi.market_week) AS previous_week_category1_sales, mcp.weekly_category1_sales FROM my_vendor_inventory mvi LEFT JOIN my_customer_purchases mcp ON mvi.market_year = mcp.market_year AND mvi.market_week = mcp.market_week ORDER BY mvi.market_year, mvi.market_week Now we have a detailed dataset summarized to one row per week that we could use to explore the relationships between the market weather, product availability, and fresh produce sales. Some of the columns in Figure 13.12 that were shown in previous figures have been condensed so the newly added LAG column is visible. Before you read this book, a query like this might have looked large and indecipherable, but now you know that it’s made by using various combina- tions of straightforward SQL statements you learned in other chapters, which are then combined into datasets with a progressively larger number of columns to use in analysis. Figure 13.12 210 Chapter 13 ■ Analytical Dataset Development Examples Chapter 13 ■ Analytical Dataset Development Examples 211 How Do Sales Vary by Customer Zip Code, Market Distance, and Demographic Data? We have a couple of core questions here. How do sales vary by customer zip code and distance from the market? Can we integrate demographic data into this analysis? We will need to pull together several pieces of information to answer these questions. We could group all sales by zip code, but we might get more meaningful answers if we group sales by customer, then look at the summary statistics and distributions of per-­customer sales totals by zip code. We have the 5-­digit zip (postal) code of each customer in the customer table, but we don’t have their full address or 9-­digit zip code, so the closest we can get to a “distance from the market” calculation is by calculating the distances between the market location and some location associated with each zip code, such as a centrally located latitude and longitude (keeping in mind that zip codes can be any kind of shape, so the “center” of the area isn’t a great repre- sentation of the location of most residences in the postal code). One source of latitudes and longitudes by zip is https://public.opendatasoft.com/explore/ dataset/us-­zip-­code-­latitude-­and-­longitude/table/?q=22821. If we import this data into our database, we can join it to our queries to add a latitude and longitude per zip code to our dataset. There is also plenty of demographic data online related to zip codes, such", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 124 + }, + { + "text": "source of latitudes and longitudes by zip is https://public.opendatasoft.com/explore/ dataset/us-­zip-­code-­latitude-­and-­longitude/table/?q=22821. If we import this data into our database, we can join it to our queries to add a latitude and longitude per zip code to our dataset. There is also plenty of demographic data online related to zip codes, such as census data summarized by ZCTAs (Zip Code Tabulation Areas), so we can pull in age distributions, wealth statistics, and other demographics summa- rized by zip. For this example, I decided to summarize the sales per customer first, then join in the demographic data to every customer’s record, even though it’s not customer-­specific. Then, if I were to train a model based on the behavior of customers, I could use the zip code data as an input into the customer-­level model, or as a dimension to summarize the other per-­customer fields by for reporting purposes. So I’ll first summarize sales per customer, including for how long they’ve been a customer, the count of market dates at which they’ve made a purchase, and the total each customer has spent to date. I’ll also join the purchase summary to the customer table, in order to include each customer’s zip code. The output of the following query is shown in Figure 13.13: SELECT c.customer_id, c.customer_zip, DATEDIFF(MAX(market_date), MIN(market_date)) customer_duration_days, COUNT(DISTINCT market_date) number_of_markets, ROUND(SUM(quantity * cost_to_customer_per_qty), 2) total_spent, Continues 212 Chapter 13 ■ Analytical Dataset Development Examples ROUND(SUM(quantity * cost_to_customer_per_qty) / COUNT(DISTINCT market_date), 2) average_spent_per_market FROM farmers_market.customer c LEFT JOIN farmers_market.customer_purchases cp ON cp.customer_id = c.customer_id GROUP BY c.customer_id Note that because of the nature of the sample data, the customers in this case have pretty similar values in this output, which isn’t typical when you look at data collected from real-­world scenarios. I have loaded some example demographic data into a new table in the data- base I called zip_data, which is shown in Figure 13.14: SELECT * FROM zip_data I will join this table to my existing query by the zip code fields, so every cus- tomer in a zip code will have the same demographic data added to their record, as shown in Figure 13.15. I condensed the columns that were already shown in Figure 13.13 in order to fit the newly added columns in view: SELECT c.customer_id, DATEDIFF(MAX(market_date), MIN(market_date)) AS customer_duration_days, COUNT(DISTINCT market_date) AS number_of_markets, ROUND(SUM(quantity * cost_to_customer_per_qty), 2) AS total_spent, ROUND(SUM(quantity * cost_to_customer_per_qty) / COUNT(DISTINCT market_date), 2) AS average_spent_per_market, c.customer_zip, z.median_household_income AS zip_median_household_income, z.percent_high_income AS zip_percent_high_income, z.percent_under_18 AS zip_percent_under_18, z.percent_over_65 AS zip_percent_over_65, z.people_per_sq_mile AS zip_people_per_sq_mile, z.latitude, z.longitude FROM farmers_market.customer c LEFT JOIN farmers_market.customer_purchases cp ON cp.customer_id = c.customer_id LEFT JOIN zip_data z ON c.customer_zip = z.zip_code_5 GROUP BY c.customer_id (continued) Figure 13.13 Figure 13.14 Chapter 13 ■ Analytical Dataset Development Examples 213 214 Chapter 13 ■ Analytical Dataset Development Examples One part of the analysis question was about distance from the market, which the data we have doesn’t exactly tell us. There is a calculation for distance between two latitudes and longitudes that I found on Dayne Batten’s blog at", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 125 + }, + { + "text": "Examples 213 214 Chapter 13 ■ Analytical Dataset Development Examples One part of the analysis question was about distance from the market, which the data we have doesn’t exactly tell us. There is a calculation for distance between two latitudes and longitudes that I found on Dayne Batten’s blog at https://daynebatten.com/2015/09/latitude-­longitude-­distance-­sql/. If the Farmer’s Market is at the coordinates 38.4463, –78.8712, the calculation for distance between the latitude and longitude fields of a record in the dataset and the Farmer’s Market location is: ROUND(2 * 3961 * ASIN(SQRT(POWER(SIN(RADIANS((latitude -­ 38.4463) / 2)),2) + COS(RADIANS(38.4463)) * COS(RADIANS(latitude)) * POWER((SIN(RADIANS((longitude -­ -­ 78.8712) / 2))), 2)))) Don’t worry about understanding how that calculation works, just know that it returns the distance between two pairs of latitude and longitude rounded to the nearest mile. Replacing the latitude and longitude fields in our query with this calculation, we now have the query: SELECT c.customer_id, DATEDIFF(MAX(market_date), MIN(market_date)) AS customer_duration_days, COUNT(DISTINCT market_date) AS number_of_markets, ROUND(SUM(quantity * cost_to_customer_per_qty), 2) AS total_spent, ROUND(SUM(quantity * cost_to_customer_per_qty) / COUNT(DISTINCT market_date), 2) AS average_spent_per_market, c.customer_zip, z.median_household_income AS zip_median_household_income, z.percent_high_income AS zip_percent_high_income, z.percent_under_18 AS zip_percent_under_18, z.percent_over_65 AS zip_percent_over_65, z.people_per_sq_mile AS zip_people_per_sq_mile, ROUND(2 * 3961 * ASIN(SQRT(POWER(SIN(RADIANS((z.latitude -­ 38.4463) / 2)),2) + COS(RADIANS(38.4463)) * COS(RADIANS(z.latitude)) * POWER((SIN(RADIANS((z.longitude -­ -­ 78.8712) / 2))), 2)))) AS zip_miles_from_market FROM farmers_market.customer AS c LEFT JOIN farmers_market.customer_purchases AS cp ON cp.customer_id = c.customer_id LEFT JOIN zip_data AS z ON c.customer_zip = z.zip_code_5 GROUP BY c.customer_id Again, I have condensed the fields shown in Figures 13.13 and 13.15 in order to display the new fields fully in Figure 13.16. This will allow me to analyze customer metrics like total amount spent and number of markets attended by customer zip code or by calculated distance from the market. I could now build a scatterplot of zip_miles_from_market Figure 13.15 Figure 13.16 Chapter 13 ■ Analytical Dataset Development Examples 215 216 Chapter 13 ■ Analytical Dataset Development Examples versus total_spent (though all of the customer data points in each zip code would overlap on the distance axis, so I would have to add some jitter or size the dots in order to indicate how many people were represented by that point). With the newly added information per zip, I could create a rural versus urban flag based on the population density of the zip code and look at ­customer behavior based on that value. I could assess customer longevity by zip code, or look at the distributions of amount spent or customer durations by zip code. I could correlate the percent of high-­income residents per zip code with the cus- tomers’ total amount spent at the market. There are a wide variety of analyses I could do with just this dataset. Some other ideas of additional customer summary fields that could be added to this dataset if we also joined in details about the products include total pur- chases by product category, top vendor purchased from, number of different vendors purchased from, number of different products purchased, most frequent time of day", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 126 + }, + { + "text": "other ideas of additional customer summary fields that could be added to this dataset if we also joined in details about the products include total pur- chases by product category, top vendor purchased from, number of different vendors purchased from, number of different products purchased, most frequent time of day of purchase, etc. Here is one example usage of this dataset where I put the preceding query into a CTE and select from it to get the count of customers and the average total spent per customer for each zip code. Note that in this case, the zip_miles_from_market is the same per row, so I could take the min, max, or average and would get the same value returned because all of the values are identical per zip, which we’re grouping by. I could also add zip_miles_from_market to the GROUP BY statement, or even change the structure of my query so I’m only joining in the zip code data at this point, when I’m summarizing the customers by zip, instead of joining the zip code data to the query inside the CTE. For the purposes of summary, either approach is fine. The output of the following approach is shown in Figure 13.17: WITH customer_and_zip_data AS ( SELECT c.customer_id, DATEDIFF(MAX(market_date), MIN(market_date)) AS customer_duration_days, COUNT(DISTINCT market_date) AS number_of_markets, ROUND(SUM(quantity * cost_to_customer_per_qty), 2) AS total_spent, ROUND(SUM(quantity * cost_to_customer_per_qty) / COUNT(DISTINCT market_date), 2) AS average_spent_per_market, c.customer_zip, z.median_household_income AS zip_median_household_income, z.percent_high_income AS zip_percent_high_income, z.percent_under_18 AS zip_percent_under_18, z.percent_over_65 AS zip_percent_over_65, z.people_per_sq_mile AS zip_people_per_sq_mile, Chapter 13 ■ Analytical Dataset Development Examples 217 ROUND(2 * 3961 * ASIN(SQRT(POWER(SIN(RADIANS((z.latitude -­ 38.4463) / 2)),2) + COS(RADIANS(38.4463)) * COS(RADIANS(z.latitude)) * POWER((SIN(RADIANS((z.longitude -­ - ­78.8712) / 2))), 2)))) AS zip_miles_from_market FROM farmers_market.customer AS c LEFT JOIN farmers_market.customer_purchases AS cp ON cp.customer_id = c.customer_id LEFT JOIN zip_data AS z ON c.customer_zip = z.zip_code_5 GROUP BY c.customer_id ) SELECT cz.customer_zip, COUNT(cz.customer_id) AS customer_count, ROUND(AVG(cz.total_spent)) AS average_total_spent, MIN(cz.zip_miles_from_market) AS zip_miles_from_market FROM customer_and_zip_data AS cz GROUP BY cz.customer_zip One caveat to this analysis that we would want to inform the recipient about is that we only store the customer’s current zip code in this database, so if a cus- tomer used to live in a different zip code, that past connection is lost, and all of their purchase history is now associated with the current zip code on their cus- tomer record. To maintain the changes in zip code over time, we would need to add another table to the database in which to store the customer zip code history. How Does Product Price Distribution Affect Market Sales? What if a manager of the market asks, “What does our distribution of product prices look like at the market? Are our low-­priced items or high-­priced items generating more sales for the market?” You might have guessed that in order to answer these questions, I’m going to be using window functions. Figure 13.17 218 Chapter 13 ■ Analytical Dataset Development Examples One clarifying question I would have for the requester in this case before trying to answer these", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 127 + }, + { + "text": "more sales for the market?” You might have guessed that in order to answer these questions, I’m going to be using window functions. Figure 13.17 218 Chapter 13 ■ Analytical Dataset Development Examples One clarifying question I would have for the requester in this case before trying to answer these questions is related to time, because I know that the dis- tribution of prices can change over time, and the answer to the second question might have changed at some point as well. So, should we answer these ques- tions only for the most recent market season? Compare year over year? Or just look at all sales for the entire history we have tracked, ignoring any possible changes over time? Let’s say that the requester replied to clarify that they want to look at prod- uct price distributions for each market season over time (because the types of products sold can be very different in the heat of summer versus at the winter holidays, for example, as well as changing over the years). So first, I want to get the product pricing details prior to completing the level of summarization required to answer the questions. The first step in this analysis is to get raw data on the price per product per market date. But that seemingly simple task then raises another question: What do we mean by “product”? Is each product_id in the database a product? Like “Carrots” sold by weight? Or do the products differ enough by vendor that I should consider each product_id sold by each vendor as a separate “product”? We were asked to look at the distribution of product prices over time, and dif- ferent vendors do charge different amounts for the same products if we go by product_id, so I will choose to look at the average price per product per vendor per season. I will start with the original_price per product specified by the vendor in the vendor_inventory table, and won’t consider special discounts given to customers, which would appear in the customer_purchases table. This query’s results are shown in Figure 13.18: SELECT p.product_id, p.product_name, p.product_category_id, p.product_qty_type, vi.vendor_id, vi.market_date, SUM(vi.quantity), AVG(vi.original_price) FROM product AS p LEFT JOIN vendor_inventory AS vi ON vi.product_id = p.product_id GROUP BY p.product_id, p.product_name, p.product_category_id, p.product_qty_type, vi.vendor_id, vi.market_date Figure 13.18 Chapter 13 ■ Analytical Dataset Development Examples 219 220 Chapter 13 ■ Analytical Dataset Development Examples Next, since it was determined that we’re looking at prices per season over time, I need to pull in the market_season from the market_date_info table. I can also use the year so we’re talking about seasons over time. And here’s another challenge I can anticipate might arise later when I’m building the reports: The seasons are strings, so how do I know what order they should go in when sort- ing by season and year? I will keep the minimum month value in the dataset so it can later be used to sort the seasons in the correct order. See Figure 13.19 for", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 128 + }, + { + "text": "The seasons are strings, so how do I know what order they should go in when sort- ing by season and year? I will keep the minimum month value in the dataset so it can later be used to sort the seasons in the correct order. See Figure 13.19 for the output of this query: SELECT p.product_id, p.product_name, p.product_category_id, p.product_qty_type, vi.vendor_id, MIN(MONTH(vi.market_date)) AS month_market_season_sort, mdi.market_season, mdi.market_year, SUM(vi.quantity) AS quantity_available, AVG(vi.original_price) AS avg_original_price FROM product AS p LEFT JOIN vendor_inventory AS vi ON vi.product_id = p.product_id LEFT JOIN market_date_info AS mdi ON vi.market_date = mdi.market_date GROUP BY p.product_id, p.product_name, p.product_category_id, p.product_qty_type, vi.vendor_id, mdi.market_year, mdi.market_season One thing you might have noticed is that my attempt to pull in a sortable value to order the market seasons has resulted in multiple month_market_ season_sort values per season, because we are getting the minimum month per season per product, and some products aren’t offered in the earliest month of each season. These differing values could have detrimental effects to our results later on, depending on how we use the sorting field, so we’ll have to be careful how items group and sort with this value involved. We could also use a window function to get the minimum month per market_season across all rows for that season, ignoring the product_id values, which we’ll switch to in the next version of the query. Figure 13.19 Chapter 13 ■ Analytical Dataset Development Examples 221 222 Chapter 13 ■ Analytical Dataset Development Examples At this point, I also realized that I do need to pull in the customer_purchases data, because the second question is about sales, and the vendor_inventory table only contains available items, not items sold or the total amount gener- ated by sales of each item. So I’ll join that table in and use it to calculate the aggregate sales data for the second question. Note that we need to join on all three matching keys: product_id, vendor_id, and market_date, as shown here, and reflected in the results in Figure 13.20: SELECT p.product_id, p.product_name, p.product_category_id, p.product_qty_type, vi.vendor_id, MIN(MONTH(vi.market_date)) OVER (PARTITION BY market_season) AS month_market_season_sort, mdi.market_season, mdi.market_year, AVG(vi.original_price) AS avg_original_price, SUM(cp.quantity) AS quantity_sold, SUM(cp.quantity * cp.cost_to_customer_per_qty) AS total_sales FROM product AS p LEFT JOIN vendor_inventory AS vi ON vi.product_id = p.product_id LEFT JOIN market_date_info AS mdi ON vi.market_date = mdi.market_date LEFT JOIN customer_purchases AS cp ON vi.product_id = cp.product_id AND vi.vendor_id = cp.vendor_id AND vi.market_date = cp.market_date GROUP BY p.product_id, p.product_name, p.product_category_id, p.product_qty_type, vi.vendor_id, mdi.market_year, mdi.market_season It would be good at this point to pick a few products and look through the detailed availability and purchase data, to quality check these summarized results before continuing. Figure 13.20 Chapter 13 ■ Analytical Dataset Development Examples 223 224 Chapter 13 ■ Analytical Dataset Development Examples Now that I have a sale price of each item (the average price each vendor offered each product for each season), I could start exploring the distribution of prices. When developing these queries, I actually tried several different approaches at this point and landed on", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 129 + }, + { + "text": "■ Analytical Dataset Development Examples Now that I have a sale price of each item (the average price each vendor offered each product for each season), I could start exploring the distribution of prices. When developing these queries, I actually tried several different approaches at this point and landed on this one, partly because the small number of products in the database can expose an issue with using NTILEs: two different items of the same price can end up in different NTILEs if the number of NTILEs you choose splits the set at that point. After attempting multiple NTILE number options, I realized that if I wanted to end up with high and low price points in order to answer the second question, I probably shouldn’t be ranking the products anyway. I should be ranking the prices. In that case, I decided to modify my approach to group the records by market_year, market_season, and original_price. I used an NTILE value of 3, which then gives me groupings of the top, middle, and bottom 1/3 of prices. I can then summarize the sales of products that fall into each of these price points. Here’s the query that creates three price groupings per season: SELECT mdi.market_season, mdi.market_year, MIN(MONTH(vi.market_date)) OVER (PARTITION BY market_season) AS month_market_season_sort, vi.original_price, NTILE(3) OVER (PARTITION BY market_year, market_season ORDER BY original_price) AS price_ntile, NTILE(3) OVER (PARTITION BY market_year, market_season ORDER BY original_price DESC) AS price_ntile_desc, COUNT(DISTINCT CONCAT(vi.product_id, vi.vendor_id)) product_count, SUM(cp.quantity) AS quantity_sold, SUM(cp.quantity * cp.cost_to_customer_per_qty) AS total_sales FROM product AS p LEFT JOIN vendor_inventory AS vi ON vi.product_id = p.product_id LEFT JOIN market_date_info AS mdi ON vi.market_date = mdi.market_date LEFT JOIN customer_purchases AS cp ON vi.product_id = cp.product_id AND vi.vendor_id = cp.vendor_id AND vi.market_date = cp.market_date WHERE market_year IS NOT NULL GROUP BY mdi.market_year, mdi.market_season, vi.original_price The results of this query are shown in Figure 13.21. Figure 13.21 Chapter 13 ■ Analytical Dataset Development Examples 225 226 Chapter 13 ■ Analytical Dataset Development Examples Note that the price_ntile break points vary by season. In Summer 2019 and Summer 2020, a $4.00 item is in the second of the three NTILE groups, so it’s in the middle price grouping. In Spring, the distribution of product prices changes, so a $4.00 item is in price_ntile 1, or the low price grouping. Also note that I created a column using the NTILE window function with the same number of groups as price_ntile, but sorting the price descend- ing. This way, if you are using the output and don’t know how many groups there are, you can filter to price_ntile = 1 to get the lowest price group, and price_ntile_desc = 1 to get the highest price group. Another thing I want to point out in the previous query is that I didn’t COUNT(DISTINCT product_id) values, but first concatenated product_id and vendor_id and did a distinct count of the combined values. The reason is because of how we’re defining “product,” where the same product_id sold by different vendors, possibly for different prices, is", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 130 + }, + { + "text": "point out in the previous query is that I didn’t COUNT(DISTINCT product_id) values, but first concatenated product_id and vendor_id and did a distinct count of the combined values. The reason is because of how we’re defining “product,” where the same product_id sold by different vendors, possibly for different prices, is considered a different product for our purposes. One caveat with these results is that we’re summing up different types of quantities, so we’re counting an ounce, pound, or unit product as “an item sold.” So our quantity isn’t exactly apples-­to-­apples across seasons, but gives us a quick sales volume measure for rough comparison. The output in Figure 13.21 also illustrates why I wanted to create a month_market_season_sort, because alphabetically, the seasons sort out of order as “Late Fall,” “Spring,” and “Summer.” We will make use of the sort values in the next query. Now we’ll use the previous query as a CTE and summarize it: WITH product_prices AS ( SELECT mdi.market_season, mdi.market_year, MIN(MONTH(vi.market_date)) OVER (PARTITION BY market_season) AS month_market_season_sort, vi.original_price, NTILE(3) OVER (PARTITION BY market_year, market_season ORDER BY original_price) AS price_ntile, NTILE(3) OVER (PARTITION BY market_year, market_season ORDER BY original_price DESC) AS price_ntile_desc, COUNT(DISTINCT CONCAT(vi.product_id, vi.vendor_id)) AS product_count, SUM(cp.quantity) AS quantity_sold, SUM(cp.quantity * cp.cost_to_customer_per_qty) AS total_sales FROM product AS p LEFT JOIN vendor_inventory AS vi ON vi.product_id = p.product_id LEFT JOIN market_date_info AS mdi ON vi.market_date = mdi.market_date Chapter 13 ■ Analytical Dataset Development Examples 227 LEFT JOIN customer_purchases AS cp ON vi.product_id = cp.product_id AND vi.vendor_id = cp.vendor_id AND vi.market_date = cp.market_date WHERE market_year IS NOT NULL GROUP BY mdi.market_year, mdi.market_season, vi.original_price ) SELECT market_year, market_season, price_ntile, SUM(product_count) AS product_count, SUM(quantity_sold) AS quantity_sold, MIN(original_price) AS min_price, MAX(original_price) AS max_price, SUM(total_sales) AS total_sales FROM product_prices GROUP BY market_year, market_season, price_ntile ORDER BY market_year, month_market_season_sort, price_ntile In Figure 13.22, you’ll see that we’re now sorting the seasons in the correct order (even though we’re not outputting the sort value), and we’re displaying the minimum and maximum price per grouping, as well as the total_sales. This gives us the data to answer the second question, and we can see that with this sample data, the total_sales is highest for price_ntile 3 in every season, so the higher-­priced items are generating the most sales for the market. Figure 13.22 228 Chapter 13 ■ Analytical Dataset Development Examples Hopefully these dataset creation walkthroughs have given you a sense of the variety of ways you can combine and summarize data to create a dataset to answer analytical questions, and the kinds of clarifying questions an expe- rienced analyst might ask while going through this process. Keep in mind that each dataset can now be refreshed to pull in the latest data by rerunning the queries, and the results can be reused to answer many questions, not just the ones initially asked! C H A P T E R 229 14 We have covered many aspects of developing datasets for machine learning that involve selecting data from a database and preparing it for machine learning models,", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 131 + }, + { + "text": "and the results can be reused to answer many questions, not just the ones initially asked! C H A P T E R 229 14 We have covered many aspects of developing datasets for machine learning that involve selecting data from a database and preparing it for machine learning models, but what do you do once you have designed your query and are ready to start analyzing the results? Your SQL editor will often allow you to write the results of your query to a CSV file to be imported into Business Intelligence (BI) software such as Tableau or machine learning scripts in a language like Python. However, sometimes for data governance, data security, teamwork, or file size and processing speed purposes, it is preferable to store the dataset within the database. In this chapter, we’ll cover some types of SQL queries beyond SELECT state- ments, such as INSERT statements, which allow you to store the results of your query in a new table in the database. Storing SQL Datasets as Tables and Views In most databases, you can store the results of a query as either a table or a view. Storing results as a table takes a snapshot of whatever the results are at the time the query is run and saves the data returned as a new table object, or as new rows appended to an existing table, depending on how you write your SQL statement. A database view instead stores the SQL itself and runs it on-­demand when you write a query that references the name of the view, to dynamically Storing and Modifying Data 230 Chapter 14 ■ Storing and Modifying Data generate a new dataset based on the state of the referenced database objects at the time you run the query. (You may have also heard of the term materialized view, which is more like a stored snapshot and is not what I’m referring to here.) If you have database storage space available, permissions to create tables or insert records into tables in your database, and it is not cost-­prohibitive to do so, it can be good practice to store snapshots of the datasets you are using in your machine learning applications. You can check with your database administrator to determine whether you can and should create and modify tables, and which schema(s) you have permission to write to. When you’re iteratively testing various combinations of fields and parame- ters in your machine learning algorithm, you’ll want to test multiple different approaches with the same static dataset, so you can be sure your input data isn’t changing each time you run your script by writing the results to a table. You might also decide to store a copy of the dataset for later reference if the dataset you’re querying could change over time and you want to keep a record of the exact values that were run through your model at the time you ran it. One way to store the results of a", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 132 + }, + { + "text": "to store a copy of the dataset for later reference if the dataset you’re querying could change over time and you want to keep a record of the exact values that were run through your model at the time you ran it. One way to store the results of a query is to use a CREATE TABLE statement. The syntax is CREATE TABLE [schema_name].[new_table_name] AS ( [your query here] ) As with the SELECT statements, the indentation and line breaks in these queries don’t matter to the database and are just used to format for readability. The table name used in a CREATE TABLE statement must be new and unique within the schema. If you try to run the same CREATE TABLE statement twice in a row, you will get an error stating that the table already exists. Once you create the table, you can query it like any other table or view, referencing the new name you gave it. If you created a table by accident or want to re-­create it with a different name or definition, you can DROP the table. WARNING Be very careful when using the DROP TABLE statement, or you might accidentally delete something that should not have been be deleted! Depending on the database settings and backup frequency, the data you delete may not be recover- able! I usually ensure that I am only granted database permissions to create and drop tables in a personal schema, which is separate from the schema used to run applica- tions or where tables that others are using are stored, so I can’t accidentally delete a table I did not create or that is used in a production application. The syntax for dropping a table is simply: DROP TABLE [schema_name].[table_name] Chapter 14 ■ Storing and Modifying Data 231 So, to create, select from, and drop a table that contains a snapshot of the data that is currently in the Farmer’s Market database product table, filtered to prod- ucts with a quantity type “unit,” run the following three queries in sequence: CREATE TABLE farmers_market.product_units AS ( SELECT * FROM farmers_market.product WHERE product_qty_type = \"unit\" ) ; SELECT * FROM farmers_market.product_units ; DROP TABLE farmers_market.product_units ; The semicolons are used to separate multiple queries in the same file. TIP If you don’t want to accidentally run a DROP TABLE statement that is included in a file with other SQL statements (since many SQL editors have a “run all” command), com- ment it out immediately after running it, and save the file with that query commented out, so you don’t accidentally run it the next time you open the file and drop a table you didn’t intend to! In MySQL Workbench, you can comment out code by preceding each line with two dashes and a space or by surrounding a block of code with /* and */. Database views are created and dropped the same exact way as tables, though when you create a view, you are not actually storing the", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 133 + }, + { + "text": "you can comment out code by preceding each line with two dashes and a space or by surrounding a block of code with /* and */. Database views are created and dropped the same exact way as tables, though when you create a view, you are not actually storing the data, but storing the query to be run when you query the view. So when you drop a view, you are not actually deleting any data, since the data isn’t stored; you are just dropping the named reference to the query: CREATE VIEW farmers_market.product_units_vw AS ( SELECT * FROM farmers_market.product WHERE product_qty_type = \"unit\" ) ; SELECT * FROM farmers_market.product_units_vw ; DROP VIEW farmers_market.product_units_vw ; Note that some database systems, like SQL Server, support a SELECT INTO syntax, which operates much like the CREATE TABLE statement previously 232 Chapter 14 ■ Storing and Modifying Data ­demonstrated, and is often used to create backups of existing tables. Check your database’s documentation online to determine which syntax to use. Adding a Timestamp Column When you create or modify a database table, you might want to keep a record of when each row in the table was created or last modified. You can do this by adding a timestamp column to your CREATE TABLE or UPDATE statement. The syntax for creating a timestamp varies by database system, but in MySQL, the function that returns the current date and time is called CURRENT_TIMESTAMP. You can give the timestamp column an alias like any calculated column. Keep in mind that the timestamp is generated by the database server, so if the database is in another time zone, the timestamp returned by the function may be different than the current time at your physical location. Many databases use Coordinated Universal Time (UTC) as their default timestamp time, which is a global time standard that is synchronized using atomic clocks, aligns in hour offset with the Greenwich Mean Time time zone, and doesn’t change for Daylight Savings Time. Eastern Standard Time, which is observed in the Eastern Time Zone in North America during the winter, can be signified as UTC-­05:00, meaning it is five hours behind UTC. Eastern Daylight Time, observed during the summer, has an offset of UTC-­04:00, because the Eastern Time Zone observes Daylight Savings Time and shifts by an hour, while UTC does not shift. You can see how time zone math can quickly get complicated, which is why many databases simplify by using a standard UTC clock time instead of a developer’s local time. We can modify the preceding CREATE TABLE example to include a timestamp column as follows: CREATE TABLE farmers_market.product_units AS ( SELECT p.*, CURRENT_TIMESTAMP AS snapshot_timestamp FROM farmers_market.product AS p WHERE product_qty_type = \"unit\" ) Example output from this query is shown in Figure 14.1. Figure 14.1 Chapter 14 ■ Storing and Modifying Data 233 Inserting Rows and Updating Values in Database Tables If you want to modify data in an existing database table, you can use an INSERT statement", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 134 + }, + { + "text": "WHERE product_qty_type = \"unit\" ) Example output from this query is shown in Figure 14.1. Figure 14.1 Chapter 14 ■ Storing and Modifying Data 233 Inserting Rows and Updating Values in Database Tables If you want to modify data in an existing database table, you can use an INSERT statement to add a new row or an UPDATE statement to modify an existing row of data in a table. In this chapter, we’re specifically inserting results of a query into another table, which is a specific kind of INSERT statement called INSERT INTO SELECT. The syntax is INSERT INTO [schema_name].[table_name] ([comma-­separated list of column names]) [your SELECT query here] So if we wanted to add rows to our product_units table created earlier, we would write: INSERT INTO farmers_market.product_units (product_id, product_name, product_size, product_category_id, product_qty_type, snapshot_timestamp) SELECT product_id, product_name, product_size, product_category_id, product_qty_type, CURRENT_TIMESTAMP FROM farmers_market.product AS p WHERE product_id = 23 It is important that the columns in both queries are in the same order. The corresponding fields may not have identical names, but the system will attempt to insert the returned values from the SELECT statement in the column order listed in parentheses. Now when we query the product_units table, we’ll have a snapshot of the same product row at two different times, as shown in Figure 14.2. If you make a mistake when inserting a row and want to delete it, the syntax is simply DELETE FROM [schema_name].[table_name] WHERE [set of conditions that uniquely identifies the row] Figure 14.2 234 Chapter 14 ■ Storing and Modifying Data You may want to start with SELECT * instead of DELETE so you can see what rows will be deleted before running the DELETE statement! The product_id and snapshot_timestamp uniquely identify rows in the product_units table, so we can run the following statement to delete the row added by our previous INSERT INTO: DELETE FROM farmers_market.product_units WHERE product_id = 23 AND snapshot_timestamp = '2021-­04-­18 00:49:24' Sometimes you want to update a value in an existing row instead of inserting a totally new row. The syntax for an UPDATE statement is as follows: UPDATE [schema_name].[table_name] SET [column_name] = [new value] WHERE [set of conditions that uniquely identifies the rows you want to change] Let’s say that you’ve already entered all of the farmer’s market vendor booth assignments for the next several months, but vendor 4 informs you that they can’t make it on October 10, so you decide to upgrade vendor 8 to vendor 4’s booth, which is larger and closer to the entrance, for the day. Before making any changes, let’s snapshot the existing vendor booth assign- ments, along with the vendor name and booth type, into a new table using the following SQL: CREATE TABLE farmers_market.vendor_booth_log AS ( SELECT vba.*, b.booth_type, v.vendor_name, CURRENT_TIMESTAMP AS snapshot_timestamp FROM farmers_market.vendor_booth_assignments vba INNER JOIN farmers_market.vendor v ON vba.vendor_id = v.vendor_id INNER JOIN farmers_market.booth b ON vba.booth_number = b.booth_number WHERE market_date >= '2020-­10-­01' ) Selecting all records from this new log table produces the results shown in", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 135 + }, + { + "text": "SQL: CREATE TABLE farmers_market.vendor_booth_log AS ( SELECT vba.*, b.booth_type, v.vendor_name, CURRENT_TIMESTAMP AS snapshot_timestamp FROM farmers_market.vendor_booth_assignments vba INNER JOIN farmers_market.vendor v ON vba.vendor_id = v.vendor_id INNER JOIN farmers_market.booth b ON vba.booth_number = b.booth_number WHERE market_date >= '2020-­10-­01' ) Selecting all records from this new log table produces the results shown in Figure 14.3. Chapter 14 ■ Storing and Modifying Data 235 To update vendor 8’s booth assignment, we can run the following SQL: UPDATE farmers_market.vendor_booth_assignments SET booth_number = 7 WHERE vendor_id = 8 and market_date = '2020-­10-­10' And we can delete vendor 4’s booth assignment with the following SQL: DELETE FROM farmers_market.vendor_booth_assignments WHERE vendor_id = 4 and market_date = '2020-­10-­10' Now, when we query the vendor_booth_assignments table, there is no record that vendor 4 had a booth assignment on that date, or that vendor 8’s booth assignment used to be different. But we do have a record of the previous assign- ments in the vendor_booth_log we created! Now we can insert new records into the log table to record the latest changes: INSERT INTO farmers_market.vendor_booth_log (vendor_id, booth_number, market_date, booth_type, vendor_name, snapshot_timestamp) SELECT vba.vendor_id, vba.booth_number, vba.market_date, b.booth_type, v.vendor_name, CURRENT_TIMESTAMP AS snapshot_timestamp FROM farmers_market.vendor_booth_assignments vba INNER JOIN farmers_market.vendor v ON vba.vendor_id = v.vendor_id INNER JOIN farmers_market.booth b ON vba.booth_number = b.booth_number WHERE market_date >= '2020-­10-­01' So now even though the original vendor_booth_assignments table doesn’t contain the original booth assignments for these two vendors on October 10, if we had run an analysis at an earlier date and wanted to see what the database Figure 14.3 236 Chapter 14 ■ Storing and Modifying Data values were at that time, we could query this vendor_booth_log table to look at the values at different points time, as shown in Figure 14.4. Using SQL Inside Scripts Importing the datasets you develop into your machine learning script is beyond the scope of this book, but you can search the internet for the combination of SQL and your chosen scripting language and packages to find tutorials. For example, searching “import SQL python pandas dataframe” will lead you to tutorials for connecting to a database from within your Python script, running a SQL query, and importing the results into a pandas dataframe for analysis. You can usually either paste the SQL query into your script and store it in a string variable or reference the SQL stored in a text document. Keep in mind that some special characters in your query will need to be “escaped.” For example, if you surround your SQL query in double quotes to store it in a string variable in Python, it will interpret any double quotes within your query as ending the string and raise an error. To use quotes inside quoted strings in Python, you can precede them with a backslash. Representing this SQL query as a string in Python SELECT * FROM farmers_market.product WHERE product_qty_type = \"unit\" requires escaping the internal double quotes, like so: my_query = \"SELECT * FROM farmers_market.product WHERE product_qty_type = \\\"unit\\\"\" This is common in programming languages where", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 136 + }, + { + "text": "Python, you can precede them with a backslash. Representing this SQL query as a string in Python SELECT * FROM farmers_market.product WHERE product_qty_type = \"unit\" requires escaping the internal double quotes, like so: my_query = \"SELECT * FROM farmers_market.product WHERE product_qty_type = \\\"unit\\\"\" This is common in programming languages where strings are surrounded by quotes, since many strings contain quotes in the enclosed text, so you can search for “string escape characters” along with your chosen machine learning script tool or language to find out how to modify your SQL for use within your script. You can also write data from your script back to the database using SQL, but the approach varies based on the scripting language you’re using and the type of database you’re connecting to. In Python, for example, there are packages Figure 14.4 Chapter 14 ■ Storing and Modifying Data 237 available to help you connect and write to a variety of databases without even needing to write dynamic SQL INSERT statements—­the packages generate the SQL for you to insert values from an object in Python like a dataframe into a database table for persistent storage. Another approach is to programmatically create a file to temporarily store your data, transfer the file to a location that is accessible from your script and your database, and load the results from the file into your table. For example, you might use Python to write data from a pandas dataframe to a CSV file, transfer the CSV file to an Amazon Web Services S3 bucket, then access the file from the database and copy the records into an existing table in a Redshift database. All of these steps can be automated from your script. One machine learning use case for writing data from your script to the data- base is if you want to store your transformed dataset after you have completed feature engineering and data preprocessing steps in your script that weren’t completed in the original dataset-­generating SQL. Another use case for writing values generated within your script back to the database is when you want to store the results that your predictive model generates and associate them with the original dataset. You can create a table that stores the unique identifiers for the rows in your input dataset for which you have scores, a timestamp, the name or ID of your model, and the score or classification generated by the model for each record. You can insert new rows into the table each time you refresh your model scores. Then, use a SQL query to filter this model score log table to a specific date and model identifier and join it to the table with your input dataset, joining on the unique row identifiers. This will allow you to analyze the results of the model alongside the input data used to generate the predictions at the time. There are many ways to connect to and interact with data from a database in your other scripts that may", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 137 + }, + { + "text": "dataset, joining on the unique row identifiers. This will allow you to analyze the results of the model alongside the input data used to generate the predictions at the time. There are many ways to connect to and interact with data from a database in your other scripts that may or may not require SQL. In Closing Now that you know SQL basics, you should have the foundation needed to create datasets for your machine learning models, even if you need to search the internet for functions and syntax that were not covered in this book. I have been a data scientist for five years now, and all of the queries I have written to generate my datasets for machine learning have been variations of the SQL I originally learned in school 20 years ago. I hope that this book has given you the SQL skills you need to achieve your analysis goals faster and more independently, and that you find pulling and modifying your own datasets as empowering as I do! 238 Chapter 14 ■ Storing and Modifying Data Exercises 1. If you include a CURRENT_TIMESTAMP column when you create a view, what would you expect the values of that column to be when you query the view? 2. Write a query to determine what the data from the vendor_booth_ assignment table looked like on October 3, 2020 by querying the ven- dor_booth_log table created in this chapter. (Assume that records have been inserted into the log table any time changes were made to the vendor_booth_assignment table.) A P P E N D I X 239 Chapter 1: Data Sources Answers 1. If the “Author Full Name” field is updated (overwritten) in the existing Authors table record for the author, then when a query is run to retrieve a list of authors and their books, all past books associated with the author will now be associated with the author’s new name in the database, even if that wasn’t the name printed on the cover of the book. If instead a new row is added to the Authors table to record the new name (leaving the existing books associated with the prior name), then there might be no way to know that the two authors, who now have different Author IDs, are actually the same person. There are solutions to this problem that include designing the database tables and relationships to allow multiple names per Author ID, with start and stop dates, or adding a field to the Authors table such as “prior Author ID” that associates an Authors table record with another record in the same table, if one exists. Understanding these relationships and when and how data is updated in the database you’re querying is important for understanding and explain- ing the results of your queries. Answers to Exercises 2. One example might be tracking personal exercise routines. You could have a table of workout sessions and a table of exercises, which would be a many to many", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 138 + }, + { + "text": "updated in the database you’re querying is important for understanding and explain- ing the results of your queries. Answers to Exercises 2. One example might be tracking personal exercise routines. You could have a table of workout sessions and a table of exercises, which would be a many to many relationship: each workout could contain multiple exercises, and each exercise could be part of multiple workouts. If you included a table of workout session locations, that could be designed as a “one to many” relationship with the workout sessions table, assuming each work- out could only take place in one location (say, at home or at the gym), but each location could be the site of many workout sessions. Chapter 2: The SELECT Statement Answers 1. This query returns everything in the customer table: SELECT * FROM farmers_market.customer 2. This query displays all of the columns and 10 rows from the customer table, sorted by customer_last_name, then customer_first_name: SELECT * FROM farmers_market.customer ORDER BY customer_last_name, customer_first_name LIMIT 10 3. This query lists all customer IDs and first names in the customer table, sorted by first_name: SELECT customer_id, customer_first_name FROM farmers_market.customer ORDER BY customer_first_name Chapter 3: The WHERE Clause Answers There are multiple answers to most SQL questions, but here are some possible solutions for the exercises in Chapter 3: 1. Remember that even though the English phrasing is “product ids 4 and 9,” using AND between the conditions in the query will not return any results, because there is only one product_id per customer_purchase. Use 240 Appendix ■ Answers to Exercises Appendix ■ Answers to Exercises 241 an OR between the conditions in the WHERE clause to return every row that has a product_id of either 4 or 9: SELECT * FROM farmers_market.customer_purchases WHERE product_id = 4 OR product_id = 9 2. Note that the first query uses >= and <= to establish the inclusive range, while the second query uses BETWEEN to achieve the same result: SELECT * FROM farmers_market.customer_purchases WHERE vendor_id >= 8 AND vendor_id <= 10 SELECT * FROM farmers_market.customer_purchases WHERE vendor_id BETWEEN 8 AND 10 3. One approach is to filter to market dates that are not in the “rainy dates” list, by using the NOT operator to negate the IN condition. This will return TRUE for the rows in the customer_purchases table with a market_date that is NOT IN the query in the WHERE clause: SELECT market_date, customer_id, vendor_id, quantity * cost_to_customer_per_qty AS price FROM farmers_market.customer_purchases WHERE market_date NOT IN ( SELECT market_date FROM farmers_market.market_date_info WHERE market_rain_flag = 1 ) Another option is to keep the IN condition but change the query in the WHERE clause to return dates where it was not raining, when market_rain_ flag is set to 0: SELECT market_date, customer_id, vendor_id, quantity * cost_to_customer_per_qty AS price FROM farmers_market.customer_purchases Continues 242 Appendix ■ Answers to Exercises WHERE market_date IN ( SELECT market_date FROM farmers_market.market_date_info WHERE market_rain_flag = 0 ) Chapter 4: CASE Statements Answers 1. Look back at Figure 2.1 for sample", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 139 + }, + { + "text": "market_rain_ flag is set to 0: SELECT market_date, customer_id, vendor_id, quantity * cost_to_customer_per_qty AS price FROM farmers_market.customer_purchases Continues 242 Appendix ■ Answers to Exercises WHERE market_date IN ( SELECT market_date FROM farmers_market.market_date_info WHERE market_rain_flag = 0 ) Chapter 4: CASE Statements Answers 1. Look back at Figure 2.1 for sample data and column names for the product table referenced in this exercise. This query outputs the product_id and product_name columns from product, with a column called prod_qty_ type_condensed that displays the word “unit” if the product_qty_type is “unit,” and otherwise displays the word “bulk”: SELECT product_id, product_name, CASE WHEN product_qty_type = \"Unit\" THEN \"unit\" ELSE \"bulk\" END AS prod_qty_type_condensed FROM farmers_market.product 2. To add a column to the previous query called pepper_flag that outputs a 1 if the product_name contains the word “pepper” (regardless of capi- talization), and otherwise outputs 0, do the following: SELECT product_id, product_name, CASE WHEN product_qty_type = \"Unit\" THEN \"per unit\" ELSE \"bulk\" END AS prod_qty_type_condensed, CASE WHEN LOWER(product_name) LIKE '%pepper%' THEN 1 ELSE 0 END AS pepper_flag FROM farmers_market.product 3. If the product name doesn’t include the word “pepper,” spelled exactly that way, it won’t be flagged. For example, a product might only be labeled as “Jalapeno” instead of Jalapeno pepper. (continued) Appendix ■ Answers to Exercises 243 Chapter 5: SQL JOINs Answers 1. This query INNER JOINs the vendor table to the vendor_booth_ assignments table and sorts the result by vendor_name, then market_date: SELECT * FROM vendor AS v INNER JOIN vendor_booth_assignments AS vba ON v.vendor_id = vba.vendor_id ORDER BY v.vendor_name, vba.market_date 2. The following query uses a LEFT JOIN to produce output identical to the output of this exercise’s query: SELECT c.*, cp.* FROM customer_purchases AS cp LEFT JOIN customer AS c ON cp.customer_id = c.customer_id This could have been written with SELECT * and be considered correct. Using the table aliases in this way allows you to control which table’s columns are displayed first, so in addition to returning the same data, it’s also returned with the same column order as the given query. 3. One approach is to INNER JOIN the product table and the product_ category table, to get the category of every product (a new category with no products in it yet wouldn’t need to be included here, and there shouldn’t be any products without categories), then LEFT JOIN the vendor_inventory table to the product table. I chose a LEFT JOIN instead of an INNER JOIN because we might want to know if products exist in the database that are never in season because they have never been offered by a vendor at the farmer’s market. There are acceptable answers that include all types of JOINs, as long as the reason for each choice is explained. Because we haven’t learned about aggregation (summarization) yet, the dataset you can create using the information included in this chapter will have one row per product per vendor who offered it per market date it was offered, labeled with the product category. Because", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 140 + }, + { + "text": "the reason for each choice is explained. Because we haven’t learned about aggregation (summarization) yet, the dataset you can create using the information included in this chapter will have one row per product per vendor who offered it per market date it was offered, labeled with the product category. Because the vendor_inventory table includes the date the product was offered for sale, you could sort by product_category, product, and market_date, and scroll through the query results to determine when each type of item is in season. 244 Appendix ■ Answers to Exercises Chapter 6: Aggregating Results for Analysis Answers 1. This query determines how many times each vendor has rented a booth at the farmer’s market: SELECT vendor_id, count(*) AS count_of_booth_assignments FROM farmers_market.vendor_booth_assignments GROUP BY vendor_id 2. This query displays the product category name, product name, earliest date available, and latest date available for every product in the “Fresh Fruits & Vegetables” product category: SELECT pc.product_category_name, p.product_name, min(market_date) AS first_date_available, max(market_date) AS last_date_available FROM farmers_market.vendor_inventory vi INNER JOIN farmers_market.product p ON vi.product_id = p.product_id INNER JOIN farmers_market.product_category pc ON p.product_category_id = pc.product_category_id WHERE product_category_name = 'Fresh Fruits & Vegetables' 3. This query joins two tables, uses an aggregate function, and uses the HAVING keyword to generate a list of customers who have spent more than $50, sorted by last name, then first name: SELECT cp.customer_id, c.customer_first_name, c.customer_last_name, SUM(quantity * cost_to_customer_per_qty) AS total_spent FROM farmers_market.customer c LEFT JOIN farmers_market.customer_purchases cp ON c.customer_id = cp.customer_id GROUP BY cp.customer_id, c.customer_first_name, c.customer_last_name HAVING total_spent > 50 ORDER BY c.customer_last_name, c.customer_first_name Appendix ■ Answers to Exercises 245 Chapter 7: Window Functions and Subqueries Answers 1. Here are the answers to the two parts of this exercise: a. These queries use DENSE_RANK() or ROW_NUMBER() to select from the customer_purchases table and numbers each customer’s visits to the Farmer’s Market using DENSE_RANK(): select cp.*, DENSE_RANK() OVER (PARTITION BY customer_id ORDER BY market_date) AS visit_number FROM farmers_market.customer_purchases AS cp ORDER BY customer_id, market_date or select customer_id, market_date, ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY market_date) AS visit_number FROM farmers_market.customer_purchases GROUP BY customer_id, market_date ORDER BY customer_id, market_date b. This is how to reverse the numbering of the preceding query so each customer’s most recent visit is labeled 1, and then use another query to filter the results to only the customer’s most recent visit: SELECT * FROM ( select customer_id, market_date, ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY market_ date DESC) AS visit_number FROM farmers_market.customer_purchases GROUP BY customer_id, market_date ORDER BY customer_id, market_date ) x where x.visit_number = 1 Or SELECT * FROM ( select cp.*, DENSE_RANK() OVER (PARTITION BY customer_id ORDER BY market_ date DESC) AS visit_number FROM farmers_market.customer_purchases AS cp ORDER BY customer_id, market_date ) x where x.visit_number = 1 246 Appendix ■ Answers to Exercises 2. Here’s how to use a COUNT() window function and include a value along with each row of the customer_purchases table that indicates how many different times that customer has purchased that product_id: select cp.*, COUNT(product_id) OVER (PARTITION BY customer_id,", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 141 + }, + { + "text": "where x.visit_number = 1 246 Appendix ■ Answers to Exercises 2. Here’s how to use a COUNT() window function and include a value along with each row of the customer_purchases table that indicates how many different times that customer has purchased that product_id: select cp.*, COUNT(product_id) OVER (PARTITION BY customer_id, product_id) AS product_purchase_count FROM farmers_market.customer_purchases AS cp ORDER BY customer_id, product_id, market_date 3. If you swap out LEAD for LAG, you’re looking at the next row instead of the previous, so to get the same output, you just have to sort market_date in descending order, so everything is reversed! SELECT market_date, SUM(quantity * cost_to_customer_per_qty) AS market_date_total_sales, LEAD(SUM(quantity * cost_to_customer_per_qty), 1) OVER (ORDER BY market_date DESC) AS previous_market_date_total_sales FROM farmers_market.customer_purchases GROUP BY market_date ORDER BY market_date Chapter 8: Date and Time Functions Answers 1. Here is how to get the customer_id, month, and year (in separate columns) of every purchase in the farmers_market.customer_purchases table: SELECT customer_id, EXTRACT(MONTH FROM market_date) AS purchase_month, EXTRACT(YEAR FROM market_date) AS purchase_year FROM farmers_market.customer_purchases 2. Here is an example of filtering and summing purchases made in the past two weeks. Using March 31, 2019 as the reference date: SELECT MIN(market_date) AS sales_since_date, SUM(quantity * cost_to_customer_per_qty) AS total_sales FROM farmers_market.customer_purchases WHERE DATEDIFF('2019-­03-­31', market_date) <= 14 Appendix ■ Answers to Exercises 247 Using CURDATE(), which will result in NULL results on the sample data- base, since all dates are more than two weeks ago: SELECT MIN(market_date) AS sales_since_date, SUM(quantity * cost_to_customer_per_qty) AS total_sales FROM farmers_market.customer_purchases WHERE DATEDIFF(CURDATE(), market_date) <= 14 3. This is an example of using a quality control query to check manually entered data for correctness: SELECT market_date, market_day, DAYNAME(market_date) AS calculated_market_day, CASE WHEN market_day <> DAYNAME(market_date) then \"INCORRECT\" ELSE \"CORRECT\" END AS entered_correctly FROM farmers_market.market_date_info Chapter 9: Exploratory Data Analysis with SQL Answers 1. The following query gets the earliest and latest dates in the customer_ purchases table: SELECT MIN(market_date), MAX(market_date) FROM farmers_market.customer_purchases 2. Here is how to use the DAYNAME() and EXTRACT() functions to select and group by the weekday and hour of the day, and count the distinct number of customers during each hour of the Wednesday and Saturday markets: SELECT DAYNAME(market_date), EXTRACT(HOUR FROM transaction_time), COUNT(DISTINCT customer_id) FROM farmers_market.customer_purchases GROUP BY DAYNAME(market_date), EXTRACT(HOUR FROM transaction_time) ORDER BY DAYNAME(market_date), EXTRACT(HOUR FROM transaction_time) 3. A variety of answers would be acceptable. Two examples are shown here. How many customers made purchases at each market? SELECT market_date, COUNT(DISTINCT customer_id) FROM customer_purchases GROUP BY market_date ORDER BY market_date 248 Appendix ■ Answers to Exercises What is the total value of the inventory each vendor brought to each market? SELECT market_date, vendor_id, ROUND(SUM(quantity * original_price),2) AS inventory_value FROM vendor_inventory GROUP BY market_date, vendor_id ORDER BY market_date, vendor_id Chapter 10: Building SQL Datasets for Analytical Reporting Answers 1. Sales per vendor per market week: SELECT market_week, vendor_id, vendor_name, SUM(sales) AS weekly_sales FROM farmers_market.vw_sales_by_day_vendor AS s GROUP BY market_week, vendor_id, vendor_name ORDER BY market_date 2. Subquery rewritten using a WITH clause: WITH x AS ( SELECT market_date, vendor_id, booth_number, LAG(booth_number,1) OVER", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 142 + }, + { + "text": "Building SQL Datasets for Analytical Reporting Answers 1. Sales per vendor per market week: SELECT market_week, vendor_id, vendor_name, SUM(sales) AS weekly_sales FROM farmers_market.vw_sales_by_day_vendor AS s GROUP BY market_week, vendor_id, vendor_name ORDER BY market_date 2. Subquery rewritten using a WITH clause: WITH x AS ( SELECT market_date, vendor_id, booth_number, LAG(booth_number,1) OVER (PARTITION BY vendor_id ORDER BY market_ date, vendor_id) AS previous_booth_number FROM farmers_market.vendor_booth_assignments ORDER BY market_date, vendor_id, booth_number ) SELECT * FROM x WHERE x.market_date = '2020-­03-­13' AND (x.booth_number <> x.previous_booth_number OR x.previous_booth_number IS NULL) 3. There is one vendor booth assignment per vendor per market date, so we don’t need to change the granularity of our dataset in order to summarize by booth type, but we do need to pull that booth type into the dataset. We can accomplish that by LEFT JOINing in the vendor_booth_assignments Appendix ■ Answers to Exercises 249 and booth tables, and including the booth_number and booth_type columns in our SELECT statement: SELECT cp.market_date, md.market_day, md.market_week, md.market_year, cp.vendor_id, v.vendor_name, v.vendor_type, vba.booth_number, b.booth_type, ROUND(SUM(cp.quantity * cp.cost_to_customer_per_qty),2) AS sales FROM farmers_market.customer_purchases AS cp LEFT JOIN farmers_market.market_date_info AS md ON cp.market_date = md.market_date LEFT JOIN farmers_market.vendor AS v ON cp.vendor_id = v.vendor_id LEFT JOIN farmers_market.vendor_booth_assignments AS vba ON cp.vendor_id = vba.vendor_id AND cp.market_date = vba.market_date LEFT JOIN farmers_market.booth AS b ON vba.booth_number = b.booth_number GROUP BY cp.market_date, cp.vendor_id ORDER BY cp.market_date, cp.vendor_id Chapter 11: More Advanced Query Structures Answers 1. There are multiple possible solutions. Here is one: WITH sales_per_market_date AS ( SELECT market_date, ROUND(SUM(quantity * cost_to_customer_per_qty),2) AS sales FROM farmers_market.customer_purchases GROUP BY market_date ORDER BY market_date ), record_sales_per_market_date AS ( SELECT cm.market_date, cm.sales, Continues 250 Appendix ■ Answers to Exercises MAX(pm.sales) AS previous_max_sales, CASE WHEN cm.sales > MAX(pm.sales) THEN \"YES\" ELSE \"NO\" END sales_record_set FROM sales_per_market_date AS cm LEFT JOIN sales_per_market_date AS pm ON pm.market_date < cm.market_date GROUP BY cm.market_date, cm.sales ) SELECT market_date, sales FROM record_sales_per_market_date WHERE sales_record_set = 'YES' ORDER BY market_date DESC LIMIT 1 2. This may be more challenging than you initially anticipated! First, we need to add vendor_id to the output and the partition in the CTE, so we are ranking the first purchase date per customer per vendor. Then, we need to count the distinct customers per market per vendor, so we add the vendor_id to the GROUP BY in the outer query, and also modify the CASE statements to use the field we have re-­aliased to first_purchase_from_vendor_date: WITH customer_markets_vendors AS ( SELECT DISTINCT customer_id, vendor_id, market_date, MIN(market_date) OVER(PARTITION BY cp.customer_id, cp.vendor_id) AS first_purchase_from_vendor_date FROM farmers_market.customer_purchases cp ) SELECT md.market_year, md.market_week, cmv.vendor_id, COUNT(customer_id) AS customer_visit_count, COUNT(DISTINCT customer_id) AS distinct_customer_count, COUNT(DISTINCT CASE WHEN cmv.market_date = cmv.first_purchase_from_vendor_date THEN customer_id ELSE NULL (continued) Appendix ■ Answers to Exercises 251 END) AS new_customer_count, COUNT(DISTINCT CASE WHEN cmv.market_date = cmv.first_purchase_from_vendor_date THEN customer_id ELSE NULL END) / COUNT(DISTINCT customer_id) AS new_customer_percent FROM customer_markets_vendors AS cmv LEFT JOIN farmers_market.market_date_info AS md ON cmv.market_date = md.market_date GROUP BY md.market_year, md.market_week, cmv.vendor_id ORDER BY md.market_year, md.market_week, cmv.vendor_id 3. Again, there are many possible solutions, but here is one where the sales", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 143 + }, + { + "text": "cmv.market_date = cmv.first_purchase_from_vendor_date THEN customer_id ELSE NULL END) / COUNT(DISTINCT customer_id) AS new_customer_percent FROM customer_markets_vendors AS cmv LEFT JOIN farmers_market.market_date_info AS md ON cmv.market_date = md.market_date GROUP BY md.market_year, md.market_week, cmv.vendor_id ORDER BY md.market_year, md.market_week, cmv.vendor_id 3. Again, there are many possible solutions, but here is one where the sales per market date are ranked ascending and descending, and then the top results from each of those rankings are selected and unioned together: WITH sales_per_market AS ( SELECT market_date, ROUND(SUM(quantity * cost_to_customer_per_qty),2) AS sales FROM farmers_market.customer_purchases GROUP BY market_date ), market_dates_ranked_by_sales AS ( SELECT market_date, sales, RANK() OVER (ORDER BY sales) AS sales_rank_asc, RANK() OVER (ORDER BY sales DESC) AS sales_rank_desc FROM sales_per_market ) SELECT market_date, sales, sales_rank_desc AS sales_rank FROM market_dates_ranked_by_sales WHERE sales_rank_asc = 1 UNION SELECT market_date, sales, sales_rank_desc AS sales_rank FROM market_dates_ranked_by_sales WHERE sales_rank_desc = 1 252 Appendix ■ Answers to Exercises Chapter 12: Creating Machine Learning Datasets Using SQL Answers 1. This can be accomplished by duplicating the customer_markets_ attended_30days_count feature and replacing each “30” with 14: (SELECT COUNT(market_date) FROM customer_markets_attended cma WHERE cma.customer_id = cp.customer_id AND cma.market_date < cp.market_date AND DATEDIFF(cp.market_date, cma.market_date) <= 14) AS customer_markets_attended_14days_count, 2. The query is already grouped by customer_id and market_date, so we just need to add a column that determines if any underlying row has an item with a price over $10, and if so, return a 1, then use the MAX function to get the highest number per group, which will be a 1 if any row met the criteria: MAX(CASE WHEN cp.cost_to_customer_per_qty > 10 THEN 1 ELSE 0 END) purchased_item_over_10_dollars, 3. This is a tricky one. One way to accomplish it is to add the purchase_total per market date to the CTE, then add up all purchase_total values for dates prior to the row’s market_date. Both total_spent_to_date and customer_has_spent_over_200 fields have been added to the following query, which also includes the fields from exercises 1 and 2: WITH customer_markets_attended AS ( SELECT customer_id, market_date, SUM(quantity * cost_to_customer_per_qty) AS purchase_total, ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY market_ date) AS market_count FROM farmers_market.customer_purchases GROUP BY customer_id, market_date ORDER BY customer_id, market_date ) SELECT cp.customer_id, cp.market_date, Appendix ■ Answers to Exercises 253 EXTRACT(MONTH FROM cp.market_date) AS market_month, SUM(cp.quantity * cp.cost_to_customer_per_qty) AS purchase_total, COUNT(DISTINCT cp.vendor_id) AS vendors_patronized, MAX(CASE WHEN cp.vendor_id = 7 THEN 1 ELSE 0 END) AS purchased_ from_vendor_7, MAX(CASE WHEN cp.vendor_id = 8 THEN 1 ELSE 0 END) AS purchased_ from_vendor_8, COUNT(DISTINCT cp.product_id) AS different_products_purchased, DATEDIFF(cp.market_date, (SELECT MAX(cma.market_date) FROM customer_markets_attended AS cma WHERE cma.customer_id = cp.customer_id AND cma.market_date < cp.market_date GROUP BY cma.customer_id)) days_since_last_customer_market_date, (SELECT MAX(market_count) FROM customer_markets_attended cma WHERE cma.customer_id = cp.customer_id AND cma.market_date <= cp.market_date) AS customer_ markets_attended_count, (SELECT COUNT(market_date) FROM customer_markets_attended cma WHERE cma.customer_id = cp.customer_id AND cma.market_date < cp.market_date AND DATEDIFF(cp.market_date, cma.market_date) <= 30) AS customer_markets_attended_30days_count, (SELECT COUNT(market_date) FROM customer_markets_attended cma WHERE cma.customer_id = cp.customer_id AND cma.market_date < cp.market_date AND DATEDIFF(cp.market_date, cma.market_date) <= 14) AS customer_markets_attended_14days_count, MAX(CASE WHEN cp.cost_to_customer_per_qty > 10 THEN 1 ELSE 0 END) AS purchased_item_over_10_dollars, (SELECT SUM(purchase_total) FROM customer_markets_attended cma WHERE", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 144 + }, + { + "text": "cp.customer_id AND cma.market_date < cp.market_date AND DATEDIFF(cp.market_date, cma.market_date) <= 30) AS customer_markets_attended_30days_count, (SELECT COUNT(market_date) FROM customer_markets_attended cma WHERE cma.customer_id = cp.customer_id AND cma.market_date < cp.market_date AND DATEDIFF(cp.market_date, cma.market_date) <= 14) AS customer_markets_attended_14days_count, MAX(CASE WHEN cp.cost_to_customer_per_qty > 10 THEN 1 ELSE 0 END) AS purchased_item_over_10_dollars, (SELECT SUM(purchase_total) FROM customer_markets_attended cma WHERE cma.customer_id = cp.customer_id AND cma.market_date <= cp.market_date) AS total_spent_to_ date, CASE WHEN (SELECT SUM(purchase_total) FROM customer_markets_attended cma WHERE cma.customer_id = cp.customer_id AND cma.market_date <= cp.market_date) > 200 THEN 1 ELSE 0 END AS customer_has_spent_over_200, CASE WHEN DATEDIFF( (SELECT MIN(cma.market_date) FROM customer_markets_attended AS cma WHERE cma.customer_id = cp.customer_id AND cma.market_date > cp.market_date Continues 254 Appendix ■ Answers to Exercises GROUP BY cma.customer_id), cp.market_date) <=30 THEN 1 ELSE 0 END AS purchased_ again_within_30_days FROM farmers_market.customer_purchases AS cp GROUP BY cp.customer_id, cp.market_date ORDER BY cp.customer_id, cp.market_date Chapter 14: Storing and Modifying Data Answers 1. The timestamp returned when you query the view will be the current time (on the server), because unlike with a table, the view isn’t storing any data and is generating the results of the query when it is run. 2. There are multiple correct answers, but one approach is to filter to records prior to October 4, 2020 (so if a change was made at any time on October 3, it is retrieved), and include a window function that returns the maximum timestamp per vendor and booth pair, indicating the most recent record of each booth assignment on or before the filtered date range. Then, the results of the query are embedded inside an outer query that filters to rows in the subquery where the snapshot timestamp matches the maxi- mum timestamp calculated in the window function: SELECT x.* FROM ( SELECT vendor_id, booth_number, market_date, snapshot_timestamp, MAX(snapshot_timestamp) OVER (PARTITION BY vendor_id, booth_ number) AS max_timestamp_in_filter FROM farmers_market.vendor_booth_log WHERE DATE(snapshot_timestamp) <= '2020-­10-­04' ) AS x WHERE x.snapshot_timestamp = x.max_timestamp_in_filter (continued) 255 A Access (Microsoft), 2 ad-­hoc reporting, 143 aggregation AVG (average) function within, 91–93 CASE statement inside, 94–96 COUNT DISTINCT function within, 90–91 COUNT function within, 90–91 date functions within, 119–125 displaying group summaries, 80–83 exercise using, 244–245 granularity and, 138 GROUP BY within, 79–80 on joined tables, 86 LISTAGG for, 111 MAX function within, 88–90 MIN function within, 88–90 performing calculations inside, 84–88 underlying data prior to, 183 window functions and, 97, 103–108 algorithms, 177, 185 alias WITH clause (CTE), 204–205 FROM clause and, 66 columns, 21–22, 65–66 within Exploratory Data Analysis (EDA), 130 of tables, 66, 89 Amazon Redshift, 2, 115, 119 analytical dataset custom, 149–153 requirements of, 144–148 reusing of, 153–157 AND NOT operator, 34, 38 AND operator, 34, 37, 38 AS keyword, 21, 22, 66, 149 ASC (ascending) keyword, 18–19 asterisk symbol, 16, 21 attribute, 3–4 averaging, 92 AVG (average) function, 91–93, 97, 103–104 B Batten, Dayne, 214 BETWEEN keyword, 41–42 binary classification algorithms, 185 binary classification model, 173, 176–178 binary flag field, 52–53, 177–178, 181, 190 binning, 53–56 blank, within database, 44 C calculations aggregate, 103 averaging, 92 date, 116 with DATEDIFF function, 123 of", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 145 + }, + { + "text": "averaging, 92 AVG (average) function, 91–93, 97, 103–104 B Batten, Dayne, 214 BETWEEN keyword, 41–42 binary classification algorithms, 185 binary classification model, 173, 176–178 binary flag field, 52–53, 177–178, 181, 190 binning, 53–56 blank, within database, 44 C calculations aggregate, 103 averaging, 92 date, 116 with DATEDIFF function, 123 of distance, 214 engineered features, 180 LAG and LEAD functions for, 111 performing inside aggregate functions, 84–88 ROUND function and, 92 rounding, 22–23 of sales, 155–156 simple inline, 20–22 total spent, 87 window functions and, 97 Index 256 Index ■ C–C CASE statement aggregate functions and, 94–96 binary flags using, 52–53, 181 binning using, 53–56 categorical encoding using, 56–59 COUNT () function within, 170 example of, 202 grouping using, 53–56 overview of, 49 summary of, 59–60 syntax of, 50–52 categorical encoding, 56–59 categorical text column, 185 classification algorithms, 176 classification model, 173, 177 clauses FROM, 66, 80 ON, 197–198 FROM, 199 WITH clause analytical dataset and, 149–153 defined, 124–125 example of, 162 exercise using, 190 ROW_NUMBER window function and, 183 subquery using, 248 use of, 151, 161, 164, 168, 204–205, 209 HAVING clause example of, 125 exercise using, 244–245 filtering with, 93 GROUP BY statement and, 93 syntax of, 16, 80 use of, 132 LIMIT, 16–17, 19, 25, 26 ORDER BY clause within CONCAT function, 24–25 example of, 85, 140 LIMIT clause and, 19 location of, 32 NTILE groups and, 103 within the partition, 98 PARTITION BY window function and, 107–108 sort order specification within, 108 sorting column and, 100 sorting results using, 18–20 SUM window function and, 106–107 syntax of, 16, 29–30, 80 RANGE, 111 SELECT, 16 TOP, 17 WHERE clause condition of, 87 date filtering in, 93 defined, 100 differences between, 161 error from, 101 example of, 70 exercise using, 241 filtering uses within, 41–44, 84, 197 location of, 32 multi-­column conditional filtering within, 40–41 multiple condition filtering within, 34–40 overview of, 31 querying databases using, 17 removing, 141, 166 row removal using, 122 ROW_NUMBER () window function to, 184–185 syntax of, 16, 80, 233, 234 TOP clause and, 17 COALESCE function, 199 Cognos, 79 column alias of, 65–66 categorical text, 185 defined, 3–4 demonstration, 182 as dummy variables, 57 engineered features of, 180 input, 13 listing, 18 multi conditional filtering, 40–41 one-­hot encoded flag, 185 selecting, 16–18 sorting, 100 timestamp, 232 value possibilities within, 131–133 comment out code, 231 Common Table Expression (CTE) analytical dataset and, 149–153 WITH clause and, 161, 164, 204–205, 209 creation of, 179 defined, 124–125 exercise using, 190 select from, 216 subquery of, 180 summary of, 226–227 UNION queries and, 160–161 comparison operator, 163 CONCAT function, 24–26, 114 concatenating strings, 24–26 concatenation parameters, 25 conditional filtering, multi-­column, 40–41 conditional statements, 33 Coordinated Universal Time (UTC), 232 COUNT DISTINCT function, 90–91, 120, 183 COUNT () function, 80–81, 90–91, 102, 170 Index ■ C–D 257 COUNT () window function, 112, 246 COVID-­19 tracking dashboard, 163 CREATE TABLE statement, 230, 232 CREATE VIEW AS statement, 151 crow’s feet symbol, 62 CURDATE () function, 121, 126, 247 CURRENT_DATE", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 146 + }, + { + "text": "(UTC), 232 COUNT DISTINCT function, 90–91, 120, 183 COUNT () function, 80–81, 90–91, 102, 170 Index ■ C–D 257 COUNT () window function, 112, 246 COVID-­19 tracking dashboard, 163 CREATE TABLE statement, 230, 232 CREATE VIEW AS statement, 151 crow’s feet symbol, 62 CURDATE () function, 121, 126, 247 CURRENT_DATE function, 121 CURRENT_TIMESTAMP function, 232, 238 customer table calculations within, 20–22 columns within, 24–26 demographic data and, 211–217 example of, 59 exercises for, 30 joining with, 86–87, 211 IN keyword within, 42–43 LIMIT clause and, 26–29 new versus returning counting, 167–171 WHERE clause within, 33–38, 40–41 customer_purchases table calculated views within, 154–155 WITH clause and, 167–171 custom analytical datasets within, 149–153 demographic data and, 211–217 example using, 80–81, 82–86, 120–125, 193, 194–196, 204–208 exercise of, 126, 142 feature engineering and, 186–188 feature set and, 181–185 GROUP BY clause and, 179–181 joining with, 71–74, 94 market_date_info table and, 174–175 sales tracking within, 146–148 vendor_inventory table and, 136–142 D dashboard, 141, 163 data, 1, 7–9, 10–11, 174, 211 data leakage, 181 data point, 13 data sources, 1–3, 9–11 data warehouses, 7–9 database defined, 4 direct connection to, 3 doctor’s office, 4–6, 8–9 exercises for, 13, 30, 47, 76–77, 96, 111–112 inserting rows within, 233–236 relational, 3–7 relationships, 61–71 star schema, 7, 8 tables, relationship between, 4 types of, 11 updating values within, 10, 233–236 Database (Oracle), 2 database administrators (DBAs), 9 dataset analytical, 144–157 for binary classification, 176–178 creating, 178–181 defined, 144 for predictive models, 113 refreshing of, 228 storing, 229–232 for time series model, 174–176 within time-­series analysis, 113 for weather classification model, 174 date codes for, 114–115 counting within, 167–171 exercise using, 171 exploring changes within, 134–135 filtering to, 93 greater-­than sign within, 182 maximum values of, 133 minimum values of, 133 setting field values for, 114–115 summary of, 145–146, 174–175 training data error and, 186 date filter, 166 DATE () function, 115–116 date functions, 119–125 DATE_ADD function, 116–118 DATEDIFF function, 118, 121, 123, 124 DATE_FORMAT function, 115 DATE_PART function, 115–116 DATE_SUB function, 116–118 datetime_demo table, 118 Daylight Savings Time, 232 DAYNAME () function, 126, 142, 147, 247 DELETE FROM statement, 233 demographic data, sources for, 211 demonstration column, 182 DENSE RANK () window function, 101–102 DENSE_RANK () function, 245 DESC function, 128 DESC (descending) keyword, 18–19 DESCRIBE function, 128, 132 detectable patterns, 181 dimension, 8, 144 dimension table, 8 dimensional data warehouses, 7–9 dimensional model, 9 dimensional modeling techniques, 7 distance, calculation of, 214 DISTINCT keyword duplicate removal using, 122 example of, 131 use of, 73, 82–83, 168, 183 doctor’s office database, 4–6, 8–9 258 Index ■ D–F DROP TABLE statement, 230–231 dummy variable, 57 duplicates, removing of, 122 dynamic date ranges, 113 dynamic list of values, 46 E Eastern Standard Time, 232 edge cases, 27–28 editing, in SQL, 2–3 ELSE statement, 50, 56 encoding, categorical, 56–59 engineered features, 180 entity, 3 entity-­relationship diagram (ERD), 5, 62 ETL engineers, 9 event logs, 162 Excel (Microsoft), 1–2 Exploratory Data Analysis (EDA) alias within, 130 column value possibilities within, 131–133 demonstrating", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 147 + }, + { + "text": "Eastern Standard Time, 232 edge cases, 27–28 editing, in SQL, 2–3 ELSE statement, 50, 56 encoding, categorical, 56–59 engineered features, 180 entity, 3 entity-­relationship diagram (ERD), 5, 62 ETL engineers, 9 event logs, 162 Excel (Microsoft), 1–2 Exploratory Data Analysis (EDA) alias within, 130 column value possibilities within, 131–133 demonstrating of, 128 exploring changes over time within, 134–135 multiple table exploring within, 135–138 within the predictive modeling process, 127 process of, 189 sales versus inventory within, 138–142 use of, 11 exponential smoothing, 175–176 EXTRACT () function, 115–116, 247 F fact, 8 fact table, 8 FALSE evaluation, 33, 34–40 Farmer’s Market Database, introduction to, 11–12. See also specific tables feature, 13 feature engineering, 185–188, 189 feature set, expanding, 181–185 feature vectors, 173 field, 3–4 field values, for date/time, 114–115 filtering ON clause, 197–198 date, 93 front-­end, 88 within HAVING clause, 93 JOIN statement, 71–74 BETWEEN keyword, 41–42 IN keyword, 42–43 LIKE keyword, 43 multi-­column conditional, 40–41 multiple conditions, 34–40 NULL value, 44–46, 71–72 removing, 166 from SELECT statement, 32–34 with a subquery, 104–105 using subqueries, 46–47 ways to, 41–44 WHERE clause and, 122 flags, binary, 52–53, 177–178, 181, 190 follow-­up questions, anticipation of, 144–145 forecasting functions, 175–176 foreign key, 5 FROM clause, 66, 80, 199 FROM statement, 16, 17, 29–30, 32 front-­end filtering, 88 functions. See also window functions AVG (average), 91–93, 97, 103–104 COALESCE, 199 CONCAT, 24–26, 114 COUNT (), 80–81, 90–91, 102, 170 COUNT DISTINCT, 90–91, 120, 183 CURDATE (), 121, 126, 247 CURRENT_DATE, 121 CURRENT_TIMESTAMP, 238 DATE (), 115–116 DATE_ADD, 116–118 DATEDIFF, 118, 121, 123, 124 DATE_FORMAT, 115 DATE_PART, 115–116 DATE_SUB, 116–118 DAYNAME (), 126, 142, 147, 247 DENSE_RANK (), 245 DESC, 128 DESCRIBE, 128, 132 GETDATE, 121 LAG, 108–111, 121–122, 209, 246 LEAD, 108–111, 123, 124 LOWER, 51 MAX () DATEDIFF function and, 121 example of, 120, 140 exercise using, 252 overview of, 88–90 use of, 163, 165, 199 MIN (), 88–90, 120, 121, 140, 167–168 NTILE, 102–103, 224–225 ROUND, 22–23 ROUND function COALESCE function and, 199 defined, 97 example of, 87, 104–105, 108 use of, 22–23, 92 SQL, defined, 22 STR_TO_DATE, 114, 115 SYSDATE, 121 TIME (), 115–116 Index ■ F–K 259 TIMESTAMPDIFF, 119 TODAY, 121 TRIM, 44 UPPER, 25, 51 WEEK, 147 YEAR, 147 G GETDATE function, 121 grain, 9 granularity within aggregation, 138 changing of, 189 choosing of, 145 of customer_purchases table, 81 for forecasting, 176 table, 129 with target variable, 178–179 understanding, 10, 82 graphical use interface (GUI), 2–3 greater-­than sign, 182 Greenwich Mean Time, 232 GROUP BY clause COUNT function and, 90 defined, 97 example of, 84, 85, 95–96, 110, 216 HAVING clause and, 93, 125 ROW_NUMBER window function and, 183 within summary, 145 syntax of, 16, 79–80 use of, 161, 165, 174–175 group summaries, displaying, 80–83 grouping, continuous values with CASE statements, 53–56 H hard-­coded list of values, within IN keyword, 46 HAVING clause example of, 125 exercise using, 244–245 filtering with, 93 GROUP BY statement and, 93 syntax of, 16, 80 use of, 132 heart disease classification model, 173 histogram,", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 148 + }, + { + "text": "174–175 group summaries, displaying, 80–83 grouping, continuous values with CASE statements, 53–56 H hard-­coded list of values, within IN keyword, 46 HAVING clause example of, 125 exercise using, 244–245 filtering with, 93 GROUP BY statement and, 93 syntax of, 16, 80 use of, 132 heart disease classification model, 173 histogram, 11, 141–142 human-­readable report labels, joining within, 144 I IBM DB2, 2 \"IF\" statements, 31 IN keyword, 42–43, 46, 241 infinity symbol, 62 inline calculations, 20–23, 24–26 INNER JOIN, 68–71, 76, 194, 243 inner subquery, 100 input column, 13 input parameters, 22–23 input variable, 13 INSERT INTO SELECT statement, 233 INSERT statement, 233 inside scripts, 236–237 instance, 13, 173 integer overflow, 119 Integrated Development Environment (IDE), 2–3 inventory, sales versus, 138–142 IS NULL keyword, 44 J JOIN statement aggregation using, 86 ON clause, 197–198 database relationships and, 61–71 error within, 164 example of, 94, 166 exercises for, 77, 243, 248–249 filtering pitfalls within, 71–74 INNER JOIN, 68–71, 76, 194, 243 LEFT JOIN example of, 74–76, 87, 139, 148, 166 exercises for, 77, 243, 248–249 filtering pitfalls within, 71–74 overview of, 63–67 lookup tables and, 140 of multiple tables, 74–76 RIGHT JOIN, 67–68, 139, 194–197 self-­joining, 163–167 Jupyter notebook, 127 K keywords AS, 21, 22, 66, 149 BETWEEN, 41–42 IN, 42–43, 46, 241 AS, 66, 149 ASC (ascending), 18–19 defined, 15 DESC (descending), 18–19 DISTINCT duplicate removal using, 122 example of, 131 use of, 73, 82–83, 168, 183 IS NULL, 44 LIKE, 43, 52 NOT IN, 241 260 Index ■ L–P L LAG function, 108–111, 121–122, 209, 246 latitudes, distance calculation of, 214 LEAD function, 108–111, 123, 124 LEFT JOIN example of, 74–76, 87, 139, 148, 166 exercises for, 77, 243, 248–249 filtering pitfalls within, 71–74 overview of, 63–67 less-­than sign, 182 LIKE keyword, 43, 52 likelihood score, 177 LIMIT clause, 16–17, 19, 25, 26–27 line chart, 175–176 longitudes, distance calculation of, 214 lookup tables, 140 LOWER function, 51 M machine learning, dataset terminology, 12–13 machine learning algorithm, 13 machine learning applications, 230 many-­to-­many relationship, 6 market_date_info table binary flags within, 52–53 customer_purchases table and, 174–175 datetime field values within, 114–118 example using, 220 exercise of, 126 filtering within, 46–47 UNION query within, 159–160 market_vendor_inventory table, 160–161 materialized view, 230 MAX () function DATEDIFF function and, 121 example of, 120, 140 exercise using, 252 overview of, 88–90 use of, 163, 165, 199 measures, 8, 144 medical database table, 4–6 medical history, algorithms and, 177 Microsoft Access, 2 Microsoft Excel, 1–2 Microsoft SQL Server, 2 MIN () function, 88–90, 120, 121, 140, 167–168 modeling techniques, dimensional, 7 multiplication, 93 MySQL Workbench Community Edition, 3, 132 N N symbol, 62 negative correlation, of variables, 192 NOT IN keyword, 241 NOT operator, 241 NTILE window function, 102–103, 224–225 NULL value within ascending order, 18 CASE statement and, 50 COALESCE function and, 199 defined, 5 as edge cases, 27–28 example of, 123 filtering of, 71–72 IS NULL keyword, 44–46 within STR_TO_DATE function, 115 warning regarding, 44–46 numeric inputs, for binary classification algorithms, 185 numeric price", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 149 + }, + { + "text": "window function, 102–103, 224–225 NULL value within ascending order, 18 CASE statement and, 50 COALESCE function and, 199 defined, 5 as edge cases, 27–28 example of, 123 filtering of, 71–72 IS NULL keyword, 44–46 within STR_TO_DATE function, 115 warning regarding, 44–46 numeric inputs, for binary classification algorithms, 185 numeric price output, 33 O observations, 173 ODBC (Open Database Connectivity), 3 ON clause, 197–198 one-­hot encoded flag column, 185 one-­hot encoding, 57–59 one-­to-­many relationship, 5, 62, 63 Open Database Connectivity (ODBC), 3 OR operator, 34, 38 Oracle Database, 2, 111, 115, 119 ORDER BY clause within CONCAT function, 24–25 example of, 85, 140 LIMIT clause and, 19 location of, 32 NTILE groups and, 103 within the partition, 98 PARTITION BY window function and, 107–108 sort order specification within, 108 sorting column and, 100 sorting results using, 18–20 SUM window function and, 106–107 syntax of, 16, 29–30, 80 outer subquery, 100, 122 output, query, evaluating, 26–29 P pandas package (Python), 189 parameters, 22–23, 25 parentheses symbol, 22–23 partition, 105, 168 Index ■ P–S 261 PARTITION BY window function, 98, 101, 107–108 passing example rows of data, training by, 173 percent symbol, 43 positive correlation, of variables, 192 PostgreSQL, 2, 111 predictive modeling, 113, 127, 144, 186 price, formatting, 33 primary key, 5, 132 probability score, from algorithms, 177 product table example of, 17, 18 exercise of, 77, 142 exercises for, 60 exploring, 128–131 IS NULL keyword within, 44–46 joining with, 61–71, 94 LIMIT clause and, 17 output comparison within, 39–40 possible column values within, 131–133 snapshot from, 231 sorting within, 19 UNION query within, 160–161, 162 WHERE clause within, 32, 39–40 product_category table, 61–71, 77, 129–131, 192–194 product_units table, 233–236 purchase table, 53–56 Python connecting to, 3 Exploratory Data Analysis (EDA) within, 127 packages within, 236–237 pandas package within, 189 scripting language within, 49 special characters within, 236 Q query, 26–29, 100, 149, 156, 160 query_alias, 149 quotes symbol, 21, 236 R R, connecting to, 3 RANGE clause, 111 RANK () window function, 100, 101–102, 121–122, 161, 162 raw data, 7 record, 3 record high to-­date indicator, 163 Redshift (Amazon), 2, 115, 119 refreshing, of datasets, 228 relational database, 3–7 Relational Database Management Systems (RDBMS), 2, 5, 6, 62, 63 reporting, ad-­hoc, 143 Result Grid, 27 RIGHT JOIN, 67–68, 70, 77, 139, 194–197 ROUND function COALESCE function and, 199 defined, 97 example of, 87, 104–105, 108 use of, 22–23, 92 rounding, as inline calculation, 22–23 ROW_NUMBER () window function, 98–101, 102, 183, 184–185, 245 rows COUNT function and, 90 defined, 3 FALSE evaluation within, 33, 34–40 inserting, 233–236 LAG function and, 108 LIMIT clause and, 16–17, 26–27 limiting, 16–18 passing example of data, 173 summary within, 86–87 TRUE evaluation within, 33, 34–40 running total, SUM function as, 107–108 S sales affects to, 192 calculations of, 155–156 comparison of, 192 exercises regarding, 157, 171 forecasting of, 176 inventory versus, 138–142 summary of, 145–146, 174–175, 211 trend comparison of, 192 variable changes and, 192 variation changes of, 211–217 saving, of queries, 156 scatterplot, 192", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 150 + }, + { + "text": "running total, SUM function as, 107–108 S sales affects to, 192 calculations of, 155–156 comparison of, 192 exercises regarding, 157, 171 forecasting of, 176 inventory versus, 138–142 summary of, 145–146, 174–175, 211 trend comparison of, 192 variable changes and, 192 variation changes of, 211–217 saving, of queries, 156 scatterplot, 192 SELECT INTO statement, 231–232 SELECT statement, 15, 16, 29–30, 32–34, 79 self-­join, to-­date maximum and, 163–167 SET statement, 234 signal patterns, 181 simple inline calculations, 20–22 snapshots, storing, 230 Snowflake, 2 sort order, 19, 108 sortable value, 220 sorting column, adding, 100 special characters, escaping of, 236 SQL function, 22 SQL Server (Microsoft), 2, 17, 115, 119 SQLite, 2 star schema, 7, 8 262 Index ■ S–T statements FROM, 16, 17, 29–30, 32 conditional, 33 CREATE TABLE, 230, 232 CREATE VIEW AS, 151 DELETE FROM, 233 DROP TABLE, 230–231 ELSE, 50, 56 \"IF,\" 31 INSERT, 233 INSERT INTO SELECT, 233 JOIN statement aggregation using, 86 ON clause, 197–198 database relationships and, 61–71 error within, 164 example of, 94, 166 exercises for, 77, 243, 248–249 filtering pitfalls within, 71–74 INNER JOIN, 68–71, 76, 194, 243 LEFT JOIN, 63–67, 71–76, 77, 87, 139, 148, 166, 243, 248–249 lookup tables and, 140 of multiple tables, 74–76 RIGHT JOIN, 67–68, 139, 194–197 self-­joining, 163–167 SELECT, 15, 16, 29–30, 32–34, 79 SELECT INTO, 231–232 SET, 234 THEN, 50 UPDATE, 232, 233, 234 WHEN, 50 stock prices, time series model for, 174 storage, within a dimensional model, 9 storing, dataset, 229–232 strings categorical encoding of, 56–59 concatenating, 24–26 as edge cases, 27–28 merging, 24 quotes for, 236 searching, 43 wildcard comparison of, 41 STR_TO_DATE function, 114, 115 structured data, 1 Structured Query Language (SQL), tools for connecting to, 2–3 subject matter experts (SMEs), 9–11 subquery CTE reference by, 180 defined, 100 filtering using, 46–47, 104–105 inner, 100 limiting of, 180–181 outer, 100 use of, 122 using a WITH clause, 248 SUM () function CASE statement and, 95 COALESCE function and, 199 example of, 84, 110 ORDER BY clause of, 106–107 as running total, 107–108 use of, 80–81, 146, 163 summary COUNT function and, 80–81 of CTE, 226–227 of date, 145–146, 174–175 displaying group, 80–83 within dynamic date ranges, 113 efficiency within, 189 granularity within, 145 GROUP BY statement and, 79 within rows, 86–87 of sales, 145–146, 174–175, 211 SUM function and, 80–81 of time, 145–146 of time period, 167–171 of variables, 192 summary tables, 7 supervised learning model, 173 symbols, 16, 21, 22–23, 43, 62, 236 syntax, of SELECT query, 16 syntax-­highlighting of SQL, 3 SYSDATE function, 121 T table alias of, 66, 89 Book, 4, 6–7, 13 connector symbols within, 62 customer table calculations within, 20–22 columns within, 24–26 demographic data and, 211–217 example of, 59 exercises for, 30 joining with, 86–87, 211 IN keyword within, 42–43 LIMIT clause and, 26–29 new versus returning counting, 167–171 WHERE clause within, 33–38, 40–41 customer_purchases table calculated views within, 154–155 WITH clause and, 167–171 custom analytical datasets within, 149–153 demographic data and, 211–217 example using, 80–81, 82–86, 120–125,", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 151 + }, + { + "text": "59 exercises for, 30 joining with, 86–87, 211 IN keyword within, 42–43 LIMIT clause and, 26–29 new versus returning counting, 167–171 WHERE clause within, 33–38, 40–41 customer_purchases table calculated views within, 154–155 WITH clause and, 167–171 custom analytical datasets within, 149–153 demographic data and, 211–217 example using, 80–81, 82–86, 120–125, 193, 194–196, 204–208 exercise of, 126, 142 Index ■ T–T 263 feature engineering and, 186–188 feature set and, 181–185 GROUP BY clause and, 179–181 joining with, 71–74, 94 market_date_info table and, 174–175 sales tracking within, 146–148 vendor_inventory table and, 136–142 datetime_demo table, 118 dimension, 8 EDA exploring over multiple, 135–138 granularity of, 82, 129 JOINS of, 74–76, 86 lookup, 140 market_date_info table binary flags within, 52–53 customer_purchases table and, 174–175 datetime field values within, 114–118 example using, 220 exercise of, 126 filtering within, 46–47 UNION query within, 159–160 market_vendor_inventory table, 160–161 medical database, 4–6, 8–9 product table example of, 17, 18 exercise of, 77, 142 exercises for, 60 exploring, 128–131 IS NULL keyword within, 44–46 joining with, 61–71, 94 LIMIT clause and, 17 output comparison within, 39–40 possible column values within, 131–133 snapshot from, 231 sorting within, 19 UNION query within, 160–161, 162 WHERE clause within, 32, 39–40 product_category table, 61–71, 77, 129–131, 192–194 product_units table, 233–236 purchase table, 53–56 relationship between, 4 storing datasets as, 229–232 summary, 7 updating, 10 vendor table categorical encoding within, 56–59 custom analytical datasets within, 149–153 example of, 29–30, 59–60 exercise of, 76 joining with, 74–76, 86–87, 148 BETWEEN keyword within, 41–42 types within, 51 vendor booth assignment example within, 18, 20 WHERE clause within, 41 vendor_booth_assignments table, 75–76, 108–111 vendor_inventory table aggregate window functions within, 103–108 averaging within, 92 AVG () window function within, 103–104 calculated views within, 154–155 changes over time within, 134–135 customer_purchases table and, 136–142 DENSE RANK function within, 101–102 example using, 88–89, 90–91, 132–133, 193, 194, 217–219 exercise of, 77, 142 granularity of, 132 HAVING clause within, 93–94 NTILE function within, 102–103 query from, 201–204 RANK function within, 101–102 ROUND () function within, 104–105 ROW_NUMBER () within, 98–99 UNION query within, 162 Tableau dataset into, 141 Exploratory Data Analysis (EDA) within, 127 forecasting function of, 175–176 front-­end filtering within, 88 SQL into, 79, 152–153, 156–157 summary within, 145 target variable, 13, 177, 178–179 THEN statement, 50 time, 114–115, 134–135, 145–146, 167–171 time bounding, for target variables, 178 TIME () function, 115–116 time of year/season, comparison of, 192 time series model, 174–176 time zones, 116, 121, 232 time-­bound predictive models, 113 time-­series analysis, 113 timestamp, 115, 119, 232 TIMESTAMPDIFF function, 119 to-­date maximum, 163–167 TODAY function, 121 TOP clause, 17 trained model, testing of, 173 training data, 174, 177, 186 training dataset, creating, 178–181 training example, 13 transactional data, 33 TRIM function, 44 TRUE evaluation, 33, 34–40, 42, 50, 241 tuple, 3 264 Index ■ U–Z U UNION query, 159–163, 171 unstructured data, 1 UPDATE statement, 232, 233, 234 UPPER function, 25, 51 UTC (Coordinated Universal Time), 232 V value, distribution of, 10–11 value, updating, 233–236 variables, 13, 56–59, 177,", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 152 + }, + { + "text": "TRIM function, 44 TRUE evaluation, 33, 34–40, 42, 50, 241 tuple, 3 264 Index ■ U–Z U UNION query, 159–163, 171 unstructured data, 1 UPDATE statement, 232, 233, 234 UPPER function, 25, 51 UTC (Coordinated Universal Time), 232 V value, distribution of, 10–11 value, updating, 233–236 variables, 13, 56–59, 177, 178–179, 191–192 vendor table categorical encoding within, 56–59 custom analytical datasets within, 149–153 example of, 29–30, 59–60 exercise of, 76 joining with, 74–76, 86–87, 148 BETWEEN keyword within, 41–42 types within, 51 vendor booth assignment example within, 18, 20 WHERE clause within, 41 vendor_booth_assignments table, 75–76, 108–111 vendor_inventory table aggregate window functions within, 103–108 averaging within, 92 AVG () window function within, 103–104 calculated views within, 154–155 changes over time within, 134–135 customer_purchases table and, 136–142 DENSE RANK function within, 101–102 example using, 88–89, 90–91, 132–133, 193, 194, 217–219 exercise of, 77, 142 granularity of, 132 HAVING clause within, 93–94 NTILE function within, 102–103 query from, 201–204 RANK function within, 101–102 ROUND () function within, 104–105 ROW_NUMBER () within, 98–99 UNION query within, 162 view, 149–153, 204–205, 229–232 vw_ prefix, 151 W weather classification model, 173, 174 WEEK function, 147 WHEN statement, 50 WHERE clause condition of, 87 date filtering in, 93 defined, 100 differences between, 161 error from, 101 example of, 70 exercise using, 241 filtering uses within, 41–44, 84, 197 location of, 32 multi-­column conditional filtering within, 40–41 multiple condition filtering within, 34–40 overview of, 31 querying databases using, 17 removing, 141, 166 row removal using, 122 ROW_NUMBER () window function to, 184–185 syntax of, 16, 80, 233, 234 TOP clause and, 17 wildcard character, 43 wildcard comparison, 41 window functions. See also functions aggregate, 103–108 COUNT (), 112, 246 date functions within, 119–125 defined, 97 DENSE RANK (), 101–102 NTILE, 102–103, 224–225 within outer subquery, 122 PARTITION BY, 98, 101, 107–108 partitioning by, 168 RANK (), 100, 101–102, 121–122, 161, 162 ROW_NUMBER (), 98–101, 102, 183, 184–185, 245 use examples of, 97 window naming, 111 WITH clause analytical dataset and, 149–153 defined, 124–125 example of, 162 exercise using, 190 ROW_NUMBER window function and, 183 subquery using, 248 use of, 151, 161, 164, 168, 204–205, 209 Workbench Community Edition (MySQL), 3, 132 Y YEAR function, 147 Z Zip Code Tabulation Areas (ZCTAs), 211 WILEY END USER LICENSE AGREEMENT Go to www.wiley.com/go/eula to access Wiley’s ebook EULA.", + "source": "Renee-M-Teate-SQL-for-Data-Scientists-Wiley-2021.pdf", + "chunk_id": 153 + }, + { + "text": "SQL for Data Analysis Advanced Techniques for Transforming Data into Insights Cathy Tanimura Cathy Tanimura SQL for Data Analysis Advanced Techniques for Transforming Data into Insights Boston Farnham Sebastopol Tokyo Beijing Boston Farnham Sebastopol Tokyo Beijing 978-1-492-08878-3 [LSI] SQL for Data Analysis by Cathy Tanimura Copyright © 2021 Cathy Tanimura. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Andy Kwan Development Editors Amelia Blevins and Shira Evans Production Editor: Kristen Brown Copyeditor: Arthur Johnson Proofreader: Paula L. Fleming Indexer: Ellen Troutman-Zaig Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea September 2021: First Edition Revision History for the First Edition 2021-09-09: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781492088783 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. SQL for Data Analysis, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix 1. Analysis with SQL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 What Is Data Analysis? 1 Why SQL? 4 What Is SQL? 4 Benefits of SQL 7 SQL Versus R or Python 8 SQL as Part of the Data Analysis Workflow 9 Database Types and How to Work with Them 12 Row-Store Databases 13 Column-Store Databases 15 Other Types of", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 0 + }, + { + "text": "1 What Is Data Analysis? 1 Why SQL? 4 What Is SQL? 4 Benefits of SQL 7 SQL Versus R or Python 8 SQL as Part of the Data Analysis Workflow 9 Database Types and How to Work with Them 12 Row-Store Databases 13 Column-Store Databases 15 Other Types of Data Infrastructure 16 Conclusion 17 2. Preparing Data for Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Types of Data 20 Database Data Types 20 Structured Versus Unstructured 22 Quantitative Versus Qualitative Data 22 First-, Second-, and Third-Party Data 23 Sparse Data 24 SQL Query Structure 25 Profiling: Distributions 27 Histograms and Frequencies 28 Binning 31 n-Tiles 33 iii Profiling: Data Quality 35 Detecting Duplicates 36 Deduplication with GROUP BY and DISTINCT 38 Preparing: Data Cleaning 39 Cleaning Data with CASE Transformations 39 Type Conversions and Casting 42 Dealing with Nulls: coalesce, nullif, nvl Functions 45 Missing Data 47 Preparing: Shaping Data 52 For Which Output: BI, Visualization, Statistics, ML 52 Pivoting with CASE Statements 53 Unpivoting with UNION Statements 55 pivot and unpivot Functions 57 Conclusion 59 3. Time Series Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 Date, Datetime, and Time Manipulations 62 Time Zone Conversions 62 Date and Timestamp Format Conversions 64 Date Math 68 Time Math 71 Joining Data from Different Sources 72 The Retail Sales Data Set 74 Trending the Data 75 Simple Trends 75 Comparing Components 77 Percent of Total Calculations 86 Indexing to See Percent Change over Time 90 Rolling Time Windows 95 Calculating Rolling Time Windows 97 Rolling Time Windows with Sparse Data 102 Calculating Cumulative Values 104 Analyzing with Seasonality 107 Period-over-Period Comparisons: YoY and MoM 109 Period-over-Period Comparisons: Same Month Versus Last Year 112 Comparing to Multiple Prior Periods 116 Conclusion 119 4. Cohort Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 Cohorts: A Useful Analysis Framework 122 The Legislators Data Set 125 iv | Table of Contents Retention 127 SQL for a Basic Retention Curve 128 Adjusting Time Series to Increase Retention Accuracy 131 Cohorts Derived from the Time Series Itself 137 Defining the Cohort from a Separate Table 142 Dealing with Sparse Cohorts 146", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 1 + }, + { + "text": "Framework 122 The Legislators Data Set 125 iv | Table of Contents Retention 127 SQL for a Basic Retention Curve 128 Adjusting Time Series to Increase Retention Accuracy 131 Cohorts Derived from the Time Series Itself 137 Defining the Cohort from a Separate Table 142 Dealing with Sparse Cohorts 146 Defining Cohorts from Dates Other Than the First Date 151 Related Cohort Analyses 153 Survivorship 154 Returnship, or Repeat Purchase Behavior 158 Cumulative Calculations 163 Cross-Section Analysis, Through a Cohort Lens 166 Conclusion 174 5. Text Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 Why Text Analysis with SQL? 175 What Is Text Analysis? 176 Why SQL Is a Good Choice for Text Analysis 176 When SQL Is Not a Good Choice 177 The UFO Sightings Data Set 178 Text Characteristics 179 Text Parsing 182 Text Transformations 187 Finding Elements Within Larger Blocks of Text 195 Wildcard Matches: LIKE, ILIKE 195 Exact Matches: IN, NOT IN 200 Regular Expressions 203 Constructing and Reshaping Text 218 Concatenation 218 Reshaping Text 222 Conclusion 226 6. Anomaly Detection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 Capabilities and Limits of SQL for Anomaly Detection 228 The Data Set 229 Detecting Outliers 230 Sorting to Find Anomalies 231 Calculating Percentiles and Standard Deviations to Find Anomalies 234 Graphing to Find Anomalies Visually 241 Forms of Anomalies 250 Anomalous Values 250 Table of Contents | v Anomalous Counts or Frequencies 254 Anomalies from the Absence of Data 258 Handling Anomalies 260 Investigation 260 Removal 260 Replacement with Alternate Values 262 Rescaling 264 Conclusion 266 7. Experiment Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 Strengths and Limits of Experiment Analysis with SQL 269 The Data Set 270 Types of Experiments 272 Experiments with Binary Outcomes: The Chi-Squared Test 272 Experiments with Continuous Outcomes: The t-Test 274 Challenges with Experiments and Options for Rescuing Flawed Experiments 276 Variant Assignment 277 Outliers 278 Time Boxing 279 Repeated Exposure Experiments 280 When Controlled Experiments Aren’t Possible: Alternative Analyses 282 Pre-/Post-Analysis 282 Natural Experiment Analysis 284 Analysis of Populations Around a Threshold 286 Conclusion 286 8. Creating Complex Data", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 2 + }, + { + "text": "274 Challenges with Experiments and Options for Rescuing Flawed Experiments 276 Variant Assignment 277 Outliers 278 Time Boxing 279 Repeated Exposure Experiments 280 When Controlled Experiments Aren’t Possible: Alternative Analyses 282 Pre-/Post-Analysis 282 Natural Experiment Analysis 284 Analysis of Populations Around a Threshold 286 Conclusion 286 8. Creating Complex Data Sets for Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 When to Use SQL for Complex Data Sets 287 Advantages of Using SQL 288 When to Build into ETL Instead 288 When to Put Logic in Other Tools 290 Code Organization 292 Commenting 292 Capitalization, Indentation, Parentheses, and Other Formatting Tricks 293 Storing Code 296 Organizing Computations 296 Understanding Order of SQL Clause Evaluation 296 Subqueries 300 Temporary Tables 302 Common Table Expressions 303 grouping sets 305 vi | Table of Contents Managing Data Set Size and Privacy Concerns 308 Sampling with %, mod 308 Reducing Dimensionality 310 PII and Data Privacy 314 Conclusion 316 9. Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317 Funnel Analysis 317 Churn, Lapse, and Other Definitions of Departure 319 Basket Analysis 323 Resources 325 Books and Blogs 325 Data Sets 326 Final Thoughts 327 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329 Table of Contents | vii Preface Over the past 20 years, I’ve spent many of my working hours manipulating data with SQL. For most of those years, I’ve worked in technology companies spanning a wide range of consumer and business-to-business industries. In that time, volumes of data have increased dramatically, and the technology I get to use has improved by leaps and bounds. Databases are faster than ever, and the reporting and visualization tools used to communicate the meaning in the data are more powerful than ever. One thing that has remained remarkably constant, however, is SQL being a key part of my toolbox. I remember when I first learned SQL. I started my career in finance, where spread‐ sheets rule, and I’d gotten pretty good at writing formulas and memorizing all those keyboard shortcuts. One day I totally geeked out and Ctrl- and Alt-clicked every key on my keyboard just", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 3 + }, + { + "text": "of my toolbox. I remember when I first learned SQL. I started my career in finance, where spread‐ sheets rule, and I’d gotten pretty good at writing formulas and memorizing all those keyboard shortcuts. One day I totally geeked out and Ctrl- and Alt-clicked every key on my keyboard just to see what would happen (and then created a cheat sheet for my peers). That was part fun and part survival: the faster I was with my spreadsheets, the more likely I would be to finish my work before midnight so I could go home and get some sleep. Spreadsheet mastery got me in the door at my next role, a startup where I was first introduced to databases and SQL. Part of my role involved crunching inventory data in spreadsheets, and thanks to early internet scale, the data sets were sometimes tens of thousands of rows. This was “big data” at the time, at least for me. I got in the habit of going for a cup of coffee or for lunch while my computer’s CPU was occupied with running its vlookup magic. One day my manager went on vacation and asked me to tend to the data warehouse he’d built on his laptop using Access. Refreshing the data involved a series of steps: running SQL queries in a portal, loading the resulting csv files into the database, and then refreshing the spreadsheet reports. After the first successful load, I started tin‐ kering, trying to understand how it worked, and pestering the engineers to show me how to modify the SQL queries. I was hooked, and even when I thought I might change directions with my career, I’ve kept coming back to data. Manipulating data, answering questions, helping my ix colleagues work better and smarter, and learning about businesses and the world through sets of data have never stopped feeling fun and exciting. When I started working with SQL, there weren’t many learning resources. I got a book on basic syntax, read it in a night, and from there mostly learned through trial and error. Back in the days when I was learning, I queried production databases directly and brought the website down more than once with my overly ambitious (or more likely just poorly written) SQL. Fortunately my skills improved, and over the years I learned to work forward from the data in tables, and backward from the out‐ put needed, solving technical and logic challenges and puzzles to write queries that returned the right data. I ended up designing and building data warehouses to gather data from different sources and avoid bringing down critical production databases. I’ve learned a lot about when and how to aggregate data before writing the SQL query and when to leave data in a more raw form. I’ve compared notes with others who got into data around the same time, and it’s clear we mostly learned in the same ad hoc way. The lucky among us had peers with whom to share", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 4 + }, + { + "text": "before writing the SQL query and when to leave data in a more raw form. I’ve compared notes with others who got into data around the same time, and it’s clear we mostly learned in the same ad hoc way. The lucky among us had peers with whom to share techniques. Most SQL texts are either introductory and basic (there’s definitely a place for these!) or else aimed at database developers. There are few resources for advanced SQL users who are focused on analysis work. Knowledge tends to be locked up in individuals or small teams. A goal of this book is to change that, giving practitioners a reference for how to solve common analysis problems with SQL, and I hope inspiring new inquiries into data using techniques you might not have seen before. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, file extensions, and keywords. Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, environment variables, and statements. Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. x | Preface This element signifies a tip or suggestion. This element signifies a general note. This element indicates a warning or caution. Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/cathytanimura/sql_book. If you have a technical question or a problem using the code examples, please send email to bookquestions@oreilly.com. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “SQL for Data Analysis by Cathy Tanimura (O’Reilly). Copyright 2021 Cathy Tanimura, 978-1-492-08878-3.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. Preface | xi O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 5 + }, + { + "text": "For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/sql-data-analysis. Email bookquestions@oreilly.com to comment or ask technical questions about this book. For news and information about our books and courses, visit http://oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://www.youtube.com/oreillymedia Acknowledgments This book wouldn’t have been possible without the efforts of a number of people at O’Reilly. Andy Kwan recruited me to this project. Amelia Blevins and Shira Evans guided me through the process and gave helpful feedback along the way. Kristen Brown shepherded the book through the production process. Arthur Johnson improved the quality and clarity of the text and inadvertently made me think more deeply about SQL keywords. xii | Preface Many colleagues over the years played an important role in my SQL journey, and I’m grateful for their tutorials, tips, and shared code, and the time spent brainstorming ways to solve analysis problems over the years. Sharon Lin opened my eyes to regular expressions. Elyse Gordon gave me lots of book-writing advice. Dave Hoch and our conversations about experiment analysis inspired Chapter 7. Dan, Jim, and Stu from the Star Chamber have long been my favorite guys to geek out with. I’m also grateful for all of the colleagues who asked hard questions over the years—and once those were answered, asked even harder ones. I’d like to thank my husband Rick, son Shea, daughters Lily and Fiona, and mom Janet for their love, encouragement, and most of all the gift of time to work on this book. Amy, Halle, Jessi, and the Den of Slack kept me sane and laughing through months of writing and pandemic lockdown. Preface | xiii CHAPTER 1 Analysis with SQL If you’re reading this book, you’re probably interested in data analysis and in using SQL to accomplish it. You may be experienced with data analysis but new to SQL, or perhaps you’re experienced with SQL but new to data analysis. Or you may be new to both topics entirely. Whatever your starting point, this chapter lays the groundwork for the topics covered in the rest of the book and makes sure we have a common vocabulary. I’ll start with a discussion of what data analysis", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 6 + }, + { + "text": "but new to data analysis. Or you may be new to both topics entirely. Whatever your starting point, this chapter lays the groundwork for the topics covered in the rest of the book and makes sure we have a common vocabulary. I’ll start with a discussion of what data analysis is and then move on to a discussion of SQL: what it is, why it’s so popular, how it compares to other tools, and how it fits into data analysis. Then, since modern data analysis is so intertwined with the technologies that have enabled it, I’ll conclude with a discussion of different types of databases that you may encounter in your work, why they’re used, and what all of that means for the SQL you write. What Is Data Analysis? Collecting and storing data for analysis is a very human activity. Systems to track stores of grain, taxes, and the population go back thousands of years, and the roots of statistics date back hundreds of years. Related disciplines, including statistical process control, operations research, and cybernetics, exploded in the 20th century. Many dif‐ ferent names are used to describe the discipline of data analysis, such as business intelligence (BI), analytics, data science, and decision science, and practitioners have a range of job titles. Data analysis is also done by marketers, product managers, busi‐ ness analysts, and a variety of other people. In this book, I’ll use the terms data ana‐ lyst and data scientist interchangeably to mean the person working with SQL to understand data. I will refer to the software used to build reports and dashboards as BI tools. Data analysis in the contemporary sense was enabled by, and is intertwined with, the history of computing. Trends in both research and commercialization have shaped it, 1 and the story includes a who’s who of researchers and major companies, which we’ll talk about in the section on SQL. Data analysis blends the power of computing with techniques from traditional statistics. Data analysis is part data discovery, part data interpretation, and part data communication. Very often the purpose of data analysis is to improve decision making, by humans and increasingly by machines through automation. Sound methodology is critical, but analysis is about more than just producing the right number. It’s about curiosity, asking questions, and the “why” behind the num‐ bers. It’s about patterns and anomalies, discovering and interpreting clues about how businesses and humans behave. Sometimes analysis is done on a data set gathered to answer a specific question, as in a scientific setting or an online experiment. Analysis is also done on data that is generated as a result of doing business, as in sales of a company’s products, or that is generated for analytics purposes, such as user interac‐ tion tracking on websites and mobile apps. This data has a wide range of possible applications, from troubleshooting to planning user interface (UI) improvements, but it often arrives in a format and volume such that the data needs processing", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 7 + }, + { + "text": "or that is generated for analytics purposes, such as user interac‐ tion tracking on websites and mobile apps. This data has a wide range of possible applications, from troubleshooting to planning user interface (UI) improvements, but it often arrives in a format and volume such that the data needs processing before yielding answers. Chapter 2 will cover preparing data for analysis, and Chapter 8 will discuss some of the ethical and privacy concerns with which all data practitioners should be familiar. It’s hard to think of an industry that hasn’t been touched by data analysis: manufac‐ turing, retail, finance, health care, education, and even government have all been changed by it. Sports teams have employed data analysis since the early years of Billy Beane’s term as general manager of the Oakland Athletics, made famous by Michael Lewis’s book Moneyball (Norton). Data analysis is used in marketing, sales, logistics, product development, user experience design, support centers, human resources, and more. The combination of techniques, applications, and computing power has led to the explosion of related fields such as data engineering and data science. Data analysis is by definition done on historical data, and it’s important to remember that the past doesn’t necessarily predict the future. The world is dynamic, and organi‐ zations are dynamic as well—new products and processes are introduced, competi‐ tors rise and fall, sociopolitical climates change. Criticisms are leveled against data analysis for being backward looking. Though that characterization is true, I have seen organizations gain tremendous value from analyzing historical data. Mining histori‐ cal data helps us understand the characteristics and behavior of customers, suppliers, and processes. Historical data can help us develop informed estimates and predicted ranges of outcomes, which will sometimes be wrong but quite often will be right. Past data can point out gaps, weaknesses, and opportunities. It allows organizations to optimize, save money, and reduce risk and fraud. It can also help organizations find opportunity, and it can become the building blocks of new products that delight customers. 2 | Chapter 1: Analysis with SQL Organizations that don’t do some form of data analysis are few and far between these days, but there are still some holdouts. Why do some organizations not use data analysis? One argument is the cost-to-value ratio. Collecting, processing, and analyzing data takes work and some level of financial investment. Some organizations are too new, or they’re too haphazard. If there isn’t a consistent pro‐ cess, it’s hard to generate data that’s consistent enough to analyze. Finally, there are ethical considerations. Collecting or storing data about certain people in certain situations may be regulated or even banned. Data about children and health-care interventions is sensi‐ tive, for example, and there are extensive regulations around its collection. Even organizations that are otherwise data driven need to take care around customer privacy and to think hard about what data should be collected, why it is needed, and how long it should be stored. Regulations such as the European Union’s General Data", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 8 + }, + { + "text": "there are extensive regulations around its collection. Even organizations that are otherwise data driven need to take care around customer privacy and to think hard about what data should be collected, why it is needed, and how long it should be stored. Regulations such as the European Union’s General Data Protection Regulation, or GDPR, and the California Consumer Pri‐ vacy Act, or CCPA, have changed the way businesses think about consumer data. We’ll discuss these regulations in more depth in Chapter 8. As data practitioners, we should always be thinking about the ethical implications of our work. When working with organizations, I like to tell people that data analysis is not a project that wraps up at a fixed date—it’s a way of life. Developing a data-informed mindset is a process, and reaping the rewards is a journey. Unknowns become known, difficult questions are chipped away at until there are answers, and the most critical information is embedded in dashboards that power tactical and strategic deci‐ sions. With this information, new and harder questions are asked, and then the pro‐ cess repeats. Data analysis is both accessible for those looking to get started and hard to master. The technology can be learned, particularly SQL. Many problems, such as optimizing marketing spend or detecting fraud, are familiar and translate across businesses. Every organization is different and every data set has quirks, so even familiar prob‐ lems can pose new challenges. Communicating results is a skill. Learning to make good recommendations and becoming a trusted partner to an organization take time. In my experience, simple analysis presented persuasively has more impact than sophisticated analysis presented poorly. Successful data analysis also requires partner‐ ship. You can have great insights, but if there is no one to execute on them, you haven’t really made an impact. Even with all the technology, it’s still about people, and relationships matter. What Is Data Analysis? | 3 Why SQL? This section describes what SQL is, the benefits of using it, how it compares to other languages commonly used for analysis, and finally how SQL fits into the analysis workflow. What Is SQL? SQL is the language used to communicate with databases. The acronym stands for Structured Query Language and is pronounced either like “sequel” or by saying each letter, as in “ess cue el.” This is only the first of many controversies and inconsisten‐ cies surrounding SQL that we’ll see, but most people will know what you mean regardless of how you say it. There is some debate as to whether SQL is or isn’t a pro‐ gramming language. It isn’t a general purpose language in the way that C or Python are. SQL without a database and data in tables is just a text file. SQL can’t build a web‐ site, but it is powerful for working with data in databases. On a practical level, what matters most is that SQL can help you get the job of data analysis done. IBM was the first to", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 9 + }, + { + "text": "and data in tables is just a text file. SQL can’t build a web‐ site, but it is powerful for working with data in databases. On a practical level, what matters most is that SQL can help you get the job of data analysis done. IBM was the first to develop SQL databases, from the relational model invented by Edgar Codd in the 1960s. The relational model was a theoretical description for man‐ aging data using relationships. By creating the first databases, IBM helped to advance the theory, but it also had commercial considerations, as did Oracle, Microsoft, and every other company that has commercialized a database since. From the beginning, there has been tension between computer theory and commercial reality. SQL became an International Organization for Standards (ISO) standard in 1987 and an American National Standards Institute (ANSI) standard in 1986. Although all major databases start from these standards in their implementation of SQL, many have var‐ iations and functions that make life easier for the users of those databases. These come at the cost of making SQL more difficult to move between databases without some modifications. SQL is used to access, manipulate, and retrieve data from objects in a database. Data‐ bases can have one or more schemas, which provide the organization and structure and contain other objects. Within a schema, the objects most commonly used in data analysis are tables, views, and functions. Tables contain fields, which hold the data. Tables may have one or more indexes; an index is a special kind of data structure that allows data to be retrieved more efficiently. Indexes are usually defined by a database administrator. Views are essentially stored queries that can be referenced in the same way as a table. Functions allow commonly used sets of calculations or procedures to be stored and easily referenced in queries. They are usually created by a database administrator, or DBA. Figure 1-1 gives an overview of the organization of databases. 4 | Chapter 1: Analysis with SQL Figure 1-1. Overview of database organization and objects in a database To communicate with databases, SQL has four sublanguages for tackling different jobs, and these are mostly standard across database types. Most people who work in data analysis don’t need to recall the names of these sublanguages on a daily basis, but they might come up in conversation with database administrators or data engineers, so I’ll briefly introduce them. The commands all work fluidly together, and some may coexist in the same SQL statement. DQL, or data query language, is what this book is mainly about. It’s used for querying data, which you can think of as using code to ask questions of a database. DQL com‐ mands include SELECT, which will be familiar to prior users of SQL, but the acronym DQL is not frequently used in my experience. SQL queries can be as short as a single line or span many tens of lines. SQL queries can access a single table (or view),", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 10 + }, + { + "text": "DQL com‐ mands include SELECT, which will be familiar to prior users of SQL, but the acronym DQL is not frequently used in my experience. SQL queries can be as short as a single line or span many tens of lines. SQL queries can access a single table (or view), can combine data from multiple tables through the use of joins, and can also query across multiple schemas in the same database. SQL queries generally cannot query across databases, but in some cases clever network settings or additional software can be used to retrieve data from multiple sources, even databases of different types. SQL queries are self-contained and, apart from tables, do not reference variables or out‐ puts from previous steps not contained in the query, unlike scripting languages. DDL, or data definition language, is used to create and modify tables, views, users, and other objects in the database. It affects the structure but not the contents. There are three common commands: CREATE, ALTER, and DROP. CREATE is used to Why SQL? | 5 make new objects. ALTER changes the structure of an object, such as by adding a col‐ umn to a table. DROP deletes the entire object and its structure. You might hear DBAs and data engineers talk about working with DDLs—this is really just shorthand for the files or pieces of code that do the creates, alters, or drops. An example of how DDL is used in the context of analysis is the code to create temporary tables. DCL, or data control language, is used for access control. Commands include GRANT and REVOKE, which give permission and remove permission, respectively. In an analysis context, GRANT might be needed to allow a colleague to query a table you created. You might also encounter such a command when someone has told you a table exists in the database but you can’t see it—permissions might need to be GRANTed to your user. DML, or data manipulation language, is used to act on the data itself. The commands are INSERT, UPDATE, and DELETE. INSERT adds new records and is essentially the “load” step in extract, transform, load (ETL). UPDATE changes values in a field, and DELETE removes rows. You will encounter these commands if you have any kind of self-managed tables—temp tables, sandbox tables—or if you find yourself in the role of both owner and analyzer of the database. These four sublanguages are present in all major databases. In this book, I’ll focus mainly on DQL. We will touch on a few DDL and DML commands in Chapter 8, and you will also see some examples in the GitHub site for the book, where they are used to create and populate the data used in examples. Thanks to this common set of com‐ mands, SQL code written for any database will look familiar to anyone used to work‐ ing with SQL. However, reading SQL from another database may feel a bit like listening to someone who speaks the same language as", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 11 + }, + { + "text": "the data used in examples. Thanks to this common set of com‐ mands, SQL code written for any database will look familiar to anyone used to work‐ ing with SQL. However, reading SQL from another database may feel a bit like listening to someone who speaks the same language as you but comes from another part of the country or the world. The basic structure of the language is the same, but the slang is different, and some words have different meanings altogether. Variations in SQL from database to database are often termed dialects, and database users will reference Oracle SQL, MSSQL, or other dialects. Still, once you know SQL, you can work with different database types as long as you pay attention to details such as the handling of nulls, dates, and timestamps; the divi‐ sion of integers; and case sensitivity. This book uses PostgreSQL, or Postgres, for the examples, though I will try to point out where the code would be meaningfully different in other types of databases. You can install Postgres on a personal computer in order to follow along with the examples. 6 | Chapter 1: Analysis with SQL Benefits of SQL There are many good reasons to use SQL for data analysis, from computing power to its ubiquity in data analysis tools and its flexibility. Perhaps the best reason to use SQL is that much of the world’s data is already in data‐ bases. It’s likely your own organization has one or more databases. Even if data is not already in a database, loading it into one can be worthwhile in order to take advan‐ tage of the storage and computing advantages, especially when compared to alterna‐ tives such as spreadsheets. Computing power has exploded in recent years, and data warehouses and data infrastructure have evolved to take advantage of it. Some newer cloud databases allow massive amounts of data to be queried in memory, speeding things up further. The days of waiting minutes or hours for query results to return may be over, though analysts may just write more complex queries in response. SQL is the de facto standard for interacting with databases and retrieving data from them. A wide range of popular software connects to databases with SQL, from spreadsheets to BI and visualization tools and coding languages such as Python and R (discussed in the next section). Due to the computing resources available, performing as much data manipulation and aggregation as possible in the database often has advantages downstream. We’ll discuss strategies for building complex data sets for downstream tools in depth in Chapter 8. The basic SQL building blocks can be combined in an endless number of ways. Start‐ ing with a relatively small number of building blocks—the syntax—SQL can accom‐ plish a wide array of tasks. SQL can be developed iteratively, and it’s easy to review the results as you go. It may not be a full-fledged programming language, but it can do a lot, from transforming data to performing", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 12 + }, + { + "text": "a relatively small number of building blocks—the syntax—SQL can accom‐ plish a wide array of tasks. SQL can be developed iteratively, and it’s easy to review the results as you go. It may not be a full-fledged programming language, but it can do a lot, from transforming data to performing complex calculations and answering questions. Last, SQL is relatively easy to learn, with a finite amount of syntax. You can learn the basic keywords and structure quickly and then hone your craft over time working with varied data sets. Applications of SQL are virtually infinite, when you take into account the range of data sets in the world and the possible questions that can be asked of data. SQL is taught in many universities, and many people pick up some skills on the job. Even employees who don’t already have SQL skills can be trained, and the learning curve may be easier than that for other programming languages. This makes storing data for analysis in relational databases a logical choice for organizations. Why SQL? | 7 SQL Versus R or Python While SQL is a popular language for data analysis, it isn’t the only choice. R and Python are among the most popular of the other languages used for data analysis. R is a statistical and graphing language, while Python is a general-purpose programming language that has strengths in working with data. Both are open source, can be installed on a laptop, and have active communities developing packages, or exten‐ sions, that tackle various data manipulation and analysis tasks. Choosing between R and Python is beyond the scope of this book, but there are many discussions online about the relative advantages of each. Here I will consider them together as coding- language alternatives to SQL. One major difference between SQL and other coding languages is where the code runs and, therefore, how much computing power is available. SQL always runs on a database server, taking advantage of all its computing resources. For doing analysis, R and Python are usually run locally on your machine, so computing resources are cap‐ ped by whatever is available locally. There are, of course, lots of exceptions: databases can run on laptops, and R and Python can be run on servers with more resources. When you are performing anything other than the simplest analysis on large data sets, pushing work onto a database server with more resources is a good option. Since databases are usually set up to continually receive new data, SQL is also a good choice when a report or dashboard needs to update periodically. A second difference is in how data is stored and organized. Relational databases always organize data into rows and columns within tables, so SQL assumes this struc‐ ture for every query. R and Python have a wider variety of ways to store data, includ‐ ing variables, lists, and dictionaries, among other options. These provide more flexibility, but at the cost of a steeper learning curve. To facilitate data", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 13 + }, + { + "text": "and columns within tables, so SQL assumes this struc‐ ture for every query. R and Python have a wider variety of ways to store data, includ‐ ing variables, lists, and dictionaries, among other options. These provide more flexibility, but at the cost of a steeper learning curve. To facilitate data analysis, R has data frames, which are similar to database tables and organize data into rows and col‐ umns. The pandas package makes DataFrames available in Python. Even when other options are available, the table structure remains valuable for analysis. Looping is another major difference between SQL and most other computer pro‐ gramming languages. A loop is an instruction or a set of instructions that repeats until a specified condition is met. SQL aggregations implicitly loop over the set of data, without any additional code. We will see later how the lack of ability to loop over fields can result in lengthy SQL statements when pivoting or unpivoting data. While deeper discussion is beyond the scope of this book, some vendors have created exten‐ sions to SQL, such as PL/SQL in Oracle and T-SQL in Microsoft SQL Server, that allow functionality such as looping. 8 | Chapter 1: Analysis with SQL 1 There are some newer technologies that allow SQL queries on data stored in nonrelational sources. A drawback of SQL is that your data must be in a database,1 whereas R and Python can import data from files stored locally or can access files stored on servers or web‐ sites. This is convenient for many one-off projects. A database can be installed on a laptop, but this does add an extra layer of overhead. In the other direction, packages such as dbplyr for R and SQLAlchemy for Python allow programs written in those languages to connect to databases, execute SQL queries, and use the results in further processing steps. In this sense, R or Python can be complementary to SQL. R and Python both have sophisticated statistical functions that are either built in or available in packages. Although SQL has, for example, functions to calculate average and standard deviation, calculations of p-values and statistical significance that are needed in experiment analysis (discussed in Chapter 7) cannot be performed with SQL alone. In addition to sophisticated statistics, machine learning is another area that is better tackled with one of these other coding languages. When deciding whether to use SQL, R, or Python for an analysis, consider: • Where is the data located—in a database, a file, a website? • What is the volume of data? • Where is the data going—into a report, a visualization, a statistical analysis? • Will it need to be updated or refreshed with new data? How often? • What does your team or organization use, and how important is it to conform to existing standards? There is no shortage of debate around which languages and tools are best for doing data analysis or data science. As with many things, there’s often more than one way", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 14 + }, + { + "text": "How often? • What does your team or organization use, and how important is it to conform to existing standards? There is no shortage of debate around which languages and tools are best for doing data analysis or data science. As with many things, there’s often more than one way to accomplish an analysis. Programming languages evolve and change in popularity, and we’re lucky to live and work in a time with so many good choices. SQL has been around for a long time and will likely remain popular for years to come. The ultimate goal is to use the best available tool for the job. This book will help you get the most out of SQL for data analysis, regardless of what else is in your toolkit. SQL as Part of the Data Analysis Workflow Now that I’ve explained what SQL is, discussed some of its benefits, and compared it to other languages, we’ll turn to a discussion of where SQL fits in the data analysis process. Analysis work always starts with a question, which may be about how many new customers have been acquired, how sales are trending, or why some users stick around for a long time while others try a service and never return. Once the question is framed, we consider where the data originated, where the data is stored, the Why SQL? | 9 analysis plan, and how the results will be presented to the audience. Figure 1-2 shows the steps in the process. Queries and analysis are the focus of this book, though I will discuss the other steps briefly in order to put the queries and analysis stage into a broader context. Figure 1-2. Steps in the data analysis process First, data is generated by source systems, a term that includes any human or machine process that generates data of interest. Data can be generated by people by hand, such as when someone fills out a form or takes notes during a doctor’s visit. Data can also be machine generated, such as when an application database records a purchase, an event-streaming system records a website click, or a marketing management tool records an email open. Source systems can generate many different types and formats of data, and Chapter 2 will discuss them, and how the type of source may impact the analysis, in more detail. The second step is moving the data and storing it in a database for analysis. I will use the terms data warehouse, which is a database that consolidates data from across an organization into a central repository, and data store, which refers to any type of data storage system that can be queried. Other terms you might come across are data mart, which is typically a subset of a data warehouse, or a more narrowly focused data warehouse; and data lake, a term that can mean either that data resides in a file storage system or that it is stored in a database but without the degree of", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 15 + }, + { + "text": "across are data mart, which is typically a subset of a data warehouse, or a more narrowly focused data warehouse; and data lake, a term that can mean either that data resides in a file storage system or that it is stored in a database but without the degree of data trans‐ formation that is common in data warehouses. Data warehouses range from small and simple to huge and expensive. A database running on a laptop will be sufficient for you to follow along with the examples in this book. What matters is having the data you need to perform an analysis together in one place. 10 | Chapter 1: Analysis with SQL Usually a person or team is responsible for getting data into the data warehouse. This process is called ETL, or extract, transform, load. Extract pulls the data from the source system. Transform optionally changes the structure of the data, performs data quality cleaning, or aggregates the data. Load puts the data into the data‐ base. This process can also be called ELT, for extract, load, trans‐ form—the difference being that, rather than transformations being done before data is loaded, all the data is loaded and then transfor‐ mations are performed, usually using SQL. You might also hear the terms source and target in the context of ETL. The source is where the data comes from, and the target is the destination, i.e., the data‐ base and the tables within it. Even when SQL is used to do the transforming, another language such as Python or Java is used to glue the steps together, coordinate scheduling, and raise alerts when something goes wrong. There are a number of commercial products as well as open source tools available, so teams don’t have to create an ETL system entirely from scratch. Once the data is in a database, the next step is performing queries and analysis. In this step, SQL is applied to explore, profile, clean, shape, and analyze the data. Figure 1-3 shows the general flow of the process. Exploring the data involves becom‐ ing familiar with the topic, where the data was generated, and the database tables in which it is stored. Profiling involves checking the unique values and distribution of records in the data set. Cleaning involves fixing incorrect or incomplete data, adding categorization and flags, and handling null values. Shaping is the process of arranging the data into the rows and columns needed in the result set. Finally, analyzing the data involves reviewing the output for trends, conclusions, and insights. Although this process is shown as linear, in practice it is often cyclical—for example, when shaping or analysis reveals data that should be cleaned. Figure 1-3. Stages within the queries and analysis step of the analysis workflow Why SQL? | 11 Presentation of the data into a final output form is the last step in the overall work‐ flow. Businesspeople won’t appreciate receiving a file of SQL code; they expect you to present graphs, charts, and", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 16 + }, + { + "text": "Stages within the queries and analysis step of the analysis workflow Why SQL? | 11 Presentation of the data into a final output form is the last step in the overall work‐ flow. Businesspeople won’t appreciate receiving a file of SQL code; they expect you to present graphs, charts, and insights. Communication is key to having an impact with analysis, and for that we need a way to share the results with other people. At other times, you may need to apply more sophisticated statistical analysis than is possible in SQL, or you may want to feed the data into a machine learning (ML) algorithm. For‐ tunately, most reporting and visualization tools have SQL connectors that allow you to pull in data from entire tables or prewritten SQL queries. Statistical software and languages commonly used for ML also usually have SQL connectors. Analysis workflows encompass a number of steps and often include multiple tools and technologies. SQL queries and analysis are at the heart of many analyses and are what we will focus on in the following chapters. Chapter 2 will discuss types of source systems and the types of data they generate. The rest of this chapter will take a look at the types of databases you are likely to encounter in your analysis journey. Database Types and How to Work with Them If you’re working with SQL, you’ll be working with databases. There is a range of database types—open source to proprietary, row-store to column-store. There are on- premises databases and cloud databases, as well as hybrid databases, where an organi‐ zation runs the database software on a cloud vendor’s infrastructure. There are also a number of data stores that aren’t databases at all but can be queried with SQL. Databases are not all created equal; each database type has its strengths and weak‐ nesses when it comes to analysis work. Unlike tools used in other parts of the analysis workflow, you may not have much say in which database technology is used in your organization. Knowing the ins and outs of the database you have will help you work more efficiently and take advantage of any special SQL functions it offers. Familiarity with other types of databases will help you if you find yourself working on a project to build or migrate to a new data warehouse. You may want to install a database on your laptop for personal, small-scale projects, or get an instance of a cloud warehouse for similar reasons. Databases and data stores have been a dynamic area of technology development since they were introduced. A few trends since the turn of the 21st century have driven the technology in ways that are really exciting for data practitioners today. First, data vol‐ umes have increased incredibly with the internet, mobile devices, and the Internet of Things (IoT). In 2020 IDC predicted that the amount of data stored globally will grow to 175 zettabytes by 2025. This scale of data is hard to even think", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 17 + }, + { + "text": "exciting for data practitioners today. First, data vol‐ umes have increased incredibly with the internet, mobile devices, and the Internet of Things (IoT). In 2020 IDC predicted that the amount of data stored globally will grow to 175 zettabytes by 2025. This scale of data is hard to even think about, and not all of it will be stored in databases for analysis. It’s not uncommon for companies to have data in the scale of terabytes and petabytes these days, a scale that would have been impossible to process with the technology of the 1990s and earlier. Second, decreases in data storage and computing costs, along with the advent of the cloud, 12 | Chapter 1: Analysis with SQL have made it cheaper and easier for organizations to collect and store these massive amounts of data. Computer memory has gotten cheaper, meaning that large amounts of data can be loaded into memory, calculations performed, and results returned, all without reading and writing to disk, greatly increasing the speed. Third, distributed computing has allowed the breaking up of workloads across many machines. This allows a large and tunable amount of computing to be pointed to complex data tasks. Databases and data stores have combined these technological trends in a number of different ways in order to optimize for particular types of tasks. There are two broad categories of databases that are relevant for analysis work: row-store and column- store. In the next section I’ll introduce them, discuss what makes them similar to and different from each other, and talk about what all of this means as far as doing analy‐ sis with data stored in them. Finally, I’ll introduce some additional types of data infra‐ structure beyond databases that you may encounter. Row-Store Databases Row-store databases—also called transactional databases—are designed to be efficient at processing transactions: INSERTs, UPDATEs, and DELETEs. Popular open source row-store databases include MySQL and Postgres. On the commercial side, Microsoft SQL Server, Oracle, and Teradata are widely used. Although they’re not really opti‐ mized for analysis, for a number of years row-store databases were the only option for companies building data warehouses. Through careful tuning and schema design, these databases can be used for analytics. They are also attractive due to the low cost of open source options and because they’re familiar to the database administrators who maintain them. Many organizations replicate their production database in the same technology as a first step toward building out data infrastructure. For all of these reasons, data analysts and data scientists are likely to work with data in a row- store database at some point in their career. We think of a table as rows and columns, but data has to be serialized for storage. A query searches a hard disk for the needed data. Hard disks are organized in a series of blocks of a fixed size. Scanning the hard disk takes both time and resources, so mini‐ mizing the amount of the disk that needs to be scanned", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 18 + }, + { + "text": "to be serialized for storage. A query searches a hard disk for the needed data. Hard disks are organized in a series of blocks of a fixed size. Scanning the hard disk takes both time and resources, so mini‐ mizing the amount of the disk that needs to be scanned to return query results is important. Row-store databases approach this problem by serializing data in a row. Figure 1-4 shows an example of row-wise data storage. When querying, the whole row is read into memory. This approach is fast when making row-wise updates, but it’s slower when making calculations across many rows if only a few columns are needed. Database Types and How to Work with Them | 13 Figure 1-4. Row-wise storage, in which each row is stored together on disk To reduce the width of tables, row-store databases are usually modeled in third nor‐ mal form, which is a database design approach that seeks to store each piece of infor‐ mation only once, to avoid duplication and inconsistencies. This is efficient for transaction processing but often leads to a large number of tables in the database, each with only a few columns. To analyze such data, many joins may be required, and it can be difficult for nondevelopers to understand how all of the tables relate to each other and where a particular piece of data is stored. When doing analysis, the goal is usually denormalization, or getting all the data together in one place. Tables typically have a primary key that enforces uniqueness—in other words, it pre‐ vents the database from creating more than one record for the same thing. Tables will often have an id column that is an auto-incrementing integer, where each new record gets the next integer after the last one inserted, or an alphanumeric value that is created by a primary key generator. There should also be a set of columns that together make the row unique; this combination of fields is called a composite key, or sometimes a business key. For example, in a table of people, the columns first_name, last_name, and birthdate together might make the row unique. Social_security_id would also be a unique identifier, in addition to the table’s person_id column. Tables also optionally have indexes that make looking up specific records faster and make joins involving these columns faster. Indexes store the values in the field or fields indexed as single pieces of data along with a row pointer, and since the indexes are smaller than the whole table, they are faster to scan. Usually the primary key is indexed, but other fields or groups of fields can be indexed as well. When working with row-store databases, it’s useful to get to know which fields in the tables you use have indexes. Common joins can be sped up by adding indexes, so it’s worth investi‐ gating whether analysis queries take a long time to run. Indexes don’t come for free: they take up storage space, and they slow down", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 19 + }, + { + "text": "to get to know which fields in the tables you use have indexes. Common joins can be sped up by adding indexes, so it’s worth investi‐ gating whether analysis queries take a long time to run. Indexes don’t come for free: they take up storage space, and they slow down loading, as new values need to be added with each insert. DBAs may not index everything that might be useful for anal‐ ysis. Beyond reporting, analysis work may not be routine enough to bother with opti‐ mizing indexes either. Exploratory and complex queries often use complex join patterns, and we may throw out one approach when we figure out a new way to solve a problem. Star schema modeling was developed in part to make row-store databases more friendly to analytic workloads. The foundations are laid out in the book The Data 14 | Chapter 1: Analysis with SQL 2 Ralph Kimball and Margy Ross, The Data Warehouse Toolkit, 3rd ed. (Indianapolis: Wiley, 2013). Warehouse Toolkit,2 which advocates modeling the data as a series of fact and dimen‐ sion tables. Fact tables represent events, such as retail store transactions. Dimensions hold descriptors such as customer name and product type. Since data doesn’t always fit neatly into fact and dimension categories, there’s an extension called the snowflake schema in which some dimensions have dimensions of their own. Column-Store Databases Column-store databases took off in the early part of the 21st century, though their the‐ oretical history goes back as far as that of row-store databases. Column-store data‐ bases store the values of a column together, rather than storing the values of a row together. This design is optimized for queries that read many records but not neces‐ sarily all the columns. Popular column-store databases include Amazon Redshift, Snowflake, and Vertica. Column-store databases are efficient at storing large volumes of data thanks to com‐ pression. Missing values and repeating values can be represented by very small marker values instead of the full value. For example, rather than storing “United Kingdom” thousands or millions of times, a column-store database will store a surro‐ gate value that takes up very little storage space, along with a lookup that stores the full “United Kingdom” value. Column-store databases also compress data by taking advantage of repetitions of values in sorted data. For example, the database can store the fact that the marker value for “United Kingdom” is repeated 100 times, and this takes up even less space than storing that marker 100 times. Column-store databases do not enforce primary keys and do not have indexes. Repeated values are not problematic, thanks to compression. As a result, schemas can be tailored for analysis queries, with all the data together in one place as opposed to being in multiple tables that need to be joined. Duplicate data can easily sneak in without primary keys, however, so understanding the source of the data and quality checking are important. Updates and deletes are expensive in most column-store databases, since data", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 20 + }, + { + "text": "data together in one place as opposed to being in multiple tables that need to be joined. Duplicate data can easily sneak in without primary keys, however, so understanding the source of the data and quality checking are important. Updates and deletes are expensive in most column-store databases, since data for a single row is distributed rather than stored together. For very large tables, a write- only policy may exist, so we also need to know something about how the data is gen‐ erated in order to figure out which records to use. The data can also be slower to read, as it needs to be uncompressed before calculations are applied. Column-store databases are generally the gold standard for fast analysis work. They use standard SQL (with some vendor-specific variations), and in many ways working with them is no different from working with a row-store database in terms of the queries you write. The size of the data matters, as do the computing and storage Database Types and How to Work with Them | 15 resources that have been allocated to the database. I have seen aggregations run across millions and billions of records in seconds. This does wonders for productivity. There are a few tricks to be aware of. Since certain types of com‐ pression rely on sorting, knowing the fields that the table is sorted on and using them to filter queries improves performance. Joining tables can be slow if both tables are large. At the end of the day, some databases will be easier or faster to work with, but there is nothing inherent in the type of database that will prevent you from performing any of the analysis in this book. As with all things, using a tool that’s properly powerful for the volume of data and complexity of the task will allow you to focus on creating meaningful analysis. Other Types of Data Infrastructure Databases aren’t the only way data can be stored, and there is an increasing variety of options for storing data needed for analysis and powering applications. File storage systems, sometimes called data lakes, are probably the main alternative to database warehouses. NoSQL databases and search-based data stores are alternative data stor‐ age systems that offer low latency for application development and searching log files. Although not typically part of the analysis process, they are increasingly part of organizations’ data infrastructure, so I will introduce them briefly in this section as well. One interesting trend to point out is that although these newer types of infra‐ structure at first aimed to break away from the confines of SQL databases, many have ended up implementing some kind of SQL interface to query the data. Hadoop, also known as HDFS (for “Hadoop distributed filesystem”), is an open source file storage system that takes advantage of the ever-falling cost of data storage and computing power, as well as distributed systems. Files are split into blocks, and Hadoop distributes them across a filesystem that is stored", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 21 + }, + { + "text": "data. Hadoop, also known as HDFS (for “Hadoop distributed filesystem”), is an open source file storage system that takes advantage of the ever-falling cost of data storage and computing power, as well as distributed systems. Files are split into blocks, and Hadoop distributes them across a filesystem that is stored on nodes, or computers, in a cluster. The code to run operations is sent to the nodes, and they process the data in parallel. Hadoop’s big breakthrough was to allow huge amounts of data to be stored cheaply. Many large internet companies, with massive amounts of often unstructured data, found this to be an advantage over the cost and storage limitations of traditional databases. Hadoop’s early versions had two major downsides: specialized coding skills were needed to retrieve and process data since it was not SQL compatible, and execu‐ tion time for the programs was often quite long. Hadoop has since matured, and vari‐ ous tools have been developed that allow SQL or SQL-like access to the data and speed up query times. 16 | Chapter 1: Analysis with SQL Other commercial and open source products have been introduced in the last few years to take advantage of cheap data storage and fast, often in-memory data process‐ ing, while offering SQL querying ability. Some of them even permit the analyst to write a single query that returns data from multiple underlying sources. This is excit‐ ing for anyone who works with large amounts of data, and it is validation that SQL is here to stay. NoSQL is a technology that allows for data modeling that is not strictly relational. It allows for very low latency storage and retrieval, critical in many online applications. The class includes key-value pair storage and graph databases, which store in a node- edge format, and document stores. Examples of these data stores that you might hear about in your organization are Cassandra, Couchbase, DynamoDB, Memcached, Gir‐ aph, and Neo4j. Early on, NoSQL was marketed as making SQL obsolete, but the acronym has more recently been marketed as “not only SQL.” For analysis purposes, using data stored in a NoSQL key-value store for analysis typically requires moving it to a more traditional SQL data warehouse, since NoSQL is not optimized for query‐ ing many records at once. Graph databases have applications such as network analy‐ sis, and analysis work may be done directly in them with special query languages. The tool landscape is always evolving, however, and perhaps someday we’ll be able to ana‐ lyze this data with SQL as well. Search-based data stores include Elasticsearch and Splunk. Elasticsearch and Splunk are often used to analyze machine-generated data, such as logs. These and similar technologies have non-SQL query languages, but if you know SQL, you can often understand them. Recognizing how common SQL skills are, some data stores, such as Elasticsearch, have added SQL querying interfaces. These tools are useful and power‐ ful for the use cases they were designed for, but they’re usually not well", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 22 + }, + { + "text": "non-SQL query languages, but if you know SQL, you can often understand them. Recognizing how common SQL skills are, some data stores, such as Elasticsearch, have added SQL querying interfaces. These tools are useful and power‐ ful for the use cases they were designed for, but they’re usually not well suited to the types of analysis tasks this book is covering. As I’ve explained to people over the years, they are great for finding needles in haystacks. They’re not as great at measur‐ ing the haystack itself. Regardless of the type of database or other data storage technology, the trend is clear: even as data volumes grow and use cases become more complex, SQL is still the stan‐ dard tool for accessing data. Its large existing user base, approachable learning curve, and power for analytical tasks mean that even technologies that try to move away from SQL come back around and accommodate it. Conclusion Data analysis is an exciting discipline with a range of applications for businesses and other organizations. SQL has many benefits for working with data, particularly any data stored in a database. Querying and analyzing data is part of the larger analysis workflow, and there are several types of data stores that a data scientist might expect to work with. Now that we’ve set the groundwork for analysis, SQL, and data stores, Conclusion | 17 the rest of the book will cover using SQL for analysis in depth. Chapter 2 focuses on data preparation, starting with an introduction to data types and then moving on to profiling, cleaning, and shaping data. Chapters 3 through 7 present applications of data analysis, focusing on time series analysis, cohort analysis, text analysis, anomaly detection, and experiment analysis. Chapter 8 covers techniques for developing com‐ plex data sets for further analysis in other tools. Finally, Chapter 9 concludes with thoughts on how types of analysis can be combined for new insights and lists some additional resources to support your analytics journey. 18 | Chapter 1: Analysis with SQL CHAPTER 2 Preparing Data for Analysis Estimates of how long data scientists spend preparing their data vary, but it’s safe to say that this step takes up a significant part of the time spent working with data. In 2014, the New York Times reported that data scientists spend from 50% to 80% of their time cleaning and wrangling their data. A 2016 survey by CrowdFlower found that data scientists spend 60% of their time cleaning and organizing data in order to prepare it for analysis or modeling work. Preparing data is such a common task that terms have sprung up to describe it, such as data munging, data wrangling, and data prep. (“Mung” is an acronym for Mash Until No Good, which I have certainly done on occasion.) Is all this data preparation work just mindless toil, or is it an important part of the process? Data preparation is easier when a data set has a data dictionary, a document or repos‐ itory", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 23 + }, + { + "text": "is an acronym for Mash Until No Good, which I have certainly done on occasion.) Is all this data preparation work just mindless toil, or is it an important part of the process? Data preparation is easier when a data set has a data dictionary, a document or repos‐ itory that has clear descriptions of the fields, possible values, how the data was collec‐ ted, and how it relates to other data. Unfortunately, this is frequently not the case. Documentation often isn’t prioritized, even by people who see its value, or it becomes out-of-date as new fields and tables are added or the way data is populated changes. Data profiling creates many of the elements of a data dictionary, so if your organiza‐ tion already has a data dictionary, this is a good time to use it and contribute to it. If no data dictionary exists currently, consider starting one! This is one of the most val‐ uable gifts you can give to your team and to your future self. An up-to-date data dic‐ tionary allows you to speed up the data-profiling process by building on profiling that’s already been done rather than replicating it. It will also improve the quality of your analysis results, since you can verify that you have used fields correctly and applied appropriate filters. Even when a data dictionary exists, you will still likely need to do data prep work as part of the analysis. In this chapter, I’ll start with a review of data types you are likely to encounter. This is followed by a review of SQL query structure. Next, I will talk 19 about profiling the data as a way to get to know its contents and check for data qual‐ ity. Then I’ll talk about some data-shaping techniques that will return the columns and rows needed for further analysis. Finally, I’ll walk through some useful tools for cleaning data to deal with any quality issues. Types of Data Data is the foundation of analysis, and all data has a database data type and also belongs to one or more categories of data. Having a firm grasp of the many forms data can take will help you be a more effective data analyst. I’ll start with the database data types most frequently encountered in analysis. Then I’ll move on to some con‐ ceptual groupings that can help us understand the source, quality, and possible appli‐ cations of the data. Database Data Types Fields in database tables all have defined data types. Most databases have good docu‐ mentation on the types they support, and this is a good resource for any needed detail beyond what is presented here. You don’t necessarily need to be an expert on the nuances of data types to be good at analysis, but later in the book we’ll encounter sit‐ uations in which considering the data type is important, so this section will cover the basics. The main types of data are strings, numeric, logical, and datetime, as", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 24 + }, + { + "text": "be an expert on the nuances of data types to be good at analysis, but later in the book we’ll encounter sit‐ uations in which considering the data type is important, so this section will cover the basics. The main types of data are strings, numeric, logical, and datetime, as summar‐ ized in Table 2-1. These are based on Postgres but are similar across most major data‐ base types. Table 2-1. A summary of common database data types Type Name Description String CHAR / VARCHAR Holds strings. A CHAR is always of fixed length, whereas a VARCHAR is of variable length, up to some maximum size (256 characters, for example). TEXT / BLOB Holds longer strings that don’t fit in a VARCHAR. Descriptions or free text entered by survey respondents might be held in these fields. Numeric INT / SMALLINT / BIGINT Holds integers (whole numbers). Some databases have SMALLINT and/or BIGINT. SMALLINT can be used when the field will only hold values with a small number of digits. SMALLINT takes less memory than a regular INT. BIGINT is capable of holding numbers with more digits than an INT, but it takes up more space than an INT. FLOAT / DOUBLE / DECIMAL Holds decimal numbers, sometimes with the number of decimal places specified. Logical BOOLEAN Holds values of TRUE or FALSE. DATETIME / TIMESTAMP Holds dates with times. Typically in a YYYY-MM-DD hh:mi:ss format, where YYYY is the four- digit year, MM is the two-digit month number, DD is the two-digit day, hh is the two-digit hour (usually 24-hour time, or values of 0 to 23), mi is the two-digit minutes, and ss is the two-digit seconds. Some databases store only timestamps without time zone, while others have specific types for timestamps with and without time zones. TIME Holds times. 20 | Chapter 2: Preparing Data for Analysis String data types are the most versatile. These can hold letters, numbers, and special characters, including unprintable characters like tabs and newlines. String fields can be defined to hold a fixed or variable number of characters. A CHAR field could be defined to allow only two characters to hold US state abbreviations, for example, whereas a field storing the full names of states would need to be a VARCHAR to allow a variable number of characters. Fields can be defined as TEXT, CLOB (Character Large Object), or BLOB (Binary Large Object, which can include additional data types such as images), depending on the database to hold very long strings, though since they often take up a lot of space, these data types tend to be used sparingly. When data is loaded, if strings arrive that are too big for the defined data type, they may be truncated or rejected entirely. SQL has a number of string functions that we will make use of for various analysis purposes. Numeric data types are all the ones that store numbers, both positive and negative. Mathematical functions and operators can be applied to numeric fields.", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 25 + }, + { + "text": "data type, they may be truncated or rejected entirely. SQL has a number of string functions that we will make use of for various analysis purposes. Numeric data types are all the ones that store numbers, both positive and negative. Mathematical functions and operators can be applied to numeric fields. Numeric data types include the INT types as well as FLOAT, DOUBLE, and DECIMAL types that allow decimal places. Integer data types are often implemented because they use less memory than their decimal counterparts. In some databases, such as Postgres, divid‐ ing integers results in an integer, rather than a value with decimal places as you might expect. We’ll discuss converting numeric data types to obtain correct results later in this chapter. The logical data type is called BOOLEAN. It has values of TRUE and FALSE and is an efficient way to store information where these options are appropriate. Operations that compare two fields return a BOOLEAN value as a result. This data type is often used to create flags, fields that summarize the presence or absence of a property in the data. For example, a table storing email data might have a BOOLEAN has_opened field. The datetime types include DATE, TIMESTAMP, and TIME. Date and time data should be stored in a field of one of these database types whenever possible, since SQL has a number of useful functions that operate on them. Timestamps and dates are very common in databases and are critical to many types of analysis, particularly time series analysis (covered in Chapter 3) and cohort analysis (covered in Chap‐ ter 4). Chapter 3 will discuss date and time formatting, transformations, and calculations. Other data types, such as JSON and geographical types, are supported by some but not all databases. I won’t go into detail on all of them here since they are generally beyond the scope of this book. However, they are a sign that SQL continues to evolve to tackle emerging analysis tasks. Beyond database data types, there are a number of conceptual ways that data is cate‐ gorized. These can have an impact both on how data is stored and on how we think about analyzing it. I will discuss these categorical data types next. Types of Data | 21 Structured Versus Unstructured Data is often described as structured or unstructured, or sometimes as semistruc‐ tured. Most databases were designed to handle structured data, where each attribute is stored in a column, and instances of each entity are represented as rows. A data model is first created, and then data is inserted according to that data model. For example, an address table might have fields for street address, city, state, and postal code. Each row would hold a particular customer’s address. Each field has a data type and allows only data of that type to be entered. When structured data is inserted into a table, each field is verified to ensure it conforms to the correct data type. Structured data is easy to", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 26 + }, + { + "text": "Each row would hold a particular customer’s address. Each field has a data type and allows only data of that type to be entered. When structured data is inserted into a table, each field is verified to ensure it conforms to the correct data type. Structured data is easy to query with SQL. Unstructured data is the opposite of structured data. There is no predetermined struc‐ ture, data model, or data types. Unstructured data is often the “everything else” that isn’t database data. Documents, emails, and web pages are unstructured. Photos, images, videos, and audio files are also examples of unstructured data. They don’t fit into the traditional data types, and thus they are more difficult for relational data‐ bases to store efficiently and for SQL to query. Unstructured data is often stored out��� side of relational databases as a result. This allows data to be loaded quickly, but lack of data validation can result in low data quality. As we saw in Chapter 1, the technol‐ ogy continues to evolve, and new tools are being developed to allow SQL querying of many types of unstructured data. Semistructured data falls in between these two categories. Much “unstructured” data has some structure that we can make use of. For example, emails have from and to email addresses, subject lines, body text, and sent timestamps that can be stored sepa‐ rately in a data model with those fields. Metadata, or data about data, can be extracted from other file types and stored for analysis. For example, music audio files might be tagged with artist, song name, genre, and duration. Generally, the structured parts of semistructured data can be queried with SQL, and SQL can often be used to parse or otherwise extract structured data for further querying. We’ll see some applications of this in the discussion of text analysis in Chapter 5. Quantitative Versus Qualitative Data Quantitative data is numeric. It measures people, things, and events. Quantitative data can include descriptors, such as customer information, product type, or device configurations, but it also comes with numeric information such as price, quantity, or visit duration. Counts, sums, average, or other numeric functions are applied to the data. Quantitative data is often machine generated these days, but it doesn’t need to be. Height, weight, and blood pressure recorded on a paper patient intake form are quantitative, as are student quiz scores typed into a spreadsheet by a teacher. 22 | Chapter 2: Preparing Data for Analysis Qualitative data is usually text based and includes opinions, feelings, and descriptions that aren’t strictly quantitative. Temperature and humidity levels are quantitative, while descriptors like “hot and humid” are qualitative. The price a customer paid for a product is quantitative; whether they like or dislike it is qualitative. Survey feed‐ back, customer support inquiries, and social media posts are qualitative. There are whole professions that deal with qualitative data. In a data analysis context, we usu‐ ally try to quantify the qualitative. One technique for this is to", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 27 + }, + { + "text": "is quantitative; whether they like or dislike it is qualitative. Survey feed‐ back, customer support inquiries, and social media posts are qualitative. There are whole professions that deal with qualitative data. In a data analysis context, we usu‐ ally try to quantify the qualitative. One technique for this is to extract keywords or phrases and count their occurrences. We’ll look at this in more detail when we delve into text analysis in Chapter 5. Another technique is sentiment analysis, in which the structure of language is used to interpret the meaning of the words used, in addition to their frequency. Sentences or other bodies of text can be scored for their level of positivity or negativity, and then counts or averages are used to derive insights that would be hard to summarize otherwise. There have been exciting advances in the field of natural language processing, or NLP, though much of this work is done with tools such as Python. First-, Second-, and Third-Party Data First-party data is collected by the organization itself. This can be done through server logs, databases that keep track of transactions and customer information, or other systems that are built and controlled by the organization and generate data of interest for analysis. Since the systems were created in-house, finding the people who built them and learning about how the data is generated is usually possible. Data ana‐ lysts may also be able to influence or have control over how certain pieces of data are created and stored, particularly when bugs are responsible for poor data quality. Second-party data comes from vendors that provide a service or perform a business function on the organization’s behalf. These are often software as a service (SaaS) products; common examples are CRM, email and marketing automation tools, ecommerce-enabling software, and web and mobile interaction trackers. The data is similar to first-party data since it is about the organization itself, created by its employees and customers. However, both the code that generates and stores the data and the data model are controlled externally, and the data analyst typically has little influence over these aspects. Second-party data is increasingly imported into an organization’s data warehouse for analysis. This can be accomplished with custom code or ETL connectors, or with SaaS vendors that offer data integration. Types of Data | 23 Many SaaS vendors provide some reporting capabilities, so the question may arise of whether to bother copying the data to a data warehouse. The department that interacts with a tool may find that reporting sufficient, such as a customer service department that reports on time to resolve issues and agent productivity from within its helpdesk software. On the other hand, customer service interactions might be an important input to a customer retention model, which would require integrating that data into a data store with sales and cancellation data. Here’s a good rule of thumb when deciding whether to import data from a particular data source: if the data will create value when combined", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 28 + }, + { + "text": "might be an important input to a customer retention model, which would require integrating that data into a data store with sales and cancellation data. Here’s a good rule of thumb when deciding whether to import data from a particular data source: if the data will create value when combined with data from other sys‐ tems, import it; if not, wait until there is a stronger case before doing the work. Third-party data may be purchased or obtained from free sources such as those pub‐ lished by governments. Unless the data has been collected specifically on behalf of the organization, data teams usually have little control over the format, frequency, and data quality. This data often lacks the granularity of first- and second-party data. For example, most third-party sources do not have user-level data, and instead data might be joined with first-party data at the postal code or city level, or at a higher level. Third-party data can have unique and useful information, however, such as aggregate spending patterns, demographics, and market trends that would be very expensive or impossible to collect otherwise. Sparse Data Sparse data occurs when there is a small amount of information within a larger set of empty or unimportant information. Sparse data might show up as many nulls and only a few values in a particular column. Null, different from a value of 0, is the absence of data; that will be covered later in the section on data cleaning. Sparse data can occur when events are rare, such as software errors or purchases of products in the long tail of a product catalog. It can also occur in the early days of a feature or product launch, when only testers or beta customers have access. JSON is one approach that has been developed to deal with sparse data from a writing and storage perspective, as it stores only the data that is present and omits the rest. This is in con‐ trast to a row-store database, which has to hold memory for a field even if there is no value in it. Sparse data can be problematic for analysis. When events are rare, trends aren’t nec‐ essarily meaningful, and correlations are hard to distinguish from chance fluctua‐ tions. It’s worth profiling your data, as discussed later in this chapter, to understand if and where your data is sparse. Some options are to group infrequent events or items into categories that are more common, exclude the sparse data or time period from 24 | Chapter 2: Preparing Data for Analysis the analysis entirely, or show descriptive statistics along with cautionary explanations that the trends are not necessarily meaningful. There are a number of different types of data and a variety of ways that data is described, many of which are overlapping or not mutually exclusive. Familiarity with these types is useful not only in writing good SQL but also for deciding how to ana‐ lyze the data in appropriate ways. You may not always know", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 29 + }, + { + "text": "of data and a variety of ways that data is described, many of which are overlapping or not mutually exclusive. Familiarity with these types is useful not only in writing good SQL but also for deciding how to ana‐ lyze the data in appropriate ways. You may not always know the data types in advance, which is why data profiling is so critical. Before we get to that, and to our first code examples, I’ll give a brief review of SQL query structure. SQL Query Structure SQL queries have common clauses and syntax, although these can be combined in a nearly infinite number of ways to achieve analysis goals. This book assumes you have some prior knowledge of SQL, but I’ll review the basics here so that we have a com‐ mon foundation for the code examples to come. The SELECT clause determines the columns that will be returned by the query. One column will be returned for each expression within the SELECT clause, and expres‐ sions are separated by commas. An expression can be a field from the table, an aggre‐ gation such as a sum, or any number of calculations, such as CASE statements, type conversions, and various functions that will be discussed later in this chapter and throughout the book. The FROM clause determines the tables from which the expressions in the SELECT clause are derived. A “table” can be a database table, a view (a type of saved query that otherwise functions like a table), or a subquery. A subquery is itself a query, wrapped in parentheses, and the result is treated like any other table by the query that refer‐ ences it. A query can reference multiple tables in the FROM clause, though they must use one of the JOIN types along with a condition that specifies how the tables relate. The JOIN condition usually specifies an equality between fields in each table, such as orders.customer_id = customers.customer_id. JOIN conditions can include mul‐ tiple fields and can also specify inequalities or ranges of values, such as ranges of dates. We’ll see a variety of JOIN conditions that achieve specific analysis goals throughout the book. An INNER JOIN returns all records that match in both tables. A LEFT JOIN returns all records from the first table, but only those records from the second table that match. A RIGHT JOIN returns all records from the second table, but only those records from the first table that match. A FULL OUTER JOIN returns all records from both tables. A Cartesian JOIN can result when each record in the first table matches more than one record in the second table. Cartesian JOINs should gen‐ erally be avoided, though there are some specific use cases, such as generating data to fill in a time series, in which we will use them intentionally. Finally, tables in the FROM clause can be aliased, or given a shorter name of one or more letters that can SQL Query Structure | 25 be referenced", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 30 + }, + { + "text": "are some specific use cases, such as generating data to fill in a time series, in which we will use them intentionally. Finally, tables in the FROM clause can be aliased, or given a shorter name of one or more letters that can SQL Query Structure | 25 be referenced in other clauses in the query. Aliases save query writers from having to type out long table names repeatedly, and they make queries easier to read. While both LEFT JOIN and RIGHT JOIN can be used in the same query, it’s much easier to keep track of your logic when you stick with only one or the other. In practice, LEFT JOIN is much more commonly used than RIGHT JOIN. The WHERE clause specifies restrictions or filters that are needed to exclude or remove rows from the result set. WHERE is optional. The GROUP BY clause is required when the SELECT clause contains aggregations and at least one nonaggregated field. An easy way to remember what should go in the GROUP BY clause is that it should have every field that is not part of an aggregation. In most databases, there are two ways to list the GROUP BY fields: either by field name or by position, such as 1, 2, 3, and so on. Some people prefer to use the field name notation, and SQL Server requires this. I prefer the position notation, particu‐ larly when the GROUP BY fields contain complex expressions or when I’m doing a lot of iteration. This book will typically use the position notation. How Not to Kill Your Database: LIMIT and Sampling Database tables can be very large, containing millions or billions of records. Querying across all of these records can cause problems at the least and crash databases at the worst. To avoid receiving cranky calls from database administrators or getting locked out, it’s a good idea to limit the results returned during profiling or while testing queries. LIMIT clauses and sampling are two techniques that should be part of your toolbox. LIMIT is added as the last line of the query, or subquery, and can take any positive integer value: SELECT column_a, column_b FROM table LIMIT 1000 ; When used in a subquery, the limit will be applied at that step, and only the restricted result set will be evaluated by the outer query: SELECT... FROM ( SELECT column_a, column_b, sum(sales) as total_sales FROM table GROUP BY 1,2 LIMIT 1000 26 | Chapter 2: Preparing Data for Analysis ) a ; SQL Server does not support the LIMIT clause, but a similar result can be obtained using top: SELECT top 1000 column_a, column_b FROM table ; Sampling can be accomplished by using a function on an ID field that has a random distribution of digits at the beginning or end. The modulus or mod function returns the remainder when one integer is divided by another. If the ID field is an integer, mod can be used to find the last one,", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 31 + }, + { + "text": "using a function on an ID field that has a random distribution of digits at the beginning or end. The modulus or mod function returns the remainder when one integer is divided by another. If the ID field is an integer, mod can be used to find the last one, two, or more digits and filter on the result: WHERE mod(integer_order_id,100) = 6 This will return every order whose last two digits are 06, which should be about 1% of the total. If the field is alphanumeric, you can use a right() function to find a certain number of digits at the end: WHERE right(alphanum_order_id,1) = 'B' This will return every order with a last digit of B, which will be about 3% of the total if all letters and numbers are equally common, an assumption worth validating. Limiting the result set also makes your work faster, but be aware that subsets of data might not contain all of the variations in values and edge cases that exist in the full data set. Remember to remove the LIMIT or sampling before running your final anal‐ ysis or report with your query, or you’ll end up with funny results! That covers the basics of SQL query structure. Chapter 8 will go into additional detail on each of these clauses, a few additional ones that are less commonly encountered but appear in this book, and the order in which each clause is evaluated. Now that we have this foundation, we can turn to one of the most important parts of the analysis process: data profiling. Profiling: Distributions Profiling is the first thing I do when I start working with any new data set. I look at how the data is arranged into schemas and tables. I look at the table names to get familiar with the topics covered, such as customers, orders, or visits. I check out the column names in a few tables and start to construct a mental model of how the tables relate to one another. For example, the tables might include an order_detail table with line-item breakouts that relate to the order table via an order_id, while the order table relates to the customer table via a customer_id. If there is a data dictio‐ nary, I review that and compare it to the data I see in a sample of rows. Profiling: Distributions | 27 1 John W. Tukey, Exploratory Data Analysis (Reading, MA: Addison-Wesley, 1977). The tables generally represent the operations of an organization, or some subset of the operations, so I think about what domain or domains are covered, such as ecom‐ merce, marketing, or product interactions. Working with data is easier when we have knowledge of how the data was generated. Profiling can provide clues about this, or about what questions to ask of the source, or of people inside or outside the organiza‐ tion responsible for the collection or generation of the data. Even when you collect the data yourself, profiling is useful. Another", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 32 + }, + { + "text": "of how the data was generated. Profiling can provide clues about this, or about what questions to ask of the source, or of people inside or outside the organiza‐ tion responsible for the collection or generation of the data. Even when you collect the data yourself, profiling is useful. Another detail I check for is how history is represented, if at all. Data sets that are replicas of production databases may not contain previous values for customer addresses or order statuses, for example, whereas a well-constructed data warehouse may have daily snapshots of changing data fields. Profiling data is related to the concept of exploratory data analysis, or EDA, named by John Tukey. In his book of that name,1 Tukey describes how to analyze data sets by computing various summaries and visualizing the results. He includes techniques for looking at distributions of data, including stem-and-leaf plots, box plots, and histograms. After checking a few samples of data, I start looking at distributions. Distributions allow me to understand the range of values that exist in the data and how often they occur, whether there are nulls, and whether negative values exist alongside positive ones. Distributions can be created with continuous or categorical data and are also called frequencies. In this section, we’ll look at how to create histograms, how bin‐ ning can help us understand the distribution of continuous values, and how to use n-tiles to get more precise about distributions. Histograms and Frequencies One of the best ways to get to know a data set, and to know particular fields within the data set, is to check the frequency of values in each field. Frequency checks are also useful whenever you have a question about whether certain values are possible or if you spot an unexpected value and want to know how commonly it occurs. Fre‐ quency checks can be done on any data type, including strings, numerics, dates, and booleans. Frequency queries are a great way to detect sparse data as well. The query is straightforward. The number of rows can be found with count(*), and the profiled field is in the GROUP BY. For example, we can check the frequency of each type of fruit in a fictional fruit_inventory table: 28 | Chapter 2: Preparing Data for Analysis SELECT fruit, count(*) as quantity FROM fruit_inventory GROUP BY 1 ; When using count, it’s worth taking a minute to consider whether there might be any duplicate records in the data set. You can use count(*) when you want the number of records, but use count distinct to find out how many unique items there are. A frequency plot is a way to visualize the number of times something occurs in the data set. The field being profiled is usually plotted on the x-axis, with the count of observations on the y-axis. Figure 2-1 shows an example of plotting the frequency of fruit from our query. Frequency graphs can also be drawn horizontally, which accom‐ modates long value names well.", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 33 + }, + { + "text": "in the data set. The field being profiled is usually plotted on the x-axis, with the count of observations on the y-axis. Figure 2-1 shows an example of plotting the frequency of fruit from our query. Frequency graphs can also be drawn horizontally, which accom‐ modates long value names well. Notice that this is categorical data without any inher‐ ent order. Figure 2-1. Frequency plot of fruit inventory A histogram is a way to visualize the distribution of numerical values in a data set and will be familiar to those with a statistics background. A basic histogram might show the distribution of ages across a group of customers. Imagine that we have a custom ers table that contains names, registration date, age, and other attributes. To create a histogram by age, GROUP BY the numerical age field and count customer_id: Profiling: Distributions | 29 SELECT age, count(customer_id) as customers FROM customers GROUP BY 1 ; The results of our hypothetical age distribution are graphed in Figure 2-2. Figure 2-2. Customers by age Another technique I’ve used repeatedly and that has become the basis for one of my favorite interview questions involves an aggregation followed by a frequency count. I give candidates a hypothetical table called orders, which has a date, customer identi‐ fier, order identifier, and an amount, and then ask them to write a SQL query that returns the distribution of orders per customer. This can’t be solved with a simple query; it requires an intermediate aggregation step, which can be accomplished with a subquery. First, count the number of orders placed by each customer_id in the sub‐ query. The outer query uses the number of orders as a category and counts the num‐ ber of customers: SELECT orders, count(*) as num_customers FROM ( SELECT customer_id, count(order_id) as orders FROM orders GROUP BY 1 ) a GROUP BY 1 ; 30 | Chapter 2: Preparing Data for Analysis This type of profiling can be applied whenever you need to see how frequently certain entities or attributes appear in the data. In these examples, count has been used, but the other basic aggregations (sum, avg, min, and max) can be used to create histograms as well. For instance, we might want to profile customers by the sum of all their orders, their avg order size, their min order date, or their max (most recent) order date. Binning Binning is useful when working with continuous values. Rather than the number of observations or records for each value being counted, ranges of values are grouped together, and these groups are called bins or buckets. The number of records that fall into each interval is then counted. Bins can be variable in size or have a fixed size, depending on whether your goal is to group the data into bins that have particular meaning for the organization, are roughly equal width, or contain roughly equal numbers of records. Bins can be created with CASE statements, rounding, and logarithms. A CASE statement allows for", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 34 + }, + { + "text": "or have a fixed size, depending on whether your goal is to group the data into bins that have particular meaning for the organization, are roughly equal width, or contain roughly equal numbers of records. Bins can be created with CASE statements, rounding, and logarithms. A CASE statement allows for conditional logic to be evaluated. These statements are very flexible, and we will come back to them throughout the book, applying them to data profiling, cleaning, text analysis, and more. The basic structure of a CASE state‐ ment is: case when condition1 then return_value_1 when condition2 then return_value_2 ... else return_value_default end The WHEN condition can be an equality, inequality, or other logical condition. The THEN return value can be a constant, an expression, or a field in the table. Any num‐ ber of conditions can be included, but the statement will stop executing and return the result the first time a condition evaluates to TRUE. ELSE tells the database what to use as a default value if no matches are found and can also be a constant or field. ELSE is optional, and if it is not included, any nonmatches will return null. CASE statements can also be nested so that the return value is another CASE statement. The return values following THEN must all be the same data type (strings, numeric, BOOLEAN, etc.), or else you’ll get an error. Consider casting to a common data type such as string if you encounter this. A CASE statement is a flexible way to control the number of bins, the range of values that fall into each bin, and how the bins are named. I find them particularly useful when there is a long tail of very small or very large values that I want to group Profiling: Distributions | 31 together rather than have empty bins in part of the distribution. Certain ranges of values have a business meaning that needs to be re-created in the data. Many B2B companies separate their customers into “enterprise” and “SMB” (small- and medium-sized businesses) categories based on number of employees or revenue, because their buying patterns are different. As an example, imagine we are consider‐ ing discounted shipping offers and we want to know how many customers will be affected. We can group order_amount into three buckets using a CASE statement: SELECT case when order_amount <= 100 then 'up to 100' when order_amount <= 500 then '100 - 500' else '500+' end as amount_bin ,case when order_amount <= 100 then 'small' when order_amount <= 500 then 'medium' else 'large' end as amount_category ,count(customer_id) as customers FROM orders GROUP BY 1,2 ; Arbitrary-sized bins can be useful, but at other times bins of fixed size are more appropriate for the analysis. Fixed-size bins can be accomplished in a few ways, including with rounding, logarithms, and n-tiles. To create equal-width bins, round‐ ing is useful. Rounding reduces the precision of the values, and we usually think about rounding as reducing the number of decimal places", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 35 + }, + { + "text": "size are more appropriate for the analysis. Fixed-size bins can be accomplished in a few ways, including with rounding, logarithms, and n-tiles. To create equal-width bins, round‐ ing is useful. Rounding reduces the precision of the values, and we usually think about rounding as reducing the number of decimal places or removing them alto‐ gether by rounding to the nearest integer. The round function takes the form: round(value,number_of_decimal_places) The number of decimal places can also be a negative number, allowing this function to round to the nearest tens, hundreds, thousands, and so on. Table 2-2 demonstrates the results of rounding with arguments ranging from –3 to 2. Table 2-2. The number 123,456.789 rounded with various decimal places Decimal places Formula Result 2 round(123456.789,2) 123456.79 1 round(123456.789,1) 123456.8 0 round(123456.789,0) 123457 -1 round(123456.789,-1) 123460 -2 round(123456.789,-2) 123500 -3 round(123456.789,-3) 123000 SELECT round(sales,-1) as bin ,count(customer_id) as customers FROM table GROUP BY 1 ; 32 | Chapter 2: Preparing Data for Analysis Logarithms are another way to create bins, particularly in data sets in which the larg‐ est values are orders of magnitude greater than the smallest values. The distribution of household wealth, the number of website visitors across different properties on the internet, and the shaking force of earthquakes are all examples of phenomena that have this property. While they don’t create bins of equal width, logarithms create bins that increase in size with a useful pattern. To refresh your memory, a logarithm is the exponent to which 10 must be raised to produce that number: log(number) = exponent In this case, 10 is called the base, and this is usually the default implementation in databases, but technically the base can be any number. Table 2-3 shows the loga‐ rithms for several powers of 10. Table 2-3. Results of log function on powers of 10 Formula Result log(1) 0 log(10) 1 log(100) 2 log(1000) 3 log(10000) 4 In SQL, the log function returns the logarithm of its argument, which can be a con‐ stant or a field: SELECT log(sales) as bin ,count(customer_id) as customers FROM table GROUP BY 1 ; The log function can be used on any positive value, not just multiples of 10. How‐ ever, the logarithm function does not work when values can be less than or equal to 0; it will return null or an error, depending on the database. n-Tiles You’re probably familiar with the median, or middle value, of a data set. This is the 50th percentile value. Half of the values are larger than the median, and the other half are smaller. With quartiles, we fill in the 25th and 75th percentile values. A quarter of the values are smaller and three quarters are larger for the 25th percentile; three quar‐ ters are smaller and one quarter are larger at the 75th percentile. Deciles break the data set into 10 equal parts. Making this concept generic, n-tiles allow us to calculate any percentile of the data set: 27th percentile, 50.5th percentile, and so on.", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 36 + }, + { + "text": "larger for the 25th percentile; three quar‐ ters are smaller and one quarter are larger at the 75th percentile. Deciles break the data set into 10 equal parts. Making this concept generic, n-tiles allow us to calculate any percentile of the data set: 27th percentile, 50.5th percentile, and so on. Profiling: Distributions | 33 Window Functions The n-tiles functions are part of a group of SQL functions called window or analytic functions. Unlike most SQL functions, which can operate only on the current row of data, window functions perform calculations that span multiple rows. Window func‐ tions have special syntax that includes the function name and an OVER clause that is used to determine the rows on which to operate and the ordering of those rows. The general format of a window function is: function(field_name) over (partition by field_name order by field_name) The function can be any of the normal aggregations (count, sum, avg, min, max) as well as a number of special functions, including rank, first_value, and ntile. The PAR‐ TITION BY clause can include zero or more fields. When no fields are specified, the function operates over the entire table, but when one or more fields are specified, the function will operate only on that section of rows. For example, we might PARTI‐ TION BY a customer_id to perform calculations about all of the records per cus‐ tomer, restarting the calculation for each customer. The ORDER BY clause determines the ordering of the rows for functions that rely on this; for example, to rank custom‐ ers, we need to specify a field by which to order them, such as number of orders. All of the major database types have window functions, except for versions of MySQL prior to 8.0.2. We will see these useful functions throughout the book, along with additional explanations of how they work and how to set up the arguments correctly. Many databases have a median function built in but rely on more generic n-tile func‐ tions for the rest. These functions are window functions, computing across a range of rows to return a value for a single row. They take an argument that specifies the num‐ ber of bins to split the data into and, optionally, a PARTITION BY and/or an ORDER BY clause: ntile(num_bins) over (partition by... order by...) As an example, imagine we had 12 transactions with order_amounts of $19.99, $9.99, $59.99, $11.99, $23.49, $55.98, $12.99, $99.99, $14.99, $34.99, $4.99, and $89.99. Per‐ forming an ntile calculation with 10 bins sorts each order_amount and assigns a bin from 1 to 10: order_amount ntile ------------ ----- 4.99 1 9.99 1 11.99 2 12.99 2 14.99 3 19.99 4 23.49 5 34.99 6 34 | Chapter 2: Preparing Data for Analysis 55.98 7 59.99 8 89.99 9 99.99 10 This can be used to bin records in practice by first calculating the ntile of each row in a subquery and then wrapping it in an outer query that uses min and max to find", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 37 + }, + { + "text": "| Chapter 2: Preparing Data for Analysis 55.98 7 59.99 8 89.99 9 99.99 10 This can be used to bin records in practice by first calculating the ntile of each row in a subquery and then wrapping it in an outer query that uses min and max to find the upper and lower boundaries of the value range: SELECT ntile ,min(order_amount) as lower_bound ,max(order_amount) as upper_bound ,count(order_id) as orders FROM ( SELECT customer_id, order_id, order_amount ,ntile(10) over (order by order_amount) as ntile FROM orders ) a GROUP BY 1 ; A related function is percent_rank. Instead of returning the bins that the data falls into, percent_rank returns the percentile. It takes no argument but requires paren‐ theses and optionally takes a PARTITION BY and/or an ORDER BY clause: percent_rank() over (partition by... order by...) While not as useful as ntile for binning, percent_rank can be used to create a con‐ tinuous distribution, or it can be used as an output itself for reporting or further anal‐ ysis. Both ntile and percent_rank can be expensive to compute over large data sets, since they require sorting all the rows. Filtering the table to only the data set you need helps. Some databases have implemented approximate versions of the functions that are faster to compute and generally return high-quality results if absolute precision is not required. We will look at additional uses for n-tiles in the discussion of anomaly detection in Chapter 6. In many contexts, there is no single correct or objectively best way to look at distribu‐ tions of data. There is significant leeway for analysts to use the preceding techniques to understand data and present it to others. However, data scientists need to use judg‐ ment and must bring their ethical radar along whenever sharing distributions of sen‐ sitive data. Profiling: Data Quality Data quality is absolutely critical when it comes to creating good analysis. Although this may seem obvious, it has been one of the hardest lessons I’ve learned in my years of working with data. It’s easy to get overly focused on the mechanics of processing Profiling: Data Quality | 35 the data, finding clever query techniques and just the right visualization, only to have stakeholders ignore all of that and point out the one data inconsistency. Ensuring data quality can be one of the hardest and most frustrating parts of analysis. The saying “garbage in, garbage out” captures only part of the problem. Good ingredients in plus incorrect assumptions can also lead to garbage out. Comparing data against ground truth, or what is otherwise known to be true, is ideal though not always possible. For example, if you are working with a replica of a pro‐ duction database, you could compare the row counts in each system to verify that all rows arrived in the replica database. In other cases, you might know the dollar value and count of sales in a particular month and thus can query for this information in the database to make", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 38 + }, + { + "text": "duction database, you could compare the row counts in each system to verify that all rows arrived in the replica database. In other cases, you might know the dollar value and count of sales in a particular month and thus can query for this information in the database to make sure the sum of sales and count of records match. Often the dif‐ ference between your query results and the expected value comes down to whether you applied the correct filters, such as excluding cancelled orders or test accounts; how you handled nulls and spelling anomalies; and whether you set up correct JOIN conditions between tables. Profiling is a way to uncover data quality issues early on, before they negatively impact results and conclusions drawn from the data. Profiling reveals nulls, categori‐ cal codings that need to be deciphered, fields with multiple values that need to be parsed, and unusual datetime formats. Profiling can also uncover gaps and step changes in the data that have resulted from tracking changes or outages. Data is rarely perfect, and it’s often only through its use in analysis that data quality issues are uncovered. Detecting Duplicates A duplicate is when you have two (or more) rows with the same information. Dupli‐ cates can exist for any number of reasons. A mistake might have been made during data entry, if there is some manual step. A tracking call might have fired twice. A pro‐ cessing step might have run multiple times. You might have created it accidentally with a hidden many-to-many JOIN. However they come to be, duplicates can really throw a wrench in your analysis. I can recall times early in my career when I thought I had a great finding, only to have a product manager point out that my sales figure was twice the actual sales. It’s embarrassing, it erodes trust, and it requires rework and sometimes painstaking reviews of the code to find the problem. I’ve learned to check for duplicates as I go. Fortunately, it’s relatively easy to find duplicates in our data. One way is to inspect a sample, with all columns ordered: SELECT column_a, column_b, column_c... FROM table ORDER BY 1,2,3... ; 36 | Chapter 2: Preparing Data for Analysis This will reveal whether the data is full of duplicates, for example, when looking at a brand-new data set, when you suspect that a process is generating duplicates, or after a possible Cartesian JOIN. If there are only a few duplicates, they might not show up in the sample. And scrolling through data to try to spot duplicates is taxing on your eyes and brain. A more systematic way to find duplicates is to SELECT the col‐ umns and then count the rows (this might look familiar from the discussion of histograms!): SELECT count(*) FROM ( SELECT column_a, column_b, column_c... , count(*) as records FROM... GROUP BY 1,2,3... ) a WHERE records > 1 ; This will tell you whether there are any cases of duplicates. If the", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 39 + }, + { + "text": "and then count the rows (this might look familiar from the discussion of histograms!): SELECT count(*) FROM ( SELECT column_a, column_b, column_c... , count(*) as records FROM... GROUP BY 1,2,3... ) a WHERE records > 1 ; This will tell you whether there are any cases of duplicates. If the query returns 0, you’re good to go. For more detail, you can list out the number of records (2, 3, 4, etc.): SELECT records, count(*) FROM ( SELECT column_a, column_b, column_c..., count(*) as records FROM... GROUP BY 1,2,3... ) a WHERE records > 1 GROUP BY 1 ; As an alternative to a subquery, you can use a HAVING clause and keep everything in a single main query. Since it is evaluated after the aggregation and GROUP BY, HAVING can be used to filter on the aggregation value: SELECT column_a, column_b, column_c..., count(*) as records FROM... GROUP BY 1,2,3... HAVING count(*) > 1 ; I prefer to use subqueries, because I find that they’re a useful way to organize my logic. Chapter 8 will discuss order of evaluation and strategies for keeping your SQL queries organized. Profiling: Data Quality | 37 For full detail on which records have duplicates, you can list out all the fields and then use this information to chase down which records are problematic: SELECT * FROM ( SELECT column_a, column_b, column_c..., count(*) as records FROM... GROUP BY 1,2,3... ) a WHERE records = 2 ; Detecting duplicates is one thing; figuring out what to do about them is another. It’s almost always useful to understand why duplicates are occurring and, if possible, fix the problem upstream. Can a data process be improved to reduce or remove duplica‐ tion? Is there an error in an ETL process? Have you failed to account for a one-to- many relationship in a JOIN? Next, we’ll turn to some options for handling and removing duplicates with SQL. Deduplication with GROUP BY and DISTINCT Duplicates happen, and they’re not always a result of bad data. For example, imagine we want to find a list of all the customers who have successfully completed a transac‐ tion so we can send them a coupon for their next order. We might JOIN the custom ers table to the transactions table, which would restrict the records returned to only those customers that appear in the transactions table: SELECT a.customer_id, a.customer_name, a.customer_email FROM customers a JOIN transactions b on a.customer_id = b.customer_id ; This will return a row for each customer for each transaction, however, and there are hopefully at least a few customers who have transacted more than once. We have accidentally created duplicates, not because there is any underlying data quality prob‐ lem but because we haven’t taken care to avoid duplication in the results. Fortunately, there are several ways to avoid this with SQL. One way to remove duplicates is to use the keyword DISTINCT: SELECT distinct a.customer_id, a.customer_name, a.customer_email FROM customers a JOIN transactions b on a.customer_id = b.customer_id ; Another option", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 40 + }, + { + "text": "because we haven’t taken care to avoid duplication in the results. Fortunately, there are several ways to avoid this with SQL. One way to remove duplicates is to use the keyword DISTINCT: SELECT distinct a.customer_id, a.customer_name, a.customer_email FROM customers a JOIN transactions b on a.customer_id = b.customer_id ; Another option is to use a GROUP BY, which, although typically seen in connection with an aggregation, will also deduplicate in the same way as DISTINCT. I remember the first time I saw a colleague use GROUP BY without an aggregation dedupe—I 38 | Chapter 2: Preparing Data for Analysis didn’t even realize it was possible. I find it somewhat less intuitive than DISTINCT, but the result is the same: SELECT a.customer_id, a.customer_name, a.customer_email FROM customers a JOIN transactions b on a.customer_id = b.customer_id GROUP BY 1,2,3 ; Another useful technique is to perform an aggregation that returns one row per entity. Although technically not deduping, it has a similar effect. For example, if we have a number of transactions by the same customer and need to return one record per customer, we could find the min (first) and/or the max (most recent) transac tion_date: SELECT customer_id ,min(transaction_date) as first_transaction_date ,max(transaction_date) as last_transaction_date ,count(*) as total_orders FROM table GROUP BY customer_id ; Duplicate data, or data that contains multiple records per entity even if they techni‐ cally are not duplicates, is one of the most common reasons for incorrect query results. You can suspect duplicates as the cause if all of a sudden the number of cus‐ tomers or total sales returned by a query is many times greater than what you were expecting. Fortunately, there are several techniques that can be applied to prevent this from occurring. Another common problem is missing data, which we’ll turn to next. Preparing: Data Cleaning Profiling often reveals where changes can make the data more useful for analysis. Some of the steps are CASE transformations, adjusting for null, and changing data types. Cleaning Data with CASE Transformations CASE statements can be used to perform a variety of cleaning, enrichment, and sum‐ marization tasks. Sometimes the data exists and is accurate, but it would be more use‐ ful for analysis if values were standardized or grouped into categories. The structure of CASE statements was presented earlier in this chapter, in the section on binning. Nonstandard values occur for a variety of reasons. Values might come from different systems with slightly different lists of choices, system code might have changed, Preparing: Data Cleaning | 39 options might have been presented to the customer in different languages, or the cus‐ tomer might have been able to fill out the value rather than pick from a list. Imagine a field containing information about the gender of a person. Values indicat‐ ing a female person exist as “F,” “female,” and “femme.” We can standardize the values like this: CASE when gender = 'F' then 'Female' when gender = 'female' then 'Female' when gender = 'femme' then 'Female' else gender end", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 41 + }, + { + "text": "field containing information about the gender of a person. Values indicat‐ ing a female person exist as “F,” “female,” and “femme.” We can standardize the values like this: CASE when gender = 'F' then 'Female' when gender = 'female' then 'Female' when gender = 'femme' then 'Female' else gender end as gender_cleaned CASE statements can also be used to add categorization or enrichment that does not exist in the original data. As an example, many organizations use a Net Promoter Score, or NPS, to monitor customer sentiment. NPS surveys ask respondents to rate, on a scale of 0 to 10, how likely they are to recommend a company or product to a friend or colleague. Scores of 0 to 6 are considered detractors, 7 and 8 are passive, and 9 and 10 are promoters. The final score is calculated by subtracting the percentage of detractors from the percentage of promoters. Survey result data sets usually include optional free text comments and are sometimes enriched with information the orga‐ nization knows about the person surveyed. Given a data set of NPS survey responses, the first step is to group the responses into the categories of detractor, passive, and promoter: SELECT response_id ,likelihood ,case when likelihood <= 6 then 'Detractor' when likelihood <= 8 then 'Passive' else 'Promoter' end as response_type FROM nps_responses ; Note that the data type can differ between the field being evaluated and the return data type. In this case, we are checking an integer and returning a string. Listing out all the values with an IN list is also an option. The IN operator allows you to specify a list of items rather than having to write an equality for each one separately. It is useful when the input isn’t continuous or when values in order shouldn’t be grouped together: case when likelihood in (0,1,2,3,4,5,6) then 'Detractor' when likelihood in (7,8) then 'Passive' when likelihood in (9,10) then 'Promoter' end as response_type CASE statements can consider multiple columns and can contain AND/OR logic. They can also be nested, though often this can be avoided with AND/OR logic: 40 | Chapter 2: Preparing Data for Analysis case when likelihood <= 6 and country = 'US' and high_value = true then 'US high value detractor' when likelihood >= 9 and (country in ('CA','JP') or high_value = true ) then 'some other label' ... end Alternatives for Cleaning Data Cleaning or enriching data with a CASE statement works well as long as there is a relatively short list of variations, you can find them all in the data, and the list of val‐ ues isn’t expected to change. For longer lists and ones that change frequently, a lookup table can be a better option. A lookup table exists in the database and is either static or populated with code that checks for new values periodically. The query will JOIN to the lookup table to get the cleaned data. In this way, the cleaned values can be maintained outside your code", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 42 + }, + { + "text": "be a better option. A lookup table exists in the database and is either static or populated with code that checks for new values periodically. The query will JOIN to the lookup table to get the cleaned data. In this way, the cleaned values can be maintained outside your code and used by many queries, without your having to worry about maintaining consistency between them. An example of this might be a lookup table that maps state abbreviations to full state names. In my own work, I often start with a CASE statement and create a lookup table only after the list becomes unruly, or once it’s clear that my team or I will need to use this cleaning step repeatedly. Of course, it’s worth investigating whether the data can be cleaned upstream. I once started with a CASE statement of 5 or so lines that grew to 10 lines and then eventu‐ ally to more than 100 lines, at which point the list was unruly and difficult to main‐ tain. The insights were valuable enough that I was able to convince engineers to change the tracking code and send the meaningful categorizations in the data stream in the first place. Another useful thing you can do with CASE statements is to create flags indicating whether a certain value is present, without returning the actual value. This can be useful during profiling for understanding how common the existence of a particular attribute is. Another use for flagging is during preparation of a data set for statistical analysis. In this case, a flag is also known as a dummy variable, taking a value of 0 or 1 and indicating the presence or absence of some qualitative variable. For example, we can create is_female and is_promoter flags with CASE statements on gender and likelihood (to recommend) fields: SELECT customer_id ,case when gender = 'F' then 1 else 0 end as is_female ,case when likelihood in (9,10) then 1 else 0 end as is_promoter Preparing: Data Cleaning | 41 FROM ... ; If you are working with a data set that has multiple rows per entity, such as with line items in an order, you can flatten the data with a CASE statement wrapped in an aggregate and turn it into a flag at the same time by using 1 and 0 as the return value. We saw previously that a BOOLEAN data type is often used to create flags (fields that represent the presence or absence of some attribute). Here, 1 is substituted for TRUE and 0 is substituted for FALSE so that a max aggregation can be applied. The way this works is that for each customer, the CASE statement returns 1 for any row with a fruit type of “apple.” Then max is evaluated and will return the largest value from any of the rows. As long as a customer bought an apple at least once, the flag will be 1; if not, it will be 0: SELECT customer_id ,max(case", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 43 + }, + { + "text": "1 for any row with a fruit type of “apple.” Then max is evaluated and will return the largest value from any of the rows. As long as a customer bought an apple at least once, the flag will be 1; if not, it will be 0: SELECT customer_id ,max(case when fruit = 'apple' then 1 else 0 end) as bought_apples ,max(case when fruit = 'orange' then 1 else 0 end) as bought_oranges FROM ... GROUP BY 1 ; You can also construct more complex conditions for flags, such as requiring a thres‐ hold or amount of something before labeling with a value of 1: SELECT customer_id ,max(case when fruit = 'apple' and quantity > 5 then 1 else 0 end) as loves_apples ,max(case when fruit = 'orange' and quantity > 5 then 1 else 0 end) as loves_oranges FROM ... GROUP BY 1 ; CASE statements are powerful, and as we saw, they can be used to clean, enrich, and flag or add dummy variables to data sets. In the next section, we’ll look at some spe‐ cial functions related to CASE statements that handle null values specifically. Type Conversions and Casting Every field in a database is defined with a data type, which we reviewed at the begin‐ ning of this chapter. When data is inserted into a table, values that aren’t of the field’s type are rejected by the database. Strings can’t be inserted into integer fields, and boo‐ leans are not allowed in date fields. Most of the time, we can take the data types for 42 | Chapter 2: Preparing Data for Analysis granted and apply string functions to strings, date functions to dates, and so on. Occasionally, however, we need to override the data type of the field and force it to be something else. This is where type conversions and casting come in. Type conversion functions allow pieces of data with the appropriate format to be changed from one data type to another. The syntax comes in a few forms that are basically equivalent. One way to change the data type is with the cast function, cast (input as data_type), or two colons, input :: data_type. Both of these are equivalent and convert the integer 1,234 to a string: cast (1234 as varchar) 1234::varchar Converting an integer to a string can be useful in CASE statements when categorizing numeric values with some unbounded upper or lower value. For example, in the fol‐ lowing code, leaving the values that are less than or equal to 3 as integers while returning the string “4+” for higher values would result in an error: case when order_items <= 3 then order_items else '4+' end Casting the integers to the VARCHAR type solves the problem: case when order_items <= 3 then order_items::varchar else '4+' end Type conversions also come in handy when values that should be integers are parsed out of a string, and then we want to aggregate the values or use mathematical func‐ tions on them.", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 44 + }, + { + "text": "to the VARCHAR type solves the problem: case when order_items <= 3 then order_items::varchar else '4+' end Type conversions also come in handy when values that should be integers are parsed out of a string, and then we want to aggregate the values or use mathematical func‐ tions on them. Imagine we have a data set of prices, but the values include the dollar sign ($), and so the data type of the field is VARCHAR. We can remove the $ charac‐ ter with a function called replace, which will be discussed more during our look at text analysis in Chapter 5: SELECT replace('$19.99','$',''); replace ------- 9.99 The result is still a VARCHAR, however, so trying to apply an aggregation will return an error. To fix this, we can cast the result as a FLOAT: replace('$19.99','$','')::float cast(replace('$19.99','$','')) as float Dates and datetimes can come in a bewildering array of formats, and understanding how to cast them to the desired format is useful. I’ll show a few examples on type conversion here, and Chapter 3 will go into more detail on date and datetime calcula‐ tions. As a simple example, imagine that transaction or event data often arrives in the Preparing: Data Cleaning | 43 database as a TIMESTAMP, but we want to summarize some value such as transac‐ tions by day. Simply grouping by the timestamp will result in more rows than necessary. Casting the TIMESTAMP to a DATE reduces the size of the results and achieves our summarization goal: SELECT tx_timestamp::date, count(transactions) as num_transactions FROM ... GROUP BY 1 ; Likewise, a DATE can be cast to a TIMESTAMP when a SQL function requires a TIMESTAMP argument. Sometimes the year, month, and day are stored in separate columns, or they end up as separate elements because they’ve been parsed out of a longer string. These then need to be assembled back into a date. To do this, we use the concatenation operator || (double pipe) or concat function and then cast the result to a DATE. Any of these syntaxes works and returns the same value: (year || ',' || month|| '-' || day)::date Or equivalently: cast(concat(year, '-', month, '-', day) as date) Yet another way to convert between string values and dates is by using the date func‐ tion. For example, we can construct a string value as above and convert it into a date: date(concat(year, '-', month, '-', day)) The to_datatype functions can take both a value and a format string and thus give you more control over how the data is converted. Table 2-4 summarizes the functions and their purposes. They are particularly useful when converting in and out of DATE or DATETIME formats, as they allow you to specify the order of the date and time elements. Table 2-4. The to_datatype functions Function Purpose to_char Converts other types to string to_number Converts other types to numeric to_date Converts other types to date, with specified date parts to_timestamp Converts other types to date, with specified date", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 45 + }, + { + "text": "allow you to specify the order of the date and time elements. Table 2-4. The to_datatype functions Function Purpose to_char Converts other types to string to_number Converts other types to numeric to_date Converts other types to date, with specified date parts to_timestamp Converts other types to date, with specified date and time parts Sometimes the database automatically converts a data type. This is called type coer‐ cion. For example, INT and FLOAT numerics can usually be used together in mathe‐ matical functions or aggregations without explicitly changing the type. CHAR and VARCHAR values can usually be mixed. Some databases will coerce BOOLEAN fields to 0 and 1 values, where 0 is FALSE and 1 is TRUE, but some databases require you to convert the values explicitly. Some databases are pickier than others about 44 | Chapter 2: Preparing Data for Analysis mixing dates and datetimes in result sets and functions. You can read through the documentation, or you can do some simple query experiments to learn how the database you’re working with handles data types implicitly and explicitly. There is usually a way to accomplish what you want, though sometimes you need to get crea‐ tive in using functions in your queries. Dealing with Nulls: coalesce, nullif, nvl Functions Null was one of the stranger concepts I had to get used to when I started working with data. Null just isn’t something we think about in daily life, where we’re used to dealing in concrete quantities of things. Null has a special meaning in databases and was introduced by Edgar Codd, the inventor of the relational database, to ensure that databases have a way to represent missing information. If someone asks me how many parachutes I have, I can answer “zero.” But if the question is never asked, I have null parachutes. Nulls can represent fields for which no data was collected or that aren’t applicable for that row. When new columns are added to a table, the values for previously created rows will be null unless explicitly filled with some other value. When two tables are joined via an OUTER JOIN, nulls will appear in any fields for which there is no matching record in the second table. Nulls are problematic for certain aggregations and groupings, and different types of databases handle them in different ways. For example, imagine I have five records, with 5, 10, 15, 20, and null. The sum of these is 50, but the average is either 10 or 12.5 depending on whether the null value is counted in the denominator. The whole ques‐ tion may also be considered invalid since one of the values is null. For most database functions, a null input will return a null output. Equalities and inequalities involving null also return null. A variety of unexpected and frustrating results can be output from your queries if you are not on the lookout for nulls. When tables are defined, they can either allow nulls, reject nulls, or populate a default value", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 46 + }, + { + "text": "return a null output. Equalities and inequalities involving null also return null. A variety of unexpected and frustrating results can be output from your queries if you are not on the lookout for nulls. When tables are defined, they can either allow nulls, reject nulls, or populate a default value if the field would otherwise be left null. In practice, this means that you can’t always rely on a field to show up as null if the data is missing, because it may have been filled with a default value such as 0. I once had a long debate with a data engi‐ neer when it turned out that null dates in the source system were defaulting to “1970-01-01” in our data warehouse. I insisted that the dates should be null instead, to reflect the fact that they were unknown or not applicable. The engineer pointed out that I could remember to filter those dates or change them back to null with a CASE statement. I finally prevailed by pointing out that one day another user who wasn’t as aware of the nuances of default dates would come along, run a query, and get the puz‐ zling cluster of customers about a year before the company was even founded. Preparing: Data Cleaning | 45 Nulls are often inconvenient or inappropriate for the analysis you want to do. They can also make output confusing to the intended audience for your analysis. Business‐ people don’t necessarily understand how to interpret a null value or may assume that null values represent a problem with data quality. Empty Strings A concept related to but slightly different from nulls is empty string, where there is no value but the field is not technically null. One reason an empty string might be used is to indicate that a field is known to be blank, as opposed to null, where the value might be missing or unknown. For example, the database might have a name_suffix field that can be used to hold a value such as “Jr.” Many people do not have a name_suffix, so an empty string is appropriate. Empty string can also be used as a default value instead of null, or as a way to overcome a NOT NULL constraint by inserting a value, even if empty. An empty string can be specified in a query with two quote marks: WHERE my_field = '' or my_field <> 'apple' Profiling the frequencies of values should reveal whether your data includes nulls, empty strings, or both. There are a few ways to replace nulls with alternate values: CASE statements, and the specialized coalesce and nullif functions. We saw previously that CASE statements can check a condition and return a value. They can also be used to check for a null and, if one is found, replace it with another value: case when num_orders is null then 0 else num_orders end case when address is null then 'Unknown' else address end case when column_a is null then", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 47 + }, + { + "text": "and return a value. They can also be used to check for a null and, if one is found, replace it with another value: case when num_orders is null then 0 else num_orders end case when address is null then 'Unknown' else address end case when column_a is null then column_b else column_a end The coalesce function is a more compact way to achieve this. It takes two or more arguments and returns the first one that is not null: coalesce(num_orders,0) coalesce(address,'Unknown') coalesce(column_a,column_b) coalesce(column_a,column_b,column_c) The function nvl exists in some databases and is similar to coalesce, but it allows only two arguments. 46 | Chapter 2: Preparing Data for Analysis The nullif function compares two numbers, and if they are not equal, it returns the first number; if they are equal, the function returns null. Running this code: nullif(6,7) returns 6, whereas null is returned by: nullif(6,6) nullif is equivalent to the following, more wordy case statement: case when 6 = 7 then 6 when 6 = 6 then null end This function can be useful for turning values back into nulls when you know a cer‐ tain default value has been inserted into the database. For example, with my default time example, we could change it back to null by using: nullif(date,'1970-01-01') Nulls can be problematic when filtering data in the WHERE clause. Returning values that are null is fairly straightforward: WHERE my_field is null However, imagine that my_field contains some nulls and also some names of fruits. I would like to return all rows that are not apples. It seems like this should work: WHERE my_field <> 'apple' However, some databases will exclude both the “apple” rows and all rows with null values in my_field. To correct this, the SQL should both filter out “apple” and explicitly include nulls by connecting the conditions with OR: WHERE my_field <> 'apple' or my_field is null Nulls are a fact of life when working with data. Regardless of why they occur, we often need to consider them in profiling and as targets for data cleaning. Fortunately, there are a number of ways to detect them with SQL, as well as several useful functions that allow us to replace nulls with alternate values. Next we’ll look at missing data, a prob‐ lem that can cause nulls but has even wider implications and thus deserves a section of its own. Missing Data Data can be missing for a variety of reasons, each with its own implications for how you decide to handle the data’s absence. A field might not have been required by the system or process that collected it, as with an optional “how did you hear about us?” field in an ecommerce checkout flow. Requiring this field might create friction for the Preparing: Data Cleaning | 47 customer and decrease successful checkouts. Alternatively, data might normally be required but wasn’t collected due to a code bug or human error, such as in a medical questionnaire where the interviewer missed the second", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 48 + }, + { + "text": "ecommerce checkout flow. Requiring this field might create friction for the Preparing: Data Cleaning | 47 customer and decrease successful checkouts. Alternatively, data might normally be required but wasn’t collected due to a code bug or human error, such as in a medical questionnaire where the interviewer missed the second page of questions. A change in the way the data was collected can result in records before or after the change hav‐ ing missing values. A tool tracking mobile app interactions might add an additional field recording whether the interaction was a tap or a scroll, for example, or remove another field due to functionality change. Data can be orphaned when a table refer‐ ences a value in another table, and that row or the entire table has been deleted or is not yet loaded into the data warehouse. Finally, data may be available but not at the level of detail, or granularity, needed for the analysis. An example of this comes from subscription businesses, where customers pay on an annual basis for a monthly prod‐ uct and we want to analyze monthly revenue. In addition to profiling the data with histograms and frequency analysis, we can often detect missing data by comparing values in two tables. For example, we might expect that each customer in the transactions table also has a record in the customer table. To check this, query the tables using a LEFT JOIN and add a WHERE condition to find the customers that do not exist in the second table: SELECT distinct a.customer_id FROM transactions a LEFT JOIN customers b on a.customer_id = b.customer_id WHERE b.customer_id is null ; Missing data can be an important signal in and of itself, so don’t assume that it always needs to be fixed or filled. Missing data can reveal the underlying system design or biases in the data collection process. Records with missing fields can be filtered out entirely, but often we want to keep them and instead make some adjustments based on what we know about expected or typical values. We have some options, called imputation techniques, for filling in missing data. These include filling with an average or median of the data set, or with the previous value. Documenting the missing data and how it was replaced is impor‐ tant, as this may impact the downstream interpretation and use of the data. Imputed values can be particularly problematic when the data is used in machine learning, for example. A common option is to fill missing data with a constant value. Filling with a constant value can be useful when the value is known for some records even though they were not populated in the database. For example, imagine there was a software bug that prevented the population of the price for an item called “xyz,” but we know the price is always $20. A CASE statement can be added to the query to handle this: case when price is null and item_name = 'xyz' then 20 else", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 49 + }, + { + "text": "imagine there was a software bug that prevented the population of the price for an item called “xyz,” but we know the price is always $20. A CASE statement can be added to the query to handle this: case when price is null and item_name = 'xyz' then 20 else price end as price 48 | Chapter 2: Preparing Data for Analysis Another option is to fill with a derived value, either a mathematical function on other columns or a CASE statement. For example, imagine we have a field for the net_sales amount for each transaction. Due to a bug, some rows don’t have this field populated, but they do have the gross_sales and discount fields populated. We can calculate net_sales by subtracting discount from gross_sales: SELECT gross_sales - discount as net_sales... Missing values can also be filled with values from other rows in the data set. Carrying over a value from the previous row is called fill forward, while using a value from the next row is called fill backward. These can be accomplished with the lag and lead window functions, respectively. For example, imagine that our transaction table has a product_price field that stores the undiscounted price a customer pays for a prod uct. Occasionally this field is not populated, but we can make an assumption that the price is the same as the price paid by the last customer to buy that product. We can fill with the previous value using the lag function, PARTITION BY the product to ensure the price is pulled only from the same product, and ORDER BY the appropri‐ ate date to ensure the price is pulled from the most recent prior transaction: lag(product_price) over (partition by product order by order_date) The lead function could be used to fill with product_price for the following transac‐ tion. Alternatively, we could take the avg of prices for the product and use that to fill in the missing value. Filling with previous, next, or average values involves making some assumptions about typical values and what’s reasonable to include in an analy‐ sis. It’s always a good idea to check the results to make sure they are plausible and to note that you have interpolated the data when not available. For data that is available but not at the granularity needed, we often have to create additional rows in the data set. For example, imagine we have a customer_subscrip tions table with the fields subscription_date and annual_amount. We can spread this annual subscription amount into 12 equal monthly revenue amounts by dividing by 12, effectively converting ARR (annual recurring revenue) into MRR (monthly recurring revenue): SELECT customer_id ,subscription_date ,annual_amount ,annual_amount / 12 as month_1 ,annual_amount / 12 as month_2 ... ,annual_amount / 12 as month_12 FROM customer_subscriptions ; Preparing: Data Cleaning | 49 This gets a bit tedious, particularly if subscription periods can be two, three, or five years as well as one year. It’s also not helpful if what we want is the actual dates", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 50 + }, + { + "text": "12 as month_2 ... ,annual_amount / 12 as month_12 FROM customer_subscriptions ; Preparing: Data Cleaning | 49 This gets a bit tedious, particularly if subscription periods can be two, three, or five years as well as one year. It’s also not helpful if what we want is the actual dates of the months. In theory we could write a query like this: SELECT customer_id ,subscription_date ,annual_amount ,annual_amount / 12 as '2020-01' ,annual_amount / 12 as '2020-02' ... ,annual_amount / 12 as '2020-12' FROM customer_subscriptions ; However, if the data includes orders from customers across time, hardcoding the month names won’t be accurate. We could use CASE statements in combination with hardcoded month names, but again this is tedious and is likely to be error-prone as you add more convoluted logic. Instead, creating new rows through a JOIN to a table such as a date dimension provides an elegant solution. A date dimension is a static table that has one row per day, with optional extended date attributes, such as day of the week, month name, end of month, and fiscal year. The dates extend far enough into the past and far enough into the future to cover all anticipated uses. Because there are only 365 or 366 days per year, tables covering even 100 years don’t take up a lot of space. Figure 2-3 shows a sample of the data in a date dimension table. Sample code to create a date dimension using SQL functions is on the book’s GitHub site. Figure 2-3. A date dimension table with date attributes If you’re using a Postgres database, the generate_series function can be used to cre‐ ate a date dimension either to populate the table initially or if creating a table is not an option. It takes the following form: 50 | Chapter 2: Preparing Data for Analysis generate_series(start, stop, step interval) In this function, start is the first date you want in the series, stop is the last date, and step interval is the time period between values. The step interval can take any value, but one day is appropriate for a date dimension: SELECT * FROM generate_series('2000-01-01'::timestamp,'2030-12-31', '1 day') The generate_series function requires at least one of the arguments to be a TIME‐ STAMP, so “2000-01-01” is cast as a TIMESTAMP. We can then create a query that results in a row for every day, regardless of whether a customer ordered on a particu‐ lar day. This is useful when we want to ensure that a customer is counted for each day, or when we specifically want to count or otherwise analyze days on which a customer did not make a purchase: SELECT a.generate_series as order_date, b.customer_id, b.items FROM ( SELECT * FROM generate_series('2020-01-01'::timestamp,'2020-12-31','1 day') ) a LEFT JOIN ( SELECT customer_id, order_date, count(item_id) as items FROM orders GROUP BY 1,2 ) b on a.generate_series = b.order_date ; Returning to our subscription example, we can use the date dimension to create a record for each month by JOINing the date", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 51 + }, + { + "text": "SELECT * FROM generate_series('2020-01-01'::timestamp,'2020-12-31','1 day') ) a LEFT JOIN ( SELECT customer_id, order_date, count(item_id) as items FROM orders GROUP BY 1,2 ) b on a.generate_series = b.order_date ; Returning to our subscription example, we can use the date dimension to create a record for each month by JOINing the date dimension on dates that are between the subscription_date and 11 months later (for 12 total months): SELECT a.date ,b.customer_id ,b.subscription_date ,b.annual_amount / 12 as monthly_subscription FROM date_dim a JOIN customer_subscriptions b on a.date between b.subscription_date and b.subscription_date + interval '11 months' ; Data can be missing for various reasons, and understanding the root cause is impor‐ tant in deciding how to deal with it. There are a number of options for finding and replacing missing data. These include using CASE statements to set default values, deriving values by performing calculations on other fields in the same row, and inter‐ polating from other values in the same column. Data cleaning is an important part of the data preparation process. Data may need to be cleaned for many different reasons. Some data cleaning needs to be done to fix Preparing: Data Cleaning | 51 poor data quality, such as when there are inconsistent or missing values in the raw data, while other data cleaning is done to make further analysis easier or more mean‐ ingful. The flexibility of SQL allows us to perform cleaning tasks in a variety of ways. After data is cleaned, a common next step in the preparation process is shaping the data set. Preparing: Shaping Data Shaping data refers to manipulating the way the data is represented in columns and rows. Each table in the database has a shape. The result set of each query has a shape. Shaping data may seem like a rather abstract concept, but if you work with enough data, you will come to see its value. It is a skill that can be learned, practiced, and mastered. One of the most important concepts in shaping data is figuring out the granularity of data that you need. Just as rocks can range in size from giant boulders down to grains of sand, and even further down to microscopic dust, so too can data have varying lev‐ els of detail. For example, if the population of a country is a boulder, then the popula‐ tion of a city is a small rock, and that of a household is a grain of sand. Data at a smaller level of detail might include individual births and deaths, or moves from one city or country to another. Flattening data is another important concept in shaping. This refers to reducing the number of rows that represent an entity, including down to a single row. Joining mul‐ tiple tables together to create a single output data set is one way to flatten data. Another way is through aggregation. In this section, we’ll first cover some considerations for choosing data shapes. Then we’ll look at some common use cases: pivoting and", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 52 + }, + { + "text": "down to a single row. Joining mul‐ tiple tables together to create a single output data set is one way to flatten data. Another way is through aggregation. In this section, we’ll first cover some considerations for choosing data shapes. Then we’ll look at some common use cases: pivoting and unpivoting. We’ll see examples of shaping data for specific analyses throughout the remaining chapters. Chapter 8 will go into more detail on keeping complex SQL organized when creating data sets for further analysis. For Which Output: BI, Visualization, Statistics, ML Deciding how to shape your data with SQL depends a lot on what you are planning to do with the data afterward. It’s generally a good idea to output a data set that has as few rows as possible while still meeting your need for granularity. This will leverage the computing power of the database, reduce the time it takes to move data from the database to somewhere else, and reduce the amount of processing you or someone else needs to do in other tools. Some of the other tools that your output might go to are a BI tool for reporting and dashboarding, a spreadsheet for business users to examine, a statistics tool such as R, or a machine learning model in Python—or you might output the data straight to a visualization created with a range of tools. 52 | Chapter 2: Preparing Data for Analysis 2 Hadley Wickham, “Tidy Data,” Journal of Statistical Software 59, no. 10 (2014): 1–23, https://doi.org/10.18637/ jss.v059.i10. When outputting data to a business intelligence tool for reports and dashboards, it’s important to understand the use case. Data sets may need to be very detailed to enable exploration and slicing by end users. They may need to be small and aggrega‐ ted and include specific calculations to enable fast loading and response times in executive dashboards. Understanding how the tool works, and whether it performs better with smaller data sets or is architected to perform its own aggregations across larger data sets, is important. There is no “one size fits all” answer. The more you know about how the data will be used, the better prepared you will be to shape the data appropriately. Smaller, aggregated, and highly specific data sets often work best for visualizations, whether they are created in commercial software or using a programming language like R, Python, or JavaScript. Think about the level of aggregation and slices, or vari‐ ous elements, the end users will need to filter on. Sometimes the data sets require a row for each slice, as well as an “everything” slice. You may need to UNION together two queries—one at the detail level and one at the “everything” level. When creating output for statistics packages or machine learning models, it’s impor‐ tant to understand the core entity being studied, the level of aggregation desired, and the attributes or features needed. For example, a model might need one record per customer with several attributes, or a record per", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 53 + }, + { + "text": "“everything” level. When creating output for statistics packages or machine learning models, it’s impor‐ tant to understand the core entity being studied, the level of aggregation desired, and the attributes or features needed. For example, a model might need one record per customer with several attributes, or a record per transaction with its associated attributes as well as customer attributes. Generally, the output for modeling will fol‐ low the notion of “tidy data” proposed by Hadley Wickham.2 Tidy data has these properties: 1. Each variable forms a column. 2. Each observation forms a row. 3. Each value is a cell. We will next look at how to use SQL to transform data from the structure in which it exists in your database into any other pivoted or unpivoted structure that is needed for analysis. Pivoting with CASE Statements A pivot table is a way to summarize data sets by arranging the data into rows, accord‐ ing to the values of an attribute, and columns, according to the values of another attribute. At the intersection of each row and column, a summary statistic such as sum, count, or avg is calculated. Pivot tables are often a good way to summarize data for business audiences, since they reshape the data into a more compact and easily Preparing: Shaping Data | 53 understandable form. Pivot tables are widely known from their implementation in Microsoft Excel, which has a drag-and-drop interface to create the summaries of data. Pivot tables, or pivoted output, can be created in SQL using a CASE statement along with one or more aggregation functions. We’ve seen CASE statements several times so far, and reshaping data is another major use case for them. For example, imagine we have an orders table with a row for each purchase made by customers. To flatten the data, GROUP BY the customer_id and sum the order_amount: SELECT customer_id ,sum(order_amount) as total_amount FROM orders GROUP BY 1 ; customer_id total_amount ----------- ------------ 123 59.99 234 120.55 345 87.99 ... ... To create a pivot, we will additionally create columns for each of the values of an attribute. Imagine the orders table also has a product field that contains the type of item purchased and the order_date. To create pivoted output, GROUP BY the order_date, and sum the result of a CASE statement that returns the order_amount whenever the row meets the product name criteria: SELECT order_date ,sum(case when product = 'shirt' then order_amount else 0 end) as shirts_amount ,sum(case when product = 'shoes' then order_amount else 0 end) as shoes_amount ,sum(case when product = 'hat' then order_amount else 0 end) hats_amount FROM orders GROUP BY 1 ; order_date shirts_amount shoes_amount hats_amount ---------- ------------- ------------ ----------- 2020-05-01 5268.56 1211.65 562.25 2020-05-02 5533.84 522.25 325.62 2020-05-03 5986.85 1088.62 858.35 ... ... ... ... Note that with the sum aggregation, you can optionally use “else 0” to avoid nulls in the result set. With count or count distinct, however, you should not include an 54 | Chapter 2: Preparing", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 54 + }, + { + "text": "5268.56 1211.65 562.25 2020-05-02 5533.84 522.25 325.62 2020-05-03 5986.85 1088.62 858.35 ... ... ... ... Note that with the sum aggregation, you can optionally use “else 0” to avoid nulls in the result set. With count or count distinct, however, you should not include an 54 | Chapter 2: Preparing Data for Analysis 3 US Census Bureau, “International Data Base (IDB),” last updated December 2020, https://www.census.gov/ data-tools/demo/idb. ELSE statement, as doing so would inflate the result set. This is because the database won’t count a null, but it will count a substitute value such as zero. Pivoting with CASE statements is quite handy, and having this ability opens up data warehouse table designs that are long and narrow rather than wide, which can be bet‐ ter for storing sparse data, because adding columns to a table can be an expensive operation. For example, rather than storing various customer attributes in many dif‐ ferent columns, a table could contain multiple records per customer, with each attribute in a separate row, and with attribute_name and attribute_value fields specifying what the attribute is and its value. The data can then be pivoted as needed to assemble a customer record with the desired attributes. This design is efficient when there are many sparse attributes (only a subset of customers have values for many of the attributes). Pivoting data with a combination of aggregation and CASE statements works well when there are a finite number of items to pivot. For people who have worked with other programming languages, it’s essentially looping, but written out explicitly line by line. This gives you a lot of control, such as if you want to calculate different met‐ rics in each column, but it can also be tedious. Pivoting with case statements doesn’t work well when new values arrive constantly or are rapidly changing, since the SQL code would need to be constantly updated. In those cases, pushing the computing to another layer of your analysis stack, such as a BI tool or statistical language, may be more appropriate. Unpivoting with UNION Statements Sometimes we have the opposite problem and need to move data stored in columns into rows instead to create tidy data. This operation is called unpivoting. Data sets that may need unpivoting are those that are in a pivot table format. As an example, the populations of North American countries at 10-year intervals starting in 1980 are shown in Figure 2-4. Figure 2-4. Country population by year (in thousands)3 Preparing: Shaping Data | 55 To turn this into a result set with a row per country per year, we can use a UNION operator. UNION is a way to combine data sets from multiple queries into a single result set. There are two forms, UNION and UNION ALL. When using UNION or UNION ALL, the numbers of columns in each component query must match. The data types must match or be compatible (integers and floats can be mixed, but inte‐ gers and strings cannot). The column", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 55 + }, + { + "text": "a single result set. There are two forms, UNION and UNION ALL. When using UNION or UNION ALL, the numbers of columns in each component query must match. The data types must match or be compatible (integers and floats can be mixed, but inte‐ gers and strings cannot). The column names in the result set come from the first query. Aliasing the fields in the remaining queries is therefore optional but can make a query easier to read: SELECT country ,'1980' as year ,year_1980 as population FROM country_populations UNION ALL SELECT country ,'1990' as year ,year_1990 as population FROM country_populations UNION ALL SELECT country ,'2000' as year ,year_2000 as population FROM country_populations UNION ALL SELECT country ,'2010' as year ,year_2010 as population FROM country_populations ; country year population ------------- ---- ---------- Canada 1980 24593 Mexico 1980 68347 United States 1980 227225 ... ... ... In this example, we use a constant to hardcode the year, in order to keep track of the year that the population value corresponds to. The hardcoded values can be of any type, depending on your use case. You may need to explicitly cast certain hardcoded values, such as when entering a date: '2020-01-01'::date as date_of_interest 56 | Chapter 2: Preparing Data for Analysis What is the difference between UNION and UNION ALL? Both can be used to append or stack data together in this fashion, but they are slightly different. UNION removes duplicates from the result set, whereas UNION ALL retains all records, whether duplicates or not. UNION ALL is faster, since the database doesn’t have to do a pass over the data to find duplicates. It also ensures that every record ends up in the result set. I tend to use UNION ALL, using UNION only when I have a reason to sus‐ pect duplicate data. UNIONing data can also be useful for bringing together data from different sources. For example, imagine we have a populations table with yearly data per country, and another gdp table with yearly gross domestic product, or GDP. One option is to JOIN the tables and obtain a result set with one column for population and another for GDP: SELECT a.country, a.population, b.gdp FROM populations a JOIN gdp b on a.country = b.country ; Another option is to UNION ALL the data sets so that we end up with a stacked data set: SELECT country, 'population' as metric, population as metric_value FROM populations UNION ALL SELECT country, 'gdp' as metric, gdp as metric_value FROM gdp ; Which approach you use largely depends on the output that you need for your analy‐ sis. The latter option can be useful when you have a number of different metrics in different tables and no single table has a full set of entities (in this case, countries). This is an alternative approach to a FULL OUTER JOIN. pivot and unpivot Functions Recognizing that the pivot and unpivot use cases are common, some database ven‐ dors have implemented functions to do this with fewer", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 56 + }, + { + "text": "and no single table has a full set of entities (in this case, countries). This is an alternative approach to a FULL OUTER JOIN. pivot and unpivot Functions Recognizing that the pivot and unpivot use cases are common, some database ven‐ dors have implemented functions to do this with fewer lines of code. Microsoft SQL Server and Snowflake have pivot functions that take the form of extra expressions in the WHERE clause. Here, aggregation is any aggregation function, such as sum or avg, the value_column is the field to be aggregated, and a column will be created for each value of the label_column listed as a label: SELECT... FROM... pivot(aggregation(value_column) for label_column in (label_1, label_2, ...) ; Preparing: Shaping Data | 57 We could rewrite the earlier pivoting example that used CASE statements as follows: SELECT * FROM orders pivot(sum(order_amount) for product in ('shirt','shoes')) GROUP BY order_date ; Although this syntax is more compact than the CASE construction we saw earlier, the desired columns still need to be specified. As a result, pivot doesn’t solve the problem of newly arriving or rapidly changing sets of fields that need to be turned into col‐ umns. Postgres has a similar crosstab function, available in the tablefunc module. Microsoft SQL Server and Snowflake also have unpivot functions that work in a sim‐ ilar fashion to expressions in the WHERE clause and transform rows into columns: SELECT... FROM... unpivot( value_column for label_column in (label_1, label_2, ...)) ; For example, the country_populations data from the previous example could be reshaped in the following manner: SELECT * FROM country_populations unpivot(population for year in (year_1980, year_1990, year_2000, year_2010)) ; Here again the syntax is more compact than the UNION or UNION ALL approach we looked at earlier, but the list of columns must be specified in the query. Postgres has an unnest array function that can be used to unpivot data, thanks to its array data type. An array is a collection of elements, and in Postgres you can list the elements of an array in square brackets. The function can be used in the SELECT clause and takes this form: unnest(array[element_1, element_2, ...]) Returning to our earlier example with countries and populations, this query returns the same result as the query with the repeated UNION ALL clauses: SELECT country ,unnest(array['1980', '1990', '2000', '2010']) as year ,unnest(array[year_1980, year_1990, year_2000, year_2010]) as pop FROM country_populations ; 58 | Chapter 2: Preparing Data for Analysis country year pop ------- ---- ----- Canada 1980 24593 Canada 1990 27791 Canada 2000 31100 ... ... ... Data sets arrive in many different formats and shapes, and they aren’t always in the format needed in our output. There are several options for reshaping data through pivoting or unpivoting it, either with CASE statements or UNIONs, or with database- specific functions. Understanding how to manipulate your data in order to shape it in the way you want will give you greater flexibility in your analysis and in the way you present your results. Conclusion", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 57 + }, + { + "text": "data through pivoting or unpivoting it, either with CASE statements or UNIONs, or with database- specific functions. Understanding how to manipulate your data in order to shape it in the way you want will give you greater flexibility in your analysis and in the way you present your results. Conclusion Preparing data for analysis can feel like the work you do before you get to the real work of analysis, but it is so fundamental to understanding the data that I always find it is time well spent. Understanding the different types of data you’re likely to encounter is critical, and you should take the time to understand the data types in each table you work with. Profiling data helps us learn more about what is in the data set and examine it for quality. I often return to profiling throughout my analysis projects, as I learn more about the data and need to check my query results along the way as I build in complexity. Data quality will likely never stop being a problem, so we’ve looked at some ways to handle the cleaning and enhancement of data sets. Finally, knowing how to shape the data to create the right output format is essential. We’ll see these topics recur in the context of various analyses throughout the book. The next chapter, on time series analysis, starts our journey into specific analysis techniques. Conclusion | 59 CHAPTER 3 Time Series Analysis Now that I’ve covered SQL and databases and the key steps in preparing data for analysis, it’s time to turn to specific types of analysis that can be done with SQL. There are a seemingly unending number of data sets in the world, and correspond‐ ingly infinite ways in which they could be analyzed. In this and the following chap‐ ters, I have organized types of analysis into themes that I hope will be helpful as you build your analysis and SQL skills. Many of the techniques to be discussed build on those shown in Chapter 2 and then on the preceding chapters as the book progresses. Time series of data are so prevalent and so important that I’ll start the series of analy‐ sis themes here. Time series analysis is one of the most common types of analysis done with SQL. A time series is a sequence of measurements or data points recorded in time order, often at regularly spaced intervals. There are many examples of time series data in daily life, such as the daily high temperature, the closing value of the S&P 500 stock index, or the number of daily steps recorded by your fitness tracker. Time series analysis is used in a wide variety of industries and disciplines, from statistics and engineering to weather forecasting and business planning. Time series analysis is a way to under‐ stand and quantify how things change over time. Forecasting is a common goal of time series analysis. Since time only marches for‐ ward, future values can be expressed as", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 58 + }, + { + "text": "industries and disciplines, from statistics and engineering to weather forecasting and business planning. Time series analysis is a way to under‐ stand and quantify how things change over time. Forecasting is a common goal of time series analysis. Since time only marches for‐ ward, future values can be expressed as a function of past values, while the reverse is not true. However, it’s important to note that the past doesn’t perfectly predict the future. Any number of changes to wider market conditions, popular trends, product introductions, or other large changes make forecasting difficult. Still, looking at his‐ torical data can lead to insights, and developing a range of plausible outcomes is use‐ ful for planning. As I’m writing this, the world is in the midst of a global COVID-19 pandemic, the likes of which haven’t been seen in 100 years—predating all but the most long-lived organizations’ histories. Thus many current organizations haven’t 61 seen this specific event before, but they have existed through other economic crises, such as those following the dot-com burst and the 9/11 attacks in 2001, as well as the global financial crisis of 2007–2008. With careful analysis and understanding of con‐ text, we can often extract useful insights. In this chapter, we’ll first cover the SQL building blocks of time series analysis: syntax and functions for working with dates, timestamps, and time. Next, I’ll introduce the retail sales data set used for examples throughout the rest of the chapter. A discussion of methods for trending analysis follows, and then I’ll cover calculating rolling time windows. Next are period-over-period calculations to analyze data with seasonality components. Finally, we’ll wrap up with some additional techniques that are useful for time series analysis. Date, Datetime, and Time Manipulations Dates and times come in a wide variety of formats, depending on the data source. We often need or want to transform the raw data format for our output, or to perform calculations to arrive at new dates or parts of dates. For example, the data set might contain transaction timestamps, but the goal of the analysis is to trend monthly sales. At other times, we might want to know how many days or months have elapsed since a particular event. Fortunately, SQL has powerful functions and formatting capabili‐ ties that can transform just about any raw input to almost any output we might need for analysis. In this section, I’ll show you how to convert between time zones, and then I’ll go into depth on formatting dates and datetimes. Next, I’ll explore date math and time manipulations, including those that make use of intervals. An interval is a data type that holds a span of time, such as a number of months, days, or hours. Although data can be stored in a database table as an interval type, in practice I rarely see this done, so I will talk about intervals alongside the date and time functions that you can use them with. Last, I’ll discuss some special considerations when", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 59 + }, + { + "text": "of months, days, or hours. Although data can be stored in a database table as an interval type, in practice I rarely see this done, so I will talk about intervals alongside the date and time functions that you can use them with. Last, I’ll discuss some special considerations when joining or otherwise combining data from different sources. Time Zone Conversions Understanding the standard time zone used in a data set can prevent misunderstand‐ ings and mistakes further into the analysis process. Time zones split the world into north-south regions that observe the same time. Time zones allow different parts of the world to have similar clock times for daytime and nighttime—so, for example, the sun is overhead at 12 p.m. wherever you are in the world. The zones follow irregular boundaries that are as much political as geographic ones. Most are one hour apart, but some are offset only 30 or 45 minutes, and so there are more than 30 time zones spanning the globe. Many countries that are distant from the equator observe day‐ light savings time for parts of the year as well, but there are exceptions, such as in the 62 | Chapter 3: Time Series Analysis United States and Australia, where some states observe daylight savings time and oth‐ ers do not. Each time zone has a standard abbreviation, such as PST for Pacific Stan‐ dard Time and PDT for Pacific Daylight Time. Many databases are set to Coordinated Universal Time (UTC), the global standard used to regulate clocks, and record events in this time zone. It replaced Greenwich Mean Time (GMT), which you might still see if your data comes from an older data‐ base. UTC does not have daylight savings time, so it stays consistent all year long. This turns out to be quite useful for analysis. I remember one time a panicked prod‐ uct manager asked me to figure out why sales on a particular Sunday dropped so much compared to the prior Sunday. I spent hours writing queries and investigating possible causes before eventually figuring out that our data was recorded in Pacific Time (PT). Daylight savings started early Sunday morning, the database clock moved ahead 1 hour, and the day had only 23 hours instead of 24, and thus sales appeared to drop. Half a year later we had a corresponding 25-hour day, when sales appeared unusually high. Often timestamps in the database are not encoded with the time zone, and you will need to consult with the source or developer to figure out how your data was stored. UTC has become most com‐ mon in the data sets I see, but that is certainly not universal. One drawback to UTC, or really to any logging of machine time, is that we lose infor‐ mation about the local time for the human doing the actions that generated the event recorded in the database. I might want to know whether people tend to use my mobile app more during the workday", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 60 + }, + { + "text": "UTC, or really to any logging of machine time, is that we lose infor‐ mation about the local time for the human doing the actions that generated the event recorded in the database. I might want to know whether people tend to use my mobile app more during the workday or during nights and weekends. If my audience is clustered in one time zone, it’s not hard to figure this out. But if the audience spans multiple time zones or is international, then it becomes a calculation task of convert‐ ing each recorded time to its local time zone. All local time zones have a UTC offset. For example, the offset for PDT is UTC – 7 hours, while the offset for PST is UTC – 8 hours. Timestamps in databases are stored in the format YYYY-MM-DD hh:mi:ss (for years-months-days hours:minutes:sec‐ onds). Timestamps with the time zone have an additional piece of information for the UTC offset, expressed as a positive or negative number. Converting from one time zone to another can be accomplished with at time zone followed by the destination time zone’s abbreviation. For example, we can convert a timestamp in UTC (offset – 0) to PST: SELECT '2020-09-01 00:00:00 -0' at time zone 'pst'; timezone ------------------- 2020-08-31 16:00:00 Date, Datetime, and Time Manipulations | 63 The destination time zone name can be a constant, or a database field, allowing this conversion to be dynamic to the data set. Some databases have a convert_timezone or convert_tz function that works similarly. One argument is the time zone of the result, and the other argument is the time zone from which to convert: SELECT convert_timezone('pst','2020-09-01 00:00:00 -0'); timezone ------------------- 2020-08-31 16:00:00 Check your database’s documentation for the exact name and ordering of the target time zone and the source timestamp arguments. Many databases contain a list of time zones and their abbreviations in a system table. Some common ones are seen in Table 3-1. These can be queried with SELECT * FROM the table name. Wikipedia also has a useful list of standard time zone abbreviations and their UTC offsets. Table 3-1. Time zone information system tables in common databases Postgres pg_timezone_names MySQL mysql.time_zone_names SQL Server sys.time_zone_info Redshift pg_timezone_names Time zones are an innate part of working with timestamps. With time zone conver‐ sion functions, moving between the time zone in which the data was recorded and any other world time zone is possible. Next, I’ll show you a variety of techniques for manipulating dates and timestamps with SQL. Date and Timestamp Format Conversions Dates and timestamps are key to time series analysis. Due to the wide variety of ways in which dates and times can be represented in source data, it is almost inevitable that you will need to convert date formats at some point. In this section, I’ll cover several of the most common conversions and how to accomplish them with SQL: changing the data type, extracting parts of a date or timestamp, and creating a date or", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 61 + }, + { + "text": "data, it is almost inevitable that you will need to convert date formats at some point. In this section, I’ll cover several of the most common conversions and how to accomplish them with SQL: changing the data type, extracting parts of a date or timestamp, and creating a date or time‐ stamp from parts. I’ll begin by introducing some handy functions that return the current date and/or time. Returning the current date or time is a common analysis task—for example, to include a timestamp for the result set or to use in date math, covered in the next sec‐ tion. The current date and time are referred to as system time, and while returning them is easy to do with SQL, there are some syntax differences between databases. 64 | Chapter 3: Time Series Analysis To return the current date, some databases have a current_date function, with no parentheses: SELECT current_date; There is a wider variety of functions to return the current date and time. Check your database’s documentation or just experiment by typing into a SQL window to see whether a function returns a value or an error. The functions with parentheses do not take arguments, but it is important to include the parentheses: current_timestamp localtimestamp get_date() now() Finally, there are functions to return only the timestamp portion of the current system time. Again, consult documentation or experiment to figure out which func‐ tion(s) to use with your database: current_time localtime timeofday() SQL has a number of functions for changing the format of dates and times. To reduce the granularity of a timestamp, use the date_trunc function. The first argument is a text value indicating the time period level to which to truncate the timestamp in the second argument. The result is a timestamp value: date_trunc (text, timestamp) SELECT date_trunc('month','2020-10-04 12:33:35'::timestamp); date_trunc ------------------- 2020-10-01 00:00:00 Standard arguments that can be used are listed in Table 3-2. They range all the way from microseconds to millennia, providing plenty of flexibility. Databases that don’t support date_trunc, such as MySQL, have an alternate function called date_format that can be used in a similar way: SELECT date_format('2020-10-04 12:33:35','%Y-%m-01') as date_trunc; date_trunc ------------------- 2020-10-01 00:00:00 Date, Datetime, and Time Manipulations | 65 Table 3-2. Standard time period arguments Time period arguments microsecond millisecond second minute hour day week month quarter year decade century millennium Rather than returning dates or timestamps, sometimes our analysis calls for parts of dates or times. For example, we might want to group sales by month, day of the week, or hour of the day. SQL provides a few functions for returning just the part of the date or timestamp required. Dates and timestamps are usually interchangeable, except when the request is to return a time part. In those cases, time is of course required. The date_part function takes a text value for the part to be returned and a date or timestamp value. The returned value is a FLOAT, which is a numeric value with a decimal part; depending", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 62 + }, + { + "text": "request is to return a time part. In those cases, time is of course required. The date_part function takes a text value for the part to be returned and a date or timestamp value. The returned value is a FLOAT, which is a numeric value with a decimal part; depending on your needs, you may want to cast the value to an integer data type: SELECT date_part('day',current_timestamp); SELECT date_part('month',current_timestamp); SELECT date_part('hour',current_timestamp); Another function that works similarly is extract, which takes a part name and a date or timestamp value and returns a FLOAT value: SELECT extract('day' from current_timestamp); date_part --------- 27.0 SELECT extract('month' from current_timestamp); 66 | Chapter 3: Time Series Analysis date_part --------- 5.0 SELECT extract('hour' from current_timestamp); date_part --------- 14.0 The functions date_part and extract can be used with intervals, but note that the requested part must match the units of the interval. So, for example, requesting days from an interval stated in days returns the expected value of 30: SELECT date_part('day',interval '30 days'); SELECT extract('day' from interval '30 days'); date_part --------- 30.0 However, requesting days from an interval stated in months returns a value of 0.0: SELECT extract('day' from interval '3 months'); date_part --------- 0.0 A full list of date parts can be found in your database’s documenta‐ tion or by searching online, but some of the most common are “day,” “month,” and “year” for dates, and “second,” “minute,” and “hour” for timestamps. To return text values of the date parts, use the to_char function, which takes the input value and the output format as arguments: SELECT to_char(current_timestamp,'Day'); SELECT to_char(current_timestamp,'Month'); If you ever encounter timestamps stored as Unix epochs (the num‐ ber of seconds that have elapsed since January 1, 1970, at 00:00:00 UTC), you can convert them to timestamps using the to_time stamp function. Sometimes analysis calls for creating a date from parts from different sources. This can occur when the year, month, and day values are stored in different columns in the Date, Datetime, and Time Manipulations | 67 database. It can also be necessary when the parts have been parsed out of text, a topic I’ll cover in more depth in Chapter 5. A simple way to create a timestamp from separate date and time components is to concatenate them together with a plus sign (+): SELECT date '2020-09-01' + time '03:00:00' as timestamp; timestamp ------------------- 2020-09-01 03:00:00 A date can be assembled using the make_date, makedate, date_from_parts, or date fromparts function. These are equivalent, but different databases name the functions differently. The function takes arguments for the year, month, and day parts and returns a value with a date format: SELECT make_date(2020,09,01); make_date ---------- 2020-09-01 The arguments can be constants or reference field names and must be integers. Yet another way to assemble a date or timestamp is to concatenate the values together and then cast the result to a date format using one of the casting syntaxes or the to_date function: SELECT to_date(concat(2020,'-',09,'-',01), 'yyyy-mm-dd'); to_date ---------- 2020-09-01 SELECT cast(concat(2020,'-',09,'-',01) as date); to_date", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 63 + }, + { + "text": "names and must be integers. Yet another way to assemble a date or timestamp is to concatenate the values together and then cast the result to a date format using one of the casting syntaxes or the to_date function: SELECT to_date(concat(2020,'-',09,'-',01), 'yyyy-mm-dd'); to_date ---------- 2020-09-01 SELECT cast(concat(2020,'-',09,'-',01) as date); to_date ---------- 2020-09-01 SQL has a number of ways to format and convert dates and timestamps and retrieve system dates and times. In the next section, I will start putting them to use in date math. Date Math SQL allows us to do various mathematical operations on dates. This might be surpris‐ ing since, strictly speaking, dates are not numeric data types, but the concept should be familiar if you’ve ever tried to figure out what day it will be four weeks from now. Date math is useful for a variety of analytics tasks. For example, we can use it to find 68 | Chapter 3: Time Series Analysis the age or tenure of a customer, how much time elapsed between two events, and how many things occurred within a window of time. Date math involves two types of data: the dates themselves and intervals. We need the concept of intervals because date and time components don’t behave like integers. One-tenth of 100 is 10; one-tenth of a year is 36.5 days. Half of 100 is 50; half of a day is 12 hours. Intervals allow us to move smoothly between units of time. Intervals come in two types: year-month intervals and day-time ones. We’ll start with a few operations that return integer values and then move on to functions that work with or return intervals. First, let’s find the days elapsed between two dates. There are several ways to do this in SQL. The first way is by using a mathematical operator, the minus sign (–): SELECT date('2020-06-30') - date('2020-05-31') as days; days ---- 30 This returns the number of days between these two dates. Note that the answer is 30 days and not 31. The number of days is inclusive of only one of the endpoints. Sub‐ tracting the dates in the reverse also works and returns an interval of –30 days: SELECT date('2020-05-31') - date('2020-06-30') as days; days ---- -30 Finding the difference between two dates can also be accomplished with the datediff function. Postgres does not support it, but many other popular databases do, includ‐ ing SQL Server, Redshift, and Snowflake, and it’s quite handy, particularly when the goal is to return an interval other than the number of days. The function takes three arguments—the time period units you want to return, a starting timestamp or date, and an ending timestamp or date: datediff(interval_name, start_timestamp, end_timestamp) So our previous example would look like this: SELECT datediff('day',date('2020-05-31'), date('2020-06-30')) as days; days ---- 30 We can also find the number of months between two dates, and the database will do the correct math even though month lengths differ throughout the year: Date, Datetime, and Time Manipulations | 69 SELECT", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 64 + }, + { + "text": "our previous example would look like this: SELECT datediff('day',date('2020-05-31'), date('2020-06-30')) as days; days ---- 30 We can also find the number of months between two dates, and the database will do the correct math even though month lengths differ throughout the year: Date, Datetime, and Time Manipulations | 69 SELECT datediff('month' ,date('2020-01-01') ,date('2020-06-30') ) as months; months ------ 5 In Postgres, this can be accomplished using the age function, which calculates the interval between two dates: SELECT age(date('2020-06-30'),date('2020-01-01')); age -------------- 5 mons 29 days We can then find the number of months component of the interval with the date_part() function: SELECT date_part('month',age('2020-06-30','2020-01-01')) as months; months ------ 5.0 Subtracting dates to find the time elapsed between them is quite powerful. Adding dates does not work in the same way. To do addition with dates, we need to leverage intervals or special functions. For example, we can add seven days to a date by adding the interval '7 days': SELECT date('2020-06-01') + interval '7 days' as new_date; new_date ------------------- 2020-06-08 00:00:00 Some databases don’t require the interval syntax and instead automatically convert the provided number to days, although it’s generally good practice to use the interval notation, both for cross-database compatibility and to make your code easier to read: SELECT date('2020-06-01') + 7 as new_date; new_date ------------------- 2020-06-08 00:00:00 If you want to add a different unit of time, use the interval notation with months, years, hours, or another date or time period. Note that this can also be used to sub‐ tract intervals from dates by using a “-” instead of a “+.” Many but not all databases 70 | Chapter 3: Time Series Analysis have a date_add or dateadd function that takes the desired interval, a value, and the starting date and does the math: SELECT date_add('month',1,'2020-06-01') as new_date; new_date ---------- 2020-07-01 Consult your database’s documentation, or just experiment with queries, to figure out the syntax and functions that are available and appropriate for your project. Any of these formulations can be used in the WHERE clause in addition to the SELECT clause. For example, we can filter to records that occurred at least three months ago: WHERE event_date < current_date - interval '3 months' They can also be used in JOIN conditions, but note that database performance will usually be slower when the JOIN condition contains a calculation rather than an equality or inequality between dates. Using date math is common in analysis with SQL, both to find the time elapsed between dates or timestamps and to calculate new dates based on an interval from a known date. There are several ways to find the elapsed time between two dates, add intervals to dates, and subtract intervals from dates. Next, we’ll turn to time manipu‐ lations, which are similar. Time Math Time math is less common in many areas of analysis, but it can be useful in some situations. For example, we might want to know how long it takes for a support rep‐ resentative to answer a phone", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 65 + }, + { + "text": "Next, we’ll turn to time manipu‐ lations, which are similar. Time Math Time math is less common in many areas of analysis, but it can be useful in some situations. For example, we might want to know how long it takes for a support rep‐ resentative to answer a phone call in a call center or respond to an email requesting assistance. Whenever the elapsed time between two events is less than a day, or when rounding the result to a number of days doesn’t provide enough information, time manipulation comes into play. Time math works similarly to date math, by leveraging intervals. We can add time intervals to times: SELECT time '05:00' + interval '3 hours' as new_time; new_time -------- 08:00:00 Date, Datetime, and Time Manipulations | 71 We can subtract intervals from times: SELECT time '05:00' - interval '3 hours' as new_time; new_time -------- 02:00:00 We can also subtract times, resulting in an interval: SELECT time '05:00' - time '03:00' as time_diff; time_diff --------- 02:00:00 Times, unlike dates, can be multiplied: SELECT time '05:00' * 2 as time_multiplied; time_multiplied --------------- 10:00:00 Intervals can also be multiplied, resulting in a time value: SELECT interval '1 second' * 2000 as interval_multiplied; interval_multiplied ------------------- 00:33:20 SELECT interval '1 day' * 45 as interval_multiplied; interval_multiplied ------------------- 45 days These examples use constant values, but you can include database field names or cal‐ culations in the SQL query as well to make the calculations dynamic. Next, I’ll discuss special date considerations to keep in mind when combining data sets from different source systems. Joining Data from Different Sources Combining data from different sources is one of the most compelling use cases for a data warehouse. However, different source systems can record dates and times in dif‐ ferent formats or different time zones or even just be off slightly due to issues with the internal clock time of the server. Even tables from the same data source can have dif‐ ferences, though this is less common. Reconciling and standardizing dates and time‐ stamps is an important step before moving further in the analysis. 72 | Chapter 3: Time Series Analysis Dates and timestamps that are in different formats can be standardized with SQL. JOINing on dates or including date fields in UNIONs generally requires that the dates or timestamps be in the same format. Earlier in the chapter, I showed techniques for formatting dates and timestamps that will serve well with these problems. Take care with time zones when combining data from different sources. For example, an inter‐ nal database may use UTC time, but data from a third party could be in a local time zone. I have seen data sourced from software as a service (SaaS) that was recorded in a variety of local times. Note that the timestamp values themselves won’t necessarily have the time zone embedded. You may need to consult the vendor’s documentation and convert the data to UTC if the rest of your data is stored that way.", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 66 + }, + { + "text": "as a service (SaaS) that was recorded in a variety of local times. Note that the timestamp values themselves won’t necessarily have the time zone embedded. You may need to consult the vendor’s documentation and convert the data to UTC if the rest of your data is stored that way. Another option is to store the time zone in a field so that the timestamp value can be con‐ verted as needed. Another thing to look out for when working with data from different sources is time‐ stamps that are slightly out of sync. This can happen when timestamps are recorded from client devices—for example, from a laptop or mobile phone in one data source and a server in the other data source. I once saw a series of experiment results be mis‐ calculated because the client mobile device that recorded a user’s action was offset by a few minutes from the server that recorded the treatment group to which the user was assigned. Data from the mobile clients appeared to arrive before the treatment group timestamp, so some events were inadvertently excluded. A fix for something like this is relatively straightforward: rather than filter for action timestamps greater than the treatment group timestamp, allow events within a short interval or window of time prior to the treatment timestamp to be included in the results. This can be accomplished with a BETWEEN clause and date math, as seen in the last section. When working with data from mobile apps, pay particular attention to whether the timestamps represent when the action happened on the device or when the event arrived in the database. The difference can range from negligible all the way up to days, depending on whether the mobile app allows offline usage and on how it han‐ dles sending data during periods of low signal strength. Data from mobile apps can be late-arriving or may make its way into the database days after it occurred on the device. Dates and timestamps can also become corrupted en route, and you may see ones that are impossibly distant in the past or future as a result. Now that I’ve shown how to manipulate dates, datetimes, and time by changing the formats, converting time zones, performing date math, and working across data sets from different sources, we’re ready to get into some time series examples. First, I’ll introduce the data set for examples in the rest of the chapter. Date, Datetime, and Time Manipulations | 73 The Retail Sales Data Set The examples in the rest of this chapter use a data set of monthly US retail sales from the Monthly Retail Trade Report: Retail and Food Services Sales: Excel (1992– present), available on the Census.gov website. The data in this report is used as an economic indicator to understand trends in US consumer spending patterns. While gross domestic product (GDP) figures are published quarterly, this retail sales data is published monthly, so it is also used to help predict GDP. For", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 67 + }, + { + "text": "available on the Census.gov website. The data in this report is used as an economic indicator to understand trends in US consumer spending patterns. While gross domestic product (GDP) figures are published quarterly, this retail sales data is published monthly, so it is also used to help predict GDP. For both of these reasons, the latest figures are usually covered in the business press when they are released. The data spans from 1992 to 2020 and includes both total sales as well as details for subcategories of retail sales. It contains both unadjusted and seasonally adjusted numbers. This chapter will use the unadjusted numbers, since one of the goals is ana‐ lyzing seasonality. Sales figures are in millions of US dollars. The original file format is an Excel file, with a tab for each year and with months as columns. The GitHub site for this book has the data in a format that’s easier to import into a database, along with code specifically for importing into Postgres. Figure 3-1 shows a sample of the retail_sales table. Figure 3-1. Preview of the US retail sales data set 74 | Chapter 3: Time Series Analysis Trending the Data With time series data, we often want to look for trends in the data. A trend is simply the direction in which the data is moving. It may be moving up or increasing over time, or it may be moving down or decreasing over time. It can remain more or less flat, or there could be so much noise, or movement up and down, that it’s hard to determine a trend at all. This section will cover several techniques for trending time series data, from simple trends for graphing to comparing components of a trend, using percent of total calculations to compare parts to the whole, and finally indexing to see the percent change from a reference time period. Simple Trends Creating a trend may be a step in profiling and understanding data, or it may be the final output. The result set is a series of dates or timestamps and a numerical value. When graphing a time series, the dates or timestamps will become the x-axis, and the numerical value will be the y-axis. For example, we can check the trend of total retail and food services sales in the US: SELECT sales_month ,sales FROM retail_sales WHERE kind_of_business = 'Retail and food services sales, total' ; sales_month sales ----------- ------ 1992-01-01 146376 1992-02-01 147079 1992-03-01 159336 ... ... The results are graphed in Figure 3-2. Trending the Data | 75 Figure 3-2. Trend of monthly retail and food services sales This data clearly has some patterns, but it also has some noise. Transforming the data and aggregating at the yearly level can help us gain a better understanding. First, we’ll use the date_part function to return just the year from the sales_month field and then sum the sales. The results are filtered to the “Retail and food services sales, total” kind_of_business in the", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 68 + }, + { + "text": "the data and aggregating at the yearly level can help us gain a better understanding. First, we’ll use the date_part function to return just the year from the sales_month field and then sum the sales. The results are filtered to the “Retail and food services sales, total” kind_of_business in the WHERE clause: SELECT date_part('year',sales_month) as sales_year ,sum(sales) as sales FROM retail_sales WHERE kind_of_business = 'Retail and food services sales, total' GROUP BY 1 ; sales_year sales ---------- ------- 1992.0 2014102 1993.0 2153095 1994.0 2330235 ... ... After graphing this data, as in Figure 3-3, we now have a smoother time series that is generally increasing over time, as might be expected, since the sales values are not adjusted for inflation. Sales for all retail and food services fell in 2009, during the 76 | Chapter 3: Time Series Analysis global financial crisis. After growing every year throughout the 2010s, sales were flat in 2020 compared to 2019, due to the impact of the COVID-19 pandemic. Figure 3-3. Trend of yearly total retail and food services sales Graphing time series data at different levels of aggregation, such as weekly, monthly, or yearly, is a good way to understand trends. This step can be used to simply profile the data, but it can also be the final output, depending on the goals of the analysis. Next, we’ll turn to using SQL to compare components of a time series. Comparing Components Often data sets contain not just a single time series but multiple slices or components of a total across the same time range. Comparing these slices often reveals interesting patterns. In the retail sales data set, there are values for total sales but also a number of subcategories. Let’s compare the yearly sales trend for a few categories that are associated with leisure activities: book stores, sporting goods stores, and hobby stores. This query adds kind_of_business in the SELECT clause and, since it is another attribute rather than an aggregation, adds it to the GROUP BY clause as well: SELECT date_part('year',sales_month) as sales_year ,kind_of_business ,sum(sales) as sales FROM retail_sales Trending the Data | 77 WHERE kind_of_business in ('Book stores' ,'Sporting goods stores','Hobby, toy, and game stores') GROUP BY 1,2 ; sales_year kind_of_business sales ---------- --------------------------- ----- 1992.0 Book stores 8327 1992.0 Hobby, toy, and game stores 11251 1992.0 Sporting goods stores 15583 ... ... ... The results are graphed in Figure 3-4. Sales at sporting goods retailers started the highest among the three categories and grew much faster during the time period, and by the end of the time series, those sales were substantially higher. Sales at sporting goods stores started declining in 2017 but had a big rebound in 2020. Sales at hobby, toy, and game stores were relatively flat over this time span, with a slight dip in the mid-2000s and another slight decline prior to a rebound in 2020. Sales at book stores grew until the mid-2000s and have been on the decline since then. All of these cate‐", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 69 + }, + { + "text": "at hobby, toy, and game stores were relatively flat over this time span, with a slight dip in the mid-2000s and another slight decline prior to a rebound in 2020. Sales at book stores grew until the mid-2000s and have been on the decline since then. All of these cate‐ gories have been impacted by the growth of online retailers, but the timing and mag‐ nitude seem to differ. Figure 3-4. Trend of yearly retail sales for sporting goods stores; hobby, toy, and game stores; and book stores 78 | Chapter 3: Time Series Analysis In addition to looking at simple trends, we might want to perform more complex comparisons between parts of the time series. For the next few examples, we’ll look at sales at women’s clothing stores and at men’s clothing stores. Note that since the names contain apostrophes, the character otherwise used to indicate the beginning and end of strings, we need to escape them with an extra apostrophe. This lets the database know that the apostrophe is part of the string rather than the end. Although we might consider adding a step in a data-loading pipeline that removes extra apos‐ trophes in names, I’ve left them in here as a demonstration of the types of code adjustments that are often needed in the real world. First, we’ll trend the data for each type of store by month: SELECT sales_month ,kind_of_business ,sales FROM retail_sales WHERE kind_of_business in ('Men''s clothing stores' ,'Women''s clothing stores') ; sales_month kind_of_business sales ----------- ----------------------- ----- 1992-01-01 Men's clothing stores 701 1992-01-01 Women's clothing stores 1873 1992-02-01 Women's clothing stores 1991 ... ... ... The results are graphed in Figure 3-5. Sales at women’s clothing retailers are much higher than those at men’s clothing retailers. Both types of stores exhibit seasonality, a topic I’ll cover in depth in “Analyzing with Seasonality” on page 107. Both experi‐ enced significant drops in 2020 due to store closures and a reduction in shopping because of the COVID-19 pandemic. Trending the Data | 79 Figure 3-5. Monthly trend of sales at women’s and men’s clothing stores The monthly data has intriguing patterns but is noisy, so we’ll use yearly aggregates for the next few examples. We’ve seen this query format previously when rolling up total sales and sales for leisure categories: SELECT date_part('year',sales_month) as sales_year ,kind_of_business ,sum(sales) as sales FROM retail_sales WHERE kind_of_business in ('Men''s clothing stores' ,'Women''s clothing stores') GROUP BY 1,2 ; Are sales at women’s clothing stores uniformly higher than those at men’s clothing stores? In the yearly trend shown in Figure 3-6, the gap between men’s and women’s sales does not appear constant but rather was increasing during the early to mid-2000s. Women’s clothing sales in particular dipped during the global financial crisis of 2008–2009, and sales in both categories dropped a lot during the pandemic in 2020. 80 | Chapter 3: Time Series Analysis Figure 3-6. Yearly trend of sales at women’s and men’s clothing stores We don’t need to rely on", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 70 + }, + { + "text": "clothing sales in particular dipped during the global financial crisis of 2008–2009, and sales in both categories dropped a lot during the pandemic in 2020. 80 | Chapter 3: Time Series Analysis Figure 3-6. Yearly trend of sales at women’s and men’s clothing stores We don’t need to rely on visual estimation, however. For more precision on this gap, we can calculate the gap between the two categories, the ratio, and the percent differ‐ ence between them. To do this, the first step is to arrange the data so that there is a single row for each month, with a column for each category. Pivoting the data with aggregate functions combined with CASE statements accomplishes this: SELECT date_part('year',sales_month) as sales_year ,sum(case when kind_of_business = 'Women''s clothing stores' then sales end) as womens_sales ,sum(case when kind_of_business = 'Men''s clothing stores' then sales end) as mens_sales FROM retail_sales WHERE kind_of_business in ('Men''s clothing stores' ,'Women''s clothing stores') GROUP BY 1 ; sales_year womens_sales mens_sales ---------- ------------ ---------- 1992.0 31815 10179 1993.0 32350 9962 1994.0 30585 10032 ... ... ... Trending the Data | 81 1 October and November 2020 data points were suppressed by the publisher of the data, due to concerns about the data quality. Collecting the data likely became more difficult due to store closures during the 2020 pan‐ demic. With this building block calculation, we can find the difference, ratio, and percent difference between time series in the data set. The difference can be calculated by subtracting one value from the other using the mathematical “–” operator. Depend‐ ing on the goals of the analysis, either finding the difference from men’s sales or find‐ ing the difference from women’s sales might be appropriate. Both are shown here and are equivalent except for the sign: SELECT sales_year ,womens_sales - mens_sales as womens_minus_mens ,mens_sales - womens_sales as mens_minus_womens FROM ( SELECT date_part('year',sales_month) as sales_year ,sum(case when kind_of_business = 'Women''s clothing stores' then sales end) as womens_sales ,sum(case when kind_of_business = 'Men''s clothing stores' then sales end) as mens_sales FROM retail_sales WHERE kind_of_business in ('Men''s clothing stores' ,'Women''s clothing stores') and sales_month <= '2019-12-01' GROUP BY 1 ) a ; sales_year womens_minus_mens mens_minus_womens ---------- ----------------- ----------------- 1992.0 21636 -21636 1993.0 22388 -22388 1994.0 20553 -20553 ... ... ... The subquery is not required from a query execution standpoint, since aggregations can be added to or subtracted from each other. A subquery is often more legible but does add more lines to the code. Depending on how long or complex the rest of your SQL query is, you might prefer to place the intermediate calculation in a subquery, or just calculate it in the main query. Here is an example without the subquery, subtract‐ ing men’s sales from women’s sales, with an added WHERE clause filter to remove 2020, since a few months have null values:1 SELECT date_part('year',sales_month) as sales_year ,sum(case when kind_of_business = 'Women''s clothing stores' then sales end) 82 | Chapter 3: Time Series Analysis - sum(case when kind_of_business = 'Men''s", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 71 + }, + { + "text": "subtract‐ ing men’s sales from women’s sales, with an added WHERE clause filter to remove 2020, since a few months have null values:1 SELECT date_part('year',sales_month) as sales_year ,sum(case when kind_of_business = 'Women''s clothing stores' then sales end) 82 | Chapter 3: Time Series Analysis - sum(case when kind_of_business = 'Men''s clothing stores' then sales end) as womens_minus_mens FROM retail_sales WHERE kind_of_business in ('Men''s clothing stores' ,'Women''s clothing stores') and sales_month <= '2019-12-01' GROUP BY 1 ; sales_year womens_minus_mens ---------- ----------------- 1992.0 21636 1993.0 22388 1994.0 20553 ... ... Figure 3-7 shows that the gap decreased between 1992 and about 1997, began a long increase through about 2011 (with a brief dip in 2007), and then was more or less flat through 2019. Figure 3-7. Yearly difference between sales at women’s and men’s clothing stores Trending the Data | 83 Let’s continue our investigation and look at the ratio of these categories. We’ll use men’s sales as the baseline or denominator, but note that we could just as easily use women’s store sales instead: SELECT sales_year ,womens_sales / mens_sales as womens_times_of_mens FROM ( SELECT date_part('year',sales_month) as sales_year ,sum(case when kind_of_business = 'Women''s clothing stores' then sales end) as womens_sales ,sum(case when kind_of_business = 'Men''s clothing stores' then sales end) as mens_sales FROM retail_sales WHERE kind_of_business in ('Men''s clothing stores' ,'Women''s clothing stores') and sales_month <= '2019-12-01' GROUP BY 1 ) a ; sales_year womens_times_of_mens ---------- -------------------- 1992.0 3.1255526083112290 1993.0 3.2473398915880345 1994.0 3.0487440191387560 ... ... SQL returns a lot of decimal digits when performing division. You should generally consider rounding the result before presenting the analysis. Use the level of precision (number of decimal places) that tells the story. Plotting the result, shown in Figure 3-8, reveals that the trend is similar to the differ‐ ence trend, but while there was a drop in the difference in 2009, the ratio actually increased. 84 | Chapter 3: Time Series Analysis Figure 3-8. Yearly ratio of women’s to men’s clothing sales Next, we can calculate the percent difference between sales at women’s and men’s clothing stores: SELECT sales_year ,(womens_sales / mens_sales - 1) * 100 as womens_pct_of_mens FROM ( SELECT date_part('year',sales_month) as sales_year ,sum(case when kind_of_business = 'Women''s clothing stores' then sales end) as womens_sales ,sum(case when kind_of_business = 'Men''s clothing stores' then sales end) as mens_sales FROM retail_sales WHERE kind_of_business in ('Men''s clothing stores' ,'Women''s clothing stores') and sales_month <= '2019-12-01' GROUP BY 1 ) a ; Trending the Data | 85 sales_year womens_pct_of_mens ---------- -------------------- 1992.0 212.5552608311229000 1993.0 224.7339891588034500 1994.0 204.8744019138756000 ... ... Although the units for this output are different from those in the previous example, the shape of this graph is the same as that of the ratio graph. The choice of which to use depends on your audience and the norms in your domain. All of these statements are accurate: in 2009, sales at women’s clothing stores were $28.7 billion higher than sales at men’s stores; in 2009, sales at women’s clothing stores were 4.9 times the sales at", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 72 + }, + { + "text": "choice of which to use depends on your audience and the norms in your domain. All of these statements are accurate: in 2009, sales at women’s clothing stores were $28.7 billion higher than sales at men’s stores; in 2009, sales at women’s clothing stores were 4.9 times the sales at men’s stores; in 2009, sales at women’s stores were 390% higher than sales at men’s stores. Which version to select depends on the story you want to tell with the analysis. The transformations we’ve seen in this section allow us to analyze time series by com‐ paring related parts. The next section will continue the theme of comparing time ser‐ ies by showing ways to analyze series that represent parts of a whole. Percent of Total Calculations When working with time series data that has multiple parts or attributes that consti‐ tute a whole, it’s often useful to analyze each part’s contribution to the whole and whether that has changed over time. Unless the data already contains a time series of the total values, we’ll need to calculate the overall total in order to calculate the per‐ cent of total for each row. This can be accomplished with a self-JOIN, or a window function, which as we saw in Chapter 2 is a special kind of SQL function that can ref‐ erence any row within a specified partition of the table. First I’ll show the self-JOIN method. A self-JOIN is any time a table is joined to itself. As long as each instance of the table in the query is given a different alias, the data‐ base will treat them all as distinct tables. For example, to find the percent of com‐ bined men’s and women’s clothing sales that each series represents, we can JOIN retail_sales, aliased as a, to retail_sales, aliased as b, on the sales_month field. We then SELECT the individual series name (kind_of_business) and sales values from alias a. Then, from alias b we sum the sales for both categories and call the result total_sales. Note that the JOIN between the tables on the sales_month field creates a partial Cartesian JOIN, which results in two rows from alias b for each row in alias a. Grouping by a.sales_month, a.kind_of_business, and a.sales and aggregating b.sales returns exactly the results needed, however. In the outer query, the percent of total for each row is calculated by dividing sales by total_sales: SELECT sales_month ,kind_of_business ,sales * 100 / total_sales as pct_total_sales 86 | Chapter 3: Time Series Analysis FROM ( SELECT a.sales_month, a.kind_of_business, a.sales ,sum(b.sales) as total_sales FROM retail_sales a JOIN retail_sales b on a.sales_month = b.sales_month and b.kind_of_business in ('Men''s clothing stores' ,'Women''s clothing stores') WHERE a.kind_of_business in ('Men''s clothing stores' ,'Women''s clothing stores') GROUP BY 1,2,3 ) aa ; sales_month kind_of_business pct_total_sales ----------- ----------------------- ------------------- 1992-01-01 Men's clothing stores 27.2338772338772339 1992-01-01 Women's clothing stores 72.7661227661227661 1992-02-01 Men's clothing stores 24.8395620989052473 ... ... ... The subquery isn’t required here, as the same result could be obtained without it, but", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 73 + }, + { + "text": "clothing stores' ,'Women''s clothing stores') GROUP BY 1,2,3 ) aa ; sales_month kind_of_business pct_total_sales ----------- ----------------------- ------------------- 1992-01-01 Men's clothing stores 27.2338772338772339 1992-01-01 Women's clothing stores 72.7661227661227661 1992-02-01 Men's clothing stores 24.8395620989052473 ... ... ... The subquery isn’t required here, as the same result could be obtained without it, but it makes the code a little easier to follow. A second way to calculate the percent of total sales for each category is to use the sum window function and PARTITION BY the sales_month. Recall that the PARTITION BY clause indicates the section of the table within which the function should calculate. The ORDER BY clause is not required in this sum window function, because the order of calculation doesn’t matter. Additionally, the query does not need a GROUP BY clause, because window functions look across multiple rows, but they do not reduce the number of rows in the result set: SELECT sales_month, kind_of_business, sales ,sum(sales) over (partition by sales_month) as total_sales ,sales * 100 / sum(sales) over (partition by sales_month) as pct_total FROM retail_sales WHERE kind_of_business in ('Men''s clothing stores' ,'Women''s clothing stores') ; sales_month kind_of_business sales total_sales pct_total ----------- ----------------------- ----- ----------- --------- 1992-01-01 Men's clothing stores 701 2574 27.233877 1992-01-01 Women's clothing stores 1873 2574 72.766122 1992-02-01 Women's clothing stores 1991 2649 75.160437 ... ... ... ... ... Graphing this data, as in Figure 3-9, reveals some interesting trends. First, starting in the late 1990s, women’s clothing store sales became an increasing percentage of the total. Second, early in the series a seasonal pattern is evident, where men’s sales spike as a percent of total sales in December and January. In the first decade of the 21st Trending the Data | 87 century, two seasonal peaks appear, in the summer and the winter, but by the late 2010s, the seasonal patterns are dampened almost to the point of randomness. We’ll take a look at analyzing seasonality in greater depth later in this chapter. Figure 3-9. Men’s and women’s clothing store sales as percent of monthly total Another percent of total we might want to find is the percent of sales within a longer time period, such as the percent of yearly sales each month represents. Again, either a self-JOIN or a window function will do the job. In this example, we’ll use a self-JOIN in the subquery: SELECT sales_month ,kind_of_business ,sales * 100 / yearly_sales as pct_yearly FROM ( SELECT a.sales_month, a.kind_of_business, a.sales ,sum(b.sales) as yearly_sales FROM retail_sales a JOIN retail_sales b on date_part('year',a.sales_month) = date_part('year',b.sales_month) and a.kind_of_business = b.kind_of_business and b.kind_of_business in ('Men''s clothing stores' 88 | Chapter 3: Time Series Analysis ,'Women''s clothing stores') WHERE a.kind_of_business in ('Men''s clothing stores' ,'Women''s clothing stores') GROUP BY 1,2,3 ) aa ; sales_month kind_of_business pct_yearly ----------- --------------------- ------------------ 1992-01-01 Men's clothing stores 6.8867275763827488 1992-02-01 Men's clothing stores 6.4642892229099126 1992-03-01 Men's clothing stores 7.1814520090382159 ... ... ... Alternatively, the window function method can be used: SELECT sales_month, kind_of_business, sales ,sum(sales) over (partition by date_part('year',sales_month) ,kind_of_business ) as yearly_sales ,sales", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 74 + }, + { + "text": "1,2,3 ) aa ; sales_month kind_of_business pct_yearly ----------- --------------------- ------------------ 1992-01-01 Men's clothing stores 6.8867275763827488 1992-02-01 Men's clothing stores 6.4642892229099126 1992-03-01 Men's clothing stores 7.1814520090382159 ... ... ... Alternatively, the window function method can be used: SELECT sales_month, kind_of_business, sales ,sum(sales) over (partition by date_part('year',sales_month) ,kind_of_business ) as yearly_sales ,sales * 100 / sum(sales) over (partition by date_part('year',sales_month) ,kind_of_business ) as pct_yearly FROM retail_sales WHERE kind_of_business in ('Men''s clothing stores' ,'Women''s clothing stores') ; sales_month kind_of_business pct_yearly ----------- --------------------- ------------------ 1992-01-01 Men's clothing stores 6.8867275763827488 1992-02-01 Men's clothing stores 6.4642892229099126 1992-03-01 Men's clothing stores 7.1814520090382159 ... ... ... The results, zoomed in to 2019, are shown in Figure 3-10. The two time series track fairly closely, but men’s stores had a greater percentage of their sales in January than did women’s stores. Men’s stores had a summer dip in July, while the corresponding dip in women’s store sales wasn’t until September. Trending the Data | 89 Figure 3-10. Percent of yearly sales for 2019 for women’s and men’s clothing sales Now that I’ve shown how to use SQL for percent of total calculations and the types of analysis that can be accomplished, I’ll turn to indexing and calculating percent change over time. Indexing to See Percent Change over Time The values in time series usually fluctuate over time. Sales increase with growing pop‐ ularity and availability of a product, while web page response time decreases with engineers’ efforts to optimize code. Indexing data is a way to understand the changes in a time series relative to a base period (starting point). Indices are widely used in economics as well as business settings. One of the most famous indices is the Con‐ sumer Price Index (CPI), which tracks the change in the prices of items that a typical consumer purchases and is used to track inflation, to decide salary increases, and for many other applications. The CPI is a complex statistical measure using various weights and data inputs, but the basic premise is straightforward. Pick a base period and compute the percent change in value from that base period for each subsequent period. 90 | Chapter 3: Time Series Analysis Indexing time series data with SQL can be done with a combination of aggregations and window functions, or self-JOINs. As an example, we index women’s clothing store sales to the first year in the series, 1992. The first step is to aggregate the sales by sales_year in a subquery, as we’ve done previously. In the outer query, the first_value window function finds the value associated with the first row in the PARTITION BY clause, according to the sort in the ORDER BY clause. In this exam‐ ple, we can omit the PARTITION BY clause, because we want to return the sales value for the first row in the entire data set returned by the subquery: SELECT sales_year, sales ,first_value(sales) over (order by sales_year) as index_sales FROM ( SELECT date_part('year',sales_month) as sales_year ,sum(sales) as sales FROM retail_sales WHERE kind_of_business = 'Women''s clothing", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 75 + }, + { + "text": "PARTITION BY clause, because we want to return the sales value for the first row in the entire data set returned by the subquery: SELECT sales_year, sales ,first_value(sales) over (order by sales_year) as index_sales FROM ( SELECT date_part('year',sales_month) as sales_year ,sum(sales) as sales FROM retail_sales WHERE kind_of_business = 'Women''s clothing stores' GROUP BY 1 ) a ; sales_year sales index_sales ---------- ----- ----------- 1992.0 31815 31815 1993.0 32350 31815 1994.0 30585 31815 ... ... ... With this sample of data, we can visually verify that the index value is correctly set at the value for 1992. Next, find the percent change from this base year for each row: SELECT sales_year, sales ,(sales / first_value(sales) over (order by sales_year) - 1) * 100 as pct_from_index FROM ( SELECT date_part('year',sales_month) as sales_year ,sum(sales) as sales FROM retail_sales WHERE kind_of_business = 'Women''s clothing stores' GROUP BY 1 ) a ; sales_year sales pct_from_index ---------- ----- -------------- 1992.0 31815 0 1993.0 32350 1.681596731101 1994.0 30585 -3.86610089580 ... ... ... Trending the Data | 91 The percent change can be either positive or negative, and we’ll see that does in fact occur in this time series. The last_value window function could be substituted for first_value in this query. Indexing from the last value in a series is much less com‐ mon, however, since analysis questions more often relate to change from a starting point rather than looking back from an arbitrary ending point; still, the option is there. Additionally, the sort order can be used to achieve indexing from the first or last value by switching between ASC and DESC: first_value(sales) over (order by sales_year desc) Window functions provide a lot of flexibility. Indexing can be accomplished without them through a series of self-JOINs, though more lines of code are required: SELECT sales_year, sales ,(sales / index_sales - 1) * 100 as pct_from_index FROM ( SELECT date_part('year',aa.sales_month) as sales_year ,bb.index_sales ,sum(aa.sales) as sales FROM retail_sales aa JOIN ( SELECT first_year, sum(a.sales) as index_sales FROM retail_sales a JOIN ( SELECT min(date_part('year',sales_month)) as first_year FROM retail_sales WHERE kind_of_business = 'Women''s clothing stores' ) b on date_part('year',a.sales_month) = b.first_year WHERE a.kind_of_business = 'Women''s clothing stores' GROUP BY 1 ) bb on 1 = 1 WHERE aa.kind_of_business = 'Women''s clothing stores' GROUP BY 1,2 ) aaa ; sales_year sales pct_from_index ---------- ----- -------------- 1992.0 31815 0 1993.0 32350 1.681596731101 1994.0 30585 -3.86610089580 ... ... ... 92 | Chapter 3: Time Series Analysis Notice the unusual JOIN clause on 1 = 1 between alias aa and subquery bb. Since we want the index_sales value to populate for every row in the result set, we can’t JOIN on the year or any other value, which would restrict the results. However, the data‐ base will return an error if no JOIN clause is specified. We can fool the database by using any expression that evaluates to TRUE in order to create the desired Cartesian JOIN. Any other TRUE statement, such as on 2 = 2 or on 'apples' = 'apples', could be", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 76 + }, + { + "text": "data‐ base will return an error if no JOIN clause is specified. We can fool the database by using any expression that evaluates to TRUE in order to create the desired Cartesian JOIN. Any other TRUE statement, such as on 2 = 2 or on 'apples' = 'apples', could be used instead. Beware of zeros in the denominator of division operations such as sales / index_sales in the last example. Databases return an error when they encounter division by zero, which can be frustrat‐ ing. Even when you think a zero in the denominator field is unlikely, it’s good practice to prevent this by telling the database to return an alternate default value when it encounters a zero. This can be done with a CASE statement. The examples in this section do not have zeros in the denominator, so I will omit this extra code for the sake of legibility. To wrap up this section, let’s look at a graph of the indexed time series for men’s and women’s clothing stores, shown in Figure 3-11. The SQL code looks like: SELECT sales_year, kind_of_business, sales ,(sales / first_value(sales) over (partition by kind_of_business order by sales_year) - 1) * 100 as pct_from_index FROM ( SELECT date_part('year',sales_month) as sales_year ,kind_of_business ,sum(sales) as sales FROM retail_sales WHERE kind_of_business in ('Men''s clothing stores' ,'Women''s clothing stores') and sales_month <= '2019-12-31' GROUP BY 1,2 ) a ; Trending the Data | 93 Figure 3-11. Men’s and women’s clothing store sales, indexed to 1992 sales It’s apparent from this graph that 1992 was something of a high-water mark for sales at men’s clothing stores. After 1992 sales dropped, then returned briefly to the same level in 1998, and have been declining ever since. This is striking since the data set is not adjusted for inflation, the tendency for prices to rise over time. Sales at women’s clothing stores decreased from 1992 levels initially, but they returned to the 1992 level by 2003. They have increased since, with the exception of the drop during the finan‐ cial crisis that decreased sales in 2009 and 2010. One explanation for these trends is that men simply decreased spending on clothes over time, perhaps becoming less fashion conscious relative to women. Perhaps men’s clothing simply became less expensive as global supply chains decreased costs. Yet another explanation might be that men shifted their clothing purchases from retailers categorized as “men’s clothing stores” to other types of retailers, such as sporting goods stores or online retailers. Indexing time series data is a powerful analysis technique, allowing us to find a range of insights in the data. SQL is well suited to this task, and I’ve shown how to construct indexed time series with and without window functions. Next, I’ll show you how to analyze data by using rolling time windows to find patterns in noisy time series. 94 | Chapter 3: Time Series Analysis Rolling Time Windows Time series data is often noisy, a challenge for one of our primary goals of", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 77 + }, + { + "text": "with and without window functions. Next, I’ll show you how to analyze data by using rolling time windows to find patterns in noisy time series. 94 | Chapter 3: Time Series Analysis Rolling Time Windows Time series data is often noisy, a challenge for one of our primary goals of finding patterns. We’ve seen how aggregating data, such as from monthly to yearly, can smooth out the results and make them easier to interpret. Another technique for smoothing data is rolling time windows, also known as moving calculations, that take into account multiple periods. Moving averages are probably the most common, but with the power of SQL, any aggregate function is available for analysis. Rolling time windows are used in a wide variety of analysis areas, including stock markets, macro‐ economic trends, and audience measurement. Some calculations are so commonly used that they have their own acronyms: last twelve months (LTM), trailing twelve months (TTM), and year-to-date (YTD). Figure 3-12 shows an example of a rolling time window and a cumulative calculation, relative to the month of October in the time series. Figure 3-12. Example of LTM and YTD rolling sum of sales There are several important pieces of any rolling time series calculation. First is the size of the window, which is the number of periods to include in the calculation. Larger windows with more time periods have a greater smoothing effect, but at the risk of losing sensitivity to important short-term changes in the data. Shorter win‐ dows with fewer time periods do less smoothing and thus are more sensitive to short- term changes, but at the risk of too little noise reduction. The second piece of time series calculations is the aggregate function used. As noted previously, moving averages are probably the most common. Moving sums, counts, Rolling Time Windows | 95 minimums, and maximums can also be calculated with SQL. Moving counts are use‐ ful in user population metrics (see the following sidebar). Moving minimums and maximums can help in understanding the extremes of the data, useful for planning analyses. The third piece of time series calculations is choosing the partitioning, or grouping, of the data that is included in the window. The analysis might call for resetting the window every year. Or the analysis might need a different moving series for each component or user group. Chapter 4 will go into more detail on cohort analysis of user groups, where we will consider how retention and cumulative values such as spend differ between populations over time. Partitioning will be controlled through grouping as well as the PARTITION BY statement of window functions. With these three pieces in mind, we’ll move into the SQL code and calculations for moving time periods, continuing with the US retail sales data set for examples. Measuring “Active Users”: DAU, WAU, and MAU Many consumer and some B2B SaaS applications use active user calculations such as daily active users (DAU), weekly active users (WAU), and monthly active users (MAU) to estimate", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 78 + }, + { + "text": "calculations for moving time periods, continuing with the US retail sales data set for examples. Measuring “Active Users”: DAU, WAU, and MAU Many consumer and some B2B SaaS applications use active user calculations such as daily active users (DAU), weekly active users (WAU), and monthly active users (MAU) to estimate their audience size. Since each of these are rolling windows, they can be calculated on a daily basis. I’ve often been asked what is the right or best met‐ ric to use, and my answer is always “it depends.” DAU helps companies with capacity planning, such as estimating how much load to expect on servers. Depending on the service, however, even more detailed data might be needed, such as peak hourly or even minute-by-minute concurrent user information. MAU is commonly used to estimate relative sizes of applications or services. It is use‐ ful for measuring fairly stable or growing user populations that have regular usage patterns that aren’t necessarily daily, such as higher use on the weekend for leisure products, or higher weekday use for work- or school-related products. MAU is not as well suited to detecting changes in underlying churn from users who stop using an application. Since it takes a user 30 days, the most common window, to pass through MAU, a user can have been absent from the product for 29 days before they trigger a drop in MAU. WAU, calculated over 7 days, can be a happy medium between DAU and MAU. WAU is more sensitive to short-term fluctuations, alerting teams to changes in churn more quickly than MAU while smoothing over day of week fluctuations that are tracked by DAU. A drawback to WAU is that it is still sensitive to short-term fluctuations driven by events such as holidays. 96 | Chapter 3: Time Series Analysis Calculating Rolling Time Windows Now that we know what rolling time windows are, how they’re useful, and their key components, let’s get into calculating them using the US retail sales data set. We’ll start with the simpler case, when the data set contains a record for each period that should be in the window, and then in the next section we’ll look at what to do when this is not the case. There are two main methods for calculating a rolling time window: a self-JOIN, which can be used in any database, and a window function, which as we’ve seen isn’t available in some databases. In both cases we need the same result: a date and a num‐ ber of data points that corresponds to the size of the window to which we will apply an average or another aggregate function. For this example, we’ll use a window of 12 months to get rolling annual sales, since the data is at a monthly level of granularity. We’ll then apply an average to get a 12- month moving average of retail sales. First, let’s develop the intuition for what will go into the calculation. In this query, alias a of", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 79 + }, + { + "text": "12 months to get rolling annual sales, since the data is at a monthly level of granularity. We’ll then apply an average to get a 12- month moving average of retail sales. First, let’s develop the intuition for what will go into the calculation. In this query, alias a of the table is our “anchor” table, the one from which we gather the dates. To start, we’ll look at a single month, December 2019. From alias b, the query gathers the 12 individual months of sales that will go into the moving average. This is accomplished with the JOIN clause b.sales_month between a.sales_month - interval '11 months' and a.sales_month, which cre‐ ates an intentional Cartesian JOIN: SELECT a.sales_month ,a.sales ,b.sales_month as rolling_sales_month ,b.sales as rolling_sales FROM retail_sales a JOIN retail_sales b on a.kind_of_business = b.kind_of_business and b.sales_month between a.sales_month - interval '11 months' and a.sales_month and b.kind_of_business = 'Women''s clothing stores' WHERE a.kind_of_business = 'Women''s clothing stores' and a.sales_month = '2019-12-01' ; sales_month sales rolling_sales_month rolling_sales ----------- ----- ------------------- ------------- 2019-12-01 4496 2019-01-01 2511 2019-12-01 4496 2019-02-01 2680 2019-12-01 4496 2019-03-01 3585 2019-12-01 4496 2019-04-01 3604 2019-12-01 4496 2019-05-01 3807 2019-12-01 4496 2019-06-01 3272 2019-12-01 4496 2019-07-01 3261 2019-12-01 4496 2019-08-01 3325 2019-12-01 4496 2019-09-01 3080 2019-12-01 4496 2019-10-01 3390 Rolling Time Windows | 97 2019-12-01 4496 2019-11-01 3850 2019-12-01 4496 2019-12-01 4496 Notice that the sales_month and sales figures from alias a are repeated for each row of the 12 months in the window. Remember that the dates in a BETWEEN clause are inclusive (both will be returned in the result set). It’s a common mistake to use 12 instead of 11 in the preceding query. When in doubt, check the intermediate query results as I’ve done here to make sure the intended number of periods ends up in the window calculation. The next step is to apply the aggregation—in this case, avg, since we want a rolling average. The count of records returned from alias b is included to confirm that each row averages 12 data points, a useful data quality check. Alias a also has a filter on sales_month. Since this data set starts in 1992, months in that year, except for December, have fewer than 12 historical records: SELECT a.sales_month ,a.sales ,avg(b.sales) as moving_avg ,count(b.sales) as records_count FROM retail_sales a JOIN retail_sales b on a.kind_of_business = b.kind_of_business and b.sales_month between a.sales_month - interval '11 months' and a.sales_month and b.kind_of_business = 'Women''s clothing stores' WHERE a.kind_of_business = 'Women''s clothing stores' and a.sales_month >= '1993-01-01' GROUP BY 1,2 ; sales_month sales moving_avg records_count ----------- ----- ---------- ------------- 1993-01-01 2123 2672.08 12 1993-02-01 2005 2673.25 12 1993-03-01 2442 2676.50 12 ... ... ... ... The results are graphed in Figure 3-13. While the monthly trend is noisy, the smoothed moving average trend makes detecting changes such as the increase from 2003 to 2007 and the subsequent dip through 2011 easier to spot. Notice that the extreme drop in early 2020 pulls the moving average down even after sales start", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 80 + }, + { + "text": "Figure 3-13. While the monthly trend is noisy, the smoothed moving average trend makes detecting changes such as the increase from 2003 to 2007 and the subsequent dip through 2011 easier to spot. Notice that the extreme drop in early 2020 pulls the moving average down even after sales start to rebound later in the year. 98 | Chapter 3: Time Series Analysis Figure 3-13. Monthly sales and 12-month moving average sales for women’s clothing stores Adding the filter kind_of_business = 'Women''s clothing stores' to each alias isn’t strictly necessary. Since the query uses an INNER JOIN, filtering on one table will automatically filter on the other. However, filtering on both tables often makes queries run faster, particularly when the tables are large. Window functions are another way to calculate rolling time windows. To make a roll‐ ing window, we need to use another optional part of a window calculation: the frame clause. The frame clause allows you to specify which records to include in the win‐ dow. By default, all records in the partition are included, and for many cases this works just fine. However, controlling the included records at a more fine-grained level is useful for cases like moving window calculations. The syntax is simple and yet can be confusing when encountering it for the first time. The frame clause can be specified as: { RANGE | ROWS | GROUPS } BETWEEN frame_start AND frame_end Rolling Time Windows | 99 Within the curly braces are three options for the frame type: range, rows, and groups. These are the ways you can specify which records to include in the result, relative to the current row. Records are always chosen from the current partition and follow the ORDER BY specified. The default sorting is ascending (ASC), but it can be changed to descending (DESC). Rows is the most straightforward and will allow you to specify the exact number of rows that should be returned. Range includes records that are within some boundary of values relative to the current row. Groups can be used when there are multiple records with the same ORDER BY value, such as when a data set includes multiple lines per sales month, one for each customer. The frame_start and frame_end can be any of the following: UNBOUNDED PRECEDING offset PRECEDING CURRENT ROW offset FOLLOWING UNBOUNDED FOLLOWING Preceding means to include rows before the current row, according to the ORDER BY sorting. Current row is just that, and following means to include rows that occur after the current row according to the ORDER BY sorting. The UNBOUNDED keyword means to include all records in the partition before or after the current row. The offset is the number of records, often just an integer constant, though a field or an expres‐ sion that returns an integer could also be used. Frame clauses also have an optional frame_exclusion option, which is beyond the scope of the discussion here. Figure 3-14 shows an example of the rows that each of", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 81 + }, + { + "text": "of records, often just an integer constant, though a field or an expres‐ sion that returns an integer could also be used. Frame clauses also have an optional frame_exclusion option, which is beyond the scope of the discussion here. Figure 3-14 shows an example of the rows that each of the window frame options will pick up. Figure 3-14. Window frame clauses and the rows they include 100 | Chapter 3: Time Series Analysis From partition to ordering to window frames, window functions have a variety of options that control the calculations, making them incredibly powerful and well suited to tackling complex calculations with relatively simple syntax. Returning to our retail sales example, the moving average that we calculated using a self-JOIN can be accomplished with window functions in fewer lines of code: SELECT sales_month ,avg(sales) over (order by sales_month rows between 11 preceding and current row ) as moving_avg ,count(sales) over (order by sales_month rows between 11 preceding and current row ) as records_count FROM retail_sales WHERE kind_of_business = 'Women''s clothing stores' ; sales_month moving_avg records_count ----------- ---------- ------------- 1992-01-01 1873.00 1 1992-02-01 1932.00 2 1992-03-01 2089.00 3 ... ... ... 1993-01-01 2672.08 12 1993-02-01 2673.25 12 1993-03-01 2676.50 12 ... ... ... In this query, the window orders the sales by month (ascending) to ensure that the window records are in chronological order. The frame clause is rows between 11 preceding and current row, since I know that I have one record for each month and I want the 11 prior months and the month from the current row included in the average and count calculations. The query returns all months, including those that don’t have 11 prior months, and we might want to filter these out by placing this query in a subquery and filtering by month or number of records in the outer query. While calculating moving averages from prior time periods is com‐ mon in many business contexts, SQL window functions are flexible enough to include future time periods as well. They can also be used in any scenario in which the data has some ordering, not just in time series analysis. Calculating rolling averages or other moving aggregations can be accomplished with self-JOINs or window functions when records exist in the data set for each time period in the window. There may be performance differences between the two meth‐ ods, depending on the type of database and the size of the data set. Unfortunately, it’s difficult to predict which one will be performant or to give general advice on which to Rolling Time Windows | 101 use. It’s worth trying both methods and paying attention to how long it takes to return your query results; then make whichever one seems to run faster your default choice. Now that we’ve seen how to calculate rolling time windows, I’ll show how to calculate rolling windows with sparse data sets. Rolling Time Windows with Sparse Data Data sets in the real world may not contain a record", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 82 + }, + { + "text": "results; then make whichever one seems to run faster your default choice. Now that we’ve seen how to calculate rolling time windows, I’ll show how to calculate rolling windows with sparse data sets. Rolling Time Windows with Sparse Data Data sets in the real world may not contain a record for every time period that falls within the window. The measurement of interest might be seasonal or intermittent by nature. For example, customers might return to purchase from a website at irregular intervals, or a particular product might go in and out of stock. This results in sparse data. In the last section, I showed how to calculate a rolling window with a self-JOIN and a date interval in the JOIN clause. You might be thinking that this will pick up any records within the 12-month time window, whether all were in the data set or not, and you’d be correct. The problem with this approach comes when there is no record for the month (or day or year) itself. For example, imagine I want to calculate the roll‐ ing 12-month sales for each model of shoe my store stocks as of December 2019. Some of the shoes went out of stock prior to December, however, and so don’t have sales records in that month. Using a self-JOIN or window function will return a data set of rolling sales for all the shoes that sold in December, but the data will be missing the shoes that went out of stock. Fortunately, we have a way to solve this problem: by using a date dimension. The date dimension, a static table that contains a row for each calendar date, was introduced in Chapter 2. With such a table we can ensure that a query returns a result for every date of interest, whether or not there was a data point for that date in the underlying data set. Since the retail_sales data does include rows for all months, I’ve simulated a sparse data set by adding a subquery to filter the table to only sales_months from January and July (1 and 7). Let’s look at the results when JOINed to the date_dim, but before aggregation, to develop intuition about the data before applying calculations: SELECT a.date, b.sales_month, b.sales FROM date_dim a JOIN ( SELECT sales_month, sales FROM retail_sales WHERE kind_of_business = 'Women''s clothing stores' and date_part('month',sales_month) in (1,7) ) b on b.sales_month between a.date - interval '11 months' and a.date WHERE a.date = a.first_day_of_month and a.date between '1993-01-01' and '2020-12-01' ; 102 | Chapter 3: Time Series Analysis date sales_month sales ---------- ----------- ----- 1993-01-01 1992-07-01 2373 1993-01-01 1993-01-01 2123 1993-02-01 1992-07-01 2373 1993-02-01 1993-01-01 2123 1993-03-01 1992-07-01 2373 ... ... ... Notice that the query returns results for February and March dates in addition to January, even though there are no sales for these months in the subquery results. This is possible because the date dimension contains records for all months. The filter a.date = a.first_day_of_month restricts the result set", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 83 + }, + { + "text": "... Notice that the query returns results for February and March dates in addition to January, even though there are no sales for these months in the subquery results. This is possible because the date dimension contains records for all months. The filter a.date = a.first_day_of_month restricts the result set to one value per month, instead of the 28 to 31 rows per month that would result from joining to every date. The construction of this query is otherwise very similar to the self-JOIN query in the last section, with the JOIN clause on b.sales_month between a.date - interval '11 months' and a.date of the same form as the JOIN clause in the self-JOIN. Now that we have developed an understanding of what the query will return, we can go ahead and apply the avg aggregation to get the moving average: SELECT a.date ,avg(b.sales) as moving_avg ,count(b.sales) as records FROM date_dim a JOIN ( SELECT sales_month, sales FROM retail_sales WHERE kind_of_business = 'Women''s clothing stores' and date_part('month',sales_month) in (1,7) ) b on b.sales_month between a.date - interval '11 months' and a.date WHERE a.date = a.first_day_of_month and a.date between '1993-01-01' and '2020-12-01' GROUP BY 1 ; date moving_avg records ---------- ---------- ------- 1993-01-01 2248.00 2 1993-02-01 2248.00 2 1993-03-01 2248.00 2 ... ... ... As we saw above, the result set includes a row for every month; however, the moving average stays constant until a new data point (in this case, a January or July) is added. Each moving average consists of two underlying data points. In a real use case, the number of underlying data points is likely to vary. To return the current month’s value when using a data dimension, an aggregation with a CASE statement can be used—for example: Rolling Time Windows | 103 ,max(case when a.date = b.sales_month then b.sales end) as sales_in_month The conditions inside the CASE statement can be changed to return any of the underlying records that the analysis requires through use of equality, inequality, or offsets with date math. If a date dimension is not available in your database, then another technique can be used to simulate one. In a subquery, SELECT the DIS‐ TINCT dates needed and JOIN them to your table in the same way as in the preceding examples: SELECT a.sales_month, avg(b.sales) as moving_avg FROM ( SELECT distinct sales_month FROM retail_sales WHERE sales_month between '1993-01-01' and '2020-12-01' ) a JOIN retail_sales b on b.sales_month between a.sales_month - interval '11 months' and a.sales_month and b.kind_of_business = 'Women''s clothing stores' GROUP BY 1 ; sales_month moving_avg ----------- ---------- 1993-01-01 2672.08 1993-02-01 2673.25 1993-03-01 2676.50 ... ... In this example, I used the same underlying table because I know it contains all the months. However, in practice any database table that contains the needed dates can be used, whether or not it is related to the table from which you want to calculate the rolling aggregation. Calculating rolling time windows with sparse or missing data can be done in SQL with controlled application", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 84 + }, + { + "text": "months. However, in practice any database table that contains the needed dates can be used, whether or not it is related to the table from which you want to calculate the rolling aggregation. Calculating rolling time windows with sparse or missing data can be done in SQL with controlled application of Cartesian JOINs. Next, we’ll look at how to calculate cumulative values that are often used in analysis. Calculating Cumulative Values Rolling window calculations, such as moving averages, typically use fixed-size win‐ dows, such as 12 months, as we saw in the last section. Another commonly used type of calculation is the cumulative value, such as YTD, quarter-to-date (QTD), and month-to-date (MTD). Rather than a fixed-length window, these rely on a common starting point, with the window size growing with each row. 104 | Chapter 3: Time Series Analysis The simplest way to calculate cumulative values is with a window function. In this example, sum is used to find total sales YTD as of each month. Other analyses might call for a monthly average YTD or a monthly maximum YTD, which can be accom‐ plished by swapping sum for avg or max. The window resets according to the PARTI‐ TION BY clause, in this case the year of the sales month. The ORDER BY clause typically includes a date field in time series analysis. Omitting the ORDER BY can lead to incorrect results due to the way the data is sorted in the underlying table, so it’s a good idea to include it even if you think the data is already sorted by date: SELECT sales_month, sales ,sum(sales) over (partition by date_part('year',sales_month) order by sales_month ) as sales_ytd FROM retail_sales WHERE kind_of_business = 'Women''s clothing stores' ; sales_month sales sales_ytd ----------- ----- --------- 1992-01-01 1873 1873 1992-02-01 1991 3864 1992-03-01 2403 6267 ... ... ... 1992-12-01 4416 31815 1993-01-01 2123 2123 1993-02-01 2005 4128 ... ... ... The query returns a record for each sales_month, the sales for that month, and the running total sales_ytd. The series starts in 1992 and then resets in January 1993, as it will for every year in the data set. The results for years 2016 through 2020 are graphed in Figure 3-15. The first four years show similar patterns through the year, but of course 2020 looks very different. Rolling Time Windows | 105 Figure 3-15. Monthly sales and cumulative annual sales for women’s clothing stores The same results can be achieved without window functions, by using a self-JOIN that leverages a Cartesian JOIN. In this example, the two table aliases are JOINed on the year of the sales_month to ensure that the aggregated values are for the same year, resetting each year. The JOIN clause also specifies that the results should include sales_months from alias b that are less than or equal to the sales_month in alias a. In January 1992, only the January 1992 row from alias b meets this criterion; in February 1992, both January and February 1992 do; and so", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 85 + }, + { + "text": "JOIN clause also specifies that the results should include sales_months from alias b that are less than or equal to the sales_month in alias a. In January 1992, only the January 1992 row from alias b meets this criterion; in February 1992, both January and February 1992 do; and so on: SELECT a.sales_month, a.sales ,sum(b.sales) as sales_ytd FROM retail_sales a JOIN retail_sales b on date_part('year',a.sales_month) = date_part('year',b.sales_month) and b.sales_month <= a.sales_month and b.kind_of_business = 'Women''s clothing stores' WHERE a.kind_of_business = 'Women''s clothing stores' GROUP BY 1,2 ; 106 | Chapter 3: Time Series Analysis sales_month sales sales_ytd ----------- ----- --------- 1992-01-01 1873 1873 1992-02-01 1991 3864 1992-03-01 2403 6267 ... ... ... 1992-12-01 4416 31815 1993-01-01 2123 2123 1993-02-01 2005 4128 ... ... ... Window functions require fewer characters of code, and it’s usually easier to keep track of exactly what they are calculating once you are familiar with the syntax. There’s often more than one way to approach a problem in SQL, and rolling time windows are a good example of that. I find it useful to know multiple approaches, because every once in a while I run into a tricky problem that is actually better solved with an approach that seems less efficient in other contexts. Now that we’ve covered rolling time windows, we’ll move on to our final topic in time series analysis with SQL: seasonality. Analyzing with Seasonality Seasonality is any pattern that repeats over regular intervals. Unlike other noise in the data, seasonality can be predicted. The word seasonality brings to mind the four sea‐ sons of the year—spring, summer, fall, winter—and some data sets include these pat‐ terns. Shopping patterns change with the seasons, from the clothes and food people buy to the money spent on leisure and travel. The winter holiday shopping season can be make-or-break for many retailers. Seasonality can also exist at other time scales, from years down to minutes. Presidential elections in the United States happen every four years, leading to distinct patterns in media coverage. Day of week cyclicality is common, as work and school dominate Monday to Friday, while chores and leisure activities dominate the weekend. Time of day is another type of seasonality that res‐ taurants experience, with rushes around lunch and dinner time and slower sales in between. To understand whether seasonality exists in a time series, and at what scale, it’s useful to graph it and then visually inspect for patterns. Try aggregating at different levels, from hourly to daily, weekly, and monthly. You should also incorporate knowledge about the data set. Are there patterns that you can guess based on what you know about the entity or process it represents? Consult subject matter experts, if available. Let’s take a look at some seasonal patterns in the retail sales data set, shown in Figure 3-16. Jewelry stores have a highly seasonal pattern, with annual peaks in December related to holiday gift giving. Book stores have two peaks each year: one peak is in August, corresponding", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 86 + }, + { + "text": "if available. Let’s take a look at some seasonal patterns in the retail sales data set, shown in Figure 3-16. Jewelry stores have a highly seasonal pattern, with annual peaks in December related to holiday gift giving. Book stores have two peaks each year: one peak is in August, corresponding with back-to-school time in the United States; the other peak starts in December and lasts through January, including both the holiday Analyzing with Seasonality | 107 gift period and back-to-school time for the spring semester. A third example is gro‐ cery stores, which have much less monthly seasonality than the other two time series (although they likely have seasonality at the day of week and time of day level). This isn’t surprising: people need to eat year-round. Grocery store sales increase a bit in December for the holidays, and they decline in February, since that month simply has fewer days. Figure 3-16. Examples of seasonality patterns in book store, grocery store, and jewelry store sales Seasonality can take many forms, though there are some common approaches to ana‐ lyzing it regardless. One way to deal with seasonality is to smooth it out, either by aggregating the data to a less granular time period or by using rolling windows, as we saw previously. Another way to work with seasonal data is to benchmark against sim‐ ilar time periods and analyze the difference. I’ll show several ways to accomplish this next. 108 | Chapter 3: Time Series Analysis Period-over-Period Comparisons: YoY and MoM Period-over-period comparisons can take multiple forms. The first one is to compare a time period to the previous value in the series, a practice so common in analysis that there are acronyms for the most often-used comparisons. Depending on the level of aggregation the comparison might be year-over-year (YoY), month-over-month (MoM), day-over-day (DoD), and so on. For these calculations we’ll use the lag function, another one of the window func‐ tions. The lag function returns a previous or lagging value from a series. The lag function has the following form: lag(return_value [,offset [,default]]) The return_value is any field from the data set and thus can be any data type. The optional OFFSET indicates how many rows back in the partition to take the return_value from. The default is 1, but any integer value can be used. You can also optionally specify a default value to use if there is no lagging record to retrieve a value from. Like other window functions, lag is also calculated over a partition, with sorting determined by the ORDER BY clause. If no PARTITION BY clause is speci‐ fied, lag looks back over the whole data set, and likewise if no ORDER BY clause is specified, the database order is used. It’s usually a good idea to at least include an ORDER BY clause in a lag window function to control the output. The lead window function works in the same way as the lag func‐ tion, except that it returns a subsequent value", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 87 + }, + { + "text": "is specified, the database order is used. It’s usually a good idea to at least include an ORDER BY clause in a lag window function to control the output. The lead window function works in the same way as the lag func‐ tion, except that it returns a subsequent value as determined by the offset. Changing the ORDER BY from ascending (ASC) to descend‐ ing (DESC) in a time series has the effect of turning a lag statement into the equivalent of a lead statement. Alternatively, a negative integer can be used as the OFFSET value to return a value from a subsequent row. Let’s apply this to our retail sales data set to calculate MoM and YoY growth. In this section, we’ll focus on book store sales, since I’m a real book store nerd. First, we’ll develop our intuition about what is returned by the lag function by returning both the lagging month and the lagging sales values: SELECT kind_of_business, sales_month, sales ,lag(sales_month) over (partition by kind_of_business order by sales_month ) as prev_month ,lag(sales) over (partition by kind_of_business order by sales_month ) as prev_month_sales FROM retail_sales WHERE kind_of_business = 'Book stores' Analyzing with Seasonality | 109 ; kind_of_business sales_month sales prev_month prev_month_sales ---------------- ----------- ----- ---------- ---------------- Book stores 1992-01-01 790 (null) (null) Book stores 1992-02-01 539 1992-01-01 790 Book stores 1992-03-01 535 1992-02-01 539 ... ... ... ... ... For each row, the previous sales_month is returned, as well as the sales for that month, and we can confirm this by inspecting the first few lines of the result set. The first row has null for prev_month and prev_month_sales since there is no earlier record in this data set. With an understanding of the values returned by the lag func‐ tion, we can calculate the percent change from the previous value: SELECT kind_of_business, sales_month, sales ,(sales / lag(sales) over (partition by kind_of_business order by sales_month) - 1) * 100 as pct_growth_from_previous FROM retail_sales WHERE kind_of_business = 'Book stores' ; kind_of_business sales_month sales pct_growth_from_previous ---------------- ----------- ----- ------------------------ Book stores 1992-01-01 790 (null) Book stores 1992-02-01 539 -31.77 Book stores 1992-03-01 535 -0.74 ... ... ... ... Sales dropped 31.8% from January to February, due at least in part to the seasonal decline after the holidays and the return to school for the spring semester. Sales were down only 0.7% from February to March. The calculation for the YoY comparison is similar, but first we need to aggregate sales to the yearly level. Since we’re looking at only one kind_of_business, I’ll drop that field from the rest of the examples to simplify the code: SELECT sales_year, yearly_sales ,lag(yearly_sales) over (order by sales_year) as prev_year_sales ,(yearly_sales / lag(yearly_sales) over (order by sales_year) -1) * 100 as pct_growth_from_previous FROM ( SELECT date_part('year',sales_month) as sales_year ,sum(sales) as yearly_sales FROM retail_sales WHERE kind_of_business = 'Book stores' GROUP BY 1 ) a ; 110 | Chapter 3: Time Series Analysis sales_year yearly_sales prev_year_sales pct_growth_from_previous ---------- ------------ --------------- ------------------------ 1992.0 8327 (null) (null) 1993.0 9108", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 88 + }, + { + "text": "by sales_year) -1) * 100 as pct_growth_from_previous FROM ( SELECT date_part('year',sales_month) as sales_year ,sum(sales) as yearly_sales FROM retail_sales WHERE kind_of_business = 'Book stores' GROUP BY 1 ) a ; 110 | Chapter 3: Time Series Analysis sales_year yearly_sales prev_year_sales pct_growth_from_previous ---------- ------------ --------------- ------------------------ 1992.0 8327 (null) (null) 1993.0 9108 8327 9.37 1994.0 10107 9108 10.96 ... ... ... ... Sales grew more than 9.3% from 1992 to 1993, and almost 11% from 1993 to 1994. These period-over-period calculations are useful, but they don’t quite allow us to ana‐ lyze the seasonality in the data set. For example, in Figure 3-17 the MoM percent growth values are plotted, and they contain just as much seasonality as the original time series. Figure 3-17. Percent growth from previous month for US retail book store sales To tackle this, the next section will demonstrate how to use SQL to compare current values to the values for the same month in the previous year. Analyzing with Seasonality | 111 Period-over-Period Comparisons: Same Month Versus Last Year Comparing data for one time period to data for a similar previous time period can be a useful way to control for seasonality. The previous time period may be the same day of the week in the previous week, the same month in the previous year, or another variation that makes sense for the data set. To accomplish this comparison, we can use the lag function along with some clever partitioning: the unit of time with which we want to compare the current value. In this case, we will compare monthly sales to the sales for the same month in the pre‐ vious year. For example, January sales will be compared to prior year January sales, February sales will be compared to prior year February sales, and so on. First, recall that the date_part function returns a numeric value when used with the “month” argument: SELECT sales_month ,date_part('month',sales_month) FROM retail_sales WHERE kind_of_business = 'Book stores' ; sales_month date_part ----------- --------- 1992-01-01 1.0 1992-02-01 2.0 1992-03-01 3.0 ... ... Next, we include the date_part in the PARTITION BY clause so that the window function looks up the value for the matching month number from the prior year. This is an example of how window function clauses can include calculations in addi‐ tion to database fields, giving them even more versatility. I find it useful to check intermediate results to build intuition about what the final query will return, so first we’ll confirm that the lag function with partition by date_part('month', sales_month) returns the intended values: SELECT sales_month, sales ,lag(sales_month) over (partition by date_part('month',sales_month) order by sales_month ) as prev_year_month ,lag(sales) over (partition by date_part('month',sales_month) order by sales_month ) as prev_year_sales FROM retail_sales WHERE kind_of_business = 'Book stores' ; 112 | Chapter 3: Time Series Analysis sales_month sales prev_year_month prev_year_sales ----------- ----- --------------- --------------- 1992-01-01 790 (null) (null) 1993-01-01 998 1992-01-01 790 1994-01-01 1053 1993-01-01 998 ... ... ... ... 1992-02-01 539 (null) (null) 1993-02-01 568 1992-02-01 539 1994-02-01 635", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 89 + }, + { + "text": "prev_year_sales FROM retail_sales WHERE kind_of_business = 'Book stores' ; 112 | Chapter 3: Time Series Analysis sales_month sales prev_year_month prev_year_sales ----------- ----- --------------- --------------- 1992-01-01 790 (null) (null) 1993-01-01 998 1992-01-01 790 1994-01-01 1053 1993-01-01 998 ... ... ... ... 1992-02-01 539 (null) (null) 1993-02-01 568 1992-02-01 539 1994-02-01 635 1993-02-01 568 ... ... ... ... The first lag function returns the same month for the prior year, which we can verify by looking at the prev_year_month value. The row for the 1993-01-01 sales_month returns 1992-01-01 for the prev_year_month as intended, and the prev_year_sales of 790 match the sales we can see in the 1992-01-01 row. Notice that the prev_year_month and prev_year_sales are null for 1992 since there are no prior records in the data set. Now that we’re confident the lag function as written returns the correct values, we can calculate comparison metrics such as absolute difference and percent change from previous: SELECT sales_month, sales ,sales - lag(sales) over (partition by date_part('month',sales_month) order by sales_month ) as absolute_diff ,(sales / lag(sales) over (partition by date_part('month',sales_month) order by sales_month) - 1) * 100 as pct_diff FROM retail_sales WHERE kind_of_business = 'Book stores' ; sales_month sales absolute_diff pct_diff ----------- ----- ------------- -------- 1992-01-01 790 (null) (null) 1993-01-01 998 208 26.32 1994-01-01 1053 55 5.51 ... ... ... ... We can now graph the results in Figure 3-18 and more easily see the months where growth was unusually high, such as January 2002, or unusually low, such as December 2001. Analyzing with Seasonality | 113 Figure 3-18. Book store sales, YoY absolute difference in sales, and YoY percent growth Another useful analysis tool is to create a graph that lines up the same time period— in this case, months—with a line for each time series—in this case, years. To do this, we’ll create a result set that has a row for each month number or name, and a column for each of the years we want to consider. To get the month, we can use either the date_part or the to_char function, depending on whether we want numeric or text values for the months. Then we’ll pivot the data using an aggregate function. This example uses the max aggregate, but depending on the analysis, a sum, count, or other aggregation might be appropriate. We’ll zoom in on 1992 through 1994 for this example: SELECT date_part('month',sales_month) as month_number ,to_char(sales_month,'Month') as month_name ,max(case when date_part('year',sales_month) = 1992 then sales end) as sales_1992 ,max(case when date_part('year',sales_month) = 1993 then sales end) as sales_1993 ,max(case when date_part('year',sales_month) = 1994 then sales end) as sales_1994 FROM retail_sales 114 | Chapter 3: Time Series Analysis WHERE kind_of_business = 'Book stores' and sales_month between '1992-01-01' and '1994-12-01' GROUP BY 1,2 ; month_number month_name sales_1992 sales_1993 sales_1994 ------------ ---------- ---------- ---------- ---------- 1.0 January 790 998 1053 2.0 February 539 568 635 3.0 March 535 602 634 4.0 April 523 583 610 5.0 May 552 612 684 6.0 June 589 618 724 7.0 July 592 607 678 8.0 August 894", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 90 + }, + { + "text": "1,2 ; month_number month_name sales_1992 sales_1993 sales_1994 ------------ ---------- ---------- ---------- ---------- 1.0 January 790 998 1053 2.0 February 539 568 635 3.0 March 535 602 634 4.0 April 523 583 610 5.0 May 552 612 684 6.0 June 589 618 724 7.0 July 592 607 678 8.0 August 894 983 1154 9.0 September 861 903 1022 10.0 October 645 669 732 11.0 November 642 692 772 12.0 December 1165 1273 1409 By lining the data up in this way, we can see some trends immediately. December sales are the highest monthly sales of the year. Sales in 1994 were higher every month than sales in 1992 and 1993. The August-to-September sales bump is visible, and par‐ ticularly easy to spot in 1994. With a graph of the data, as in Figure 3-19, the trends are much easier to identify. Sales increased year to year in every month, though the increases were larger in some months than others. With this data and graph in hand, we can start to construct a story about book store sales that might help with inventory planning or scheduling of marketing promotions or might serve as a piece of evidence in a wider story about US retail sales. With SQL there are a number of techniques for cutting through the noise of seasonal‐ ity to compare data in time series. In this section, we’ve seen how to compare current values to prior comparable periods using lag functions and how to pivot the data with date_part, to_char, and aggregate functions. Next, I’ll show some techniques for comparing multiple prior periods in order to further control for noisy time series data. Analyzing with Seasonality | 115 Figure 3-19. Book store sales for 1992–1994, aligned by month Comparing to Multiple Prior Periods Comparing data to prior comparable periods is a useful way to reduce the noise that arises from seasonality. Sometimes comparing to a single prior period is insufficient, particularly if that prior period was impacted by unusual events. Comparing a Mon‐ day to the previous Monday is difficult if one of them was a holiday. The month in the prior year might be unusual due to economic events, severe weather, or a site outage that changed typical behavior. Comparing current values to an aggregate of multiple prior periods can help smooth out these fluctuations. These techniques also combine what we’ve learned about using SQL to calculate rolling time periods and comparable prior period results. The first technique uses the lag function, as in the last section, but here we’ll take advantage of the optional offset value. Recall that when no offset is provided to lag, the function returns the immediate prior value according to the PARTITION BY and ORDER BY clauses. An offset value of 2 skips over the immediate prior value and returns the value prior to that, an offset value of 3 returns the value from 3 rows back, and so on. For this example, we’ll compare the current month’s sales to the same month’s", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 91 + }, + { + "text": "and ORDER BY clauses. An offset value of 2 skips over the immediate prior value and returns the value prior to that, an offset value of 3 returns the value from 3 rows back, and so on. For this example, we’ll compare the current month’s sales to the same month’s sales over three prior years. As usual, first we’ll inspect the returned values to confirm the SQL is working as expected: 116 | Chapter 3: Time Series Analysis SELECT sales_month, sales ,lag(sales,1) over (partition by date_part('month',sales_month) order by sales_month ) as prev_sales_1 ,lag(sales,2) over (partition by date_part('month',sales_month) order by sales_month ) as prev_sales_2 ,lag(sales,3) over (partition by date_part('month',sales_month) order by sales_month ) as prev_sales_3 FROM retail_sales WHERE kind_of_business = 'Book stores' ; sales_month sales prev_sales_1 prev_sales_2 prev_sales_3 ----------- ----- ------------ ------------ ------------ 1992-01-01 790 (null) (null) (null) 1993-01-01 998 790 (null) (null) 1994-01-01 1053 998 790 (null) 1995-01-01 1308 1053 998 790 1996-01-01 1373 1308 1053 998 ... ... ... ... ... Null is returned where no prior record exists, and we can confirm that the correct same month, prior year value appears. From here we can calculate whatever compari‐ son metric the analysis calls for—in this case, the percent of the rolling average of three prior periods: SELECT sales_month, sales ,sales / ((prev_sales_1 + prev_sales_2 + prev_sales_3) / 3) as pct_of_3_prev FROM ( SELECT sales_month, sales ,lag(sales,1) over (partition by date_part('month',sales_month) order by sales_month ) as prev_sales_1 ,lag(sales,2) over (partition by date_part('month',sales_month) order by sales_month ) as prev_sales_2 ,lag(sales,3) over (partition by date_part('month',sales_month) order by sales_month ) as prev_sales_3 FROM retail_sales WHERE kind_of_business = 'Book stores' ) a ; sales_month sales pct_of_3_prev ----------- ----- ------------- 1995-01-01 1308 138.12 1996-01-01 1373 122.69 Analyzing with Seasonality | 117 1997-01-01 1558 125.24 ... ... ... 2017-01-01 1386 94.67 2018-01-01 1217 84.98 2019-01-01 1004 74.75 ... ... ... We can see from the result that book sales grew from the prior three-year rolling average in the mid-1990s, but the picture was different in the late 2010s, when sales were a shrinking percentage of that three-year rolling average each year. You might have noticed that this problem resembles one we saw earlier when calcu‐ lating rolling time windows. As an alternative to the last example, we can use an avg window function with a frame clause. To accomplish this, the PARTITION BY will use the same date_part function, and the ORDER BY is the same. A frame clause is added to include rows between 3 preceding and 1 preceding. This includes the values in the 1, 2, and 3 rows prior but excludes the value in the current row: SELECT sales_month, sales ,sales / avg(sales) over (partition by date_part('month',sales_month) order by sales_month rows between 3 preceding and 1 preceding ) as pct_of_prev_3 FROM retail_sales WHERE kind_of_business = 'Book stores' ; sales_month sales pct_of_prev_3 ----------- ----- ------------- 1995-01-01 1308 138.12 1996-01-01 1373 122.62 1997-01-01 1558 125.17 ... ... ... 2017-01-01 1386 94.62 2018-01-01 1217 84.94 2019-01-01 1004 74.73 ... ... ... The results match those", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 92 + }, + { + "text": "3 preceding and 1 preceding ) as pct_of_prev_3 FROM retail_sales WHERE kind_of_business = 'Book stores' ; sales_month sales pct_of_prev_3 ----------- ----- ------------- 1995-01-01 1308 138.12 1996-01-01 1373 122.62 1997-01-01 1558 125.17 ... ... ... 2017-01-01 1386 94.62 2018-01-01 1217 84.94 2019-01-01 1004 74.73 ... ... ... The results match those of the previous example, confirming that the alternative code is equivalent. If you look closely, you’ll notice that the decimal place values are slightly different in the result using the three lag windows and the result using the avg window function. This is due to how the data‐ base handles decimal rounding in intermediate calculations. For many analyses, the difference won’t matter, but pay careful atten‐ tion if you’re working with financial or other highly regulated data. 118 | Chapter 3: Time Series Analysis Analyzing data with seasonality often involves trying to reduce noise in order to make clear conclusions about the underlying trends in the data. Comparing data points against multiple prior time periods can give us an even smoother trend to compare to and determine what is actually happening in the current time period. This does require that the data include enough history to make these comparisons, but when we have a long enough time series, it can be insightful. Conclusion Time series analysis is a powerful way to analyze data sets. We’ve seen how to set up our data for analysis with date and time manipulations. We talked about date dimen‐ sions and saw how to apply them to calculating rolling time windows. We looked at period-over-period calculations and how to analyze data with seasonality patterns. In the next chapter, we’ll delve deep into a related topic that extends on time series anal‐ ysis: cohort analysis. Conclusion | 119 CHAPTER 4 Cohort Analysis In Chapter 3 we covered time series analysis. With those techniques in hand, we will now turn to a related type of analysis with many business and other applications: cohort analysis. I remember the first time I encountered a cohort analysis. I was working at my first data analyst job, at a small startup. I was reviewing a purchase analysis I’d worked on with the CEO, and he suggested that I break up the customer base by cohorts to see whether behavior was changing over time. I assumed it was some fancy business school thing and probably useless, but he was the CEO, so of course I humored him. Turns out it wasn’t just a lark. Breaking populations into cohorts and following them over time is a powerful way to analyze your data and avoid various biases. Cohorts can provide clues to how subpopulations differ from each other and how they change over time. In this chapter, we’ll first take a look at what cohorts are and at the building blocks of certain types of cohort analysis. After an introduction to the legislators data set used for the examples, we’ll learn how to construct a retention analysis and deal with vari‐ ous challenges such", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 93 + }, + { + "text": "In this chapter, we’ll first take a look at what cohorts are and at the building blocks of certain types of cohort analysis. After an introduction to the legislators data set used for the examples, we’ll learn how to construct a retention analysis and deal with vari‐ ous challenges such as defining the cohort and handling sparse data. Next, we’ll cover survivorship, returnship, and cumulative calculations, all of which are similar to retention analysis in the way the SQL code is structured. Finally, we’ll look at how to combine cohort analysis with cross-sectional analysis to understand the makeup of populations over time. 121 Cohorts: A Useful Analysis Framework Before we get into the code, I will define what cohorts are, consider the types of ques‐ tions we can answer with this type of analysis, and describe the components of any cohort analysis. A cohort is a group of individuals who share some characteristic of interest, described below, at the time we start observing them. Cohort members are often people but can be any type of entity we want to study: companies, products, or physical world phe‐ nomena. Individuals in a cohort may be aware of their membership, just as children in a first-grade class are aware they are part of a peer group of first graders, or partici‐ pants in a drug trial are aware they are part of a group receiving a treatment. At other times, entities are grouped into cohorts virtually, as when a software company groups all customers acquired in a certain year to study how long they remain customers. It’s always important to consider the ethical implications of cohorting entities without their awareness, if any different treatment is to be applied to them. Cohort analysis is a useful way to compare groups of entities over time. Many impor‐ tant behaviors take weeks, months, or years to occur or evolve, and cohort analysis is a way to understand these changes. Cohort analysis provides a framework for detect‐ ing correlations between cohort characteristics and these long-term trends, which can lead to hypotheses about the causal drivers. For example, customers acquired through a marketing campaign may have different long-term purchase patterns than those who were persuaded by a friend to try a company’s products. Cohort analysis can be used to monitor new cohorts of users or customers and assess how they compare to previous cohorts. Such monitoring can provide an early alert signal that something has gone wrong (or right) for new customers. Cohort analysis is also used to mine historical data. A/B tests, discussed in Chapter 7, are the gold standard for determin‐ ing causality, but we can’t go back in time and run every test for every question about the past in which we are interested. We should of course be cautious about attaching causal meaning to cohort analysis and instead use cohort analysis as a way to under‐ stand customers and generate hypotheses that can be tested rigorously in the future. Cohort analyses have three components:", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 94 + }, + { + "text": "question about the past in which we are interested. We should of course be cautious about attaching causal meaning to cohort analysis and instead use cohort analysis as a way to under‐ stand customers and generate hypotheses that can be tested rigorously in the future. Cohort analyses have three components: the cohort grouping, a time series of data over which the cohort is observed, and an aggregate metric that measures an action done by cohort members. Cohort grouping is often based on a start date: the customer’s first purchase or sub‐ scription date, the date a student started school, and so on. However, cohorts can also be formed around other characteristics that are either innate or changing over time. Innate qualities include birth year and country of origin, or the year a company was founded. Characteristics that can change over time include city of residence and mar‐ ital status. When these are used, we need to be careful to cohort only on the value on the starting date, or else entities can jump between cohort groups. 122 | Chapter 4: Cohort Analysis Cohort or Segment? These two terms are often used in similar ways, or even inter‐ changeably, but it’s worth drawing a distinction between them for the sake of clarity. A cohort is a group of users (or other entities) who have a common starting date and are followed over time. A segment is a grouping of users who share a common characteristic or set of characteristics at a point in time, regardless of their start‐ ing date. Similar to cohorts, segments can be based on innate fac‐ tors such as age or on behavioral characteristics. A segment of users that signs up in the same month can be put into a cohort and followed over time. Or different groupings of users can be explored with cohort analysis so that you can see which ones have the most valuable characteristics. The analyses we’ll cover in this chapter, such as retention, can help put concrete data behind marketing segments. The second component of any cohort analysis is the time series. This is a series of pur‐ chases, logins, interactions, or other actions that are taken by the customers or enti‐ ties to be cohorted. It’s important that the time series covers the entire life span of the entities, or there will be survivorship bias in early cohorts. Survivorship bias occurs when only customers who have stayed are in the data set; churned customers are excluded because they are no longer around, so the rest of the customers appear to be of higher quality or fit in comparison to newer cohorts (see “Survivorship Bias” on page 167). It’s also important to have a time series that is long enough for the entities to complete the action of interest. For example, if customers tend to purchase once a month, a time series of several months is needed. If, on the other hand, purchases happen only once a year, a time series of", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 95 + }, + { + "text": "have a time series that is long enough for the entities to complete the action of interest. For example, if customers tend to purchase once a month, a time series of several months is needed. If, on the other hand, purchases happen only once a year, a time series of several years would be preferable. Inevitably, more recently acquired customers will not have had as long to complete actions as those customers who were acquired further in the past. In order to normalize, cohort analysis usually measures the number of periods that have elapsed from a starting date, rather than calendar months. In this way, cohorts can be compared in period 1, period 2, and so on to see how they evolve over time, regardless of which month the action actually occurred. The intervals may be days, weeks, months, or years. The aggregate metric should be related to the actions that matter to the health of the organization, such as customers continuing to use or purchase the product. Metric values are aggregated across the cohort, usually with sum, count, or average, though any relevant aggregation works. The result is a time series that can then be used to understand changes in behavior over time. In this chapter, I’ll cover four types of cohort analysis: retention, survivorship, return‐ ship or repeat purchase behavior, and cumulative behavior. Cohorts: A Useful Analysis Framework | 123 Retention Retention is concerned with whether the cohort member has a record in the time series on a particular date, expressed as a number of periods from the starting date. This is useful in any kind of organization in which repeated actions are expected, from playing an online game to using a product or renewing a sub‐ scription, and it helps to answer questions about how sticky or engaging a prod‐ uct is and how many entities can be expected to appear on future dates. Survivorship Survivorship is concerned with how many entities remained in the data set for a certain length of time or longer, regardless of the number or frequency of actions up to that time. Survivorship is useful for answering questions about the propor‐ tion of the population that can be expected to remain—either in a positive sense by not churning or passing away, or in a negative sense by not graduating or ful‐ filling some requirement. Returnship Returnship or repeat purchase behavior is concerned with whether an action has happened more than some minimum threshold of times—often simply more than once—during a fixed window of time. This type of analysis is useful in sit‐ uations in which the behavior is intermittent and unpredictable, such as in retail, where it characterizes the share of repeat purchasers in each cohort within a fixed time window. Cumulative Cumulative calculations are concerned with the total number or amounts meas‐ ured at one or more fixed time windows, regardless of when they happened dur‐ ing that window. Cumulative calculations are often used in calculations of customer lifetime value (LTV", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 96 + }, + { + "text": "purchasers in each cohort within a fixed time window. Cumulative Cumulative calculations are concerned with the total number or amounts meas‐ ured at one or more fixed time windows, regardless of when they happened dur‐ ing that window. Cumulative calculations are often used in calculations of customer lifetime value (LTV or CLTV). The four types of cohort analysis allow us to compare subgroups and understand how they differ over time in order to make better product, marketing, and financial deci‐ sions. The calculations for the different types are similar, so we will set the stage with retention, and then I’ll show how to modify retention code to calculate the other types. Before we dive into constructing our cohort analysis, let’s take a look at the data set we’ll be using for the examples in this chapter. 124 | Chapter 4: Cohort Analysis The Legislators Data Set The SQL examples in this chapter will use a data set of past and present members of the United States Congress maintained in a GitHub repository. In the US, Congress is responsible for writing laws or legislation, so its members are also known as legisla‐ tors. Since the data set is a JSON file, I have applied some transformations to produce a more suitable data model for analysis, and I have posted data in a format suitable for following along with the examples in the book’s GitHub legislators folder. The source repository has an excellent data dictionary, so I won’t repeat all the details here. I will provide a few details, however, that should help those who aren’t familiar with the US government to follow along with the analyses in this chapter. Congress has two chambers, the Senate (“sen” in the data set) and the House of Rep‐ resentatives (“rep”). Each state has two senators, and they are elected for six-year terms. Representatives are allocated to states based on population; each representative has a district that they alone represent. Representatives are elected for two-year terms. Actual terms in either chamber can be shorter in the event that the legislator dies or is elected or appointed to a higher office. Legislators accumulate power and influence via leadership positions the longer they are in office, and thus standing for re-election is common. Finally, a legislator may belong to a political party, or they may be an “independent.” In the modern era, the vast majority of legislators are Democrats or Republicans, and the rivalry between the two parties is well known. Legislators occa‐ sionally change parties while in office. For the analyses, we’ll make use of two tables: legislators and legislators_terms. The legislators table contains a list of all the people included in the data set, with birthday, gender, and a set of ID fields that can be used to look up the person in other data sets. The legislators_terms table contains a record for each term in office for each legislator, with start and end date, and other attributes such as chamber and party. The id_bioguide", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 97 + }, + { + "text": "birthday, gender, and a set of ID fields that can be used to look up the person in other data sets. The legislators_terms table contains a record for each term in office for each legislator, with start and end date, and other attributes such as chamber and party. The id_bioguide field is used as the unique identifier of a legislator and appears in each table. Figure 4-1 shows a sample of the legislators data. Figure 4-2 shows a sample of the legislators_terms data. The Legislators Data Set | 125 Figure 4-1. Sample of the legislators table Figure 4-2. Sample of the legislators_terms table Now that we have an understanding of what cohort analysis is and of the data set we’ll be using for examples, let’s get into how to write SQL for retention analysis. The key question SQL will help us answer is: once representatives take office, how long do they keep their jobs? 126 | Chapter 4: Cohort Analysis Retention One of the most common types of cohort analysis is retention analysis. To retain is to keep or continue something. Many skills need to be practiced to be retained. Busi‐ nesses usually want their customers to keep purchasing their products or using their services, since retaining customers is more profitable than acquiring new ones. Employers want to retain their employees, because recruiting replacements is expen‐ sive and time consuming. Elected officials seek reelection in order to continue work‐ ing on the priorities of their constituents. The main question in retention analysis is whether the starting size of the cohort— number of subscribers or employees, amount spent, or another key metric—will remain constant, decay, or increase over time. When there is an increase or a decrease, the amount and speed of change are also interesting questions. In most retention analyses, the starting size will tend to decay over time, since a cohort can lose but cannot gain new members once it is formed. Revenue is an interesting excep‐ tion, since a cohort of customers can spend more in subsequent months than they did in the first month collectively, even if some of them churn. Retention analysis uses the count of entities or sum of money or actions present in the data set for each period from the starting date, and it normalizes by dividing this number by the count or sum of entities, money, or actions in the first time period. The result is expressed as a percentage, and retention in the starting period is always 100%. Over time, retention based on counts generally declines and can never exceed 100%, whereas money- or action-based retention, while often declining, can increase and be greater than 100% in a time period. Retention analysis output is typically dis‐ played in either table or graph form, which is referred to as a retention curve. We’ll see a number of examples of retention curves later in this chapter. Graphs of retention curves can be used to compare cohorts. The first characteristic to pay attention to", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 98 + }, + { + "text": "analysis output is typically dis‐ played in either table or graph form, which is referred to as a retention curve. We’ll see a number of examples of retention curves later in this chapter. Graphs of retention curves can be used to compare cohorts. The first characteristic to pay attention to is the shape of the curve in the initial few periods, where there is often an initial steep drop. For many consumer apps, losing half a cohort in the first few months is common. A cohort with a curve that is either more or less steep than others can indicate changes in the product or customer acquisition source that merit further investigation. A second characteristic to look for is whether the curve flattens after some number of periods or continues declining rapidly to zero. A flattening curve indicates that there is a point in time from which most of the cohort that remains stays indefinitely. A retention curve that inflects upward, sometimes called a smile curve, can occur if cohort members return or reactivate after falling out of the data set for some period. Finally, retention curves that measure subscription revenue are monitored for signs of increasing revenue per customer over time, a sign of a healthy SaaS software business. Retention | 127 This section will show how to create a retention analysis, add cohort groupings from the time series itself and other tables, and handle missing and sparse data that can occur in time series data. With this framework in hand, you’ll learn in the subsequent section how to make modifications to create the other related types of cohort analysis. As a result, this section on retention will be the longest one in the chapter, as you build up code and develop your intuition about the calculations. SQL for a Basic Retention Curve For retention analysis, as with other cohort analyses, we need three components: the cohort definition, a time series of actions, and an aggregate metric that measures something relevant to the organization or process. In our case, the cohort members will be the legislators, the time series will be the terms in office for each legislator, and the metric of interest will be the count of those who are still in office each period from the starting date. We’ll start by calculating basic retention, before moving on to examples that include various cohort groupings. The first step is to find the first date each legislator took office (first_term). We will use this date to calculate the number of periods for each subsequent date in the time series. To do this, take the min of the term_start and GROUP BY each id_bioguide, the unique identifier for a legislator: SELECT id_bioguide ,min(term_start) as first_term FROM legislators_terms GROUP BY 1 ; id_bioguide first_term ----------- ---------- A000118 1975-01-14 P000281 1933-03-09 K000039 1933-03-09 ... ... The next step is to put this code into a subquery and JOIN it to the time series. The age function is applied to calculate the intervals", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 99 + }, + { + "text": "SELECT id_bioguide ,min(term_start) as first_term FROM legislators_terms GROUP BY 1 ; id_bioguide first_term ----------- ---------- A000118 1975-01-14 P000281 1933-03-09 K000039 1933-03-09 ... ... The next step is to put this code into a subquery and JOIN it to the time series. The age function is applied to calculate the intervals between each term_start and the first_term for each legislator. Applying the date_part functions to the result, with year, transforms this into the number of yearly periods. Since elections happen every two or six years, we’ll use years as the time interval to calculate the periods. We could use a shorter interval, but in this data set there is little fluctuation daily or weekly. The count of legislators with records for that period is the number retained: SELECT date_part('year',age(b.term_start,a.first_term)) as period ,count(distinct a.id_bioguide) as cohort_retained FROM ( SELECT id_bioguide, min(term_start) as first_term 128 | Chapter 4: Cohort Analysis FROM legislators_terms GROUP BY 1 ) a JOIN legislators_terms b on a.id_bioguide = b.id_bioguide GROUP BY 1 ; period cohort_retained ------ --------------- 0.0 12518 1.0 3600 2.0 3619 ... ... In databases that support the datediff function, the date_part and age construction can be replaced by this simpler function: datediff('year',first_term,term_start) Some databases, such as Oracle, place the date_part last: datediff(first_term,term_start,'year' Now that we have the periods and the number of legislators retained in each, the final step is to calculate the total cohort_size and populate it in each row so that the cohort_retained can be divided by it. The first_value window function returns the first record in the PARTITION BY clause, according to the ordering set in the ORDER BY, a convenient way to get the cohort size in each row. In this case, the cohort_size comes from the first record in the entire data set, so the PARTITION BY is omitted: first_value(cohort_retained) over (order by period) as cohort_size To find the percent retained, divide the cohort_retained value by this same calculation: SELECT period ,first_value(cohort_retained) over (order by period) as cohort_size ,cohort_retained ,cohort_retained / first_value(cohort_retained) over (order by period) as pct_retained FROM ( SELECT date_part('year',age(b.term_start,a.first_term)) as period ,count(distinct a.id_bioguide) as cohort_retained FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms GROUP BY 1 ) a JOIN legislators_terms b on a.id_bioguide = b.id_bioguide Retention | 129 GROUP BY 1 ) aa ; period cohort_size cohort_retained pct_retained ------ ----------- --------------- ------------ 0.0 12518 12518 1.0000 1.0 12518 3600 0.2876 2.0 12518 3619 0.2891 ... ... ... ... We now have a retention calculation, and we can see that there is a big drop-off between the 100% of legislators retained in period 0, or on their start date, and the share with another term record that starts a year later. Graphing the results, as in Figure 4-3, demonstrates how the curve flattens and eventually goes to zero, as even the longest-serving legislators eventually retire or die. Figure 4-3. Retention from start of first term for US legislators We can take the cohort retention result and reshape the data to show it in table for‐ mat. Pivot", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 100 + }, + { + "text": "4-3, demonstrates how the curve flattens and eventually goes to zero, as even the longest-serving legislators eventually retire or die. Figure 4-3. Retention from start of first term for US legislators We can take the cohort retention result and reshape the data to show it in table for‐ mat. Pivot and flatten the results using an aggregate function with a CASE statement; max is used in this example, but other aggregations such as min or avg would return the same result. Retention is calculated for years 0 through 4, but additional years can be added by following the same pattern: SELECT cohort_size ,max(case when period = 0 then pct_retained end) as yr0 ,max(case when period = 1 then pct_retained end) as yr1 ,max(case when period = 2 then pct_retained end) as yr2 ,max(case when period = 3 then pct_retained end) as yr3 ,max(case when period = 4 then pct_retained end) as yr4 130 | Chapter 4: Cohort Analysis FROM ( SELECT period ,first_value(cohort_retained) over (order by period) as cohort_size ,cohort_retained / first_value(cohort_retained) over (order by period) as pct_retained FROM ( SELECT date_part('year',age(b.term_start,a.first_term)) as period ,count(*) as cohort_retained FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms GROUP BY 1 ) a JOIN legislators_terms b on a.id_bioguide = b.id_bioguide GROUP BY 1 ) aa ) aaa GROUP BY 1 ; cohort_size yr0 yr1 yr2 yr3 yr4 ----------- ------ ------ ------ ------ ------ 12518 1.0000 0.2876 0.2891 0.1463 0.2564 Retention appears to be quite low, and from the graph we can see that it is jagged in the first few years. One reason for this is that a representative’s term lasts two years, and senators’ terms last six years, but the data set only contains records for the start of new terms; thus we are missing data for years in which a legislator was still in office but did not start a new term. Measuring retention each year is misleading in this case. One option is to measure retention only on a two- or six-year cycle, but there is also another strategy we can employ to fill in the “missing” data. I will cover this next before returning to the topic of forming cohort groups. Adjusting Time Series to Increase Retention Accuracy We discussed techniques for cleaning “missing” data in Chapter 2, and we will turn to those techniques in this section in order to arrive at a smoother and more truthful retention curve for the legislators. When working with time series data, such as in cohort analysis, it’s important to consider not only the data that is present but also whether that data accurately reflects the presence or absence of entities at each time period. This is particularly a problem in contexts in which an event captured in the data leads to the entity persisting for some period of time that is not captured in the data. For example, a customer buying a software subscription is represented in Retention | 131 the data at the time of the transaction, but that", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 101 + }, + { + "text": "contexts in which an event captured in the data leads to the entity persisting for some period of time that is not captured in the data. For example, a customer buying a software subscription is represented in Retention | 131 the data at the time of the transaction, but that customer is entitled to use the soft‐ ware for months or years and is not necessarily represented in the data over that span. To correct for this, we need a way to derive the span of time in which the entity is still present, either with an explicit end date or with knowledge of the length of the sub‐ scription or term. Then we can say that the entity was present at any date in between those start and end dates. In the legislators data set, we have a record for a term’s start date, but we are missing the notion that this “entitles” a legislator to serve for two or six years, depending on the chamber. To correct for this and smooth out the curve, we need to fill in the “missing” values for the years that legislators are still in office between new terms. Since this data set includes a term_end value for each term, I’ll show how to create a more accurate cohort retention analysis by filling in dates between the start and end values. Then I’ll show how you can impute end dates when the data set does not include an end date. Calculating retention using a start and end date defined in the data is the most accu‐ rate approach. For the following examples, we will consider legislators retained in a particular year if they were still in office as of the last day of the year, December 31. Prior to the Twentieth Amendment to the US Constitution, terms began on March 4, but afterward the start date moved to January 3, or to a subsequent weekday if the third falls on a weekend. Legislators can be sworn in on other days of the year due to special off-cycle elections or appointments to fill vacant seats. As a result, term_start dates cluster in January but are spread across the year. While we could pick another day, December 31 is a strategy for normalizing around these varying start dates. The first step is to create a data set that contains a record for each December 31 that each legislator was in office. This can be accomplished by JOINing the subquery that found the first_term to the legislators_terms table to find the term_start and term_end for each term. A second JOIN to the date_dim retrieves the dates that fall between the start and end dates, restricting the returned values to c.month_name = 'December' and c.day_of_month = 31. The period is calculated as the years between the date from the date_dim and the first_term. Note that even though more than 11 months may have elapsed between being sworn in in January and December 31, the first year still", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 102 + }, + { + "text": "returned values to c.month_name = 'December' and c.day_of_month = 31. The period is calculated as the years between the date from the date_dim and the first_term. Note that even though more than 11 months may have elapsed between being sworn in in January and December 31, the first year still appears as 0: SELECT a.id_bioguide, a.first_term ,b.term_start, b.term_end ,c.date ,date_part('year',age(c.date,a.first_term)) as period FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms GROUP BY 1 ) a 132 | Chapter 4: Cohort Analysis JOIN legislators_terms b on a.id_bioguide = b.id_bioguide LEFT JOIN date_dim c on c.date between b.term_start and b.term_end and c.month_name = 'December' and c.day_of_month = 31 ; id_bioguide first_term term_start term_end date period ----------- ---------- ---------- ---------- ---------- ------ B000944 1993-01-05 1993-01-05 1995-01-03 1993-12-31 0.0 B000944 1993-01-05 1993-01-05 1995-01-03 1994-12-31 1.0 C000127 1993-01-05 1993-01-05 1995-01-03 1993-12-31 0.0 ... ... ... ... ... ... If a date dimension is not available, you can create a subquery with the necessary dates in a couple of ways. If your database supports the generate_series, you can create a subquery that returns the desired dates: SELECT generate_series::date as date FROM generate_series('1770-12-31','2020-12- 31',interval '1 year') You may want to save this as a table or view for later use. Alterna‐ tively, you can query the data set or any other table in the database that has a full set of dates. In this case, the table has all of the neces‐ sary years, but we will make a December 31 date for each year using the make_date function: SELECT distinct make_date(date_part('year',term_start)::int,12,31) FROM legislators_terms There are a number of creative ways to get the series of dates needed. Use whichever method is available and simplest within your queries. We now have a row for each date (year end) for which we would like to calculate retention. The next step is to calculate the cohort_retained for each period, which is done with a count of id_bioguide. A coalesce function is used on period to set a default value of 0 when null. This handles the cases in which a legislator’s term starts and ends in the same year, giving credit for serving in that year: SELECT coalesce(date_part('year',age(c.date,a.first_term)),0) as period ,count(distinct a.id_bioguide) as cohort_retained FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms GROUP BY 1 ) a JOIN legislators_terms b on a.id_bioguide = b.id_bioguide Retention | 133 LEFT JOIN date_dim c on c.date between b.term_start and b.term_end and c.month_name = 'December' and c.day_of_month = 31 GROUP BY 1 ; period cohort_retained ------ --------------- 0.0 12518 1.0 12328 2.0 8166 ... ... The final step is to calculate the cohort_size and pct_retained as we did previously using first_value window functions: SELECT period ,first_value(cohort_retained) over (order by period) as cohort_size ,cohort_retained ,cohort_retained * 1.0 / first_value(cohort_retained) over (order by period) as pct_retained FROM ( SELECT coalesce(date_part('year',age(c.date,a.first_term)),0) as period ,count(distinct a.id_bioguide) as cohort_retained FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms GROUP BY 1 ) a JOIN legislators_terms b on a.id_bioguide = b.id_bioguide LEFT JOIN date_dim c on c.date", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 103 + }, + { + "text": "cohort_size ,cohort_retained ,cohort_retained * 1.0 / first_value(cohort_retained) over (order by period) as pct_retained FROM ( SELECT coalesce(date_part('year',age(c.date,a.first_term)),0) as period ,count(distinct a.id_bioguide) as cohort_retained FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms GROUP BY 1 ) a JOIN legislators_terms b on a.id_bioguide = b.id_bioguide LEFT JOIN date_dim c on c.date between b.term_start and b.term_end and c.month_name = 'December' and c.day_of_month = 31 GROUP BY 1 ) aa ; period cohort_size cohort_retained pct_retained ------ ----------- --------------- ------------ 0.0 12518 12518 1.0000 1.0 12518 12328 0.9848 2.0 12518 8166 0.6523 ... ... ... ... The results, graphed in Figure 4-4, are now much more accurate. Almost all legisla‐ tors are still in office in year 1, and the first big drop-off occurs in year 2, when some representatives will fail to be reelected. 134 | Chapter 4: Cohort Analysis Figure 4-4. Legislator retention after adjusting for actual years in office If the data set does not contain an end date, there are a couple of options for imputing one. One option is to add a fixed interval to the start date, when the length of a sub‐ scription or term is known. This can be done with date math by adding a constant interval to the term_start. Here, a CASE statement handles the addition for the two term_types: SELECT a.id_bioguide, a.first_term ,b.term_start ,case when b.term_type = 'rep' then b.term_start + interval '2 years' when b.term_type = 'sen' then b.term_start + interval '6 years' end as term_end FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms GROUP BY 1 ) a JOIN legislators_terms b on a.id_bioguide = b.id_bioguide ; id_bioguide first_term term_start term_end ----------- ---------- ---------- ---------- B000944 1993-01-05 1993-01-05 1995-01-05 C000127 1993-01-05 1993-01-05 1995-01-05 Retention | 135 C000141 1987-01-06 1987-01-06 1989-01-06 ... ... ... ... This block of code can then be plugged into the retention code to derive the period and pct_retained. The drawback to this method is that it fails to capture instances in which a legislator did not complete a full term, which can happen in the event of death or appointment to a higher office. A second option is to use the subsequent starting date, minus one day, as the term_end date. This can be calculated with the lead window function. This function is similar to the lag function we’ve used previously, but rather than returning a value from a row earlier in the partition, it returns a value from a row later in the partition, as determined in the ORDER BY clause. The default is one row, which we will use here, but the function has an optional argument indicating a different number of rows. Here we find the term_start date of the subsequent term using lead and then subtract the interval '1 day' to derive the term_end: SELECT a.id_bioguide, a.first_term ,b.term_start ,lead(b.term_start) over (partition by a.id_bioguide order by b.term_start) - interval '1 day' as term_end FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms GROUP BY 1 ) a JOIN legislators_terms b on a.id_bioguide = b.id_bioguide", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 104 + }, + { + "text": "and then subtract the interval '1 day' to derive the term_end: SELECT a.id_bioguide, a.first_term ,b.term_start ,lead(b.term_start) over (partition by a.id_bioguide order by b.term_start) - interval '1 day' as term_end FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms GROUP BY 1 ) a JOIN legislators_terms b on a.id_bioguide = b.id_bioguide ; id_bioguide first_term term_start term_end ----------- ---------- ---------- ---------- A000001 1951-01-03 1951-01-03 (null) A000002 1947-01-03 1947-01-03 1949-01-02 A000002 1947-01-03 1949-01-03 1951-01-02 ... ... ... ... This code block can then be plugged into the retention code. This method has a cou‐ ple of drawbacks. First, when there is no subsequent term, the lead function returns null, leaving that term without a term_end. A default value, such as a default interval shown in the last example, could be used in such cases. The second drawback is that this method assumes that terms are always consecutive, with no time spent out of office. Although most legislators tend to serve continuously until their congressional careers end, there are certainly examples of gaps between terms spanning several years. Any time we make adjustments to fill in missing data, we need to be careful about the assumptions we make. In subscription- or term-based contexts, explicit start and end 136 | Chapter 4: Cohort Analysis dates tend to be most accurate. Either of the two other methods shown—adding a fixed interval or setting the end date relative to the next start date—can be used when no end date is present and we have a reasonable expectation that most customers or users will stay for the duration assumed. Now that we’ve seen how to calculate a basic retention curve and correct for missing dates, we can start adding in cohort groups. Comparing retention between different groups is one of the main reasons to do cohort analysis. Next, I’ll discuss forming groups from the time series itself, and after that, I’ll discuss forming cohort groups from data in other tables. Cohorts Derived from the Time Series Itself Now that we have SQL code to calculate retention, we can start to split the entities into cohorts. In this section, I will show how to derive cohort groupings from the time series itself. First I’ll discuss time-based cohorts based on the first date, and I’ll explain how to make cohorts based on other attributes from the time series. The most common way to create the cohorts is based on the first or minimum date or time that the entity appears in the time series. This means that only one table is nec‐ essary for the cohort retention analysis: the time series itself. Cohorting by the first appearance or action is interesting because often groups that start at different times behave differently. For consumer services, early adopters are often more enthusiastic and retain differently than later adopters, whereas in SaaS software, later adopters may retain better because the product is more mature. Time-based cohorts can be grouped by any time granularity that is meaningful to the organization, though weekly, monthly, or", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 105 + }, + { + "text": "differently. For consumer services, early adopters are often more enthusiastic and retain differently than later adopters, whereas in SaaS software, later adopters may retain better because the product is more mature. Time-based cohorts can be grouped by any time granularity that is meaningful to the organization, though weekly, monthly, or yearly cohorts are common. If you’re not sure what grouping to use, try running the cohort analysis with different groupings, without making the cohort sizes too small, to see where meaningful patterns emerge. Fortunately, once you know how to construct the cohorts and retention analysis, substituting different time granularities is straightforward. The first example will use yearly cohorts, and then I will demonstrate swapping in centuries. The key question we will consider is whether the era in which a legislator first took office has any correlation with their retention. Political trends and the pub‐ lic mood do change over time, but by how much? To calculate yearly cohorts, we first add the year of the first_term calculated previ‐ ously to the query that finds the period and cohort_retained: SELECT date_part('year',a.first_term) as first_year ,coalesce(date_part('year',age(c.date,a.first_term)),0) as period ,count(distinct a.id_bioguide) as cohort_retained FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms Retention | 137 GROUP BY 1 ) a JOIN legislators_terms b on a.id_bioguide = b.id_bioguide LEFT JOIN date_dim c on c.date between b.term_start and b.term_end and c.month_name = 'December' and c.day_of_month = 31 GROUP BY 1,2 ; first_year period cohort_retained ---------- ------ --------------- 1789.0 0.0 89 1789.0 2.0 89 1789.0 3.0 57 ... ... ... This query is then used as the subquery, and the cohort_size and pct_retained are calculated in the outer query as previously. In this case, however, we need a PARTI‐ TION BY clause that includes first_year so that the first_value is calculated only within the set of rows for that first_year, rather than across the whole result set from the subquery: SELECT first_year, period ,first_value(cohort_retained) over (partition by first_year order by period) as cohort_size ,cohort_retained ,cohort_retained / first_value(cohort_retained) over (partition by first_year order by period) as pct_retained FROM ( SELECT date_part('year',a.first_term) as first_year ,coalesce(date_part('year',age(c.date,a.first_term)),0) as period ,count(distinct a.id_bioguide) as cohort_retained FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms GROUP BY 1 ) a JOIN legislators_terms b on a.id_bioguide = b.id_bioguide LEFT JOIN date_dim c on c.date between b.term_start and b.term_end and c.month_name = 'December' and c.day_of_month = 31 GROUP BY 1,2 ) aa ; first_year period cohort_size cohort_retained pct_retained ---------- ------ ----------- --------------- ------------ 1789.0 0.0 89 89 1.0000 1789.0 2.0 89 89 1.0000 1789.0 3.0 89 57 0.6404 ... ... ... ... ... 138 | Chapter 4: Cohort Analysis This data set includes over two hundred starting years, too many to easily graph or examine in a table. Next we’ll look at a less granular interval and cohort the legislators by the century of the first_term. This change is easily made by substituting century for year in the date_part function in subquery aa. Recall that century names are off‐ set from the years they represent, so", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 106 + }, + { + "text": "a table. Next we’ll look at a less granular interval and cohort the legislators by the century of the first_term. This change is easily made by substituting century for year in the date_part function in subquery aa. Recall that century names are off‐ set from the years they represent, so that the 18th century lasted from 1700 to 1799, the 19th century lasted from 1800 to 1899, and so on. The partitioning in the first_value function changes to the first_century field: SELECT first_century, period ,first_value(cohort_retained) over (partition by first_century order by period) as cohort_size ,cohort_retained ,cohort_retained / first_value(cohort_retained) over (partition by first_century order by period) as pct_retained FROM ( SELECT date_part('century',a.first_term) as first_century ,coalesce(date_part('year',age(c.date,a.first_term)),0) as period ,count(distinct a.id_bioguide) as cohort_retained FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms GROUP BY 1 ) a JOIN legislators_terms b on a.id_bioguide = b.id_bioguide LEFT JOIN date_dim c on c.date between b.term_start and b.term_end and c.month_name = 'December' and c.day_of_month = 31 GROUP BY 1,2 ) aa ORDER BY 1,2 ; first_century period cohort_size cohort_retained pct_retained ------------- ------ ----------- --------------- ------------ 18.0 0.0 368 368 1.0000 18.0 1.0 368 360 0.9783 18.0 2.0 368 242 0.6576 ... ... ... ... ... The results are graphed in Figure 4-5. Retention in the early years has been higher for those first elected in the 20th or 21st century. The 21st century is still underway, and thus many of those legislators have not had the opportunity to stay in office for five or more years, though they are still included in the denominator. We might want to con‐ sider removing the 21st century from the analysis, but I’ve left it here to demonstrate how the retention curve drops artificially due to this circumstance. Retention | 139 Figure 4-5. Legislator retention by century in which first term began Cohorts can be defined from other attributes in a time series besides the first date, with options depending on the values in the table. The legislators_terms table has a state field, indicating which state the person is representing for that term. We can use this to create cohorts, and we will base them on the first state in order to ensure that anyone who has represented multiple states appears in the data only once. When cohorting on an attribute that can change over time, it’s important to ensure that each entity is assigned only one value. Otherwise the entity may be represented in multiple cohorts, intro‐ ducing bias into the analysis. Usually the value from the earliest record in the data set is used. To find the first state for each legislator, we can use the first_value window func‐ tion. In this example, we’ll also turn the min function into a window function to avoid a lengthy GROUP BY clause: SELECT distinct id_bioguide ,min(term_start) over (partition by id_bioguide) as first_term ,first_value(state) over (partition by id_bioguide order by term_start) as first_state FROM legislators_terms ; 140 | Chapter 4: Cohort Analysis id_bioguide first_term first_state ----------- ---------- ----------- C000001 1893-08-07 GA R000584", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 107 + }, + { + "text": "into a window function to avoid a lengthy GROUP BY clause: SELECT distinct id_bioguide ,min(term_start) over (partition by id_bioguide) as first_term ,first_value(state) over (partition by id_bioguide order by term_start) as first_state FROM legislators_terms ; 140 | Chapter 4: Cohort Analysis id_bioguide first_term first_state ----------- ---------- ----------- C000001 1893-08-07 GA R000584 2009-01-06 ID W000215 1975-01-14 CA ... ... ... We can then plug this code into our retention code to find the retention by first_state: SELECT first_state, period ,first_value(cohort_retained) over (partition by first_state order by period) as cohort_size ,cohort_retained ,cohort_retained / first_value(cohort_retained) over (partition by first_state order by period) as pct_retained FROM ( SELECT a.first_state ,coalesce(date_part('year',age(c.date,a.first_term)),0) as period ,count(distinct a.id_bioguide) as cohort_retained FROM ( SELECT distinct id_bioguide ,min(term_start) over (partition by id_bioguide) as first_term ,first_value(state) over (partition by id_bioguide order by term_start) as first_state FROM legislators_terms ) a JOIN legislators_terms b on a.id_bioguide = b.id_bioguide LEFT JOIN date_dim c on c.date between b.term_start and b.term_end and c.month_name = 'December' and c.day_of_month = 31 GROUP BY 1,2 ) aa ; first_state period cohort_size cohort_retained pct_retained ----------- ------ ----------- --------------- ------------ AK 0.0 19 19 1.0000 AK 1.0 19 19 1.0000 AK 2.0 19 15 0.7895 ... ... ... ... ... The retention curves for the five states with the highest total number of legislators are graphed in Figure 4-6. Those elected in Illinois and Massachusetts have the highest retention, while New Yorkers have the lowest retention. Determining the reasons why would be an interesting offshoot of this analysis. Retention | 141 Figure 4-6. Legislator retention by first state: top five states by total legislators Defining cohorts from the time series is relatively straightforward using a min date for each entity and then converting that date into a month, year, or century as appropri‐ ate for the analysis. Switching between month and year or other levels of granularity also is straightforward, allowing for multiple options to be tested in order to find a grouping that is meaningful for the organization. Other attributes can be used for cohorting with the first_value window function. Next, we’ll turn to cases in which the cohorting attribute comes from a table other than that of the time series. Defining the Cohort from a Separate Table Often the characteristics that define a cohort exist in a table separate from the one that contains the time series. For example, a database might have a customer table with information such as acquisition source or registration date by which customers can be cohorted. Adding in attributes from other tables, or even subqueries, is rela‐ tively straightforward and can be done in retention analysis and related analyses dis‐ cussed later in the chapter. For this example, we’ll consider whether the gender of the legislator has any impact on their retention. The legislators table has a gender field, where F means female and M means male, that we can use to cohort the legislators. To do this, we’ll JOIN the 142 | Chapter 4: Cohort Analysis legislators table in as alias d to add", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 108 + }, + { + "text": "legislator has any impact on their retention. The legislators table has a gender field, where F means female and M means male, that we can use to cohort the legislators. To do this, we’ll JOIN the 142 | Chapter 4: Cohort Analysis legislators table in as alias d to add gender to the calculation of cohort_retained, in place of year or century: SELECT d.gender ,coalesce(date_part('year',age(c.date,a.first_term)),0) as period ,count(distinct a.id_bioguide) as cohort_retained FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms GROUP BY 1 ) a JOIN legislators_terms b on a.id_bioguide = b.id_bioguide LEFT JOIN date_dim c on c.date between b.term_start and b.term_end and c.month_name = 'December' and c.day_of_month = 31 JOIN legislators d on a.id_bioguide = d.id_bioguide GROUP BY 1,2 ; gender period cohort_retained ------ ------ --------------- F 0.0 366 M 0.0 12152 F 1.0 349 M 1.0 11979 ... ... ... It’s immediately clear that many more males than females have served legislative terms. We can now calculate the percent_retained so we can compare the retention for these groups: SELECT gender, period ,first_value(cohort_retained) over (partition by gender order by period) as cohort_size ,cohort_retained ,cohort_retained/ first_value(cohort_retained) over (partition by gender order by period) as pct_retained FROM ( SELECT d.gender ,coalesce(date_part('year',age(c.date,a.first_term)),0) as period ,count(distinct a.id_bioguide) as cohort_retained FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms GROUP BY 1 ) a JOIN legislators_terms b on a.id_bioguide = b.id_bioguide LEFT JOIN date_dim c on c.date between b.term_start and b.term_end and c.month_name = 'December' and c.day_of_month = 31 Retention | 143 JOIN legislators d on a.id_bioguide = d.id_bioguide GROUP BY 1,2 ) aa ; gender period cohort_size cohort_retained pct_retained ------ ------ ----------- --------------- ------------ F 0.0 366 366 1.0000 M 0.0 12152 12152 1.0000 F 1.0 366 349 0.9536 M 1.0 12152 11979 0.9858 ... ... ... ... ... We can see from the results graphed in Figure 4-7 that retention is higher for female legislators than for their male counterparts for periods 2 through 29. The first female legislator did not take office until 1917, when Jeannette Rankin joined the House as a Republican representative from Montana. As we saw earlier, retention has increased in more recent centuries. Figure 4-7. Legislator retention by gender To make a more fair comparison, we might restrict the legislators included in the analysis to only those whose first_term started since there have been women in Congress. We can do this by adding a WHERE filter to subquery aa. Here the results are also restricted to those who started before 2000, to ensure the cohorts have had at least 20 possible years to stay in office: 144 | Chapter 4: Cohort Analysis SELECT gender, period ,first_value(cohort_retained) over (partition by gender order by period) as cohort_size ,cohort_retained ,cohort_retained / first_value(cohort_retained) over (partition by gender order by period) as pct_retained FROM ( SELECT d.gender ,coalesce(date_part('year',age(c.date,a.first_term)),0) as period ,count(distinct a.id_bioguide) as cohort_retained FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms GROUP BY 1 ) a JOIN legislators_terms b on a.id_bioguide = b.id_bioguide LEFT JOIN date_dim c on c.date between", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 109 + }, + { + "text": "/ first_value(cohort_retained) over (partition by gender order by period) as pct_retained FROM ( SELECT d.gender ,coalesce(date_part('year',age(c.date,a.first_term)),0) as period ,count(distinct a.id_bioguide) as cohort_retained FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms GROUP BY 1 ) a JOIN legislators_terms b on a.id_bioguide = b.id_bioguide LEFT JOIN date_dim c on c.date between b.term_start and b.term_end and c.month_name = 'December' and c.day_of_month = 31 JOIN legislators d on a.id_bioguide = d.id_bioguide WHERE a.first_term between '1917-01-01' and '1999-12-31' GROUP BY 1,2 ) aa ; gender period cohort_size cohort_retained pct_retained ------ ------ ----------- --------------- ------------ F 0.0 200 200 1.0000 M 0.0 3833 3833 1.0000 F 1.0 200 187 0.9350 M 1.0 3833 3769 0.9833 ... ... ... ... ... Male legislators still outnumber female legislators, but by a smaller margin. The retention for the cohorts is graphed in Figure 4-8. With the revised cohorts, male leg‐ islators have higher retention through year 7, but starting in year 12, female legisla‐ tors have higher retention. The difference between the two gender-based cohort analyses underscores the importance of setting up appropriate cohorts and ensuring that they have comparable amounts of time to be present or complete other actions of interest. To further improve this analysis, we could cohort by both starting year or decade and gender, in order to control for additional changes in retention through the 20th century and into the 21st century. Retention | 145 Figure 4-8. Legislator retention by gender: cohorts from 1917 to 1999 Cohorts can be defined in multiple ways, from the time series and from other tables. With the framework we’ve developed, subqueries, views, or other derived tables can be swapped in, opening up a whole range of calculations to be the basis of a cohort. Multiple criteria, such as starting year and gender, can be used. One caution when dividing populations into cohorts based on multiple criteria is that this can lead to sparse cohorts, where some of the defined groups are too small and are not repre‐ sented in the data set for all time periods. The next section will discuss methods for overcoming this challenge. Dealing with Sparse Cohorts In the ideal data set, every cohort has some action or record in the time series for every period of interest. We’ve already seen how “missing” dates can occur due to subscriptions or terms lasting over multiple periods, and we looked at how to correct for them using a date dimension to infer intermediate dates. Another issue can arise when, due to grouping criteria, the cohort becomes too small and as a result is repre‐ sented only sporadically in the data. A cohort may disappear from the result set, when we would prefer it to appear with a zero retention value. This problem is called sparse cohorts, and it can be worked around with the careful use of LEFT JOINs. To demonstrate this, let’s attempt to cohort female legislators by the first state they represented to see if there are any differences in retention. We’ve already seen that", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 110 + }, + { + "text": "zero retention value. This problem is called sparse cohorts, and it can be worked around with the careful use of LEFT JOINs. To demonstrate this, let’s attempt to cohort female legislators by the first state they represented to see if there are any differences in retention. We’ve already seen that 146 | Chapter 4: Cohort Analysis there have been relatively few female legislators. Cohorting them further by state is highly likely to create some sparse cohorts in which there are very few members. Before making code adjustments, let’s add first_state (calculated in the section on deriving cohorts from the time series) into our previous gender example and look at the results: SELECT first_state, gender, period ,first_value(cohort_retained) over (partition by first_state, gender order by period) as cohort_size ,cohort_retained ,cohort_retained / first_value(cohort_retained) over (partition by first_state, gender order by period) as pct_retained FROM ( SELECT a.first_state, d.gender ,coalesce(date_part('year',age(c.date,a.first_term)),0) as period ,count(distinct a.id_bioguide) as cohort_retained FROM ( SELECT distinct id_bioguide ,min(term_start) over (partition by id_bioguide) as first_term ,first_value(state) over (partition by id_bioguide order by term_start) as first_state FROM legislators_terms ) a JOIN legislators_terms b on a.id_bioguide = b.id_bioguide LEFT JOIN date_dim c on c.date between b.term_start and b.term_end and c.month_name = 'December' and c.day_of_month = 31 JOIN legislators d on a.id_bioguide = d.id_bioguide WHERE a.first_term between '1917-01-01' and '1999-12-31' GROUP BY 1,2,3 ) aa ; first_state gender period cohort_size cohort_retained pct_retained ----------- ------ ------ ----------- --------------- ------------ AZ F 0.0 2 2 1.0000 AZ M 0.0 26 26 1.0000 AZ F 1.0 2 2 1.0000 ... ... ... ... ... ... Graphing the results for the first 20 periods, as in Figure 4-9, reveals the sparse cohorts. Alaska did not have any female legislators, while Arizona’s female retention curve disappears after year 3. Only California, a large state with many legislators, has complete retention curves for both genders. This pattern repeats for other small and large states. Retention | 147 Figure 4-9. Legislator retention by gender and first state Now let’s look at how to ensure a record for every period so that the query returns zero values for retention instead of nulls. The first step is to query for all combina‐ tions of periods and cohort attributes, in this case first_state and gender, with the starting cohort_size for each combination. This can be done by JOINing subquery aa, which calculates the cohort, with a generate_series subquery that returns all integers from 0 to 20, with the criteria on 1 = 1. This is a handy way to force a Carte‐ sian JOIN when the two subqueries don’t have any fields in common: SELECT aa.gender, aa.first_state, cc.period, aa.cohort_size FROM ( SELECT b.gender, a.first_state ,count(distinct a.id_bioguide) as cohort_size FROM ( SELECT distinct id_bioguide ,min(term_start) over (partition by id_bioguide) as first_term ,first_value(state) over (partition by id_bioguide order by term_start) as first_state FROM legislators_terms ) a JOIN legislators b on a.id_bioguide = b.id_bioguide WHERE a.first_term between '1917-01-01' and '1999-12-31' GROUP BY 1,2 148 | Chapter 4: Cohort Analysis ) aa JOIN ( SELECT generate_series as period", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 111 + }, + { + "text": "over (partition by id_bioguide) as first_term ,first_value(state) over (partition by id_bioguide order by term_start) as first_state FROM legislators_terms ) a JOIN legislators b on a.id_bioguide = b.id_bioguide WHERE a.first_term between '1917-01-01' and '1999-12-31' GROUP BY 1,2 148 | Chapter 4: Cohort Analysis ) aa JOIN ( SELECT generate_series as period FROM generate_series(0,20,1) ) cc on 1 = 1 ; gender state period cohort ------ ----- ------ ------ F AL 0 3 F AL 1 3 F AL 2 3 ... ... ... ... The next step is to JOIN this back to the actual periods in office, with a LEFT JOIN to ensure all the time periods remain in the final result: SELECT aaa.gender, aaa.first_state, aaa.period, aaa.cohort_size ,coalesce(ddd.cohort_retained,0) as cohort_retained ,coalesce(ddd.cohort_retained,0) / aaa.cohort_size as pct_retained FROM ( SELECT aa.gender, aa.first_state, cc.period, aa.cohort_size FROM ( SELECT b.gender, a.first_state ,count(distinct a.id_bioguide) as cohort_size FROM ( SELECT distinct id_bioguide ,min(term_start) over (partition by id_bioguide) as first_term ,first_value(state) over (partition by id_bioguide order by term_start) as first_state FROM legislators_terms ) a JOIN legislators b on a.id_bioguide = b.id_bioguide WHERE a.first_term between '1917-01-01' and '1999-12-31' GROUP BY 1,2 ) aa JOIN ( SELECT generate_series as period FROM generate_series(0,20,1) ) cc on 1 = 1 ) aaa LEFT JOIN ( SELECT d.first_state, g.gender ,coalesce(date_part('year',age(f.date,d.first_term)),0) as period ,count(distinct d.id_bioguide) as cohort_retained Retention | 149 FROM ( SELECT distinct id_bioguide ,min(term_start) over (partition by id_bioguide) as first_term ,first_value(state) over (partition by id_bioguide order by term_start) as first_state FROM legislators_terms ) d JOIN legislators_terms e on d.id_bioguide = e.id_bioguide LEFT JOIN date_dim f on f.date between e.term_start and e.term_end and f.month_name = 'December' and f.day_of_month = 31 JOIN legislators g on d.id_bioguide = g.id_bioguide WHERE d.first_term between '1917-01-01' and '1999-12-31' GROUP BY 1,2,3 ) ddd on aaa.gender = ddd.gender and aaa.first_state = ddd.first_state and aaa.period = ddd.period ; gender first_state period cohort_size cohort_retained pct_retained ------ ----------- ------ ----------- --------------- ------------ F AL 0 3 3 1.0000 F AL 1 3 1 0.3333 F AL 2 3 0 0.0000 ... ... ... ... ... ... We can then pivot the results and confirm that a value exists for each cohort for each period: gender first_state yr0 yr2 yr4 yr6 yr8 yr10 ------ ----------- ----- ------ ------ ------ ------ ------ F AL 1.000 0.0000 0.0000 0.0000 0.0000 0.0000 F AR 1.000 0.8000 0.2000 0.4000 0.4000 0.4000 F CA 1.000 0.9200 0.8000 0.6400 0.6800 0.6800 ... ... ... ... ... ... ... ... Notice that at this point, the SQL code has gotten quite long. One of the harder parts of writing SQL for cohort retention analysis is keeping all of the logic straight and the code organized, a topic I’ll discuss more in Chapter 8. When building up retention code, I find it helpful to go step-by-step, checking results along the way. I also spot- check individual cohorts to validate that the final result is accurate. Cohorts can be defined in many ways. So far, we’ve normalized all our cohorts to the first date they appear in the time series data.", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 112 + }, + { + "text": "find it helpful to go step-by-step, checking results along the way. I also spot- check individual cohorts to validate that the final result is accurate. Cohorts can be defined in many ways. So far, we’ve normalized all our cohorts to the first date they appear in the time series data. This isn’t the only option, however, and interesting analysis can be done starting in the middle of an entity’s life span. Before concluding our work on retention analysis, let’s take a look at this additional way to define cohorts. 150 | Chapter 4: Cohort Analysis Defining Cohorts from Dates Other Than the First Date Usually time-based cohorts are defined from the entity’s first appearance in the time series or from some other earliest date, such as a registration date. However, cohort‐ ing on a different date can be useful and insightful. For example, we might want to look at retention across all customers using a service as of a particular date. This type of analysis can be used to understand whether product or marketing changes have had a long-term impact on existing customers. When using a date other than the first date, we need to take care to precisely define the criteria for inclusion in each cohort. One option is to pick entities present on a particular calendar date. This is relatively straightforward to put into SQL code, but it can be problematic if a large share of the regular user population doesn’t show up every day, causing retention to vary depending on the exact day chosen. One option to correct for this is to calculate retention for several starting dates and then average the results. Another option is to use a window of time such as a week or month. Any entity that appears in the data set during that window is included in the cohort. While this approach is often more representative of the business or process, the trade-off is that the SQL code will become more complex, and the query time may be slower due to more intense database calculations. Finding the right balance between query perfor‐ mance and accuracy of results is something of an art. Let’s take a look at how to calculate such midstream analysis with the legislators data set by considering retention for legislators who were in office in the year 2000. We’ll cohort by the term_type, which has values of “sen” for senators and “rep” for repre‐ sentatives. The definition will include any legislator in office at any time during the year 2000: those who started prior to 2000 and whose terms ended during or after 2000 qualify, as do those who started a term in 2000. We can hardcode any date in 2000 as the first_term, since we will later check whether they were in office at some point during 2000. The min_start of the terms falling in this window is also calcula‐ ted for use in a later step: SELECT distinct id_bioguide, term_type, date('2000-01-01') as first_term ,min(term_start) as min_start FROM", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 113 + }, + { + "text": "in 2000 as the first_term, since we will later check whether they were in office at some point during 2000. The min_start of the terms falling in this window is also calcula‐ ted for use in a later step: SELECT distinct id_bioguide, term_type, date('2000-01-01') as first_term ,min(term_start) as min_start FROM legislators_terms WHERE term_start <= '2000-12-31' and term_end >= '2000-01-01' GROUP BY 1,2,3 ; id_bioguide term_type first_term min_start ----------- --------- ---------- --------- C000858 sen 2000-01-01 1997-01-07 G000333 sen 2000-01-01 1995-01-04 M000350 rep 2000-01-01 1999-01-06 ... ... ... ... Retention | 151 We can then plug this into our retention code, with two adjustments. First, an addi‐ tional JOIN criteria between subquery a and the legislators_terms table is added in order to return only terms that started on or after the min_start date. Second, an additional filter is added to the date_dim so that it only returns dates in 2000 or later: SELECT term_type, period ,first_value(cohort_retained) over (partition by term_type order by period) as cohort_size ,cohort_retained ,cohort_retained / first_value(cohort_retained) over (partition by term_type order by period) as pct_retained FROM ( SELECT a.term_type ,coalesce(date_part('year',age(c.date,a.first_term)),0) as period ,count(distinct a.id_bioguide) as cohort_retained FROM ( SELECT distinct id_bioguide, term_type ,date('2000-01-01') as first_term ,min(term_start) as min_start FROM legislators_terms WHERE term_start <= '2000-12-31' and term_end >= '2000-01-01' GROUP BY 1,2,3 ) a JOIN legislators_terms b on a.id_bioguide = b.id_bioguide and b.term_start >= a.min_start LEFT JOIN date_dim c on c.date between b.term_start and b.term_end and c.month_name = 'December' and c.day_of_month = 31 and c.year >= 2000 GROUP BY 1,2 ) aa ; term_type period cohort_size cohort_retained pct_retained --------- ------ ----------- --------------- ------------ rep 0.0 440 440 1.0000 sen 0.0 101 101 1.0000 rep 1.0 440 392 0.8909 sen 1.0 101 89 0.8812 ... ... ... ... ... Figure 4-10 shows that despite longer terms for senators, retention among the two cohorts was similar, and was actually worse for senators after 10 years. A further anal‐ ysis comparing the different years they were first elected, or other cohort attributes, might yield some interesting insights. 152 | Chapter 4: Cohort Analysis Figure 4-10. Retention by term type for legislators in office during the year 2000 A common use case for cohorting on a value other than a starting value is when try‐ ing to analyze retention after an entity has reached a threshold, such as a certain number of purchases or a certain amount spent. As with any cohort, it’s important to take care in defining what qualifies an entity to be in a cohort and which date will be used as the starting date. Cohort retention is a powerful way to understand the behavior of entities in a time series data set. We’ve seen how to calculate retention with SQL and how to cohort based on the time series itself or on other tables, and from points in the middle of entity life span. We also looked at how to use functions and JOINs to adjust dates within time series and compensate for sparse cohorts. There are several types of", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 114 + }, + { + "text": "and how to cohort based on the time series itself or on other tables, and from points in the middle of entity life span. We also looked at how to use functions and JOINs to adjust dates within time series and compensate for sparse cohorts. There are several types of anal‐ yses that are related to cohort retention: analysis, survivorship, returnship, and cumu‐ lative calculations, all of which build off of the SQL code that we’ve developed for retention. Let’s turn to them next. Related Cohort Analyses In the last section, we learned how to write SQL for cohort retention analysis. Reten‐ tion captures whether an entity was in a time series data set on a specific date or win‐ dow of time. In addition to presence on a specific date, analysis is often interested in questions of how long an entity lasted, whether an entity did multiple actions, and Related Cohort Analyses | 153 how many of those actions occurred. These can all be answered with code that is sim‐ ilar to retention and is well suited to just about any cohorting criteria you like. Let’s take a look at the first of these, survivorship. Survivorship Survivorship, also called survival analysis, is concerned with questions about how long something lasts, or the duration of time until a particular event such as churn or death. Survivorship analysis can answer questions about the share of the population that is likely to remain past a certain amount of time. Cohorts can help identify or at least provide hypotheses about which characteristics or circumstances increase or decrease the survival likelihood. This is similar to a retention analysis, but instead of calculating whether an entity was present in a certain period, we calculate whether the entity is present in that period or later in the time series. Then the share of the total cohort is calculated. Typically one or more periods are chosen depending on the nature of the data set analyzed. For example, if we want to know the share of game players who survive for a week or longer, we can check for actions that occur after a week from starting and consider those players still surviving. On the other hand, if we are concerned about the num‐ ber of students who are still in school after a certain number of years, we could look for the absence of a graduation event in a data set. The number of periods can be SELECTed either by calculating an average or typical life span or by choosing time periods that are meaningful to the organization or process analyzed, such as a month, year, or longer time period. In this example, we’ll look at the share of legislators who survived in office for a dec‐ ade or more after their first term started. Since we don’t need to know the specific dates of each term, we can start by calculating the first and last term_start dates, using min and max aggregations: SELECT id_bioguide ,min(term_start) as", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 115 + }, + { + "text": "share of legislators who survived in office for a dec‐ ade or more after their first term started. Since we don’t need to know the specific dates of each term, we can start by calculating the first and last term_start dates, using min and max aggregations: SELECT id_bioguide ,min(term_start) as first_term ,max(term_start) as last_term FROM legislators_terms GROUP BY 1 ; id_bioguide first_term last_term ----------- ---------- --------- A000118 1975-01-14 1977-01-04 P000281 1933-03-09 1937-01-05 K000039 1933-03-09 1951-01-03 ... ... ... 154 | Chapter 4: Cohort Analysis Next, we add to the query a date_part function to find the century of the min term_start, and we calculate the tenure as the number of years between the min and max term_starts found with the age function: SELECT id_bioguide ,date_part('century',min(term_start)) as first_century ,min(term_start) as first_term ,max(term_start) as last_term ,date_part('year',age(max(term_start),min(term_start))) as tenure FROM legislators_terms GROUP BY 1 ; id_bioguide first_century first_term last_term tenure ----------- ------------- ---------- --------- ------ A000118 20.0 1975-01-14 1977-01-04 1.0 P000281 20.0 1933-03-09 1937-01-05 3.0 K000039 20.0 1933-03-09 1951-01-03 17.0 ... ... ... ... ... Finally, we calculate the cohort_size with a count of all the legislators, as well as cal‐ culating the number who survived for at least 10 years by using a CASE statement and count aggregation. The percent who survived is found by dividing these two values: SELECT first_century ,count(distinct id_bioguide) as cohort_size ,count(distinct case when tenure >= 10 then id_bioguide end) as survived_10 ,count(distinct case when tenure >= 10 then id_bioguide end) / count(distinct id_bioguide) as pct_survived_10 FROM ( SELECT id_bioguide ,date_part('century',min(term_start)) as first_century ,min(term_start) as first_term ,max(term_start) as last_term ,date_part('year',age(max(term_start),min(term_start))) as tenure FROM legislators_terms GROUP BY 1 ) a GROUP BY 1 ; century cohort survived_10 pct_survived_10 ------- ------ ----------- --------------- 18 368 83 0.2255 19 6299 892 0.1416 20 5091 1853 0.3640 21 760 119 0.1566 Related Cohort Analyses | 155 Since terms may or may not be consecutive, we can also calculate the share of legisla‐ tors in each century who survived for five or more total terms. In the subquery, add a count to find the total number of terms per legislator. Then in the outer query, divide the number of legislators with five or more terms by the total cohort size: SELECT first_century ,count(distinct id_bioguide) as cohort_size ,count(distinct case when total_terms >= 5 then id_bioguide end) as survived_5 ,count(distinct case when total_terms >= 5 then id_bioguide end) / count(distinct id_bioguide) as pct_survived_5_terms FROM ( SELECT id_bioguide ,date_part('century',min(term_start)) as first_century ,count(term_start) as total_terms FROM legislators_terms GROUP BY 1 ) a GROUP BY 1 ; century cohort survived_5 pct_survived_5_terms ------- ------ ---------- -------------------- 18 368 63 0.1712 19 6299 711 0.1129 20 5091 2153 0.4229 21 760 205 0.2697 Ten years or five terms is somewhat arbitrary. We can also calculate the survivorship for each number of years or periods and display the results in graph or table form. Here, we calculate the survivorship for each number of terms from 1 to 20. This is accomplished through a Cartesian JOIN to a subquery that contains those", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 116 + }, + { + "text": "arbitrary. We can also calculate the survivorship for each number of years or periods and display the results in graph or table form. Here, we calculate the survivorship for each number of terms from 1 to 20. This is accomplished through a Cartesian JOIN to a subquery that contains those integers derived by the generate_series function: SELECT a.first_century, b.terms ,count(distinct id_bioguide) as cohort ,count(distinct case when a.total_terms >= b.terms then id_bioguide end) as cohort_survived ,count(distinct case when a.total_terms >= b.terms then id_bioguide end) / count(distinct id_bioguide) as pct_survived FROM ( SELECT id_bioguide ,date_part('century',min(term_start)) as first_century ,count(term_start) as total_terms FROM legislators_terms GROUP BY 1 ) a JOIN 156 | Chapter 4: Cohort Analysis ( SELECT generate_series as terms FROM generate_series(1,20,1) ) b on 1 = 1 GROUP BY 1,2 ; century terms cohort cohort_survived pct_survived ------- ----- ------ --------------- ------------ 18 1 368 368 1.0000 18 2 368 249 0.6766 18 3 368 153 0.4157 ... ... ... ... ... The results are graphed in Figure 4-11. Survivorship was highest in the 20th century, a result that agrees with results we saw previously in which retention was also highest in the 20th century. Figure 4-11. Survivorship for legislators: share of cohort who stayed in office for that many terms or longer Survivorship is closely related to retention. While retention counts entities present in a specific number of periods from the start, survivorship considers only whether an entity was present as of a specific period or later. As a result, the code is simpler since it needs only the first and last dates in the time series, or a count of dates. Cohorting Related Cohort Analyses | 157 is done similar to cohorting for retention, and cohort definitions can come from within the time series or be derived from another table or subquery. Next we’ll consider another type of analysis that is in some ways the inverse of survi‐ vorship. Rather than calculating whether an entity is present in the data set at a cer‐ tain time or later, we will calculate whether an entity returns or repeats an action at a certain period or earlier. This is called returnship or repeat purchase behavior. Returnship, or Repeat Purchase Behavior Survivorship is useful for understanding how long a cohort is likely to stick around. Another useful type of cohort analysis seeks to understand whether a cohort member can be expected to return within a given window of time and the intensity of activity during that window. This is called returnship or repeat purchase behavior. For example, an ecommerce site might want to know not only how many new buyers were acquired via a marketing campaign but also whether those buyers have become repeat buyers. One way to figure this out is to simply calculate total purchases per customer. However, comparing customers acquired two years ago with those acquired a month ago isn’t fair, since the former have had a much longer time in which to return. The older cohort would almost certainly appear", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 117 + }, + { + "text": "buyers. One way to figure this out is to simply calculate total purchases per customer. However, comparing customers acquired two years ago with those acquired a month ago isn’t fair, since the former have had a much longer time in which to return. The older cohort would almost certainly appear more valuable than the newer one. Although this is true in a sense, it gives an incomplete picture of how the cohorts are likely to behave across their entire life span. To make fair comparisons between cohorts with different starting dates, we need to create an analysis based on a time box, or a fixed window of time from the first date, and consider whether cohort members returned within that window. This way, every cohort has an equal amount of time under consideration, so long as we include only those cohorts for which the full window has elapsed. Returnship analysis is common for retail organizations, but it can also be applied in other domains. For example, a university might want to see how many students enrolled in a second course, or a hospital might be interested in how many patients need follow-up medical treatments after an initial incident. To demonstrate returnship analysis, we can ask a new question of the legislators data set: how many legislators have more than one term type, and specifically, what share of them start as representatives and go on to become senators (some senators later become representatives, but that is much less common). Since relatively few make this transition, we’ll cohort legislators by the century in which they first became a representative. The first step is to find the cohort size for each century, using the subquery and date_part calculations seen previously, for only those with term_type = 'rep': SELECT date_part('century',a.first_term) as cohort_century ,count(id_bioguide) as reps 158 | Chapter 4: Cohort Analysis FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms WHERE term_type = 'rep' GROUP BY 1 ) a GROUP BY 1 ; cohort_century reps -------------- ---- 18 299 19 5773 20 4481 21 683 Next we’ll perform a similar calculation, with a JOIN to the legislators_terms table, to find the representatives who later became senators. This is accomplished with the clauses b.term_type = 'sen' and b.term_start > a.first_term: SELECT date_part('century',a.first_term) as cohort_century ,count(distinct a.id_bioguide) as rep_and_sen FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms WHERE term_type = 'rep' GROUP BY 1 ) a JOIN legislators_terms b on a.id_bioguide = b.id_bioguide and b.term_type = 'sen' and b.term_start > a.first_term GROUP BY 1 ; cohort_century rep_and_sen -------------- ----------- 18 57 19 329 20 254 21 25 Finally, we JOIN these two subqueries together and calculate the percent of represen‐ tatives who became senators. A LEFT JOIN is used; this clause is typically recom‐ mended to ensure that all cohorts are included whether or not the subsequent event happened. If there is a century in which no representatives became senators, we still want to include that century in the result set: SELECT aa.cohort_century ,bb.rep_and_sen /", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 118 + }, + { + "text": "LEFT JOIN is used; this clause is typically recom‐ mended to ensure that all cohorts are included whether or not the subsequent event happened. If there is a century in which no representatives became senators, we still want to include that century in the result set: SELECT aa.cohort_century ,bb.rep_and_sen / aa.reps as pct_rep_and_sen FROM ( Related Cohort Analyses | 159 SELECT date_part('century',a.first_term) as cohort_century ,count(id_bioguide) as reps FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms WHERE term_type = 'rep' GROUP BY 1 ) a GROUP BY 1 ) aa LEFT JOIN ( SELECT date_part('century',b.first_term) as cohort_century ,count(distinct b.id_bioguide) as rep_and_sen FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms WHERE term_type = 'rep' GROUP BY 1 ) b JOIN legislators_terms c on b.id_bioguide = b.id_bioguide and c.term_type = 'sen' and c.term_start > b.first_term GROUP BY 1 ) bb on aa.cohort_century = bb.cohort_century ; cohort_century pct_rep_and_sen -------------- --------------- 18 0.1906 19 0.0570 20 0.0567 21 0.0366 Representatives from the 18th century were most likely to become senators. However, we have not yet applied a time box to ensure a fair comparison. While we can safely assume that all legislators who served in the 18th and 19th centuries are no longer living, many of those who were first elected in the 20th and 21st centuries are still in the middle of their careers. Adding the filter WHERE age(c.term_start, b.first_term) <= interval '10 years' to subquery bb creates a time box of 10 years. Note that the window can easily be made larger or smaller by changing the constant in the interval. An additional filter applied to subquery a, WHERE first_term <= '2009-12-31', excludes those who were less than 10 years into their careers when the data set was assembled: SELECT aa.cohort_century ,bb.rep_and_sen * 100.0 / aa.reps as pct_10_yrs FROM ( SELECT date_part('century',a.first_term)::int as cohort_century 160 | Chapter 4: Cohort Analysis ,count(id_bioguide) as reps FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms WHERE term_type = 'rep' GROUP BY 1 ) a WHERE first_term <= '2009-12-31' GROUP BY 1 ) aa LEFT JOIN ( SELECT date_part('century',b.first_term)::int as cohort_century ,count(distinct b.id_bioguide) as rep_and_sen FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms WHERE term_type = 'rep' GROUP BY 1 ) b JOIN legislators_terms c on b.id_bioguide = c.id_bioguide and c.term_type = 'sen' and c.term_start > b.first_term WHERE age(c.term_start, b.first_term) <= interval '10 years' GROUP BY 1 ) bb on aa.cohort_century = bb.cohort_century ; Cohort_century pct_10_yrs -------------- ---------- 18 0.0970 19 0.0244 20 0.0348 21 0.0764 With this new adjustment, the 18th century still had the highest share of representa‐ tives becoming senators within 10 years, but the 21st century has the second-highest share, and the 20th century had a higher share than the 19th. Since 10 years is somewhat arbitrary, we might also want to compare several time windows. One option is to run the query several times with different intervals and note the results. Another option is to calculate multiple windows in the same result set by using a set of CASE", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 119 + }, + { + "text": "19th. Since 10 years is somewhat arbitrary, we might also want to compare several time windows. One option is to run the query several times with different intervals and note the results. Another option is to calculate multiple windows in the same result set by using a set of CASE statements inside of count distinct aggregations to form the intervals, rather than specifying the interval in the WHERE clause: SELECT aa.cohort_century ,bb.rep_and_sen_5_yrs * 1.0 / aa.reps as pct_5_yrs ,bb.rep_and_sen_10_yrs * 1.0 / aa.reps as pct_10_yrs ,bb.rep_and_sen_15_yrs * 1.0 / aa.reps as pct_15_yrs FROM ( Related Cohort Analyses | 161 SELECT date_part('century',a.first_term) as cohort_century ,count(id_bioguide) as reps FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms WHERE term_type = 'rep' GROUP BY 1 ) a WHERE first_term <= '2009-12-31' GROUP BY 1 ) aa LEFT JOIN ( SELECT date_part('century',b.first_term) as cohort_century ,count(distinct case when age(c.term_start,b.first_term) <= interval '5 years' then b.id_bioguide end) as rep_and_sen_5_yrs ,count(distinct case when age(c.term_start,b.first_term) <= interval '10 years' then b.id_bioguide end) as rep_and_sen_10_yrs ,count(distinct case when age(c.term_start,b.first_term) <= interval '15 years' then b.id_bioguide end) as rep_and_sen_15_yrs FROM ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms WHERE term_type = 'rep' GROUP BY 1 ) b JOIN legislators_terms c on b.id_bioguide = c.id_bioguide and c.term_type = 'sen' and c.term_start > b.first_term GROUP BY 1 ) bb on aa.cohort_century = bb.cohort_century ; cohort_century pct_5_yrs pct_10_yrs pct_15_yrs -------------- --------- ---------- ---------- 18 0.0502 0.0970 0.1438 19 0.0088 0.0244 0.0409 20 0.0100 0.0348 0.0478 21 0.0400 0.0764 0.0873 With this output, we can see how the share of representatives who became senators evolved over time, both within each cohort and across cohorts. In addition to the table format, graphing the output often reveals interesting trends. In Figure 4-12, the cohorts based on century are replaced with cohorts based on the first decade, and the trends over 10 and 20 years are shown. Conversion of representatives to senators dur‐ ing the first few decades of the new US legislature was clearly different from patterns in the years since. 162 | Chapter 4: Cohort Analysis Figure 4-12. Trend of the share of representatives for each cohort, defined by starting decade, who later became senators Finding the repeat behavior within a fixed time box is a useful tool for comparing cohorts. This is particularly true when the behaviors are intermittent in nature, such as purchase behavior or content or service consumption. In the next section, we’ll look at how to calculate not only whether an entity had a subsequent action but also how many subsequent actions they had, and we’ll aggregate them with cumulative calculations. Cumulative Calculations Cumulative cohort analysis can be used to establish cumulative lifetime value, also called customer lifetime value (the acronyms CLTV and LTV are used interchangea‐ bly), and to monitor newer cohorts in order to be able to predict what their full LTV will be. This is possible because early behavior is often highly correlated with long- term behavior. Users of a service who return frequently in their first days", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 120 + }, + { + "text": "CLTV and LTV are used interchangea‐ bly), and to monitor newer cohorts in order to be able to predict what their full LTV will be. This is possible because early behavior is often highly correlated with long- term behavior. Users of a service who return frequently in their first days or weeks of using it tend to be the most likely to stay around over the long term. Customers who buy a second or third time early on are likely to continue purchasing over a longer time period. Subscribers who renew after the first month or year are often likely to stick around over many subsequent months or years. In this section, I’ll mainly talk about the revenue-generating activities of customers, but this analysis can also be applied to situations in which customers or entities incur Related Cohort Analyses | 163 costs, such as through product returns, support interactions, or use of health-care services. With cumulative calculations, we’re less concerned about whether an entity did an action on a particular date and more about the total as of a particular date. The cumulative calculations used in this type of analysis are most often counts or sums. We will again use the time box concept to ensure apples-to-apples comparisons between cohorts. Let’s look at the number of terms started within 10 years of the first term_start, cohorting the legislators by century and type of first term: SELECT date_part('century',a.first_term) as century ,first_type ,count(distinct a.id_bioguide) as cohort ,count(b.term_start) as terms FROM ( SELECT distinct id_bioguide ,first_value(term_type) over (partition by id_bioguide order by term_start) as first_type ,min(term_start) over (partition by id_bioguide) as first_term ,min(term_start) over (partition by id_bioguide) + interval '10 years' as first_plus_10 FROM legislators_terms ) a LEFT JOIN legislators_terms b on a.id_bioguide = b.id_bioguide and b.term_start between a.first_term and a.first_plus_10 GROUP BY 1,2 ; century first_type cohort terms ------- ---------- ------ ----- 18 rep 297 760 18 sen 71 101 19 rep 5744 12165 19 sen 555 795 20 rep 4473 16203 20 sen 618 1008 21 rep 683 2203 21 sen 77 118 The largest cohort is that of representatives first elected in the 19th century, but the cohort with the largest number of terms started within 10 years is that of representa‐ tives first elected in the 20th century. This type of calculation can be useful for under‐ standing the overall contribution of a cohort to an organization. Total sales or total repeat purchases can be valuable metrics. Usually, though, we want to normalize to understand the contribution on a per-entity basis. Calculations we might want to make include average actions per person, average order value (AOV), items per order, and orders per customer. To normalize by the cohort size, simply divide by the starting cohort, which we’ve done previously with retention, survivorship, and 164 | Chapter 4: Cohort Analysis returnship. Here we do that and also pivot the results into table form for easier comparisons: SELECT century ,max(case when first_type = 'rep' then cohort end) as rep_cohort ,max(case when", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 121 + }, + { + "text": "simply divide by the starting cohort, which we’ve done previously with retention, survivorship, and 164 | Chapter 4: Cohort Analysis returnship. Here we do that and also pivot the results into table form for easier comparisons: SELECT century ,max(case when first_type = 'rep' then cohort end) as rep_cohort ,max(case when first_type = 'rep' then terms_per_leg end) as avg_rep_terms ,max(case when first_type = 'sen' then cohort end) as sen_cohort ,max(case when first_type = 'sen' then terms_per_leg end) as avg_sen_terms FROM ( SELECT date_part('century',a.first_term) as century ,first_type ,count(distinct a.id_bioguide) as cohort ,count(b.term_start) as terms ,count(b.term_start) / count(distinct a.id_bioguide) as terms_per_leg FROM ( SELECT distinct id_bioguide ,first_value(term_type) over (partition by id_bioguide order by term_start ) as first_type ,min(term_start) over (partition by id_bioguide) as first_term ,min(term_start) over (partition by id_bioguide) + interval '10 years' as first_plus_10 FROM legislators_terms ) a LEFT JOIN legislators_terms b on a.id_bioguide = b.id_bioguide and b.term_start between a.first_term and a.first_plus_10 GROUP BY 1,2 ) aa GROUP BY 1 ; century rep_cohort avg_rep_terms sen_cohort avg_sen_terms ------- ---------- ------------- ---------- ------------- 18 297 2.6 71 1.4 19 5744 2.1 555 1.4 20 4473 3.6 618 1.6 21 683 3.2 77 1.5 With the cumulative terms normalized by the cohort size, we can now confirm that representatives first elected in the 20th century had the highest average number of terms, while those who started in the 19th century had the fewest number of terms on average. Senators have fewer but longer terms than their representative peers, and again those who started in the 20th century have had the highest number of terms on average. Related Cohort Analyses | 165 Cumulative calculations are often used in customer lifetime value calculations. LTV is usually calculated using monetary measures, such as total dollars spent by a customer, or the gross margin (revenue minus costs) generated by a customer across their life‐ time. To facilitate comparisons between cohorts, the “lifetime” is often chosen to reflect average customer lifetime, or periods that are convenient to analyze, such as 3, 5, or 10 years. The legislators data set doesn’t contain financial metrics, but swapping in dollar values in any of the preceding SQL code would be straightforward. Fortu‐ nately, SQL is a flexible enough language that we can adapt these templates to address a wide variety of analytical questions. Cohort analysis includes a set of techniques that can be used to answer questions related to behavior over time and how various attributes may contribute to differ‐ ences between groups. Survivorship, returnship, and cumulative calculations all shed light on these questions. With a good understanding of how cohorts behave, we often have to turn our attention back to the composition or mix of cohorts over time, understanding how that can impact total retention, survivorship, returnship, or cumulative values such that these measures differ surprisingly from the individual cohorts. Cross-Section Analysis, Through a Cohort Lens So far in this chapter, we’ve been looking at cohort analysis. We’ve followed the behavior of cohorts across time with retention, survivorship, returnship, and cumula‐ tive", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 122 + }, + { + "text": "impact total retention, survivorship, returnship, or cumulative values such that these measures differ surprisingly from the individual cohorts. Cross-Section Analysis, Through a Cohort Lens So far in this chapter, we’ve been looking at cohort analysis. We’ve followed the behavior of cohorts across time with retention, survivorship, returnship, and cumula‐ tive behavior analyses. One of the challenges with these analyses, however, is that even as they make changes within cohorts easy to spot, it can be difficult to spot changes in the overall composition of a customer or user base. Mix shifts, which are changes in the composition of the customer or user base over time, can also occur, making later cohorts different from earlier ones. Mix shifts may be due to international expansion, shifting between organic and paid acquisition strategies, or moving from a niche enthusiast audience to a broader mass market one. Creating additional cohorts, or segments, along any of these suspected lines can help diagnose whether a mix shift is happening. Cohort analysis can be contrasted with cross-sectional analysis, which compares indi‐ viduals or groups at a single point in time. Cross-sectional studies can correlate years of education with current income, for example. On the positive side, collecting data sets for cross-sectional analysis is often easier since no time series is necessary. Cross- sectional analysis can be insightful, generating hypotheses for further investigation. On the negative side, a form of selection bias called survivorship bias usually exists, which can lead to false conclusions. 166 | Chapter 4: Cohort Analysis Survivorship Bias “Let’s look at our best customers and see what they have in common.” This seemingly innocent and well-intentioned idea can lead to some very problematic conclusions. Survivorship bias is the logical error of focusing on the people or things that made it past some selection process, while ignoring those that did not. Commonly this is because the entities no longer exist in the data set at the time of selection, because they have failed, churned, or left the population for some other reason. Concentrating only on the remaining population can lead to overly optimistic conclusions, because failures are ignored. Much has been written about a few people who dropped out of college and started wildly successful technology companies. This doesn’t mean you should immediately leave college, since the vast majority of people who drop out do not go on to be suc‐ cessful CEOs. That part of the population doesn’t make for nearly as sensational head‐ lines, so it’s easy to forget about that reality. In the successful customer context, survivorship bias might show up as an observa‐ tion that the best customers tend to live in California or Texas and tend to be 18 to 30 years old. This is a large population to start with, and it may turn out that these char‐ acteristics are shared by many customers who churned prior to the analysis date. Going back to the original population might reveal that other demographics, such as 41-to-50-year-olds in Vermont, actually stick around", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 123 + }, + { + "text": "years old. This is a large population to start with, and it may turn out that these char‐ acteristics are shared by many customers who churned prior to the analysis date. Going back to the original population might reveal that other demographics, such as 41-to-50-year-olds in Vermont, actually stick around and spend more over time, even though there are fewer of them in absolute terms. Cohort analysis helps distinguish and reduce survivorship bias. Cohort analysis is a way to overcome survivorship bias by including all members of a starting cohort in the analysis. We can take a series of cross sections from a cohort analysis to understand how the mix of entities may have changed over time. On any given date, users from a variety of cohorts are present. We can use cross-sectional analysis to examine them, like layers of sediment, to reveal new insights. In the next example, we’ll create a time series of the share of legislators from each cohort for each year in the data set. The first step is to find the number of legislators in office each year by JOINing the legislators table to the date_dim, WHERE the date from the date_dim is between the start and end dates of each term. Here we use December 31 for each year to find the legislators in office at each year’s end: SELECT b.date, count(distinct a.id_bioguide) as legislators FROM legislators_terms a JOIN date_dim b on b.date between a.term_start and a.term_end and b.month_name = 'December' and b.day_of_month = 31 and b.year <= 2019 GROUP BY 1 ; Cross-Section Analysis, Through a Cohort Lens | 167 date legislators ---------- ----------- 1789-12-31 89 1790-12-31 95 1791-12-31 99 ... ... Next, we add in the century cohorting criteria by JOINing to a subquery with the first_term calculated: SELECT b.date ,date_part('century',first_term) as century ,count(distinct a.id_bioguide) as legislators FROM legislators_terms a JOIN date_dim b on b.date between a.term_start and a.term_end and b.month_name = 'December' and b.day_of_month = 31 and b.year <= 2019 JOIN ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms GROUP BY 1 ) c on a.id_bioguide = c.id_bioguide GROUP BY 1,2 ; date century legislators ---------- ------- ----------- 1789-12-31 18 89 1790-12-31 18 95 1791-12-31 18 99 ... ... ... Finally, we calculate the percent of total legislators in each year that the century cohort represents. This can be done in a couple of ways, depending on the shape of output desired. The first way is to keep a row for each date and century combination and use a sum window function in the denominator of the percentage calculation: SELECT date ,century ,legislators ,sum(legislators) over (partition by date) as cohort ,legislators / sum(legislators) over (partition by date) as pct_century FROM ( SELECT b.date ,date_part('century',first_term) as century ,count(distinct a.id_bioguide) as legislators FROM legislators_terms a JOIN date_dim b on b.date between a.term_start and a.term_end 168 | Chapter 4: Cohort Analysis and b.month_name = 'December' and b.day_of_month = 31 and b.year <= 2019 JOIN ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms GROUP BY", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 124 + }, + { + "text": "SELECT b.date ,date_part('century',first_term) as century ,count(distinct a.id_bioguide) as legislators FROM legislators_terms a JOIN date_dim b on b.date between a.term_start and a.term_end 168 | Chapter 4: Cohort Analysis and b.month_name = 'December' and b.day_of_month = 31 and b.year <= 2019 JOIN ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms GROUP BY 1 ) c on a.id_bioguide = c.id_bioguide GROUP BY 1,2 ) a ; date century legislators cohort pct_century ---------- ------- ----------- ------ ----------- 2018-12-31 20 122 539 0.2263 2018-12-31 21 417 539 0.7737 2019-12-31 20 97 537 0.1806 2019-12-31 21 440 537 0.8194 ... ... ... ... ... The second approach results in one row per year, with a column for each century, a table format that may be easier to scan for trends: SELECT date ,coalesce(sum(case when century = 18 then legislators end) / sum(legislators),0) as pct_18 ,coalesce(sum(case when century = 19 then legislators end) / sum(legislators),0) as pct_19 ,coalesce(sum(case when century = 20 then legislators end) / sum(legislators),0) as pct_20 ,coalesce(sum(case when century = 21 then legislators end) / sum(legislators),0) as pct_21 FROM ( SELECT b.date ,date_part('century',first_term) as century ,count(distinct a.id_bioguide) as legislators FROM legislators_terms a JOIN date_dim b on b.date between a.term_start and a.term_end and b.month_name = 'December' and b.day_of_month = 31 and b.year <= 2019 JOIN ( SELECT id_bioguide, min(term_start) as first_term FROM legislators_terms GROUP BY 1 ) c on a.id_bioguide = c.id_bioguide GROUP BY 1,2 ) aa GROUP BY 1 ; Cross-Section Analysis, Through a Cohort Lens | 169 date pct_18 pct_19 pct_20 pct_21 ---------- ------ ------ ------ ------ 2017-12-31 0 0 0.2305 0.7695 2018-12-31 0 0 0.2263 0.7737 2019-12-31 0 0 0.1806 0.8193 ... ... ... ... ... We can graph the output, as in Figure 4-13, to see how newer cohorts of legislators gradually overtake older cohorts, until they themselves are replaced by new cohorts. Figure 4-13. Percent of legislators each year, by century first elected Rather than cohorting on first_term, we can cohort on tenure instead. Finding the share of customers who are relatively new, are of medium tenure, or are long-term customers at various points in time can be insightful. Let’s take a look at how the ten‐ ure of legislators in Congress has changed over time. The first step is to calculate, for each year, the cumulative number of years in office for each legislator. Since there can be gaps between terms when legislators are voted out or leave office for other reasons, we’ll first find each year in which the legislator was in office at the end of the year, in the subquery. Then we’ll use a count window function, with the window covering the rows unbounded preceding, or all prior rows for that legislator, and current row: SELECT id_bioguide, date ,count(date) over (partition by id_bioguide order by date rows between unbounded preceding and current row ) as cume_years 170 | Chapter 4: Cohort Analysis FROM ( SELECT distinct a.id_bioguide, b.date FROM legislators_terms a JOIN date_dim b on b.date between a.term_start and a.term_end and b.month_name = 'December' and b.day_of_month", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 125 + }, + { + "text": "id_bioguide, date ,count(date) over (partition by id_bioguide order by date rows between unbounded preceding and current row ) as cume_years 170 | Chapter 4: Cohort Analysis FROM ( SELECT distinct a.id_bioguide, b.date FROM legislators_terms a JOIN date_dim b on b.date between a.term_start and a.term_end and b.month_name = 'December' and b.day_of_month = 31 and b.year <= 2019 ) aa ; id_bioguide date cume_years ----------- ---------- ---------- A000001 1951-12-31 1 A000001 1952-12-31 2 A000002 1947-12-31 1 A000002 1948-12-31 2 A000002 1949-12-31 3 ... ... ... Next, count the number of legislators for each combination of date and cume_years to create a distribution: SELECT date, cume_years ,count(distinct id_bioguide) as legislators FROM ( SELECT id_bioguide, date ,count(date) over (partition by id_bioguide order by date rows between unbounded preceding and current row ) as cume_years FROM ( SELECT distinct a.id_bioguide, b.date FROM legislators_terms a JOIN date_dim b on b.date between a.term_start and a.term_end and b.month_name = 'December' and b.day_of_month = 31 and b.year <= 2019 GROUP BY 1,2 ) aa ) aaa GROUP BY 1,2 ; date cume_years legislators ----------- ---------- ---------- 1789-12-31 1 89 1790-12-31 1 6 1790-12-31 2 89 1791-12-31 1 37 ... ... ... Cross-Section Analysis, Through a Cohort Lens | 171 Before calculating the percentage for each tenure per year and adjusting the presenta‐ tion format, we might want to consider grouping the tenures. A quick profiling of our results so far reveals that in some years, almost 40 different tenures are represented. This will likely be difficult to visualize and interpret: SELECT date, count(*) as tenures FROM ( SELECT date, cume_years ,count(distinct id_bioguide) as legislators FROM ( SELECT id_bioguide, date ,count(date) over (partition by id_bioguide order by date rows between unbounded preceding and current row ) as cume_years FROM ( SELECT distinct a.id_bioguide, b.date FROM legislators_terms a JOIN date_dim b on b.date between a.term_start and a.term_end and b.month_name = 'December' and b.day_of_month = 31 and b.year <= 2019 GROUP BY 1,2 ) aa ) aaa GROUP BY 1,2 ) aaaa GROUP BY 1 ; date tenures ----------- ------- 1998-12-31 39 1994-12-31 39 1996-12-31 38 ... ... As a result, we may want to group the values. There is no single right way to group tenures. If there are organizational definitions of tenure groups, go ahead and use them. Otherwise, I usually try to break them up into three to five groups of roughly equal size. Here we’ll group the tenures into four cohorts, where cume_years is less than or equal to 4 years, between 5 and 10 years, between 11 and 20 years, and equal to or more than 21 years: 172 | Chapter 4: Cohort Analysis SELECT date, tenure ,legislators / sum(legislators) over (partition by date) as pct_legislators FROM ( SELECT date ,case when cume_years <= 4 then '1 to 4' when cume_years <= 10 then '5 to 10' when cume_years <= 20 then '11 to 20' else '21+' end as tenure ,count(distinct id_bioguide) as legislators FROM ( SELECT id_bioguide, date ,count(date) over (partition by id_bioguide order by date", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 126 + }, + { + "text": "( SELECT date ,case when cume_years <= 4 then '1 to 4' when cume_years <= 10 then '5 to 10' when cume_years <= 20 then '11 to 20' else '21+' end as tenure ,count(distinct id_bioguide) as legislators FROM ( SELECT id_bioguide, date ,count(date) over (partition by id_bioguide order by date rows between unbounded preceding and current row ) as cume_years FROM ( SELECT distinct a.id_bioguide, b.date FROM legislators_terms a JOIN date_dim b on b.date between a.term_start and a.term_end and b.month_name = 'December' and b.day_of_month = 31 and b.year <= 2019 GROUP BY 1,2 ) a ) aa GROUP BY 1,2 ) aaa ; date tenure pct_legislators ---------- ------- --------------- 2019-12-31 1 to 4 0.2998 2019-12-31 5 to 10 0.3203 2019-12-31 11 to 20 0.2011 2019-12-31 21+ 0.1788 ... ... ... The graphing of the results in Figure 4-14 shows that in the early years of the country, most legislators had very little tenure. In more recent years, the share of legislators with 21 or more years in office has been increasing. There are also interesting peri‐ odic increases in 1-to-4-year-tenure legislators that may reflect shifts in political trends. Cross-Section Analysis, Through a Cohort Lens | 173 Figure 4-14. Percent of legislators by number of years in office A cross section of a population at any point in time is made up of members from multiple cohorts. Creating a time series of these cross sections is another interesting way of analyzing trends. Combining this with insights from retention can provide a more robust picture of trends in any organization. Conclusion Cohort analysis is a useful way to investigate how groups change over time, whether it be from the perspective of retention, repeat behavior, or cumulative actions. Cohort analysis is retrospective, looking back at populations using intrinsic attributes or attributes derived from behavior. Interesting and hopefully useful correlations can be found through this type of analysis. However, as the saying goes, correlation does not imply causation. To determine actual causality, randomized experiments are the gold standard. Chapter 7 will go into depth on experiment analysis. Before we turn to experimentation, however, we have a few other types of analysis to cover. Next we’ll cover text analysis: components of text analysis often show up in other analyses, and it’s an interesting facet of analysis in itself. 174 | Chapter 4: Cohort Analysis CHAPTER 5 Text Analysis In the last two chapters, we explored applications of dates and numbers with time ser‐ ies analysis and cohort analysis. But data sets are often more than just numeric values and associated timestamps. From qualitative attributes to free text, character fields are often loaded with potentially interesting information. Although databases excel at numeric calculations such as counting, summing, and averaging things, they are also quite good at performing operations on text data. I’ll begin this chapter by providing an overview of the types of text analysis tasks that SQL is good for, and of those for which another programming language is a better choice. Next, I’ll introduce", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 127 + }, + { + "text": "summing, and averaging things, they are also quite good at performing operations on text data. I’ll begin this chapter by providing an overview of the types of text analysis tasks that SQL is good for, and of those for which another programming language is a better choice. Next, I’ll introduce our data set of UFO sightings. Then we’ll get into coding, covering text characteristics and profiling, parsing data with SQL, making various transformations, constructing new text from parts, and finally finding elements within larger blocks of text, including with regular expressions. Why Text Analysis with SQL? Among the huge volumes of data generated every day, a large portion consists of text: words, sentences, paragraphs, and even longer documents. Text data used for analysis can come from a variety of sources, including descriptors populated by humans or computer applications, log files, support tickets, customer surveys, social media posts, or news feeds. Text in databases ranges from structured (where data is in different table fields with distinct meanings) to semistructured (where the data is in separate columns but may need parsing or cleaning to be useful) or mostly unstructured (where long VARCHAR or BLOB fields hold arbitrary length strings that require extensive structuring before further analysis). Fortunately, SQL has a number of use‐ ful functions that can be combined to accomplish a range of text-structuring and analysis tasks. 175 What Is Text Analysis? Text analysis is the process of deriving meaning and insight from text data. There are two broad categories of text analysis, which can be distinguished by whether the out‐ put is qualitative or quantitative. Qualitative analysis, which may also be called textual analysis, seeks to understand and synthesize the meaning from a single text or a set of texts, often applying other knowledge or unique conclusions. This work is often done by journalists, historians, and user experience researchers. Quantitative analysis of text also seeks to synthesize information from text data, but the output is quantitative. Tasks include categorization and data extraction, and analysis is usually in the form of counts or frequencies, often trended over time. SQL is much more suited to quantita‐ tive analysis, so that is what the rest of this chapter is concerned with. If you have the opportunity to work with a counterpart who specializes in the first type of text analy‐ sis, however, do take advantage of their expertise. Combining the qualitative with the quantitative is a great way to derive new insights and persuade reluctant colleagues. Text analysis encompasses several goals or strategies. The first is text extraction, where a useful piece of data must be pulled from surrounding text. Another is catego‐ rization, where information is extracted or parsed from text data in order to assign tags or categories to rows in a database. Another strategy is sentiment analysis, where the goal is to understand the mood or intent of the writer on a scale from negative to positive. Although text analysis has been around for a while, interest and research in", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 128 + }, + { + "text": "in order to assign tags or categories to rows in a database. Another strategy is sentiment analysis, where the goal is to understand the mood or intent of the writer on a scale from negative to positive. Although text analysis has been around for a while, interest and research in this area have taken off with the advent of machine learning and the computing resources that are often needed to work with large volumes of text data. Natural language processing (NLP) has made huge advances in recognizing, classifying, and even generating brand-new text data. Human language is incredibly complex, with different languages and dialects, grammars, and slang, not to mention the thousands and thousands of words, some that have overlapping meanings or subtly modify the meaning of other words. As we’ll see, SQL is good at some forms of text analysis, but for other, more advanced tasks, there are languages and tools that are better suited. Why SQL Is a Good Choice for Text Analysis There are a number of good reasons to use SQL for text analysis. One of the most obvious is when the data is already in a database. Modern databases have a lot of computing power that can be leveraged for text tasks in addition to the other tasks we’ve discussed so far. Moving data to a flat file for analysis with another language or tool is time consuming, so doing as much work as possible with SQL within the data‐ base has advantages. If the data is not already in a database, for relatively large data sets, moving the data to a database may be worthwhile. Databases are more powerful than spreadsheets for processing transformations on many records. SQL is less error-prone than 176 | Chapter 5: Text Analysis spreadsheets, since no copying and pasting is required, and the original data stays intact. Data could potentially be altered with an UPDATE command, but this is hard to do accidentally. SQL is also a good choice when the end goal is quantification of some sort. Counting how many support tickets contain a key phrase and parsing categories out of larger text that will be used to group records are good examples of when SQL shines. SQL is good at cleaning and structuring text fields. Cleaning includes removing extra charac‐ ters or whitespace, fixing capitalization, and standardizing spellings. Structuring involves creating new columns from elements extracted or derived from other fields or constructing new fields from parts stored in different places. String functions can be nested or applied to the results of other functions, allowing for almost any manip‐ ulations that might be needed. SQL code for text analysis can be simple or complex, but it is always rule based. In a rule-based system, the computer follows a set of rules or instructions—no more, no less. This can be contrasted with machine learning, in which the computer adapts based on the data. Rules are good because they are easy for humans to understand. They are written down", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 129 + }, + { + "text": "always rule based. In a rule-based system, the computer follows a set of rules or instructions—no more, no less. This can be contrasted with machine learning, in which the computer adapts based on the data. Rules are good because they are easy for humans to understand. They are written down in code form and can be checked to ensure they produce the desired output. The downside of rules is that they can become long and complicated, particularly when there are a lot of different cases to handle. This can also make them difficult to maintain. If the structure or type of data entered into the column changes, the rule set needs to be updated. On more than one occasion, I’ve started with what seemed like a simple CASE statement with 4 or 5 lines, only to have it grow to 50 or 100 lines as the application changed. Rules might still be the right approach, but keep‐ ing in sync with the development team on changes is a good idea. Finally, SQL is a good choice when you know in advance what you are looking for. There are a number of powerful functions, including regular expressions, that allow you to search for, extract, or replace specific pieces of information. “How many reviewers mention ‘short battery life’ in their reviews?” is a question SQL can help you answer. On the other hand, “Why are these customers angry?” is not going to be as easy. When SQL Is Not a Good Choice SQL essentially allows you to harness the power of the database to apply a set of rules, albeit often powerful rules, to a set of text to make it more useful for analysis. SQL is certainly not the only option for text analysis, and there are a number of use cases for which it’s not the best choice. It’s useful to be aware of these. The first category encompasses use cases for which a human is more appropriate. When the data set is very small or very new, hand labeling can be faster and more informative. Additionally, if the goal is to read all the records and come up with a qualitative summary of key themes, a human is a better choice. Why Text Analysis with SQL? | 177 The second category is when there’s a need to search for and retrieve specific records that contain text strings with low latency. Tools like Elasticsearch or Splunk have been developed to index strings for these use cases. Performance will often be an issue with SQL and databases; this is one of the main reasons that we usually try to structure the data into discrete columns that can more easily be searched by the database engine. The third category comprises tasks in the broader NLP category, where machine learning approaches and the languages that run them, such as Python, are a better choice. Sentiment analysis, used to analyze ranges of positive or negative feelings in texts, can be handled only in a", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 130 + }, + { + "text": "by the database engine. The third category comprises tasks in the broader NLP category, where machine learning approaches and the languages that run them, such as Python, are a better choice. Sentiment analysis, used to analyze ranges of positive or negative feelings in texts, can be handled only in a simplistic way with SQL. For example, “love” and “hate” could be extracted and used to categorize records, but given the range of words that can express positive and negative emotions, as well as all the ways to negate those words, it would be nearly impossible to create a rule set with SQL to handle them all. Part-of-speech tagging, where words in a text are labeled as nouns, verbs, and so on, is better handled with libraries available in Python. Language generation, or creating brand-new text based on learnings from example texts, is another example best han‐ dled in other tools. We will see how we can create new text by concatenating pieces of data together, but SQL is still bound by rules and won’t automatically learn from and adapt to new examples in the data set. Now that we’ve discussed the many good reasons to use SQL for text analysis, as well as the types of use cases to avoid, let’s take a look at the data set we’ll be using for the examples before launching into the SQL code itself. The UFO Sightings Data Set For the examples in this chapter, we’ll use a data set of UFO sightings compiled by the National UFO Reporting Center. The data set consists of approximately 95,000 reports posted between 2006 and 2020. Reports come from individuals who can enter information through an online form. The table we will work with is ufo, and it has only two columns. The first is a compo‐ site column called sighting_report that contains information about when the sight‐ ing occurred, when it was reported, and when it was posted. It also contains metadata about the location, shape, and duration of the sighting event. The second column is a text field called description that contains the full description of the event. Figure 5-1 shows a sample of the data. 178 | Chapter 5: Text Analysis Figure 5-1. Sample of the ufo table Through the examples and discussion in this chapter, I will show how to parse the first column into structured dates and descriptors. I will also show how to perform various analyses on the description field. If I were working with this data on a con‐ tinual basis, I might consider creating an ETL pipeline, a job that processes the data in the same way on a regular basis, and storing the resulting structured data in a new table. For the examples in this chapter, however, we’ll stick with the raw table. Let’s get into the code, starting with SQL to explore and characterize the text from the sightings. Text Characteristics The most flexible data type in a database is VARCHAR, because almost any data can", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 131 + }, + { + "text": "a new table. For the examples in this chapter, however, we’ll stick with the raw table. Let’s get into the code, starting with SQL to explore and characterize the text from the sightings. Text Characteristics The most flexible data type in a database is VARCHAR, because almost any data can be put in fields of this type. As a result, text data in databases comes in a variety of shapes and sizes. As with other data sets, profiling and characterizing the data is one of the first things we do. From there we can develop a game plan for the kinds of cleaning and parsing that may be necessary for the analysis. One way we can get to know the text data is to find the number of characters in each value, which can be done with the length function (or len in some databases). This function takes the string or character field as an argument and is similar to functions found in other languages and spreadsheet programs: SELECT length('Sample string'); length ------ 13 Text Characteristics | 179 We can create a distribution of field lengths to get a sense of the typical length and of whether there are any extreme outliers that might need to be handled in special ways: SELECT length(sighting_report), count(*) as records FROM ufo GROUP BY 1 ORDER BY 1 ; length records ------ ------- 90 1 91 4 92 8 ... ... We can see in Figure 5-2 that most of the records are between roughly 150 and 180 characters long, and very few are less than 140 or more than 200 characters. The lengths of the description field range from 5 to 64,921 characters. We can assume that there is much more variety in this field, even before doing any additional profiling. Figure 5-2. Distribution of field lengths in the first column of the ufo table Let’s take a look at a few sample rows of the sighting_report column. In a query tool, I might scroll through a hundred or so rows to get familiar with the contents, but these are representative of the values in the column: 180 | Chapter 5: Text Analysis Occurred : 3/4/2018 19:07 (Entered as : 03/04/18 19:07)Reported: 3/6/2018 7:05:12 PM 19:05Posted: 3/8/2018Location: Colorado Springs, COShape: LightDuration:3 minutes Occurred : 10/16/2017 21:42 (Entered as : 10/16/2017 21:42)Reported: 3/6/2018 5:09:47 PM 17:09Posted: 3/8/2018Location: North Dayton, OHShape: SphereDuration:~5 minutes Occurred : 2/15/2018 00:10 (Entered as : 2/15/18 0:10)Reported: 3/6/2018 6:19:54 PM 18:19Posted: 3/8/2018Location: Grand Forks, NDShape: SphereDuration: 5 seconds This data is what I would call semistructured, or overstuffed. It can’t be used in an analysis as is, but there are clearly distinct pieces of information stored here, and the pattern is similar between rows. For example, each row has the word “Occurred” fol‐ lowed by what looks like a timestamp, “Location” followed by a place, and “Duration” followed by an amount of time. Data can end up in overstuffed fields for a variety of reasons, but there are two common", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 132 + }, + { + "text": "is similar between rows. For example, each row has the word “Occurred” fol‐ lowed by what looks like a timestamp, “Location” followed by a place, and “Duration” followed by an amount of time. Data can end up in overstuffed fields for a variety of reasons, but there are two common ones I see. One is when there aren’t enough fields available in the source system or application to store all the attributes required, so multiple attributes are entered into the same field. Another is when the data is stored in a JSON blob in an appli‐ cation in order to accommodate sparse attributes or frequent addi‐ tions of new attributes. Although both scenarios are less than ideal from an analysis perspective, as long as there is a consistent struc‐ ture, we can usually handle these with SQL. Our next step is to make this field more usable by parsing it into several new fields, each of which contains a single piece of information. The steps in this process are: • Plan the field(s) desired as output • Apply parsing functions • Apply transformations, including data type conversions • Check the results when applied to the entire data set, since there will often be some records that don’t conform to the pattern • Repeat these steps until the data is in the desired columns and formats The new columns we will parse out of sighting_report are occurred, entered_as, reported, posted, location, shape, and duration. Next, we will learn about parsing functions and work on structuring the ufo data set. Text Characteristics | 181 Text Parsing Parsing data with SQL is the process of extracting pieces of a text value to make them more useful for analysis. Parsing splits the data into the part that we want and “every‐ thing else,” though typically our code returns only the part we want. The simplest parsing functions return a fixed number of characters from either the beginning or the end of a string. The left function returns characters from the left side or beginning of the string, while the right function returns characters from the right side or end of the string. They otherwise work in the same way, taking the value to parse as the first argument and the number of characters as the second. Either argument can be a database field or calculation, allowing for dynamic results: SELECT left('The data is about UFOs',3) as left_digits ,right('The data is about UFOs',4) as right_digits ; left_digits right_digits ----------- ----- The UFOs In the ufo data set, we can parse out the first word, “Occurred,” using the left function: SELECT left(sighting_report,8) as left_digits ,count(*) FROM ufo GROUP BY 1 ; left_digits count ----------- ----- Occurred 95463 We can confirm that all records start with this word, which is good news because it means at least this part of the pattern is consistent. However, what we really want is the values for what occurred, not the word itself, so let’s try again. In the first example", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 133 + }, + { + "text": "95463 We can confirm that all records start with this word, which is good news because it means at least this part of the pattern is consistent. However, what we really want is the values for what occurred, not the word itself, so let’s try again. In the first example record, the end of the occurred timestamp is at character 25. In order to remove “Occurred” and retain only the actual timestamp, we can return the rightmost 14 characters using the right function. Note that the right and left functions are nested—the first argument of the right function is the result of the left function: SELECT right(left(sighting_report,25),14) as occurred FROM ufo ; occurred -------------- 3/4/2018 19:07 10/16/2017 21: 182 | Chapter 5: Text Analysis 2/15/2018 00:1 ... Although this returns the correct result for the first record, it unfortunately can’t han‐ dle the records that have two-digit month or day values. We could increase the number of characters returned by the left and right functions, but the result would then include too many characters for the first record. The left and right functions are useful for extracting fixed-length parts of a string, as in our extraction of the word “Occurred,” but for more complex patterns, a func‐ tion called split_part is more useful. The idea behind this function is to split a string into parts based on a delimiter and then allow you to select a specific part. A delimiter is one or more characters that are used to specify the boundary between regions of text or other data. The comma delimiter and tab delimiter are probably the most common, as these are used in text files (with extensions such as .csv, .tsv, or .txt) to indicate where columns start and end. However, any sequence of characters can be used, which will come in handy for our parsing task. The form of the function is: split_part(string or field name, delimiter, index) The index is the position of the text to be returned, relative to the delimiter. So index = 1 returns all of the text to the left of the first instance of the delimiter, index = 2 returns the text between the first and second instance of the delimiter (or all of the text to the right of the delimiter if the delimiter appears only once), and so on. There is no zero index, and the values must be positive integers: SELECT split_part('This is an example of an example string' ,'an example' ,1); split_part ---------- This is SELECT split_part('This is an example of an example string' ,'an example' ,2); split_part ---------- of MySQL has a substring_index function instead of split_part. SQL Server does not have a split_part function at all Text Parsing | 183 Note that spaces in the text will be retained unless specified as part of the delimiter. Let’s take a look at how we can parse the elements of the sighting_report column. As a reminder, a sample value looks like this: Occurred : 6/3/2014 23:00 (Entered", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 134 + }, + { + "text": "all Text Parsing | 183 Note that spaces in the text will be retained unless specified as part of the delimiter. Let’s take a look at how we can parse the elements of the sighting_report column. As a reminder, a sample value looks like this: Occurred : 6/3/2014 23:00 (Entered as : 06/03/14 11:00)Reported: 6/3/2014 10:33:24 PM 22:33Posted: 6/4/2014Location: Bethesda, MDShape: LightDuration:15 minutes The value we want our query to return is the text between “Occurred : ” and “ (Entered”. That is, we want the string “6/3/2014 23:00”. Checking the sample text, “Occurred :” and “(Entered” both appear only once. A colon (:) appears several times, both to separate the label from the value and in the middle of timestamps. This might make parsing using the colon tricky. The open parenthesis character appears only once. We have some choices as to what to specify as the delimiter, choosing either longer strings or only the fewest characters required to split the string accurately. I tend to be a little more verbose to ensure that I get exactly the piece that I want, but it really depends on the situation. First, split sighting_report on “Occurred : ” and check the result: SELECT split_part(sighting_report,'Occurred : ',2) as split_1 FROM ufo ; split_1 -------------------------------------------------------------- 6/3/2014 23:00 (Entered as : 06/03/14 11:00)Reported: 6/3/2014 10:33:24 PM 22:33Posted: 6/4/2014Location: Bethesda, MDShape: LightDuration:15 minutes We have successfully removed the label, but we still have a lot of extra text remaining. Let’s check the result when we split on “ (Entered”: SELECT split_part(sighting_report,' (Entered',1) as split_2 FROM ufo ; split_2 ------------------------- Occurred : 6/3/2014 23:00 This is closer, but it still has the label in the result. Fortunately, nesting split_part functions will return only the desired date and time value: SELECT split_part( split_part(sighting_report,' (Entered',1) ,'Occurred : ',2) as occurred FROM ufo ; occurred --------------- 6/3/2014 23:00 184 | Chapter 5: Text Analysis 4/25/2014 21:15 5/25/2014 Now the result includes the desired values. Reviewing a few additional lines shows that two-digit day and month values are handled appropriately, as are dates that do not have a time value. It turns out that some records omit the “Entered as” value, so one additional split is required to handle records where the “Reported” label marks the end of the desired string: SELECT split_part( split_part( split_part(sighting_report,' (Entered',1) ,'Occurred : ',2) ,'Reported',1) as occurred FROM ufo ; occurred --------------- 6/24/1980 14:00 4/6/2006 02:05 9/11/2001 09:00 ... The most common occurred values parsed out with the SQL code are graphed in Figure 5-3. Figure 5-3. Top 10 most common occurred values for UFO sightings Text Parsing | 185 Finding a set of functions that works for all values in the data set is one of the hardest parts of text parsing. It often takes several rounds of trial and error and profiling the results along the way to get it right. The next step is to apply similar parsing rules to extract the other desired fields, using beginning and ending delimiters to isolate just", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 135 + }, + { + "text": "of the hardest parts of text parsing. It often takes several rounds of trial and error and profiling the results along the way to get it right. The next step is to apply similar parsing rules to extract the other desired fields, using beginning and ending delimiters to isolate just the relevant part of the string. The final query uses split_part several times, with different arguments for each value: SELECT split_part( split_part( split_part(sighting_report,' (Entered',1) ,'Occurred : ',2) ,'Reported',1) as occurred ,split_part( split_part(sighting_report,')',1) ,'Entered as : ',2) as entered_as ,split_part( split_part( split_part( split_part(sighting_report,'Post',1) ,'Reported: ',2) ,' AM',1) ,' PM',1) as reported ,split_part(split_part(sighting_report,'Location',1),'Posted: ',2) as posted ,split_part(split_part(sighting_report,'Shape',1),'Location: ',2) as location ,split_part(split_part(sighting_report,'Duration',1),'Shape: ',2) as shape ,split_part(sighting_report,'Duration:',2) as duration FROM ufo ; occurred entered_as reported posted location shape duration -------- ---------- -------- ------- ----------- --------- ----------- 7/4/2... 07/04/2... 7/5... 7/5/... Columbus... Formation 15 minutes 7/4/2... 07/04/2... 7/5... 7/5/... St. John... Circle 2-3 minutes 7/4/2... 07/7/1... 7/5... 7/5/... Royal Pa... Circle 3 minutes ... ... ... ... ... ... ... With this SQL parsing, the data is now in a much more structured and usable format. Before we finish, however, there are a few transformations that will clean up the data a little further. We’ll take a look at these string transformation functions next. 186 | Chapter 5: Text Analysis Text Transformations Transformations change string values in some way. We saw a number of date and timestamp transformation functions in Chapter 3. There is a set of functions in SQL that specifically work on string values. These are useful for working with parsed data, but also for any text data in a database that needs to be adjusted or cleaned for analysis. Among the most common transformations are the ones that change capitalization. The upper function converts all letters to their uppercase form, while the lower func‐ tion converts all letters to their lowercase form. For example: SELECT upper('Some sample text'); upper ---------------- SOME SAMPLE TEXT SELECT lower('Some sample text'); lower ---------------- some sample text These are useful for standardizing values that may have been entered in different ways. For example, any human will recognize that “California,” “caLiforNia,” and “CALIFORNIA” all refer to the same state, but a database will treat them as distinct values. If we were to count UFO sightings by states with these values, we would end up with three records for California, resulting in incorrect analysis conclusions. Con‐ verting them to all uppercase or all lowercase letters would solve this problem. Some databases, including Postgres, have an initcap function that capitalizes the first letter of each word in a string. This is useful for proper nouns, such as state names: SELECT initcap('caLiforNia'), initcap('golden gate bridge'); initcap initcap ---------- ------------------ California Golden Gate Bridge The shape field in the data set we parsed contains one value that is in all capitals, “TRIANGULAR.” To clean this and standardize it with the other values, which all have only their first letter capitalized, apply the initcap function: Text Transformations | 187 SELECT distinct shape, initcap(shape)", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 136 + }, + { + "text": "Gate Bridge The shape field in the data set we parsed contains one value that is in all capitals, “TRIANGULAR.” To clean this and standardize it with the other values, which all have only their first letter capitalized, apply the initcap function: Text Transformations | 187 SELECT distinct shape, initcap(shape) as shape_clean FROM ( SELECT split_part( split_part(sighting_report,'Duration',1) ,'Shape: ',2) as shape FROM ufo ) a ; shape shape_clean ---------- ----------- ... ... Sphere Sphere TRIANGULAR Triangular Teardrop Teardrop ... ... The number of sightings for each shape is shown in Figure 5-4. Light is by far the most common shape, followed by circle and triangle. Some sightings do not report a shape, so a count for null value appears in the graph as well. Figure 5-4. Frequency of shapes in UFO sightings 188 | Chapter 5: Text Analysis Another useful transformation function is one called trim that removes blank spaces at the beginning and end of a string. Extra whitespace characters are a common prob‐ lem when parsing values out of longer strings, or when data is created by human entry or by copying data from one application to another. As an example, we can remove the leading spaces before “California” in the following string by using the trim function: SELECT trim(' California '); trim ---------- California The trim function has a few optional parameters that make it flexible for a variety of data-cleaning challenges. First, it can remove characters from the start of a string or from the end of a string, or both. Trimming from both ends is the default, but the other options can be specified with leading or trailing. Additionally, trim can remove any character, not just whitespace. So, for example, if an application placed a dollar sign ($) at the beginning of each state name for some reason, we could remove this with trim: SELECT trim(leading '$' from '$California'); A few of the values in the duration field have leading spaces, so applying trim will result in a cleaner output: SELECT duration, trim(duration) as duration_clean FROM ( SELECT split_part(sighting_report,'Duration:',2) as duration FROM ufo ) a ; duration duration_clean --------------------- -------------------- ~2 seconds ~2 seconds 15 minutes 15 minutes 20 minutes (ongoing) 20 minutes (ongoing) The number of sightings for the most common durations are graphed in Figure 5-5. Sightings lasting between 1 and 10 minutes are common. Some sightings do not report a duration, so a count for null value appears in the graph. Text Transformations | 189 Figure 5-5. Top 10 most common durations of UFO sightings The next type of transformation is a data type conversion. This type of transforma‐ tion, discussed in Chapter 2, will be useful for ensuring that the results of our parsing have the intended data type. In our case, there are two fields that should be treated as timestamps—the occurred and reported columns—and the posted column should be a date type. The data types can be changed with casting, using either the double colon (::) operator or the", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 137 + }, + { + "text": "of our parsing have the intended data type. In our case, there are two fields that should be treated as timestamps—the occurred and reported columns—and the posted column should be a date type. The data types can be changed with casting, using either the double colon (::) operator or the CAST field as type syntax. We’ll leave the entered_as, location, shape, and duration values as VARCHAR: SELECT occurred::timestamp ,reported::timestamp as reported ,posted::date as posted FROM ( SELECT split_part( split_part( split_part(sighting_report,' (Entered',1) ,'Occurred : ',2) ,'Reported',1) as occurred ,split_part( split_part( split_part( split_part(sighting_report,'Post',1) ,'Reported: ',2) 190 | Chapter 5: Text Analysis 1 Since the data set was created in the United States, it is in mm/dd/yyyy format. Many other parts of the world use the dd/mm/yyyy format instead. It’s always worth checking your source and adjusting your code as needed. ,' AM',1),' PM',1) as reported ,split_part( split_part(sighting_report,'Location',1) ,'Posted: ',2) as posted FROM ufo ) a ; occurred reported posted ------------------- ------------------- ---------- 2015-05-24 19:30:00 2015-05-25 10:07:21 2015-05-29 2015-05-24 22:40:00 2015-05-25 09:09:09 2015-05-29 2015-05-24 22:30:00 2015-05-24 10:49:43 2015-05-29 ... ... ... A sample of the data converts to the new formats. Notice that the database adds the seconds to the timestamp, even though there were no seconds in the original value, and correctly recognizes dates that were in month/day/year (mm/dd/yyyy) format.1 There is a problem when applying these transformations to the entire data set, how‐ ever. A few records do not have values at all, appearing as an empty string, and some have the time value but no date associated with them. Although an empty string and null seem to contain the same information—nothing—databases treat them differ‐ ently. An empty string is still a string and can’t be converted to another data type. Set‐ ting all the nonconforming records to null with a CASE statement allows the type conversion to work properly. Since we know that dates must contain at least eight characters (four digits for year, one or two digits each for month and day, and two “-” or “/” characters), one way to accomplish this is by setting any record with LENGTH less than 8 equal to null with a CASE statement: SELECT case when occurred = '' then null when length(occurred) < 8 then null else occurred::timestamp end as occurred ,case when length(reported) < 8 then null else reported::timestamp end as reported ,case when posted = '' then null else posted::date end as posted FROM ( Text Transformations | 191 SELECT split_part( split_part( split_part(sighting_report,'(Entered',1) ,'Occurred : ',2) ,'Reported',1) as occurred ,split_part( split_part( split_part( split_part(sighting_report,'Post',1) ,'Reported: ',2) ,' AM',1) ,' PM',1) as reported ,split_part( split_part(sighting_report,'Location',1) ,'Posted: ',2) as posted FROM ufo ) a ; occurred reported posted ------------------- ------------------- ---------- 1991-10-01 14:00:00 2018-03-06 08:54:22 2018-03-08 2018-03-04 19:07:00 2018-03-06 07:05:12 2018-03-08 2017-10-16 21:42:00 2018-03-06 05:09:47 2018-03-08 ... ... ... The final transformation I’ll discuss in this section is the replace function. Some‐ times there is a word, phrase, or other string within a field that we would like to change to another", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 138 + }, + { + "text": "14:00:00 2018-03-06 08:54:22 2018-03-08 2018-03-04 19:07:00 2018-03-06 07:05:12 2018-03-08 2017-10-16 21:42:00 2018-03-06 05:09:47 2018-03-08 ... ... ... The final transformation I’ll discuss in this section is the replace function. Some‐ times there is a word, phrase, or other string within a field that we would like to change to another string or remove entirely. The replace function comes in handy for this task. It takes three arguments—the original text, the string to find, and the string to substitute in its place: replace(string or field, string to find, string to substitute) So, for example, if we want to change references of “unidentified flying objects” to “UFOs,” we can use the replace function: SELECT replace('Some unidentified flying objects were noticed above...','unidentified flying objects','UFOs'); replace ------------------------------- Some UFOs were noticed above... This function will find and replace every instance of the string in the second argu‐ ment, regardless of where it appears. An empty string can be used as the third argu‐ ment, which is a good way to remove parts of a string that are not wanted. Like other string functions, replace can be nested, with the output from one replace becoming the input for another. 192 | Chapter 5: Text Analysis In the parsed UFO-sighting data set we’ve been working with, some of the location values include qualifiers indicating that the sighting took place “near,” “close to,” or “outside of” a city or town. We can use replace to standardize these to “near”: SELECT location ,replace(replace(location,'close to','near') ,'outside of','near') as location_clean FROM ( SELECT split_part(split_part(sighting_report,'Shape',1) ,'Location: ',2) as location FROM ufo ) a ; location location_clean --------------------------- --------------------- Tombstone (outside of), AZ Tombstone (near), AZ Terrell (close to), TX Terrell (near), TX Tehachapie (outside of), CA Tehachapie (near), CA ... ... The top 10 sighting locations are graphed in Figure 5-6. Figure 5-6. Most common locations of UFO sightings Now we have parsed and cleaned all the elements of the sighting_report field into distinct, appropriately typed columns. The final code looks like this: SELECT case when occurred = '' then null when length(occurred) < 8 then null else occurred::timestamp end as occurred Text Transformations | 193 ,entered_as ,case when length(reported) < 8 then null else reported::timestamp end as reported ,case when posted = '' then null else posted::date end as posted ,replace(replace(location,'close to','near'),'outside of','near') as location ,initcap(shape) as shape ,trim(duration) as duration FROM ( SELECT split_part( split_part(split_part(sighting_report,' (Entered',1) ,'Occurred : ',2) ,'Reported',1) as occurred ,split_part( split_part(sighting_report,')',1) ,'Entered as : ',2) as entered_as ,split_part( split_part( split_part( split_part(sighting_report,'Post',1) ,'Reported: ',2) ,' AM',1) ,' PM',1) as reported ,split_part( split_part(sighting_report,'Location',1) ,'Posted: ',2) as posted ,split_part( split_part(sighting_report,'Shape',1) ,'Location: ',2) as location ,split_part( split_part(sighting_report,'Duration',1) ,'Shape: ',2) as shape ,split_part(sighting_report,'Duration:',2) as duration FROM ufo ) a ; occurred entered_as reported posted location shape duration -------- ---------- -------- ------- ---------- -------- ---------- 1988-... 8-8-198... 2018-... 2018... Amity, ... Unknown 4 minutes 2018-... 07/41/1... 2018-... 2018... Bakersf... Triangle 15 minutes 2018-... 08/01/1... 2018-... 2018... Naples,... Light 10 seconds ... ... ... ... ... ... ... This piece of SQL code", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 139 + }, + { + "text": "entered_as reported posted location shape duration -------- ---------- -------- ------- ---------- -------- ---------- 1988-... 8-8-198... 2018-... 2018... Amity, ... Unknown 4 minutes 2018-... 07/41/1... 2018-... 2018... Bakersf... Triangle 15 minutes 2018-... 08/01/1... 2018-... 2018... Naples,... Light 10 seconds ... ... ... ... ... ... ... This piece of SQL code can be reused in other queries, or it can be used to copy the raw UFO data into a new, cleaned-up table. Alternatively, it could be turned into a 194 | Chapter 5: Text Analysis view or put into a common table expression for reuse. Chapter 8 will discuss these strategies in more detail. We’ve seen how to apply parsing and transformation functions to clean and improve the analysis value of text data that has some amount of structure in it. Next, we’ll look at the other field in the UFO sightings data set, the free text description field, and learn how to use SQL functions to search for specific elements. Finding Elements Within Larger Blocks of Text Parsing and transformations are common operations applied to text data to prepare it for analysis. Another common operation with text data is finding strings within larger blocks of text. This can be done to filter results, categorize records, or replace the searched-for strings with alternate values. Wildcard Matches: LIKE, ILIKE SQL has several functions for matching patterns within strings. The LIKE operator matches the specified pattern within the string. In order to allow it to match a pat‐ tern and not just find an exact match, wildcard symbols can be added before, after, or in the middle of the pattern. The “%” wildcard matches zero or more characters, while the “_” wildcard matches exactly one character. If the goal is to match the “%” or “_” itself, place the backslash escape symbol (“\\”) in front of that character: SELECT 'this is an example string' like '%example%'; true SELECT 'this is an example string' like '%abc%'; false SELECT 'this is an example string' like '%this_is%'; true The LIKE operator can be used in a number of clauses within the SQL statement. It can be used to filter records in the WHERE clause. For example, some reporters men‐ tion that they were with a spouse at the time, and so we might want to find out how many reports mention the word “wife.” Since we want to find the string anywhere in the description text, we’ll place the “%” wildcard before and after “wife”: SELECT count(*) FROM ufo WHERE description like '%wife%' ; Finding Elements Within Larger Blocks of Text | 195 count ----- 6231 We can see that more than six thousand reports mention “wife.” However, this will return only matches on the lowercase string. What if some reporters mention “Wife,” or they left Caps Lock on and typed in “WIFE” instead? There are two options for making the search case insensitive. One option is to transform the field to be searched using either the upper or lower function discussed in the previous", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 140 + }, + { + "text": "lowercase string. What if some reporters mention “Wife,” or they left Caps Lock on and typed in “WIFE” instead? There are two options for making the search case insensitive. One option is to transform the field to be searched using either the upper or lower function discussed in the previous section, which has the effect of making the search case insensitive since characters are all either uppercase or lowercase: SELECT count(*) FROM ufo WHERE lower(description) like '%wife%' ; count ----- 6439 Another way to accomplish this is with the ILIKE operator, which is effectively a case- insensitive LIKE operator. The drawback is that it is not available in every database; notably, MySQL and SQL Server do not support it. However, it’s a nice, compact syn‐ tax option if you are working in a database that does support it: SELECT count(*) FROM ufo WHERE description ilike '%wife%' ; count ----- 6439 Any of these variations of LIKE and ILIKE can be negated with NOT. So, for example, to find the records that do not mention “wife,” we can use NOT LIKE: SELECT count(*) FROM ufo WHERE lower(description) not like '%wife%' ; count ----- 89024 Filtering on multiple strings is possible with AND and OR operators: SELECT count(*) FROM ufo WHERE lower(description) like '%wife%' 196 | Chapter 5: Text Analysis or lower(description) like '%husband%' ; count ----- 10571 Be careful to use parentheses to control the order of operations when using OR in conjunction with AND operators, or you might get unexpected results. For example, these WHERE clauses do not return the same result since OR is evaluated before AND: SELECT count(*) FROM ufo WHERE lower(description) like '%wife%' or lower(description) like '%husband%' and lower(description) like '%mother%' ; count ----- 6610 SELECT count(*) FROM ufo WHERE (lower(description) like '%wife%' or lower(description) like '%husband%' ) and lower(description) like '%mother%' ; count ----- 382 In addition to filtering in WHERE or JOIN...ON clauses, LIKE can be used in the SELECT clause to categorize or aggregate certain records. Let’s start with categoriza‐ tion. The LIKE operator can be used within a CASE statement to label and group records. Some of the descriptions mention an activity the observer was doing during or prior to the sighting, such as driving or walking. We can find out how many descriptions contain such terms by using a CASE statement with LIKE: SELECT case when lower(description) like '%driving%' then 'driving' when lower(description) like '%walking%' then 'walking' when lower(description) like '%running%' then 'running' when lower(description) like '%cycling%' then 'cycling' when lower(description) like '%swimming%' then 'swimming' else 'none' end as activity ,count(*) FROM ufo GROUP BY 1 Finding Elements Within Larger Blocks of Text | 197 ORDER BY 2 desc ; activity count -------- ----- none 77728 driving 11675 walking 4516 running 1306 swimming 196 cycling 42 The most common activity was driving, whereas not many people report sightings while swimming or cycling. This is perhaps not surprising, since these activities are simply less common than driving. Although values derived through text-parsing transformation", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 141 + }, + { + "text": "-------- ----- none 77728 driving 11675 walking 4516 running 1306 swimming 196 cycling 42 The most common activity was driving, whereas not many people report sightings while swimming or cycling. This is perhaps not surprising, since these activities are simply less common than driving. Although values derived through text-parsing transformation functions can be used in JOIN criteria, database performance is often a problem. Consider parsing and/or transforming in a sub‐ query and then joining the result with an exact match in the JOIN clause. Note that this CASE statement labels each description with only one of the activities and evaluates whether each record matches the pattern in the order in which the statement is written. A description that contains both “driving” and “walking” will be labeled as “driving.” This is appropriate in many cases, but particularly when analyz‐ ing longer text such as from reviews, survey comments, or support tickets, the ability to label records with multiple categories is important. For this type of use case, a ser‐ ies of binary or BOOLEAN flag columns is called for. We saw earlier that LIKE can be used to generate a BOOLEAN response of TRUE or FALSE, and we can use this to label rows. In the data set, a number of descriptions mention the direction in which the object was detected, such as north or south, and some mention more than one direction. We might want to label each record with a field indicating whether the description mentions each direction: SELECT description ilike '%south%' as south ,description ilike '%north%' as north ,description ilike '%east%' as east ,description ilike '%west%' as west ,count(*) FROM ufo GROUP BY 1,2,3,4 ORDER BY 1,2,3,4 ; 198 | Chapter 5: Text Analysis south north east west count ----- ----- ---- ----- ----- false false false false 43757 false false false true 3963 false false true false 5724 false false true true 4202 false true false false 4048 false true false true 2607 false true true false 3299 false true true true 2592 true false false false 3687 true false false true 2571 true false true false 3041 true false true true 2491 true true false false 3440 true true false true 2064 true true true false 2684 true true true true 5293 The result is a matrix of BOOLEANs that can be used to find the frequency of various combinations of directions, or to find when a direction is used without any of the other directions in the same description. All of the combinations are useful in some contexts, particularly in building data sets that will be used by others to explore the data, or in a BI or visualization tool. How‐ ever, sometimes it is more useful to summarize the data further and perform an aggregation on the records that contain a string pattern. Here we will count the records, but other aggregations such as sum and average can be used if the data set contains other numerical fields, such as sales figures: SELECT count(case", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 142 + }, + { + "text": "more useful to summarize the data further and perform an aggregation on the records that contain a string pattern. Here we will count the records, but other aggregations such as sum and average can be used if the data set contains other numerical fields, such as sales figures: SELECT count(case when description ilike '%south%' then 1 end) as south ,count(case when description ilike '%north%' then 1 end) as north ,count(case when description ilike '%west%' then 1 end) as west ,count(case when description ilike '%east%' then 1 end) as east FROM ufo ; south north west east ----- ----- ----- ----- 25271 26027 25783 29326 We now have a much more compact summary of the frequency of direction terms in the description field, and we can see that “east” is mentioned more often than other directions. The results are graphed in Figure 5-7. Finding Elements Within Larger Blocks of Text | 199 Figure 5-7. Frequency of compass directions mentioned in UFO sighting reports In the preceding query, we still allow a record that contains more than one direction to be counted more than once. However, there is no longer visibility into which spe‐ cific combinations exist. Complexity can be added into the query as needed to handle such cases, with a statement such as: count(case when description ilike '%east%' and description ilike '%north%' then 1 end) as east Pattern matching with LIKE, NOT LIKE, and ILIKE is flexible and can be used in various places in a SQL query to filter, categorize, and aggregate data for a variety of output needs. These operators can be used in combination with the text-parsing and transformation functions we discussed earlier for even more flexibility. Next, I’ll dis‐ cuss handling multiple elements when the matches are exact before returning to more patterns in a discussion of regular expressions. Exact Matches: IN, NOT IN Before we move on to more complex pattern matching with regular expressions, it’s worth looking at a couple of additional operators that are useful in text analysis. Although not strictly about pattern matching, they are often useful in combination with LIKE and its relatives in order to come up with a rule set that includes exactly the right set of results. The operators are IN and its negation, NOT IN. These allow you to specify a list of matches, resulting in more compact code. Let’s imagine we are interested in categorizing the sightings based on the first word of the description. We can find the first word using the split_part function, with a space character as the delimiter. Many reports start with a color as the first word. We 200 | Chapter 5: Text Analysis might want to filter the records in order to take a look at reports that start by naming a color. This can be done by listing each color with an OR construction: SELECT first_word, description FROM ( SELECT split_part(description,' ',1) as first_word ,description FROM ufo ) a WHERE first_word = 'Red' or first_word = 'Orange' or first_word", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 143 + }, + { + "text": "order to take a look at reports that start by naming a color. This can be done by listing each color with an OR construction: SELECT first_word, description FROM ( SELECT split_part(description,' ',1) as first_word ,description FROM ufo ) a WHERE first_word = 'Red' or first_word = 'Orange' or first_word = 'Yellow' or first_word = 'Green' or first_word = 'Blue' or first_word = 'Purple' or first_word = 'White' ; first_word description ---------- ---------------------------------------------------- Blue Blue Floating LightSaw blue light hovering... White White dot of light traveled across the sky, very... Blue Blue Beam project known seen from the high desert... ... ... Using an IN list is more compact and often less error-prone, particularly when there are other elements in the WHERE clause. IN takes a comma-separated list of items to match. The data type of elements should match the data type of the column. If the data type is numeric, the elements should be numbers; if the data type is text, the ele‐ ments should be quoted as text (even if the element is a number): SELECT first_word, description FROM ( SELECT split_part(description,' ',1) as first_word ,description FROM ufo ) a WHERE first_word in ('Red','Orange','Yellow','Green','Blue','Purple','White') ; first_word description ---------- ---------------------------------------------------- Red Red sphere with yellow light in middleMy Grandson... Blue Blue light fireball shape shifted into several... Orange Orange lights.Strange orange-yellow hovering not... ... ... The two forms are identical in their results, and the frequencies are shown in Figure 5-8. Finding Elements Within Larger Blocks of Text | 201 Figure 5-8. Frequency of select colors used as the first word in UFO sighting descriptions The main benefit of IN and NOT IN is that they make code more compact and reada‐ ble. This can come in handy when creating more complex categorizations in the SELECT clause. For example, imagine we wanted to categorize and count the records by the first word into colors, shapes, movements, or other possible words. We might come up with something like the following that combines elements of parsing, trans‐ formations, pattern matching, and IN lists: SELECT case when lower(first_word) in ('red','orange','yellow','green', 'blue','purple','white') then 'Color' when lower(first_word) in ('round','circular','oval','cigar') then 'Shape' when first_word ilike 'triang%' then 'Shape' when first_word ilike 'flash%' then 'Motion' when first_word ilike 'hover%' then 'Motion' when first_word ilike 'pulsat%' then 'Motion' else 'Other' end as first_word_type ,count(*) FROM ( SELECT split_part(description,' ',1) as first_word ,description FROM ufo ) a GROUP BY 1 ORDER BY 2 desc ; 202 | Chapter 5: Text Analysis first_word_type count --------------- ----- Other 85268 Color 6196 Shape 2951 Motion 1048 Of course, given the nature of this data set, it would likely take many more lines of code and rules to accurately categorize the reports by first word. SQL allows you to create a variety of complex and nuanced expressions to deal with text data. Next, we’ll turn to some even more sophisticated ways to work with text data in SQL, using regu‐ lar expressions. Regular Expressions There are a number of ways to match", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 144 + }, + { + "text": "by first word. SQL allows you to create a variety of complex and nuanced expressions to deal with text data. Next, we’ll turn to some even more sophisticated ways to work with text data in SQL, using regu‐ lar expressions. Regular Expressions There are a number of ways to match patterns in SQL. One of the most powerful methods, though it is also confusing, is the use of regular expressions (regex). I will admit to finding regular expressions intimidating, and I avoided using them for a long time in my data analysis career. In a pinch, I was lucky enough to have collea‐ gues who were willing to share code snippets and get my work unstuck. It was only when I ended up with a big text analysis project that I decided it was finally time to learn about them. Regular expressions are sequences of characters, many with special meanings, that define search patterns. The main challenge in learning regex, and in using and main‐ taining code that contains it, is that the syntax is not particularly intuitive. Code snip‐ pets don’t read anything like a human language, or even like computer languages such as SQL or Python. With a working knowledge of the special characters, however, the code can be written and deciphered. Like the code for all our queries, it’s a good idea to start simple, build in complexity as needed, and check results as you go. And leave comments liberally, both for other analysts and for future you. Regex is a language, but it’s one that is used only within other languages. For example, regular expressions can be called within Java, Python, and SQL, but there is no inde‐ pendent way to program with them. All major databases have some implementation of regex. The syntax isn’t always exactly the same, but as with other functions, once you have a sense of the possibilities, adjusting syntax to your environment should be possible. A full explanation, and all of the syntax and ways to use regex, is beyond the scope of this book, but I’ll show you enough for you to get started and accomplish a number of common tasks in SQL. For a more thorough introduction, Learning Regular Expressions by Ben Forta (O’Reilly) is a good choice. Here I’ll start by introducing the ways to indicate to the database that you are using a regex, and then I’ll introduce the syntax, before moving on to some examples of how regex can be useful in the UFO sighting reports analysis. Finding Elements Within Larger Blocks of Text | 203 Regex can be used in SQL statements in a couple of ways. The first is with POSIX comparators, and the second is with regex functions. POSIX stands for Portable Operating System Interface and refers to a set of IEEE standards, but you don’t need to know any more than that to use POSIX comparators in your SQL code. The first comparator is the ~ (tilde) symbol, which compares two statements", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 145 + }, + { + "text": "second is with regex functions. POSIX stands for Portable Operating System Interface and refers to a set of IEEE standards, but you don’t need to know any more than that to use POSIX comparators in your SQL code. The first comparator is the ~ (tilde) symbol, which compares two statements and returns TRUE if one string is contained in the other. As a simple example, we can check to see whether the string “The data is about UFOs” contains the string “data”: SELECT 'The data is about UFOs' ~ 'data' as comparison; comparison ---------- true The return value is a BOOLEAN, TRUE or FALSE. Note that, although it doesn’t con‐ tain any special syntax, “data” is a regex. Regular expressions can also contain normal text strings. This example is similar to what could be accomplished with a LIKE oper‐ ator. The ~ comparator is case sensitive. To make it case insensitive, similar to ILIKE, use ~* (the tilde followed by an asterisk): SELECT 'The data is about UFOs' ~* 'DATA' as comparison; comparison ---------- true To negate the comparator, place an ! (exclamation point) before the tilde or tilde- asterisk combination: SELECT 'The data is about UFOs' !~ 'alligators' as comparison; comparison ---------- true Table 5-1 summarizes the four POSIX comparators. Table 5-1. POSIX comparators Syntax What it does Case sensitive? ~ Compares two statements and returns TRUE if one is contained in the other Yes ~* Compares two statements and returns TRUE if one is contained in the other No !~ Compares two statements and returns FALSE if one is contained in the other Yes !~* Compares two statements and returns FALSE if one is contained in the other No Now that we have a way to introduce regex into our SQL, let’s get familiar with some of the special pattern-matching syntax it offers. The first special character to know is the . (period) symbol, a wildcard that is used to match any single character: 204 | Chapter 5: Text Analysis SELECT 'The data is about UFOs' ~ '. data' as comparison_1 ,'The data is about UFOs' ~ '.The' as comparison_2 ; comparison_1 comparison_2 ------------ ------------ true false Let’s break this down in order to understand what’s going on and develop our intu‐ ition about how regex works. In the first comparison, the pattern tries to match any character, indicated by the period, a space, and then the word “data.” This pattern matches the string “e data” in the example sentence, so TRUE is returned. If this seems counterintuitive, since there are additional characters before the letter “e” and after the word “data,” remember that the comparator is only looking for this pattern somewhere within the string, similar to a LIKE operator. In the second comparison, the pattern tries to match any character followed by “The.” Since in the example sen‐ tence “The” is the start of the string and there are no characters before it, the value FALSE is returned. To match multiple characters, use the * (asterisk)", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 146 + }, + { + "text": "a LIKE operator. In the second comparison, the pattern tries to match any character followed by “The.” Since in the example sen‐ tence “The” is the start of the string and there are no characters before it, the value FALSE is returned. To match multiple characters, use the * (asterisk) symbol. This will match zero or more characters, similar to using the % (percent) symbol in a LIKE statement. This use of the asterisk is different from placing it immediately after the tilde (~*), which makes the match case insensitive. Notice, however, that in this case “%” is not a wild‐ card and is instead treated as a literal character to be matched: SELECT 'The data is about UFOs' ~ 'data *' as comparison_1 ,'The data is about UFOs' ~ 'data %' as comparison_2 ; comparison_1 comparison_2 ------------ ------------ true false The next special characters to know are [ and ] (left and right brackets). These are used to enclose a set of characters, any one of which must match. The brackets match a single character even though multiple characters can be between them, though we’ll see shortly how to match more than one time. One use for the brackets is to make part of a pattern case insensitive by enclosing the uppercase and lowercase letters within the brackets (do not use a comma, as that would match the comma character itself): SELECT 'The data is about UFOs' ~ '[Tt]he' as comparison; comparison ---------- true Finding Elements Within Larger Blocks of Text | 205 In this example, the pattern will match either “the” or “The”; since this string starts the example sentence, the statement returns the value TRUE. This isn’t quite the same thing as the case-insensitive match ~*, because in this case variations such as “tHe” and “THE” do not match the pattern: SELECT 'The data is about UFOs' ~ '[Tt]he' as comparison_1 ,'the data is about UFOs' ~ '[Tt]he' as comparison_2 ,'tHe data is about UFOs' ~ '[Tt]he' as comparison_3 ,'THE data is about UFOs' ~ '[Tt]he' as comparison_4 ; comparison_1 comparison_2 comparison_3 comparison_4 ------------ ------------ ------------ ------------ true true false false Another use of the bracket set match is to match a pattern that includes a number, allowing for any number. For example, imagine we wanted to match any description that mentions “7 minutes,” “8 minutes,” or “9 minutes.” This could be accomplished with a CASE statement with several LIKE operators, but with regex the pattern syntax is more compact: SELECT 'sighting lasted 8 minutes' ~ '[789] minutes' as comparison; comparison ---------- true To match any number, we could enclose all the digits between the brackets: [0123456789] However, regex allows a range of characters to be entered with a - (dash) separator. All of the numbers can be indicated by [0-9]. Any smaller range of numbers can be used as well, such as [0-3] or [4-9]. This pattern, with a range, is equivalent to the last example that listed out each number: SELECT 'sighting lasted 8 minutes'", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 147 + }, + { + "text": "with a - (dash) separator. All of the numbers can be indicated by [0-9]. Any smaller range of numbers can be used as well, such as [0-3] or [4-9]. This pattern, with a range, is equivalent to the last example that listed out each number: SELECT 'sighting lasted 8 minutes' ~ '[7-9] minutes' as comparison; comparison ---------- true Ranges of letters can be matched in a similar way. Table 5-2 summarizes the range patterns that are most useful in SQL analysis. Nonnumber and nonletter values can also be placed between brackets, as in [$%@]. 206 | Chapter 5: Text Analysis Table 5-2. Regex range patterns Range pattern Purpose [0-9] Match any number [a-z] Match any lowercase letter [A-Z] Match any uppercase letter [A-Za-z0-9] Match any lower- or uppercase letter, or any number [A-z] Match any ASCII character; generally not used because it matches everything, including symbols If the desired pattern match contains more than one instance of a particular value or type of value, one option is to include as many ranges as needed, one after the other. For example, we can match a three-digit number by repeating the number range notation three times: SELECT 'driving on 495 south' ~ 'on [0-9][0-9][0-9]' as comparison; comparison ---------- true Another option is to use one of the optional special syntaxes for repeating a pattern multiple times. This can be useful when you don’t know exactly how many times the pattern will repeat, but be careful to check the results to make sure you don’t acciden‐ tally return more matches than intended. To match one or more times, place the + (plus) symbol after the pattern: SELECT 'driving on 495 south' ~ 'on [0-9]+' as comparison_1 ,'driving on 1 south' ~ 'on [0-9]+' as comparison_2 ,'driving on 38east' ~ 'on [0-9]+' as comparison_3 ,'driving on route one' ~ 'on [0-9]+' as comparison_4 ; comparison_1 comparison_2 comparison_3 comparison_4 ------------ ------------ ------------ ------------ true true true false Table 5-3 summarizes the other options for indicating the number of times to repeat a pattern. Finding Elements Within Larger Blocks of Text | 207 Table 5-3. Regex patterns for matching a character set multiple times; in each case, the symbol or symbols are placed immediately after the set expression Symbol Purpose + Match the character set one or more times * Match the character set zero or more times ? Match the character set zero or one time { } Match the character set the number of times specified between the curly braces; for example, {3} matches exactly three times { , } Match the character set any number of times in a range specified by the comma-separated numbers between the curly braces; for example, {3,5} matches between three and five times Sometimes rather than matching a pattern, we want to find items that do not match a pattern. This can be done by placing the ^ (caret) symbol before the pattern, which serves to negate the pattern: SELECT 'driving on 495 south' ~ 'on [0-9]+' as", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 148 + }, + { + "text": "between three and five times Sometimes rather than matching a pattern, we want to find items that do not match a pattern. This can be done by placing the ^ (caret) symbol before the pattern, which serves to negate the pattern: SELECT 'driving on 495 south' ~ 'on [0-9]+' as comparison_1 ,'driving on 495 south' ~ 'on ^[0-9]+' as comparison_2 ,'driving on 495 south' ~ '^on [0-9]+' as comparison_3 ; comparison_1 comparison_2 comparison_3 ------------ ------------ ------------ true false false We might want to match a pattern that includes one of the special characters, so we need a way to tell the database to check for that literal character and not treat it as special. To do this, we need an escape character, which is the \\ (backslash) symbol in regex: SELECT '\"Is there a report?\" she asked' ~ '\\?' as comparison_1 ,'it was filed under ^51.' ~ '^[0-9]+' as comparison_2 ,'it was filed under ^51.' ~ '\\^[0-9]+' as comparison_3 ; comparison_1 comparison_2 comparison_3 ------------ ------------ ------------ true false true In the first line, omitting the backslash before the question mark causes the database to return an “invalid regular expression” error (the exact wording of the error may be different depending on the database type). In the second line, even though ^ is fol‐ lowed by one or more digits ([0-9]+), the database interprets the ^ in the comparison '^[0-9]+' to be a negation and will evaluate whether the string does not include the specified digits. The third line escapes the caret with a backslash, and the database now interprets this as the literal ^ character. 208 | Chapter 5: Text Analysis Text data usually includes whitespace characters. These range from the space, which our eyes notice, to the subtle and sometimes unprinted tab and newline characters. We will see later how to replace these with regex, but for now let’s stick to how to match them in a regex. Tabs are matched with \\t. Newlines are matched with \\r for a carriage return or \\n for a line feed, and depending on the operating system, some‐ times both are required: \\r\\n. Experiment with your environment by running a few simple queries to see what returns the desired result. To match any whitespace char‐ acter, use \\s, but note that this also matches the space character: SELECT 'spinning flashing and whirling' ~ '\\n' as comparison_1 ,'spinning flashing and whirling' ~ '\\s' as comparison_2 ,'spinning flashing' ~ '\\s' as comparison_3 ,'spinning' ~ '\\s' as comparison_4 ; comparison_1 comparison_2 comparison_3 comparison_4 ------------ ------------ ------------ ------------ true true true false SQL query tools or SQL query parsers may have trouble interpret‐ ing new lines typed directly into them and thus may return an error. If this is the case, try copying and pasting the text from the source rather than typing it in. All SQL query tools should be able to work with newlines that exist in a database table, however. Similar to mathematical expressions, parentheses can be used to enclose expressions that should", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 149 + }, + { + "text": "If this is the case, try copying and pasting the text from the source rather than typing it in. All SQL query tools should be able to work with newlines that exist in a database table, however. Similar to mathematical expressions, parentheses can be used to enclose expressions that should be treated together. For example, we might want to match a somewhat complex pattern that repeats several times: SELECT 'valid codes have the form 12a34b56c' ~ '([0-9]{2}[a-z]){3}' as comparison_1 ,'the first code entered was 123a456c' ~ '([0-9]{2}[a-z]){3}' as comparison_2 ,'the second code entered was 99x66y33z' ~ '([0-9]{2}[a-z]){3}' as comparison_3 ; comparison_1 comparison_2 comparison_3 ------------ ------------ ------------ true false true All three lines use the same regex pattern, '([0-9]{2}[a-z]){3}', for matching. The pattern inside the parentheses, [0-9]{2}[a-z], looks for two digits followed by a Finding Elements Within Larger Blocks of Text | 209 lowercase letter. Outside of the parentheses, {3} indicates that the whole pattern should be repeated three times. The first line follows this pattern, since it contains the string 12a34b56c. The second line does not match the pattern; it does have two digits followed by a lowercase letter (23a) and then two more digits (23a45), but this second repetition is followed by a third digit rather than by another lowercase letter (23a456), so there is no match. The third line has a matching pattern, 99x66y33z. As we’ve just seen, regex can be used in any number of combinations with other expressions, both regex and normal text, to create pattern-matching code. In addition to specifying what to match, regex can be used to specify where to match. Use the spe‐ cial character \\y to match a pattern starting at the beginning or end of a word (in some databases, this might be \\b instead). As an example, imagine we were interested in finding the word “car” in the UFO sighting reports. We could write an expression like this: SELECT 'I was in my car going south toward my home' ~ 'car' as comparison; comparison ---------- true It finds “car” in the string and returns TRUE as expected. However, let’s look at a few more strings from the data set, looking for the same expression: SELECT 'I was in my car going south toward my home' ~ 'car' as comparison_1 ,'UFO scares cows and starts stampede breaking' ~ 'car' as comparison_2 ,'I''m a carpenter and married father of 2.5 kids' ~ 'car' as comparison_3 ,'It looked like a brown boxcar way up into the sky' ~ 'car' as comparison_4 ; comparison_1 comparison_2 comparison_3 comparison_4 ------------ ------------ ------------ ------------ true true true true All of these strings match the pattern “car” as well, even though “scares,” “carpenter,” and “boxcar” aren’t exactly what was intended when we went looking for mentions of cars. To fix this, we can add \\y to the beginning and end of the “car” pattern in our expression: SELECT 'I was in my car going south toward my home' ~ '\\ycar\\y' as comparison_1 ,'UFO scares cows and starts stampede", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 150 + }, + { + "text": "was intended when we went looking for mentions of cars. To fix this, we can add \\y to the beginning and end of the “car” pattern in our expression: SELECT 'I was in my car going south toward my home' ~ '\\ycar\\y' as comparison_1 ,'UFO scares cows and starts stampede breaking' ~ '\\ycar\\y' as comparison_2 210 | Chapter 5: Text Analysis ,'I''m a carpenter and married father of 2.5 kids' ~ '\\ycar\\y' as comparison_3 ,'It looked like a brown boxcar way up into the sky' ~ '\\ycar\\y' as comparison_4 ; comparison_1 comparison_2 comparison_3 comparison_4 ------------ ------------ ------------ ------------ true false false false Of course, in this simple example, we could have simply added spaces before and after the word “car” with the same result. The benefit of the pattern is that it will also pick up cases in which the pattern is at the beginning of a string and thus does not have a leading space: SELECT 'Car lights in the sky passing over the highway' ~* '\\ycar\\y' as comparison_1 ,'Car lights in the sky passing over the highway' ~* ' car ' as comparison_2 ; comparison_1 comparison_2 ------------ ------------ true false The pattern '\\ycar\\y' makes a case-insensitive match when “Car” is the first word, but the pattern ' car ' does not. To match the beginning of an entire string, use the \\A special character, and to match the end of a string, use \\Z: SELECT 'Car lights in the sky passing over the highway' ~* '\\Acar\\y' as comparison_1 ,'I was in my car going south toward my home' ~* '\\Acar\\y' as comparison_2 ,'An object is sighted hovering in place over my car' ~* '\\ycar\\Z' as comparison_3 ,'I was in my car going south toward my home' ~* '\\ycar\\Z' as comparison_4 ; comparison_1 comparison_2 comparison_3 comparison_4 ------------ ------------ ------------ ------------ true false true false In the first line, the pattern matches “Car” at the beginning of the string. The second line starts with “I,” so the pattern does not match. In the third line, the pattern is looking for “car” at the end of the string and does match it. Finally, in the fourth line, the last word is “home,” so the pattern does not match. If this is your first time working with regular expressions, it may take a few read- throughs and some experimentation in your SQL editor to get the hang of them. Finding Elements Within Larger Blocks of Text | 211 There’s nothing like working with real examples to help solidify learning, so next I’ll go through some applications to our UFO sightings analysis, and I’ll also introduce a couple of specific regex SQL functions. Regular expression implementations vary widely by database ven‐ dor. The POSIX operators in this section work in Postgres and in databases derived from Postgres such as Amazon Redshift, but not necessarily in others. An alternative to the ~ operator is the rlike or regexp_like func‐ tion (depending on the database). These have the following format: regexp_like(string, pattern, optional_parameters) The first", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 151 + }, + { + "text": "POSIX operators in this section work in Postgres and in databases derived from Postgres such as Amazon Redshift, but not necessarily in others. An alternative to the ~ operator is the rlike or regexp_like func‐ tion (depending on the database). These have the following format: regexp_like(string, pattern, optional_parameters) The first example in this section would be written as: SELECT regexp_like('The data is about UFOs','data') as comparison; The optional parameters control matching type, such as whether the match is case insensitive. Many of these databases have additional functions not covered here, such as regexp_substr to find matching substrings, and regexp_count to find the number of times a pattern is matched. Postgres supports POSIX but unfortunately does not support these other functions. Organizations that expect to do a lot of text analy‐ sis will do well to choose a database type with a robust set of regu‐ lar expression functions. Finding and replacing with regex In the previous section, we discussed regular expressions and how to construct pat‐ terns with regex to match parts of strings in our data sets. Let’s apply this technique to the UFO sightings data set to see how it works in practice. Along the way, I’ll also introduce some additional regex SQL functions. The sighting reports contain a variety of details, such as what the reporter was doing at the time of the sighting and when and where they were doing it. Another detail commonly mentioned is seeing some number of lights. As a first example, let’s find the descriptions that contain a number and the word “light” or “lights.” For the sake of display in this book, I’ll just check the first 100 characters, but this code can also work across the entire description field: SELECT left(description,50) FROM ufo WHERE left(description,50) ~ '[0-9]+ light[s ,.]' ; left -------------------------------------------------- 212 | Chapter 5: Text Analysis Was walking outside saw 5 lights in a line changed 2 lights about 5 mins apart, goin from west to eas Black triangular aircraft with 3 lights hovering a ... The regular expression pattern matches any number of digits ([0-9]+), followed by a space, then the string “light”, and finally either a letter “s,” a space, a comma, or a period. In addition to finding the relevant records, we might want to split out just the part that refers to the number and the word “lights.” To do this, we’ll use the regex function regexp_matches. Regex function support varies widely by database vendor and sometimes by database software version. SQL Server does not sup‐ port the functions, while MySQL has minimal support for them. Analytic databases such as Redshift, Snowflake, and Vertica sup‐ port a variety of useful functions. Postgres has only match and replace functions. Explore the documentation for your database for specific function availability. The regexp_matches function takes two arguments: a string to search and a regex match pattern. It returns an array of the string(s) that matched the pattern. If there are no matches, a null value is returned.", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 152 + }, + { + "text": "match and replace functions. Explore the documentation for your database for specific function availability. The regexp_matches function takes two arguments: a string to search and a regex match pattern. It returns an array of the string(s) that matched the pattern. If there are no matches, a null value is returned. Since the return value is an array, we’ll use an index of [1] to return just a single value as a VARCHAR, which will allow for addi‐ tional string manipulation as needed. If you are working in another type of database, the regexp_substr function is similar to regexp_matches, but it returns a VAR‐ CHAR value, so there is no need to add the [1] index. An array is a collection of objects stored together in the computer’s memory. In databases, arrays are enclosed in { } (curly braces), and this is a good way to spot that something in the database is not one of the regular data types we’ve been working with so far. Arrays have some advantages when storing and retrieving data, but they are not as easy to work with in SQL since they require special syn‐ tax. Elements in an array are accessed using [ ] (square brackets) notation. For our purposes here, it’s enough to know that the first element is found with [1], the second with [2], and so on. Building on our example, we can parse the desired value, the number, and the word “light(s)” from the description field and then GROUP BY this value and the most common variations: SELECT (regexp_matches(description,'[0-9]+ light[s ,.]'))[1] ,count(*) FROM ufo WHERE description ~ '[0-9]+ light[s ,.]' Finding Elements Within Larger Blocks of Text | 213 GROUP BY 1 ORDER BY 2 desc ; regexp_matches count -------------- ----- 3 lights 1263 2 lights 565 4 lights 549 ... ... The top 10 results are graphed in Figure 5-9. Figure 5-9. Number of lights mentioned at the beginning of UFO sighting descriptions Reports mentioning three lights are more than twice as common as the second most often mentioned number of lights, and from two to six lights are most commonly seen. To find the full range of the number of lights, we can parse the matched text and then find the min and max values: SELECT min(split_part(matched_text,' ',1)::int) as min_lights ,max(split_part(matched_text,' ',1)::int) as max_lights FROM ( SELECT (regexp_matches(description ,'[0-9]+ light[s ,.]') )[1] as matched_text ,count(*) FROM ufo WHERE description ~ '[0-9]+ light[s ,.]' GROUP BY 1 214 | Chapter 5: Text Analysis ) a ; min_lights max_lights ---------- ----- 0 2000 At least one report mentions two thousand lights, and the minimum value of zero lights is also mentioned. We might want to review these reports further to see if there is anything else interesting or unusual about these extreme values. In addition to finding matches, we might want to replace the matched text with some alternate text. This is particularly useful when trying to clean a data set of text that has multiple spellings for the same", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 153 + }, + { + "text": "see if there is anything else interesting or unusual about these extreme values. In addition to finding matches, we might want to replace the matched text with some alternate text. This is particularly useful when trying to clean a data set of text that has multiple spellings for the same underlying thing. The regexp_replace function can accomplish this. It is similar to the replace function discussed earlier in the chapter, but it can take a regular expression argument as the pattern to match. The syntax is similar to the replace function: regexp_replace(field or string, pattern, replacement value) Let’s put this to work to try to clean up the duration field that we parsed out of the sighting_report column earlier. This appears to be a free text entry field, and there are more than eight thousand different values. However, inspection reveals that there are common themes—most refer to some combination of seconds, minutes, and hours: SELECT split_part(sighting_report,'Duration:',2) as duration ,count(*) as reports FROM ufo GROUP BY 1 ; duration reports -------- ------- 10 minutes 4571 1 hour 1599 10 min 333 10 mins 150 >1 hour 113 ... ... Within this sample, the durations of “10 minutes,” “10 min,” and “10 mins” all repre‐ sent the same amount of time, but the database doesn’t know to combine them because the spellings are slightly different. We could use a series of nested replace functions to convert all these different spellings. However, we would also have to take into account other variations, such as capitalizations. Regex is handy in this situation, allowing us to create more compact code. The first step is to develop a pattern that matches the desired string, which we can do with the regexp_matches function. It’s a Finding Elements Within Larger Blocks of Text | 215 good idea to review this intermediate step to make sure we’re matching the correct text: SELECT duration ,(regexp_matches(duration ,'\\m[Mm][Ii][Nn][A-Za-z]*\\y') )[1] as matched_minutes FROM ( SELECT split_part(sighting_report,'Duration:',2) as duration ,count(*) as reports FROM ufo GROUP BY 1 ) a ; duration matched_minutes ------------ --------------- 10 min. min 10 minutes+ minutes 10 min min 10 minutes + minutes 10 minutes? minutes 10 minutes minutes 10 mins mins ... ... Let’s break this down. In the subquery, the duration value is split out of the sight ing_report field. Then the regexp_matches function looks for strings that match the pattern: '\\m[Mm][Ii][Nn][A-Za-z]*\\y' This pattern starts at the beginning of a word (\\m) and then looks for any sequence of the letters “m,” “i,” and “n,” regardless of capitalization ([Mm] and so on). Next, it looks for zero or more instances of any other lowercase or uppercase letter ([A- Za-z]*), and then it finally checks for the end of a word (\\y) so that only the word that includes the variation of “minutes” is included and not the rest of the string. Notice that the “+” and “?” characters are not matched. With this pattern, we can now replace all these variations with the standard value “min”: SELECT duration", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 154 + }, + { + "text": "a word (\\y) so that only the word that includes the variation of “minutes” is included and not the rest of the string. Notice that the “+” and “?” characters are not matched. With this pattern, we can now replace all these variations with the standard value “min”: SELECT duration ,(regexp_matches(duration ,'\\m[Mm][Ii][Nn][A-Za-z]*\\y') )[1] as matched_minutes ,regexp_replace(duration ,'\\m[Mm][Ii][Nn][A-Za-z]*\\y' ,'min') as replaced_text FROM ( SELECT split_part(sighting_report,'Duration:',2) as duration 216 | Chapter 5: Text Analysis ,count(*) as reports FROM ufo GROUP BY 1 ) a ; duration matched_minutes replaced_text ----------- --------------- ------------- 10 min. min 10 min. 10 minutes+ minutes 10 min+ 10 min min 10 min 10 minutes + minutes 10 min + 10 minutes? minutes 10 min? 10 minutes minutes 10 min 10 mins mins 10 min ... ... ... The values in the replaced_text column are much more standardized now. The period, plus, and question mark characters could also be replaced by enhancing the regex. From an analytical standpoint, however, we might want to consider how to represent the uncertainty that the plus and question mark represent. The regexp_replace functions can be nested in order to achieve replacement of different parts or types of strings. For example, we can standardize both the minutes and the hours: SELECT duration ,(regexp_matches(duration ,'\\m[Hh][Oo][Uu][Rr][A-Za-z]*\\y') )[1] as matched_hour ,(regexp_matches(duration ,'\\m[Mm][Ii][Nn][A-Za-z]*\\y') )[1] as matched_minutes ,regexp_replace( regexp_replace(duration ,'\\m[Mm][Ii][Nn][A-Za-z]*\\y' ,'min') ,'\\m[Hh][Oo][Uu][Rr][A-Za-z]*\\y' ,'hr') as replaced_text FROM ( SELECT split_part(sighting_report,'Duration:',2) as duration ,count(*) as reports FROM ufo GROUP BY 1 ) a ; duration matched_hour matched_minutes replaced_text ------------------- ------------ --------------- ------------- 1 Hour 15 min Hour min 1 hr 15 min 1 hour & 41 minutes hour minutes 1 hr & 41 min Finding Elements Within Larger Blocks of Text | 217 1 hour 10 mins hour mins 1 hr 10 min 1 hour 10 minutes hour minutes 1 hr 10 min ... ... ... ... The regex for hours is similar to the one for minutes, looking for case-insensitive matches of “hour” at the beginning of a word, followed by zero or more other letter characters before the end of the word. The intermediate hour and minutes matches may not be needed in the final result, but I find them helpful to review as I’m devel‐ oping my SQL code to prevent errors later. A full cleaning of the duration column would likely involve many more lines of code, and it’s all too easy to lose track and introduce a typo. The regexp_replace function can be nested any number of times, or it can be com‐ bined with the basic replace function. Another use for regexp_replace is in CASE statements, for targeted replacement when conditions in the statement are met. Regex is a powerful and flexible tool within SQL that, as we’ve seen, can be used in a number of ways within an overall SQL query. In this section, I’ve introduced a number of ways to search for, find, and replace spe‐ cific elements within longer texts, from wildcard matches with LIKE to IN lists and more complex pattern", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 155 + }, + { + "text": "that, as we’ve seen, can be used in a number of ways within an overall SQL query. In this section, I’ve introduced a number of ways to search for, find, and replace spe‐ cific elements within longer texts, from wildcard matches with LIKE to IN lists and more complex pattern matching with regex. All of these, along with the text-parsing and transformation functions introduced earlier, allow us to create customized rule sets with as much complexity as needed to handle the data sets in hand. It’s worth keeping in mind the balance between complexity and maintenance burden, however. For one-time analysis of a data set, it can be worth the trouble to create complex rule sets that perfectly clean the data. For ongoing reporting and monitoring, it’s usually worthwhile to explore options for receiving cleaner data from data sources. Next, we’ll turn to several ways to construct new text strings with SQL: using constants, existing strings, and parsed strings. Constructing and Reshaping Text We’ve seen how to parse, transform, find, and replace elements of strings in order to perform a variety of cleaning and analysis tasks with SQL. In addition to these, SQL can be used to generate new combinations of text. In this section, I’ll first discuss concatenation, which allows different fields and types of data to be consolidated into a single field. Then I’ll discuss changing text shape with functions that combine multi‐ ple columns into a single row, as well as the opposite: breaking up a single string into multiple rows. Concatenation New text can be created with SQL with concatenation. Any combination of constant or hardcoded text, database fields, and calculations on those fields can be joined 218 | Chapter 5: Text Analysis together. There are a few ways to concatenate. Most databases support the concat function, which takes as arguments the fields or values to be concatenated: concat(value1, value2) concat(value1, value2, value3...) Some databases support the concat_ws (concatenate with separator) function, which takes a separator value as the first argument, followed by the list of values to concate‐ nate. This is useful when there are multiple values that you want to put together, using a comma, dash, or similar element to separate them: concat_ws(separator, value1, value2...) Finally, || (double pipe) can be used in many databases to concatenate strings (SQL Server uses + instead): value1 || value2 If any of the values in a concatenation are null, the database will return null. Be sure to use coalesce or CASE to replace null values with a default if you suspect they can occur. Concatenation can bring together a field and a constant string. For example, imagine we wanted to label the shapes as such and add the word “reports” to the count of reports for each shape. The subquery parses the name of the shape from the sight ing_report field and counts the number of records. The outer query concatenates the shapes with the string ' (shape)' and the reports with the string ' reports': SELECT", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 156 + }, + { + "text": "the word “reports” to the count of reports for each shape. The subquery parses the name of the shape from the sight ing_report field and counts the number of records. The outer query concatenates the shapes with the string ' (shape)' and the reports with the string ' reports': SELECT concat(shape, ' (shape)') as shape ,concat(reports, ' reports') as reports FROM ( SELECT split_part( split_part(sighting_report,'Duration',1) ,'Shape: ',2) as shape ,count(*) as reports FROM ufo GROUP BY 1 ) a ; Shape reports ---------------- ------------ Changing (shape) 2295 reports Chevron (shape) 1021 reports Cigar (shape) 2119 reports ... ... Constructing and Reshaping Text | 219 We can also combine two fields together, optionally with a string separator. For example, we could unite the shape and location values into a single field: SELECT concat(shape,' - ',location) as shape_location ,reports FROM ( SELECT split_part(split_part(sighting_report,'Shape',1) ,'Location: ',2) as location ,split_part(split_part(sighting_report,'Duration',1) ,'Shape: ',2) as shape ,count(*) as reports FROM ufo GROUP BY 1,2 ) a ; shape_location reports ----------------------- ------- Light - Albuquerque, NM 58 Circle - Albany, OR 11 Fireball - Akron, OH 8 ... ... The top 10 combinations are graphed in Figure 5-10. Figure 5-10. Top combinations of shape and location in UFO sightings We saw earlier that “light” is the most common shape, so it’s not surprising that it appears in each of the top results. Phoenix is the most common location, while Las Vegas is the second most common overall. 220 | Chapter 5: Text Analysis In this case, since we went to so much trouble to parse out the different fields, it might not make as much sense to concatenate them back together. However, it can be useful to rearrange text or combine values into a single field for display in another tool. By combining various fields and text, we can also generate sentences that can function as summaries of the data, for use in emails or automated reports. In this example, subquery a parses the occurred and shape fields, as we’ve seen previously, and counts the records. Then in subquery aa, the min and max of occurred are calcu‐ lated, along with the total number of reports, and the results are GROUPed BY shape. Rows with occurred fields shorter than eight characters are excluded, to remove ones that don’t have properly formed dates and avoid errors in the min and max calculations. Finally, in the outer query, the final text is assembled with the con cat function. The format of the dates is changed to read as long dates (April 9, 1957) for the earliest and latest dates: SELECT concat('There were ' ,reports ,' reports of ' ,lower(shape) ,' objects. The earliest sighting was ' ,trim(to_char(earliest,'Month')) , ' ' , date_part('day',earliest) , ', ' , date_part('year',earliest) ,' and the most recent was ' ,trim(to_char(latest,'Month')) , ' ' , date_part('day',latest) , ', ' , date_part('year',latest) ,'.' ) FROM ( SELECT shape ,min(occurred::date) as earliest ,max(occurred::date) as latest ,sum(reports) as reports FROM ( SELECT split_part( split_part( split_part(sighting_report,' (Entered',1) ,'Occurred :", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 157 + }, + { + "text": "' , date_part('day',earliest) , ', ' , date_part('year',earliest) ,' and the most recent was ' ,trim(to_char(latest,'Month')) , ' ' , date_part('day',latest) , ', ' , date_part('year',latest) ,'.' ) FROM ( SELECT shape ,min(occurred::date) as earliest ,max(occurred::date) as latest ,sum(reports) as reports FROM ( SELECT split_part( split_part( split_part(sighting_report,' (Entered',1) ,'Occurred : ',2) ,'Reported',1) as occurred ,split_part( split_part(sighting_report,'Duration',1) ,'Shape: ',2) as shape Constructing and Reshaping Text | 221 ,count(*) as reports FROM ufo GROUP BY 1,2 ) a WHERE length(occurred) >= 8 GROUP BY 1 ) aa ; concat --------------------------------------------------------------------- There were 820 reports of teardrop objects. The earliest sighting was April 9, 1957 and the most recent was October 3, 2020. There were 7331 reports of fireball objects. The earliest sighting was June 30, 1790 and the most recent was October 5, 2020. There were 1020 reports of chevron objects. The earliest sighting was July 15, 1954 and the most recent was October 3, 2020. We could get even more creative with formatting the number of reports or adding coalesce or CASE statements to handle blank shape names, for example. Although these sentences are repetitive and are therefore no match for human (or AI) writers, they will be dynamic if the data source is frequently updated and thus can be useful in reporting applications. Along with functions and operators for creating new text with concatenation, SQL has some special functions for reshaping text, which we’ll turn to next. Reshaping Text As we saw in Chapter 2, changing the shape of the data—either pivoting from rows to columns or the reverse, changing the data from columns to rows—is sometimes use‐ ful. We saw how to do that with GROUP BY and aggregations, or with UNION state‐ ments. In SQL there are some special functions for reshaping text, however. One use case for reshaping text is when there are multiple rows with different text values for an entity and we would like to combine them into a single value. Combin‐ ing values can make them more difficult to analyze, of course, but sometimes the use case requires a single record per entity in the output. Combining the individual val‐ ues into a single field allows us to retain the detail. The string_agg function takes two arguments, a field or an expression, and a separator, which is commonly a comma but can be any separator character desired. The function aggregates only val‐ ues that are not null, and the order can be controlled with an ORDER BY clause within the function as needed: SELECT location ,string_agg(shape,', ' order by shape asc) as shapes FROM ( SELECT 222 | Chapter 5: Text Analysis case when split_part( split_part(sighting_report,'Duration',1) ,'Shape: ',2) = '' then 'Unknown' when split_part( split_part(sighting_report,'Duration',1) ,'Shape: ',2) = 'TRIANGULAR' then 'Triangle' else split_part( split_part(sighting_report,'Duration',1),'Shape: ',2) end as shape ,split_part( split_part(sighting_report,'Shape',1) ,'Location: ',2) as location ,count(*) as reports FROM ufo GROUP BY 1,2 ) a GROUP BY 1 ; location shapes -------------- ----------------------------------- Macungie, PA Fireball, Formation, Light, Unknown Kingsford, MI Circle, Light, Triangle Olivehurst,", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 158 + }, + { + "text": "split_part(sighting_report,'Duration',1) ,'Shape: ',2) = 'TRIANGULAR' then 'Triangle' else split_part( split_part(sighting_report,'Duration',1),'Shape: ',2) end as shape ,split_part( split_part(sighting_report,'Shape',1) ,'Location: ',2) as location ,count(*) as reports FROM ufo GROUP BY 1,2 ) a GROUP BY 1 ; location shapes -------------- ----------------------------------- Macungie, PA Fireball, Formation, Light, Unknown Kingsford, MI Circle, Light, Triangle Olivehurst, CA Changing, Fireball, Formation, Oval ... ... Since string_agg is an aggregate function, it requires a GROUP BY clause on the other fields in the query. In MySQL, an equivalent function is group_concat, and analytic databases such as Redshift and Snowflake have a similar function called listagg. Another use case is to do just the opposite of string_agg and instead split out a sin‐ gle field into multiple rows. There is a lot of inconsistency in how this is implemented in different databases, and even whether a function exists for this at all. Postgres has a function called regexp_split_to_table, while certain other databases have a split_to_table function that operates similarly (check documentation for availabil‐ ity and syntax in your database). The regexp_split_to_table function takes two arguments, a string value and a delimiter. The delimiter can be a regular expression, but keep in mind that a regex can also be a simple string such as a comma or space character. The function then splits the values into rows: SELECT regexp_split_to_table('Red, Orange, Yellow, Green, Blue, Purple' ,', '); regexp_split_to_table --------------------- Red Orange Yellow Constructing and Reshaping Text | 223 Green Blue Purple The string to be split can include anything and doesn’t necessarily need to be a list. We can use the function to split up any string, including sentences. We can then use this to find the most common words used in text fields, a potentially useful tool for text analysis work. Let’s take a look at the most common words used in UFO sighting report descriptions: SELECT word, count(*) as frequency FROM ( SELECT regexp_split_to_table(lower(description),'\\s+') as word FROM ufo ) a GROUP BY 1 ORDER BY 2 desc ; word frequency ---- --------- the 882810 and 477287 a 450223 The subquery first transforms the description into lowercase, since case variations are not interesting for this example. Next, the string is split using the regex '\\s+', which splits on any one or more whitespace characters. The most commonly used words are not surprising; however, they are not particu‐ larly useful since they are just commonly used words in general. To find a more meaningful list, we can remove what are called stop words. These are simply the most commonly used words in a language. Some databases have built-in lists in what are called dictionaries, but the implementations are not standard. There is also no single agreed-upon correct list of stop words, and it is common to adjust the particular list for the desired application; however, there are a number of lists of common stop words on the internet. For this example, I loaded a list of 421 common words into a table called stop_words, available on the book’s GitHub site. The", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 159 + }, + { + "text": "and it is common to adjust the particular list for the desired application; however, there are a number of lists of common stop words on the internet. For this example, I loaded a list of 421 common words into a table called stop_words, available on the book’s GitHub site. The stop words are removed from the result set with a LEFT JOIN to the stop_words table, filtered to results that are not in that table: SELECT word, count(*) as frequency FROM ( SELECT regexp_split_to_table(lower(description),'\\s+') as word FROM ufo ) a LEFT JOIN stop_words b on a.word = b.stop_word WHERE b.stop_word is null 224 | Chapter 5: Text Analysis GROUP BY 1 ORDER BY 2 desc ; word frequency ------ --------- light 97071 lights 89537 object 80785 ... ... The top 10 most common words are graphed in Figure 5-11. Figure 5-11. Most common words in UFO sighting descriptions, excluding stop words We could continue to get more sophisticated by adding additional common words to the stop_words table or by JOINing the results with the descriptions to tag them with the interesting words they contain. Note that regexp_split_to_table and similar functions in other databases can be slow, depending on the length and number of records analyzed. Constructing and reshaping text with SQL can be done in as simple or complex a way or ways as needed. Concatenation, string aggregation, and string-splitting functions can be used alone, in combination with each other, and with other SQL functions and operators to achieve the desired data output. Constructing and Reshaping Text | 225 Conclusion Although SQL isn’t always the first tool mentioned when it comes to text analysis, it has many powerful functions and operators for accomplishing a variety of tasks. From parsing and transformations, to finding and replacing, to constructing and reshaping text, SQL can be used to both clean and prepare text data as well as per‐ form analysis. In the next chapter, we’ll turn to using SQL for anomaly detection, another topic in which SQL isn’t always the first tool mentioned but for which it has surprising capabilities. 226 | Chapter 5: Text Analysis CHAPTER 6 Anomaly Detection An anomaly is something that is different from other members of the same group. In data, an anomaly is a record, an observation, or a value that differs from the remain‐ ing data points in a way that raises concerns or suspicions. Anomalies go by a number of different names, including outliers, novelties, noise, deviations, and exceptions, to name a few. I’ll use the terms anomaly and outlier interchangeably throughout this chapter, and you may see the other terms used in discussions of this topic as well. Anomaly detection can be the end goal of an analysis or a step within a broader analysis project. Anomalies typically have one of two sources: real events that are extreme or other‐ wise unusual, or errors introduced during data collection or processing. While many of the steps used to detect outliers are the same regardless of", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 160 + }, + { + "text": "goal of an analysis or a step within a broader analysis project. Anomalies typically have one of two sources: real events that are extreme or other‐ wise unusual, or errors introduced during data collection or processing. While many of the steps used to detect outliers are the same regardless of the source, how we choose to handle a particular anomaly depends on the root cause. As a result, under‐ standing the root cause and distinguishing between the two types of causes is impor‐ tant to the analysis process. Real events can generate outliers for a variety of reasons. Anomalous data can signal fraud, network intrusion, structural defects in a product, loopholes in policies, or product use that wasn’t intended or envisioned by the developers. Anomaly detection is widely used to root out financial fraud, and cybersecurity also makes use of this type of analysis. Sometimes anomalous data results not because a bad actor is trying to exploit a system but because a customer is using a product in an unexpected way. For example, I knew someone who used a fitness-tracking app, which was intended for running, cycling, walking, and similar activities, to record data from his outings at the auto race track. He hadn’t found a better option and wasn’t thinking about how anomalous the speed and distance values for a car on a track are compared to those recorded for bike rides or running. When anomalies can be tracked to a real process, deciding what to do with them requires a good understanding of the analysis to be 227 done, as well as domain knowledge, terms of use, and sometimes the legal system that governs the product. Data can also contain anomalies because of errors in collection or processing. Man‐ ually entered data is notorious for typos and incorrect data. Changes to forms, fields, or validation rules can introduce unexpected values, including nulls. Behavior track‐ ing of web and mobile applications is common; however, any change to how and when this logging is done can introduce anomalies. I’ve spent enough hours diagnos‐ ing changes in metrics that I’ve learned to ask up front whether any logging was recently changed. Data processing can introduce outliers when some values are fil‐ tered erroneously, processing steps fail to complete, or data is loaded multiple times, creating duplicates. When anomalies result from data processing, we can generally be more confident in correcting or discarding those values. Of course, fixing the upstream data entry or processing is always a good idea, if possible, to prevent future quality problems. In this chapter, I’ll first discuss some of the reasons to use SQL for this type of analy‐ sis and places in which it falls short. Then I’ll introduce the earthquakes data set that will be used in the examples in the rest of the chapter. After that, I’ll introduce the basic tools that we have at our disposal in SQL for detecting outliers. Then I’ll discuss the various forms of outliers that we can apply", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 161 + }, + { + "text": "Then I’ll introduce the earthquakes data set that will be used in the examples in the rest of the chapter. After that, I’ll introduce the basic tools that we have at our disposal in SQL for detecting outliers. Then I’ll discuss the various forms of outliers that we can apply the tools to find. Once we’ve detected and understood anomalies, the next step is to decide what to do with them. Anoma‐ lies need not always be problematic, as they are in fraud detection, cyberattack detec‐ tion, and health system monitoring. The techniques in this chapter can also be used to detect unusually good customers or marketing campaigns, or positive shifts in cus‐ tomer behavior. Sometimes the goal of anomaly detection is to pass the anomalies on to other humans or machines to deal with, but often this is a step in a wider analysis, so I’ll wrap up with various options for correcting anomalies. Capabilities and Limits of SQL for Anomaly Detection SQL is a versatile and powerful language for many data analysis tasks, though it can’t do everything. When performing anomaly detection, SQL has a number of strengths, as well as some drawbacks that make other languages or tools better choices for some tasks. SQL is worth considering when the data set is already in a database, as we previously saw with time series and text analysis in Chapters 3 and 5, respectively. SQL leverages the computational power of the database to perform calculations over many records quickly. Particularly with large tables of data, transferring out of a database and into another tool is time consuming. Working within a database makes even more sense when anomaly detection is a step in a larger analysis that will be done in SQL. Code written in SQL can be examined to understand why particular records were flagged as 228 | Chapter 6: Anomaly Detection outliers, and SQL will remain consistent over time even as the data flowing into a database changes. On the negative side, SQL does not have the statistical sophistication that is available in packages developed for languages like R and Python. SQL has several standard statistical functions, but additional, more complex statistical calculations may be too slow or intense for some databases. For use cases requiring very rapid response, such as fraud or intrusion detection, analyzing data in a database may simply not be appro‐ priate, since there is often lag in loading data, particularly to analytics databases. A common workflow is to use SQL to do the initial analysis and determine typical min‐ imum, maximum, and average values and then develop more real-time monitoring using a streaming service or special real-time data stores. Detecting types of outlier patterns and then implementing in streaming services or special real-time data stores can be an option, however. Finally, SQL code is rule based, as we saw in Chapter 5. It is very good for handling a known set of conditions or criteria, but SQL will not auto‐ matically adjust for", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 162 + }, + { + "text": "patterns and then implementing in streaming services or special real-time data stores can be an option, however. Finally, SQL code is rule based, as we saw in Chapter 5. It is very good for handling a known set of conditions or criteria, but SQL will not auto‐ matically adjust for the types of changing patterns seen with rapidly changing adver‐ saries. Machine learning approaches, and the languages associated with them, are often a better choice for these applications. Now that we’ve discussed the advantages of SQL and when to use it instead of another language or tool, let’s take a look at the data we’ll be using for examples in this chapter before moving on to the code itself. The Data Set The data for the examples in this chapter is a set of records for all earthquakes recor‐ ded by the US Geological Survey (USGS) from 2010 to 2020. The USGS provides the data in a number of formats, including real-time feeds, at https://earthquake.usgs.gov/ earthquakes/feed. The data set contains approximately 1.5 million records. Each record represents a sin‐ gle earthquake event and includes information such as the timestamp, location, mag‐ nitude, depth, and source of the information. A sample of the data is shown in Figure 6-1. A full data dictionary is available on the USGS site. The Data Set | 229 Figure 6-1. Sample of the earthquakes data Earthquakes are caused by sudden slips along faults in the tectonic plates that exist on the outer surface of the earth. Locations on the edges of these plates experience many more, and more dramatic, earthquakes than other places. The so-called Ring of Fire is a region along the rim of the Pacific Ocean in which many earthquakes occur. Vari‐ ous locations within this region, including California, Alaska, Japan, and Indonesia, will appear frequently in our analysis. Magnitude is a measure of the size of an earthquake at its source, as measured by its seismic waves. Magnitude is recorded on a logarithmic scale, meaning that the ampli‐ tude of a magnitude 5 earthquake is 10 times that of a magnitude 4 earthquake. The actual measurement of earthquakes is fascinating but beyond the scope of this book. The USGS website is a good place to start if you want to learn more. Detecting Outliers Although the idea of an anomaly or outlier—a data point that is very different from the rest—seems straightforward, actually finding one in any particular data set poses some challenges. The first challenge has to do with knowing when a value or data point is common or rare, and the second is setting a threshold for marking values on either side of this dividing line. As we go through the earthquakes data, we’ll profile the depths and magnitudes in order to develop an understanding of which values are normal and which are unusual. Generally, the larger or more complete the data set, the easier it is to make a judgment on what is truly anomalous. In some instances, we", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 163 + }, + { + "text": "the earthquakes data, we’ll profile the depths and magnitudes in order to develop an understanding of which values are normal and which are unusual. Generally, the larger or more complete the data set, the easier it is to make a judgment on what is truly anomalous. In some instances, we have labeled or “ground truth” val‐ ues to which we can refer. A label is generally a column in the data set that indicates whether the record is normal or an outlier. Ground truth can be obtained from 230 | Chapter 6: Anomaly Detection industry or scientific sources or from past analysis and might tell us, for example, that any earthquake greater than magnitude 7 is an anomaly. In other cases, we must look to the data itself and apply reasonable judgment. For the remainder of the chapter, we’ll assume that we have a large enough data set to do just that, though of course there are outside references we could consult on typical and extreme earthquake magnitudes. Our tools for detecting outliers using the data set itself fall into a few categories. First, we can sort or ORDER BY the values in the data. This can optionally be combined with various GROUP BY clauses to find outliers by frequency. Second, we can use SQL’s statistical functions to find extreme values at either end of a value range. Finally, we can graph data and inspect it visually. Sorting to Find Anomalies One of the basic tools we have for finding outliers is sorting the data, accomplished with the ORDER BY clause. The default behavior of ORDER BY is to sort ascending (ASC). To sort in descending order, add DESC after the column. An ORDER BY clause can include one or more columns, and each column can be sorted ascending or descending, independently of the others. Sorting starts with the first column speci‐ fied. If a second column is specified, the results of the first sort are then sorted by the second column (retaining the first sort), and so on through all the columns in the clause. Since ordering happens after the database has calculated the rest of the query, many databases allow you to reference the query col‐ umns by number instead of by name. SQL Server is an exception; it requires the full name. I prefer the numbering syntax because it results in more compact code, particularly when query columns include lengthy calculations or function syntax. For example, we can sort the earthquakes table by mag, the magnitude: SELECT mag FROM earthquakes ORDER BY 1 desc ; mag ------ (null) (null) (null) ... Detecting Outliers | 231 This returns a number of rows of nulls. Let’s make a note that the data set can contain null values for magnitude—a possible outlier in itself. We can exclude the null values: SELECT mag FROM earthquakes WHERE mag is not null ORDER BY 1 desc ; mag --- 9.1 8.8 8.6 8.3 There is only one value greater than 9, and there", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 164 + }, + { + "text": "that the data set can contain null values for magnitude—a possible outlier in itself. We can exclude the null values: SELECT mag FROM earthquakes WHERE mag is not null ORDER BY 1 desc ; mag --- 9.1 8.8 8.6 8.3 There is only one value greater than 9, and there are only two additional values greater than 8.5. In many contexts, these would not appear to be particularly large values. However, with a little domain knowledge about earthquakes, we can recognize that these values are in fact both very large and unusual. The USGS provides a list of the 20 largest earthquakes in the world. All of them are magnitude 8.4 or larger, while only five are magnitude 9.0 or larger, and three occurred between 2010 and 2020, the time period covered by our data set. Another way to consider whether values are anomalies within a data set is to calculate their frequency. We can count the id field and GROUP BY the mag to find the number of earthquakes per magnitude. The number of earthquakes per magnitude is then divided by the total number of earthquakes, which can be found using a sum window function. All window functions require an OVER clause with a PARTITION BY and/or ORDER BY clause. Since the denominator should count all the records, I have added a PARTITION BY 1, which is a way to force the database to make it a window function but still read from the entire table. Finally, the result set is ORDERed BY the magnitude: SELECT mag ,count(id) as earthquakes ,round(count(id) * 100.0 / sum(count(id)) over (partition by 1),8) as pct_earthquakes FROM earthquakes WHERE mag is not null GROUP BY 1 ORDER BY 1 desc ; mag earthquakes pct_earthquakes --- ----------- --------------- 9.1 1 0.00006719 8.8 1 0.00006719 8.6 1 0.00006719 8.3 2 0.00013439 232 | Chapter 6: Anomaly Detection ... ... ... 6.9 53 0.00356124 6.8 45 0.00302370 6.7 60 0.00403160 ... ... ... There is only one each of the earthquakes that are over 8.5 in magnitude, but there are two that registered 8.3. By the value 6.9, there are double digits of earthquakes, but those still represent a very small percentage of the data. In our investigation, we should also check the other end of the sorting, the smallest values, by sorting ascend‐ ing instead of descending: SELECT mag ,count(id) as earthquakes ,round(count(id) * 100.0 / sum(count(id)) over (partition by 1),8) as pct_earthquakes FROM earthquakes WHERE mag is not null GROUP BY 1 ORDER BY 1 ; mag earthquakes pct_earthquakes --- ----------- --------------- -9.99 258 0.01733587 -9 29 0.00194861 -5 1 0.00006719 -2.6 2 0.00013439 ... ... ... At the low end of values, –9.99 and –9 occur more frequently than we might expect. Although we can’t take the logarithm of zero or a negative number, a logarithm can be negative when the argument is greater than zero and less than one. For example, log(0.5) is equal to approximately –0.301. The values –9.99 and –9 represent", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 165 + }, + { + "text": "–9 occur more frequently than we might expect. Although we can’t take the logarithm of zero or a negative number, a logarithm can be negative when the argument is greater than zero and less than one. For example, log(0.5) is equal to approximately –0.301. The values –9.99 and –9 represent extremely small earthquake magnitudes, and we might question whether such small quakes could really be detected. Given the frequency of these values, I suspect they represent an unknown value rather than a truly tiny earthquake, and thus we may consider them anomalies. In addition to sorting the overall data, it can be useful to GROUP BY one or more attribute fields to find anomalies within subsets of the data. For example, we might want to check the highest and lowest magnitudes recorded for specific geographies in the place field: SELECT place, mag, count(*) FROM earthquakes WHERE mag is not null and place = 'Northern California' GROUP BY 1,2 ORDER BY 1,2 desc Detecting Outliers | 233 ; place mag count ------------------- ---- ----- Northern California 5.61 Northern California 4.73 1 Northern California 4.51 1 ... ... ... Northern California -1.1 7 Northern California -1.2 2 Northern California -1.6 1 “Northern California” is the most common place in the data set, and inspecting just the subset for it, we can see that the high and low values are not nearly as extreme as those for the data set as a whole. Earthquakes over 5.0 magnitude are not uncommon overall, but they are outliers for “Northern California.” Calculating Percentiles and Standard Deviations to Find Anomalies Sorting and optionally grouping data and then reviewing the results visually is a use‐ ful approach for spotting anomalies, particularly when the data has values that are very extreme. Without domain knowledge, however, it might not be obvious that a 9.0 magnitude earthquake is such an anomaly. Quantifying the extremity of data points adds another layer of rigor to the analysis. There are two ways to do this: with percentiles or with standard deviations. Percentiles represent the proportion of values in a distribution that are less than a particular value. The median of a distribution is the value at which half of the popula‐ tion has a lower value and half has a higher value. The median is so commonly used that it has its own SQL function, median, in many but not all databases. Other percen‐ tiles can be calculated as well. For example, we can find the 25th percentile, where 25% of the values are lower and 75% are higher, or the 89th percentile, where 89% of values are lower and 11% are higher. Percentiles are often found in academic con‐ texts, such as standardized testing, but they can be applied to any domain. SQL has a window function, percent_rank, that returns the percentile for each row within a partition. As with all window functions, the sorting direction is controlled with an ORDER BY statement. Similar to the rank function, percent_rank does not take any", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 166 + }, + { + "text": "testing, but they can be applied to any domain. SQL has a window function, percent_rank, that returns the percentile for each row within a partition. As with all window functions, the sorting direction is controlled with an ORDER BY statement. Similar to the rank function, percent_rank does not take any argument; it operates over all the rows returned by the query. The basic form is: percent_rank() over (partition by ... order by ...) Both the PARTITION BY and the ORDER BY are optional, but the function requires something in the OVER clause, and specifying the ordering is always a good idea. To find the percentile of the magnitudes of each earthquake for each place, we can first calculate the percent_rank for each row in the subquery and then count the 234 | Chapter 6: Anomaly Detection occurrences of each magnitude in the outer query. Note that it’s important to calcu‐ late the percent_rank first, before doing any aggregation, so that repeating values are taken into account in the calculation: SELECT place, mag, percentile ,count(*) FROM ( SELECT place, mag ,percent_rank() over (partition by place order by mag) as percentile FROM earthquakes WHERE mag is not null and place = 'Northern California' ) a GROUP BY 1,2,3 ORDER BY 1,2 desc ; place mag percentile count ------------------- ---- --------------------- ----- Northern California 5.6 1.0 1 Northern California 4.73 0.9999870597065141 1 Northern California 4.51 0.9999741194130283 1 ... ... ... ... Northern California -1.1 3.8820880457568775E-5 7 Northern California -1.2 1.2940293485856258E-5 2 Northern California -1.6 0.0 1 Within Northern California, the magnitude 5.6 earthquake has a percentile of 1, or 100%, indicating that all of the other values are less than this one. The magnitude –1.6 earthquake has a percentile of 0, indicating that no other data points are smaller. In addition to finding the exact percentile of each row, SQL can carve the data set into a specified number of buckets and return the bucket each row belongs to with a function called ntile. For example, we might want to carve the data set up into 100 buckets: SELECT place, mag ,ntile(100) over (partition by place order by mag) as ntile FROM earthquakes WHERE mag is not null and place = 'Central Alaska' ORDER BY 1,2 desc ; place mag ntile -------------- ---- ----- Central Alaska 5.4 100 Central Alaska 5.3 100 Central Alaska 5.2 100 ... ... ... Central Alaska 1.5 79 Detecting Outliers | 235 ... ... ... Central Alaska -0.5 1 Central Alaska -0.5 1 Central Alaska -0.5 1 Looking at the results for “Central Alaska,” we see that the three earthquakes greater than 5 are in the 100th percentile, 1.5 falls within the 79th percentile, and the smallest values of –0.5 fall in the first percentile. After calculating these values, we can then find the boundaries of each ntile, using max and min. For this example, we’ll use four ntiles to keep the display simpler, but any positive integer is allowed in the ntile argument: SELECT place, ntile ,max(mag)", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 167 + }, + { + "text": "of –0.5 fall in the first percentile. After calculating these values, we can then find the boundaries of each ntile, using max and min. For this example, we’ll use four ntiles to keep the display simpler, but any positive integer is allowed in the ntile argument: SELECT place, ntile ,max(mag) as maximum ,min(mag) as minimum FROM ( SELECT place, mag ,ntile(4) over (partition by place order by mag) as ntile FROM earthquakes WHERE mag is not null and place = 'Central Alaska' ) a GROUP BY 1,2 ORDER BY 1,2 desc ; place ntile maximum minimum -------------- ----- ------- ------- Central Alaska 4 5.4 1.4 Central Alaska 3 1.4 1.1 Central Alaska 2 1.1 0.8 Central Alaska 1 0.8 -0.5 The highest ntile, 4, which represents the 75th to 100th percentiles, has the widest range, spanning from 1.4 to 5.4. On the other hand, the middle 50 percent of values, which include ntiles 2 and 3, range only from 0.8 to 1.4. In addition to finding the percentile or ntile for each row, we can calculate specific percentiles across the entire result set of a query. To do this, we can use the percen tile_cont function or the percentile_disc function. Both are window functions, but with a slightly different syntax than other window functions discussed previously because they require a WITHIN GROUP clause. The form of the functions is: percentile_cont(numeric) within group (order by field_name) over (partition by field_name) The numeric is a value between 0 and 1 that represents the percentile to return. For example, 0.25 returns the 25th percentile. The ORDER BY clause specifies the field to return the percentile from, as well as the ordering. ASC or DESC can optionally be 236 | Chapter 6: Anomaly Detection added, with ASC the default, as in all ORDER BY clauses in SQL. The OVER (PARTI‐ TION BY...) clause is optional (and confusingly, some databases don’t support it, so check your documentation if you run into errors). The percentile_cont function will return an interpolated (calculated) value that cor‐ responds to the exact percentile but that may not exist in the data set. The percen tile_disc (discontinuous percentile) function, on the other hand, returns the value in the data set that is closest to the requested percentile. For large data sets, or for ones with fairly continuous values, there is often little practical difference between the output of the two functions, but it’s worth considering which is more appropriate for your analysis. Let’s take a look at an example to see how this looks in practice. We’ll calculate the 25th, 50th (or median), and 75th percentile magnitudes for all nonnull magnitudes in Central Alaska: SELECT percentile_cont(0.25) within group (order by mag) as pct_25 ,percentile_cont(0.5) within group (order by mag) as pct_50 ,percentile_cont(0.75) within group (order by mag) as pct_75 FROM earthquakes WHERE mag is not null and place = 'Central Alaska' ; pct_25 pct_50 pct_75 ------ ------ ------ 0.8 1.1 1.4 The query returns the requested percentiles, summarized across the data", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 168 + }, + { + "text": "as pct_25 ,percentile_cont(0.5) within group (order by mag) as pct_50 ,percentile_cont(0.75) within group (order by mag) as pct_75 FROM earthquakes WHERE mag is not null and place = 'Central Alaska' ; pct_25 pct_50 pct_75 ------ ------ ------ 0.8 1.1 1.4 The query returns the requested percentiles, summarized across the data set. Notice that the values correspond to the maximum values for ntiles 1, 2, and 3 calculated in the previous example. Percentiles for different fields can be calculated within the same query by changing the field in the ORDER BY clause: SELECT percentile_cont(0.25) within group (order by mag) as pct_25_mag ,percentile_cont(0.25) within group (order by depth) as pct_25_depth FROM earthquakes WHERE mag is not null and place = 'Central Alaska' ; pct_25_mag pct_25_depth ---------- ------------ 0.8 7.1 Unlike other window functions, percentile_cont and percentile_disc require a GROUP BY clause at the query level when other fields are present in the query. For example, if we want to consider two areas within Alaska, and so include the place field, the query must also include it in the GROUP BY, and the percentiles are calcula‐ ted per place: Detecting Outliers | 237 SELECT place ,percentile_cont(0.25) within group (order by mag) as pct_25_mag ,percentile_cont(0.25) within group (order by depth) as pct_25_depth FROM earthquakes WHERE mag is not null and place in ('Central Alaska', 'Southern Alaska') GROUP BY place ; place pct_25_mag pct_25_depth --------------- ---------- ------------ Central Alaska 0.8 7.1 Southern Alaska 1.2 10.1 With these functions, we can find any percentile required for analysis. Since the median value is so commonly calculated, a number of databases have implemented a median function that has only one argument, the field for which to calculate the median. This is a handy and certainly much simpler syntax, but note that the same can be accomplished with percentile_cont if a median function is not available. The percentile and median functions can be slow and computa‐ tionally intensive on large data sets. This is because the database must sort and rank all the records, usually in memory. Some data‐ base vendors have implemented approximate versions of the func‐ tions, such as approximate_percentile, that are much faster and return results very close to the function that calculates the entire data set. Finding the percentiles or ntiles of a data set allows us to add some quantification to anomalies. We’ll see later in the chapter how these values also give us some tools for handling anomalies in data sets. Since percentiles are always scaled between 0 and 100, however, they don’t give a sense of just how unusual certain values are. For that we can turn to additional statistical functions supported by SQL. To measure how extreme values in a data set are, we can use the standard deviation. The standard deviation is a measure of the variation in a set of values. A lower value means less variation, while a higher number means more variation. When data is normally distributed around the mean, about 68% of the values lie within", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 169 + }, + { + "text": "set are, we can use the standard deviation. The standard deviation is a measure of the variation in a set of values. A lower value means less variation, while a higher number means more variation. When data is normally distributed around the mean, about 68% of the values lie within +/– one standard deviation from the mean, and about 95% lie within two standard deviations. The standard deviation is calculated as the square root of the sum of differences from the mean, divided by the number of observations: ∑xi −μ 2/N 238 | Chapter 6: Anomaly Detection 1 https://www.mathsisfun.com/data/standard-deviation-formulas.html has a good explanation. In this formula, xi is an observation, μ is the average of all the observations, ∑ indi‐ cates that all of the values should be summed, and N is the number of observations. Refer to any good statistics text or online resource1 for more information about how the standard deviation is derived. Most databases have three standard deviation functions. The stddev_pop function finds the standard deviation of a population. If the data set represents the entire pop‐ ulation, as is often the case with a customer data set, use the stddev_pop. The stddev_samp finds the standard deviation of a sample and differs from the above for‐ mula by dividing by N – 1 instead of N. This has the effect of increasing the standard deviation, reflecting the loss of accuracy when only a sample of the entire population is used. The stddev function available in many databases is identical to the stddev_samp function and may be used simply because it is shorter. If you’re working with data that is a sample, such as from a survey or study from a larger population, use the stddev_samp or stddev. In practice, when you are working with large data sets, there is usually little difference between the stddev_pop and stddev_samp results. For example, across the 1.5 million records in the earthquakes table, the val‐ ues diverge only after five decimal places: SELECT stddev_pop(mag) as stddev_pop_mag ,stddev_samp(mag) as stddev_samp_mag FROM earthquakes ; stddev_pop_mag stddev_samp_mag -------------------- -------------------- 1.273605805569390395 1.273606233458381515 These differences are small enough that in most practical applications, it doesn’t mat‐ ter which standard deviation function you use. With this function, we can now calculate the number of standard deviations from the mean for each value in the data set. This value is known as the z-score and is a way of standardizing data. Values that are above the average have a positive z-score, and those below the average have a negative z-score. Figure 6-2 shows how z-scores and standard deviations relate to the normal distribution. Detecting Outliers | 239 Figure 6-2. Standard deviations and z-scores for a normal distribution To find the z-scores for the earthquakes, first calculate the average and standard devi‐ ation for the entire data set in a subquery. Then JOIN this back to the data set using a Cartesian JOIN, so that the average and standard deviation values are JOINed to each earthquake row. This is", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 170 + }, + { + "text": "find the z-scores for the earthquakes, first calculate the average and standard devi‐ ation for the entire data set in a subquery. Then JOIN this back to the data set using a Cartesian JOIN, so that the average and standard deviation values are JOINed to each earthquake row. This is accomplished with the 1 = 1 syntax, since most databases require that some JOIN condition be specified. In the outer query, subtract the average magnitude from each individual magnitude and then divide by the standard deviation: SELECT a.place, a.mag ,b.avg_mag, b.std_dev ,(a.mag - b.avg_mag) / b.std_dev as z_score FROM earthquakes a JOIN ( SELECT avg(mag) as avg_mag ,stddev_pop(mag) as std_dev FROM earthquakes WHERE mag is not null ) b on 1 = 1 WHERE a.mag is not null ORDER BY 2 desc ; 240 | Chapter 6: Anomaly Detection place mag avg_mag std_dev z_score -------------------------------------- --- ------- ------- ------- 2011 Great Tohoku Earthquake, Japan 9.1 1.6251 1.2736 5.8691 offshore Bio-Bio, Chile 8.8 1.6251 1.2736 5.6335 off the west coast of northern Sumatra 8.6 1.6251 1.2736 5.4765 ... ... ... ... ... Nevada -2.5 1.6251 1.2736 -3.2389 Nevada -2.6 1.6251 1.2736 -3.3174 Nevada -2.6 1.6251 1.2736 -3.3174 The largest earthquakes have a z-score of almost 6, whereas the smallest (excluding the –9 and –9.99 earthquakes that appear to be data entry anomalies) have z-scores close to 3. We can conclude that the largest earthquakes are more extreme outliers than the ones at the low end. Graphing to Find Anomalies Visually In addition to sorting the data and calculating percentiles and standard deviations to find anomalies, visualizing the data in one of several graph formats can also help in finding anomalies. As we’ve seen in previous chapters, one strength of graphs is their ability to summarize and present many data points in a compact form. By inspecting graphs, we can often spot patterns and outliers that we might otherwise miss if only considering the raw output. Finally, graphs assist in the task of describing the data, and any potential problems with the data related to anomalies, to other people. In this section, I’ll present three types of graphs that are useful for anomaly detection: bar graphs, scatter plots, and box plots. The SQL needed to generate output for these graphs is straightforward, though you might need to enlist pivoting strategies dis‐ cussed in previous chapters, depending on the capabilities and limitations of the soft‐ ware used to create the graphs. Any major BI tool or spreadsheet software, or languages such as Python or R, will be able to produce these graph types. The graphs in this section were created using Python with Matplotlib. The bar graph is used to plot a histogram or distribution of the values in a field and is useful for both characterizing the data and spotting outliers. The full extent of values are plotted along one axis, and the number of occurrences of each value is plotted on the other axis. The extreme high and low values are", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 171 + }, + { + "text": "or distribution of the values in a field and is useful for both characterizing the data and spotting outliers. The full extent of values are plotted along one axis, and the number of occurrences of each value is plotted on the other axis. The extreme high and low values are interesting, as is the shape of the plot. We can quickly determine whether the distribution is approximately normal (symmetric around a peak or average value), has another type of distribution, or has peaks at particular values. Detecting Outliers | 241 To graph a histogram for the earthquake magnitudes, first create a data set that groups the magnitudes and counts the earthquakes. Then plot the output, as in Figure 6-3. SELECT mag ,count(*) as earthquakes FROM earthquakes GROUP BY 1 ORDER BY 1 ; mag earthquakes ----- ----------- -9.99 258 -9 29 -5 1 ... ... Figure 6-3. Distribution of earthquake magnitudes The graph extends from –10.0 to +10.0, which makes sense given our previous explo‐ ration of the data. It peaks and is roughly symmetric around a value in the range of 1.1 to 1.4 with almost 40,000 earthquakes of each magnitude, but it has a second peak of almost 20,000 earthquakes around the value 4.4. We’ll explore the reason for this second peak in the next section on forms of anomalies. The extreme values are hard to spot in this graph, however, so we might want to zoom in on a subsection of the graph, as in Figure 6-4. 242 | Chapter 6: Anomaly Detection Figure 6-4. A zoomed-in view of the distribution of earthquake magnitudes, focused on the highest magnitudes Here the frequencies of these very high-intensity earthquakes are easier to see, as is the decrease in frequency from more than 10 to only 1 as the value goes from the low 7s to over 8. Thankfully these temblors are extremely rare. A second type of graph that can be used to characterize data and spot outliers is the scatter plot. A scatter plot is appropriate when the data set contains at least two numeric values of interest. The x-axis displays the range of values of the first data field, the y-axis displays the range of values of the second data field, and a dot is graphed for every pair of x and y values in the data set. For example, we can graph the magnitude against the depth of earthquakes in the data set. First, query the data to create a data set of each pair of values. Then graph the output, as in Figure 6-5: SELECT mag, depth ,count(*) as earthquakes FROM earthquakes GROUP BY 1,2 ORDER BY 1,2 ; mag depth earthquakes ----- ----- ----------- -9.99 -0.59 1 -9.99 -0.35 1 -9.99 -0.11 1 ... ... ... Detecting Outliers | 243 Figure 6-5. Scatter plot of the magnitude and depth of earthquakes In this graph, we can see the same range of magnitudes, now plotted against the depths, which range from just below zero to", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 172 + }, + { + "text": "-9.99 -0.59 1 -9.99 -0.35 1 -9.99 -0.11 1 ... ... ... Detecting Outliers | 243 Figure 6-5. Scatter plot of the magnitude and depth of earthquakes In this graph, we can see the same range of magnitudes, now plotted against the depths, which range from just below zero to around 700 kilometers. Interestingly, the high depth values, over 300, correspond to magnitudes that are roughly 4 and higher. Perhaps such deep earthquakes can be detected only after they reach a minimum magnitude. Note that, due to the volume of data, I have taken a shortcut and grouped the values by magnitude and depth combination, rather than plotting all 1.5 million data points. The count of earthquakes can be used to size each circle in the scatter, as in Figure 6-6, which is zoomed in to the range of magnitudes from 4.0 to 7.0, and depths from 0 to 50 km. 244 | Chapter 6: Anomaly Detection Figure 6-6. Scatter plot of the magnitude and depth of earthquakes, zoomed in and with circles sized by the number of earthquakes A third type of graph useful in finding and analyzing outliers is the box plot, also known as the box-and-whisker plot. These graphs summarize data in the middle of the range of values while retaining the outliers. The graph type is named for the box, or rectangle, in the middle. The line that forms the bottom of the rectangle is located at the 25th percentile value, the line that forms the top is located at the 75th percentile, and the line through the middle is located at the 50th percentile, or median, value. Percentiles should be familiar from our discussion in the preceding section. The “whiskers” of the box plot are lines that extend out from the box, typically to 1.5 times the interquartile range. The interquartile range is simply the difference between Detecting Outliers | 245 the 75th percentile value and the 25th percentile value. Any values beyond the whisk‐ ers are plotted on the graph as outliers. Whichever software or programming language you use for graph‐ ing box plots will take care of the calculations of the percentiles and interquartile range. Many also offer options to plot the whiskers based on standard deviations from the mean, or on wider percen‐ tiles such as the 10th and 90th. The calculation will always be sym‐ metric around the midpoint (such as one standard deviation above and below the mean), but the length of the upper and lower whisk‐ ers can differ based on the data. Typically, all of the values are plotted in a box plot. Since the data set is so large, for this example we’ll look at the subset of 16,036 earthquakes that include “Japan” in the place field. First, create the data set with SQL, which is a simple SELECT of all of the mag values that meet the filter criteria: SELECT mag FROM earthquakes WHERE place like '%Japan%' ORDER BY 1 ; mag --- 2.7 3.1 3.2", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 173 + }, + { + "text": "of 16,036 earthquakes that include “Japan” in the place field. First, create the data set with SQL, which is a simple SELECT of all of the mag values that meet the filter criteria: SELECT mag FROM earthquakes WHERE place like '%Japan%' ORDER BY 1 ; mag --- 2.7 3.1 3.2 ... Then create a box plot in our graphing software of choice, as shown in Figure 6-7. 246 | Chapter 6: Anomaly Detection Figure 6-7. Box plot showing magnitude distribution of earthquakes in Japan Although the graphing software will often provide this information, we can also find the key values for the box plot with SQL: SELECT ntile_25, median, ntile_75 ,(ntile_75 - ntile_25) * 1.5 as iqr ,ntile_25 - (ntile_75 - ntile_25) * 1.5 as lower_whisker ,ntile_75 + (ntile_75 - ntile_25) * 1.5 as upper_whisker FROM ( SELECT percentile_cont(0.25) within group (order by mag) as ntile_25 ,percentile_cont(0.5) within group (order by mag) as median ,percentile_cont(0.75) within group (order by mag) as ntile_75 FROM earthquakes WHERE place like '%Japan%' ) a ; ntile_25 median ntile_75 iqr lower_whisker upper_whisker -------- ------ -------- ---- ------------- ------------- 4.3 4.5 4.7 0.60 3.70 5.30 Detecting Outliers | 247 The median Japanese earthquake had a magnitude of 4.5, and the whiskers extend from 3.7 to 5.3. The plotted circles represent outlier earthquakes, both small and large. The Great Tohoku Earthquake of 2011, at 9.1, is an obvious outlier, even among the larger earthquakes Japan experienced. In my experience, box plots are one of the more difficult visualiza‐ tions to explain to those who don’t have a statistics background, or who don’t spend all day making and looking at visualizations. The interquartile range is a particularly confusing concept, though the notion of outliers seems to make sense to most people. If you’re not absolutely sure your audience knows how to interpret a box plot, take the time to explain it in clear but not overly technical terms. I keep a drawing like Figure 6-8 that explains the parts of a box plot and send it along with my work “just in case” my audience needs a refresher. Figure 6-8. Diagram of parts of a box plot Box plots can also be used to compare across groupings of the data to further identify and diagnose where outliers occur. For example, we can compare earthquakes in Japan in different years. First add the year of the time field into the SQL output and then graph, as in Figure 6-9: 248 | Chapter 6: Anomaly Detection SELECT date_part('year',time)::int as year ,mag FROM earthquakes WHERE place like '%Japan%' ORDER BY 1,2 ; year mag ---- --- 2010 3.6 2010 3.7 2010 3.7 ... ... Figure 6-9. Box plot of magnitudes of earthquakes in Japan, by year Although the median and the range of the boxes fluctuate a bit from year to year, they are consistently between 4 and 5. Japan experienced large outlier earthquakes every year, with at least one greater than 6.0, and in six of the years it experienced", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 174 + }, + { + "text": "earthquakes in Japan, by year Although the median and the range of the boxes fluctuate a bit from year to year, they are consistently between 4 and 5. Japan experienced large outlier earthquakes every year, with at least one greater than 6.0, and in six of the years it experienced at least one earthquake at or larger than 7.0. Japan is undoubtedly a very seismically active region. Bar graphs, scatter plots, and box plots are commonly used to detect and characterize outliers in data sets. They allow us to quickly absorb the complexity of large amounts of data and to start to tell the story behind it. Along with sorting, percentiles, and standard deviations, graphs are an important part of the anomaly detection toolkit. With these tools in hand, we’re ready to discuss the various forms that anomalies can take in addition to those we’ve seen so far. Detecting Outliers | 249 Forms of Anomalies Anomalies can come in all shapes and sizes. In this section, I will discuss three gen‐ eral categories of anomalies: values, counts or frequencies, and presence or absence. These are starting points for investigating any data set, either as a profiling exercise or because anomalies are suspected. Outliers and other unusual values are often specific to a particular domain, so in general the more you know about how and why the data was generated, the better. However, these patterns and techniques for spotting anomalies are good starting places for investigation. Anomalous Values Perhaps the most common type of anomaly, and the first thing that comes to mind on this topic, is when single values are either extremely high or low outliers, or when val‐ ues in the middle of the distribution are otherwise unusual. In the last section, we looked at several ways to find outliers, through sorting, percen‐ tiles and standard deviations, and graphing. We discovered that the earthquakes data set has both unusually large values for the magnitude and some values that appear to be unusually small. The magnitudes also contain varying numbers of significant dig‐ its, or digits to the right of the decimal point. For example, we can look at a subset of values around 1 and find a pattern that repeats throughout the data set: SELECT mag, count(*) FROM earthquakes WHERE mag > 1 GROUP BY 1 ORDER BY 1 limit 100 ; mag count ---------- ----- ... ... 1.08 3863 1.08000004 1 1.09 3712 1.1 39728 1.11 3674 1.12 3995 .... ... Every once in a while there is a value with 8 significant digits. Many values have two significant digits, but having only a single significant digit is more common. This is likely due to different levels of precision in the instruments collecting the magnitude data. Additionally, the database does not display a second significant digit when that digit is zero, so “1.10” appears simply as “1.1.” However, the large number of records at “1.1” indicates that this is not just a display issue. Depending on the purpose", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 175 + }, + { + "text": "precision in the instruments collecting the magnitude data. Additionally, the database does not display a second significant digit when that digit is zero, so “1.10” appears simply as “1.1.” However, the large number of records at “1.1” indicates that this is not just a display issue. Depending on the purpose of the 250 | Chapter 6: Anomaly Detection analysis, we may or may not want to adjust the values to all have the same number of significant digits by rounding. Often in addition to finding anomalous values, understanding why they happened or other attributes that are correlated with anomalies is useful. This is where creativity and data detective work come into play. For example, 1,215 records in the data set have very high depth values of more than 600 kilometers. We might want to know where these outliers occurred or how they were collected. Let’s take a look at the source, which we can find in the net (for network) field: SELECT net, count(*) FROM earthquakes WHERE depth > 600 GROUP BY 1 ; net count --- ----- us 1215 The USGS site indicates that this source is the USGS National Earthquake Informa‐ tion Center, PDE. This is not terribly informative, however, so let’s check the place values, which contain the earthquake locations: SELECT place, count(*) FROM earthquakes WHERE depth > 600 GROUP BY 1 ; place count ------------------------------ ----- 100km NW of Ndoi Island, Fiji 1 100km SSW of Ndoi Island, Fiji 1 100km SW of Ndoi Island, Fiji 1 ... ... Visual inspection suggests that many of these very deep earthquakes happen around Ndoi Island in Fiji. However, the place includes a distance and direction component, such as “100km NW of,” that makes summarization more difficult. We can apply some text parsing to focus on the place itself for better insights. For places that con‐ tain some values and then “ of ” and some more values, split on the “ of ” string and take the second part: SELECT case when place like '% of %' then split_part(place,' of ',2) else place end as place_name ,count(*) FROM earthquakes WHERE depth > 600 GROUP BY 1 Forms of Anomalies | 251 ORDER BY 2 desc ; place_name count ----------------- ----- Ndoi Island, Fiji 487 Fiji region 186 Lambasa, Fiji 140 ... ... We can now say with more confidence that the majority of the very deep values were recorded for earthquakes somewhere in Fiji, with a particular concentration around the small volcanic island of Ndoi. The analysis could continue to get more complex, for example, by parsing the text to group together all earthquakes recorded in the greater region, which would reveal that after Fiji, other very deep earthquakes have been recorded around Vanuatu and the Philippines. Anomalies can come in the form of misspellings, variations in capitalization, or other text errors. The ease of finding these depends on the number of distinct values, or cardinality, of the field. Differences in capitalization can be detected by counting both", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 176 + }, + { + "text": "earthquakes have been recorded around Vanuatu and the Philippines. Anomalies can come in the form of misspellings, variations in capitalization, or other text errors. The ease of finding these depends on the number of distinct values, or cardinality, of the field. Differences in capitalization can be detected by counting both the distinct values and the distinct values when a lower or upper function is applied: SELECT count(distinct type) as distinct_types ,count(distinct lower(type)) as distinct_lower FROM earthquakes ; distinct_types distinct_lower -------------- -------------- 25 24 There are 24 distinct values of the type field, but 25 different forms. To find the spe‐ cific types, we can use a calculation to flag those values whose lowercase form doesn’t match the actual value. Including the count of records for each form will help contex‐ tualize so that we can later decide how to handle the values: SELECT type ,lower(type) ,type = lower(type) as flag ,count(*) as records FROM earthquakes GROUP BY 1,2,3 ORDER BY 2,4 desc ; type lower flag records --------- --------- ----- ------- ... ... ... ... explosion explosion true 9887 ice quake ice quake true 10136 252 | Chapter 6: Anomaly Detection Ice Quake ice quake false 1 ... ... ... ... The anomalous value of “Ice quake” is easy to spot, since it is the only value for which the flag calculation returns false. Since there is only one record with this value, com‐ pared to 10,136 with the lowercase form, we can assume that it can be grouped together with the other records. Other text functions can be applied, such as trim if we suspect that the values contain extra leading or trailing spaces, or replace if we suspect that certain spellings have multiple forms, such as the number “2” and the word “two.” Misspellings can be more difficult to discover than other variations. If a known set of correct values and spellings exists, it can be used to validate the data either through an OUTER JOIN to a table containing the values or with a CASE statement combined with an IN list. In either case, the goal is to flag values that are unexpected or invalid. Without such a set of correct values, our options are often either to apply domain knowledge or to make educated guesses. In the earthquakes table, we can look at the type values with only a few records and then try to determine if there is another, more common value that can be substituted: SELECT type, count(*) as records FROM earthquakes GROUP BY 1 ORDER BY 2 desc ; type records -------------------------- ------- ... ... landslide 15 mine collapse 12 experimental explosion 6 building collapse 5 ... ... meteorite 1 accidental explosion 1 collapse 1 induced or triggered event 1 Ice Quake 1 rockslide 1 We looked at “Ice Quake” previously and decided it was likely the same as “ice quake.” There is only one record for “rockslide,” though we might consider this close enough to another of the values, “landslide,” which has", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 177 + }, + { + "text": "collapse 1 induced or triggered event 1 Ice Quake 1 rockslide 1 We looked at “Ice Quake” previously and decided it was likely the same as “ice quake.” There is only one record for “rockslide,” though we might consider this close enough to another of the values, “landslide,” which has 15 records. “Collapse” is more ambig‐ uous, since the data set includes both “mine collapse” and “building collapse.” What we do with these, or whether we do anything at all, depends on the goal of the analy‐ sis, as I’ll discuss later in “Handling Anomalies” on page 260. Forms of Anomalies | 253 Anomalous Counts or Frequencies Sometimes anomalies come not in the form of individual values but in the form of patterns or clusters of activity in the data. For example, a customer spending $100 on an ecommerce site may not be unusual, but that same customer spending $100 every hour over the course of 48 hours would almost certainly be an anomaly. There are a number of dimensions on which clusters of activity can indicate anoma‐ lies, many of them dependent on the context of the data. Time and location are both common across many data sets and are features of the earthquakes data set, so I will use them to illustrate the techniques in this section. Keep in mind that these techni‐ ques can often be applied to other attributes as well. Events that happen with unusual frequency over a short time span can indicate anomalous activity. This can be good, such as when a celebrity unexpectedly pro‐ motes a product, leading to a burst of sales of that product. They can also be bad, such as when unusual spikes indicate fraudulent credit card use or attempts to bring a website down with a flood of traffic. To understand these types of anomalies and whether there are deviations from the normal trend, we first apply appropriate aggre‐ gations and then use the techniques introduced earlier in this chapter, along with time series analysis techniques discussed in Chapter 3. In the following examples, I’ll go through a series of steps and queries that will help us understand the normal patterns and hunt for unusual ones. This is an iterative process that uses data profiling, domain knowledge, and insights from previous query results to guide each step. We’ll start our journey by checking the counts of earth‐ quakes by year, which we can do by truncating the time field to the year level, and counting the records. For databases that don’t support date_trunc, consider extract or trunc instead: SELECT date_trunc('year',time)::date as earthquake_year ,count(*) as earthquakes FROM earthquakes GROUP BY 1 ; earthquake_year earthquakes --------------- ----------- 2010-01-01 122322 2011-01-01 107397 2012-01-01 105693 2013-01-01 114368 2014-01-01 135247 2015-01-01 122914 2016-01-01 122420 2017-01-01 130622 2018-01-01 179304 254 | Chapter 6: Anomaly Detection 2019-01-01 171116 2020-01-01 184523 We can see that 2011 and 2012 had low numbers of earthquakes compared to other years. There was also a sharp increase in records in 2018", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 178 + }, + { + "text": "2012-01-01 105693 2013-01-01 114368 2014-01-01 135247 2015-01-01 122914 2016-01-01 122420 2017-01-01 130622 2018-01-01 179304 254 | Chapter 6: Anomaly Detection 2019-01-01 171116 2020-01-01 184523 We can see that 2011 and 2012 had low numbers of earthquakes compared to other years. There was also a sharp increase in records in 2018 that was sustained through 2019 and 2020. This seems unusual, and we can hypothesize that the earth became more seismically active suddenly, that there is an error in the data such as duplication of records, or that something changed in the data collection process. Let’s drill down to month level to see if this trend persists at a more granular level of time: SELECT date_trunc('month',time)::date as earthquake_month ,count(*) as earthquakes FROM earthquakes GROUP BY 1 ; earthquake_month earthquakes ---------------- ----------- 2010-01-01 9651 2010-02-01 7697 2010-03-01 7750 ... ... The output is displayed in Figure 6-10. We can see that although the number of earth‐ quakes varies from month to month, there does appear to be an overall increase start‐ ing in 2017. We can also see that there are three outlier months, in April 2010, July 2018, and July 2019. Figure 6-10. Number of earthquakes per month Forms of Anomalies | 255 From here we can continue checking the data at more granular time periods, perhaps optionally filtering the result set by a range of dates to focus in on these anomalous stretches of time. After narrowing in on the specific days or even times of day to pin‐ point when the spikes occurred, we might want to break the data down further by other attributes in the data set. This can help explain the anomalies or at least narrow down the conditions in which they occurred. For example, it turns out that the increase in earthquakes starting in 2017 can be at least partially explained by the status field. The status indicates whether the event has been reviewed by a human (“reviewed”) or was directly posted by a system without review (“automatic”): SELECT date_trunc('month',time)::date as earthquake_month ,status ,count(*) as earthquakes FROM earthquakes GROUP BY 1,2 ORDER BY 1 ; earthquake_month status earthquakes ---------------- -------- ----------- 2010-01-01 automatic 620 2010-01-01 reviewed 9031 2010-02-01 automatic 695 ... ... ... The trends of “automatic” and “reviewed” status are plotted in Figure 6-11. Figure 6-11. Number of earthquakes per month, split by status In the graph, we can see that the outlier counts in July 2018 and July 2019 are due to large increases in the number of “automatic”-status earthquakes, whereas the spike in 256 | Chapter 6: Anomaly Detection April 2010 was in “reviewed”-status earthquakes. A new type of automatic recording equipment may have been added to the data set in 2017, or perhaps there hasn’t been enough time to review all the recordings yet. Analyzing location in data sets that have that information can be another powerful way to find and understand anomalies. The earthquakes table contains information about many thousands of very small earthquakes, potentially obscuring our view of the", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 179 + }, + { + "text": "or perhaps there hasn’t been enough time to review all the recordings yet. Analyzing location in data sets that have that information can be another powerful way to find and understand anomalies. The earthquakes table contains information about many thousands of very small earthquakes, potentially obscuring our view of the very large, very noteworthy earthquakes. Let’s look at the locations of the biggest quakes, those of magnitude 6 or larger, and see where they cluster geographically: SELECT place, count(*) as earthquakes FROM earthquakes WHERE mag >= 6 GROUP BY 1 ORDER BY 2 desc ; place earthquakes ------------------------------------ ----------- near the east coast of Honshu, Japan 52 off the east coast of Honshu, Japan 34 Vanuatu 28 ... ... In contrast to time, where we queried at progressively more granular levels, the place values are already so granular that it’s a bit difficult to grasp the full picture, although the Honshu, Japan, region clearly stands out. We can apply some of the text analysis techniques from Chapter 5 to parse and then group the geographic information. In this case, we’ll use split_part to remove the direction text (such as “near the coast of” or “100km N of”) that often appears at the beginning of the place field: SELECT case when place like '% of %' then split_part(place,' of ',2) else place end as place ,count(*) as earthquakes FROM earthquakes WHERE mag >= 6 GROUP BY 1 ORDER BY 2 desc ; place earthquakes --------------------- ----------- Honshu, Japan 89 Vanuatu 28 Lata, Solomon Islands 28 ... ... Forms of Anomalies | 257 The region around Honshu, Japan, experienced 89 earthquakes, making it not only the location of the largest earthquake in the data set but also an outlier in the number of very large earthquakes recorded. We could continue to parse, clean, and group the place values to gain a more refined picture of where major earthquakes occur in the world. Finding anomalous counts, sums, or frequencies in data is usually an exercise that involves a number of rounds of querying different levels of granularity in succession. It’s common to start broad, then go more granular, zoom out again to compare to baseline trends, and zoom in again on specific splits or dimensions of the data. Fortu‐ nately, SQL is a great tool for this sort of rapid iteration. Combining techniques, espe‐ cially from time series analysis, discussed in Chapter 3, and text analysis, discussed in Chapter 5, will bring even more richness to the analysis. Anomalies from the Absence of Data We’ve seen how unusually high frequencies of events can signal anomalies. Keep in mind that the absence of records can also signal anomalies. For example, the heart‐ beat of a patient undergoing surgery is monitored. The absence of a heartbeat at any time generates an alert, as do irregularities in the heartbeat. In many contexts, how‐ ever, detecting the absence of data is difficult if you’re not specifically looking for it. Customers don’t always announce they are about to", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 180 + }, + { + "text": "a patient undergoing surgery is monitored. The absence of a heartbeat at any time generates an alert, as do irregularities in the heartbeat. In many contexts, how‐ ever, detecting the absence of data is difficult if you’re not specifically looking for it. Customers don’t always announce they are about to churn. They simply stop using the product or service and quietly drop out of the data set. One way to ensure that absences in data are noticed is to use techniques from cohort analysis, discussed in Chapter 4. In particular, a JOIN to a date series or data dimen‐ sion, to ensure that a record exists for every entity whether or not it was present in that time period, makes absences easier to detect. Another way to detect absence is to query for gaps, or time since last seen. Some regions are more prone to large earthquakes due to the way tectonic plates are arranged around the globe. We’ve also detected some of this in the data in our previ‐ ous examples. Earthquakes are notoriously hard to predict, even when we have a sense of where they are likely to occur. This doesn’t stop some people from speculat‐ ing about the next “big one” simply due to the amount of time that has passed since the last one. We can use SQL to find the gaps between large earthquakes and the time since the most recent one: SELECT place ,extract('days' from '2020-12-31 23:59:59' - latest) as days_since_latest ,count(*) as earthquakes ,extract('days' from avg(gap)) as avg_gap ,extract('days' from max(gap)) as max_gap FROM ( 258 | Chapter 6: Anomaly Detection SELECT place ,time ,lead(time) over (partition by place order by time) as next_time ,lead(time) over (partition by place order by time) - time as gap ,max(time) over (partition by place) as latest FROM ( SELECT replace( initcap( case when place ~ ', [A-Z]' then split_part(place,', ',2) when place like '% of %' then split_part(place,' of ',2) else place end ) ,'Region','') as place ,time FROM earthquakes WHERE mag > 5 ) a ) a GROUP BY 1,2 ; place days_since_latest earthquakes avg_gap max_gap ---------------- ----------------- ----------- ------- ------- Greece 62.0 109 36.0 256.0 Nevada 30.0 9 355.0 1234.0 Falkland Islands 2593.0 3 0.0 0.0 ... ... ... ... ... In the innermost subquery, the place field is parsed and cleaned, returning larger regions or countries, along with the time of each earthquake, for all earthquakes of magnitude 5 or greater. The second subquery uses a lead function to find the time of the next earthquake, if any, for each place and time, and the gap between each earth‐ quake and the next one. The max window function returns the most recent earth‐ quake for each place. The outer query calculates the days since the latest 5+ earthquake in the data set, using the extract function to return just the days from the interval that is returned when two dates are subtracted. Since the data set includes records only through the end of", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 181 + }, + { + "text": "quake for each place. The outer query calculates the days since the latest 5+ earthquake in the data set, using the extract function to return just the days from the interval that is returned when two dates are subtracted. Since the data set includes records only through the end of 2020, the timestamp “2020-12-31 23:59:59” is used, though current_timestamp or an equivalent expression would be appropriate if the data were refreshed on an ongoing basis. Days are extracted in a similar fashion from the average and max of the gap value. The time since the last major earthquake in a location may have little predictive power in practice, but in many domains, gaps and time since last seen metrics have practical applications. Understanding typical gaps between actions sets a baseline against which the current gap can be compared. When the current gap is within range of historical values, we might judge that a customer is retained, but when the current Forms of Anomalies | 259 gap is much longer, the risk of churn increases. The result set from a query that returns historical gaps can itself become the subject of an anomaly detection analysis, answering questions such as the longest amount of time that a customer was gone before subsequently returning. Handling Anomalies Anomalies can appear in data sets for a number of reasons and can take a number of forms, as we’ve just seen. After detecting anomalies, the next step is to handle them in some fashion. How this is done depends on both the source of the anomaly—under‐ lying process or data quality issue—and the end goal of the data set or analysis. The options include investigation without changes, removal, replacement, rescaling, and fixing upstream. Investigation Finding, or attempting to find, the cause of an anomaly is usually the first step in deciding what to do with it. This part of the process can be both fun and frustrating— fun in the sense that tracking down and solving a mystery engages our skills and crea‐ tivity, but frustrating in the sense that we’re often working under time pressure and tracking down anomalies can feel like going down an endless series of rabbit holes, leading us to wonder whether an entire analysis is flawed. When I’m investigating anomalies, my process usually involves a series of queries that bounce back and forth between searching for patterns and looking at specific exam‐ ples. A true outlier value is easy to spot. In such cases, I will usually query for the entire row that contains the outlier for clues as to the timing, source, and any other attributes that are available. Next, I’ll check records that share those attributes to see if they have values that seem unusual. For example, I might check to see whether other records on the same day have normal or unusual values. Traffic from a particular website or purchases of a particular product might reveal other anomalies. After investigating the source and attributes of anomalies when working", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 182 + }, + { + "text": "they have values that seem unusual. For example, I might check to see whether other records on the same day have normal or unusual values. Traffic from a particular website or purchases of a particular product might reveal other anomalies. After investigating the source and attributes of anomalies when working on data pro‐ duced internally in my organization, I get in touch with the stakeholders or product owners. Sometimes there is a known bug or flaw, but often enough there is a real issue in a process or system that needs to be addressed, and context information is useful. For external or public data sets, there may not be an opportunity to find the root cause. In these cases, my goal is to gather enough information to decide which of the options discussed next is appropriate. Removal One option for dealing with data anomalies is to simply remove them from the data set. If there is reason to suspect that there was an error in the data collection that 260 | Chapter 6: Anomaly Detection might affect the entire record, removal is appropriate. Removal is also a good option when the data set is large enough that dropping a few records is unlikely to affect the conclusions. Another good reason to use removal is when the outliers are so extreme that they would skew the results enough that entirely inappropriate conclusions would be drawn. We saw previously that the earthquakes data set contains a number of records with a magnitude of –9.99 and a few with –9. Since the earthquakes these values would cor‐ respond to are extremely small, we might suspect that they are erroneous values or were simply entered when the actual magnitude was unknown. Removing records with these values is straightforward in the WHERE clause: SELECT time, mag, type FROM earthquakes WHERE mag not in (-9,-9.99) limit 100 ; time mag type ------------------- ---- ---------- 2019-08-11 03:29:20 4.3 earthquake 2019-08-11 03:27:19 0.32 earthquake 2019-08-11 03:25:39 1.8 earthquake Before removing the records, however, we might want to determine whether includ‐ ing the outliers actually makes a difference to the output. For example, we might want to know if removing the outliers affects the average magnitude, since averages can easily be skewed by outliers. We can do this by calculating the average across the entire data set, as well as the average excluding the extreme low values, using a CASE statement to exclude them: SELECT avg(mag) as avg_mag ,avg(case when mag > -9 then mag end) as avg_mag_adjusted FROM earthquakes ; avg_mag avg_mag_adjusted ------------------ ------------------ 1.6251015161530643 1.6273225642983641 The averages are different only at the third significant digit (1.625 versus 1.627), which is a fairly small difference. However, if we filter just to Yellowstone National Park, where many of the –9.99 values occur, the difference is more dramatic: SELECT avg(mag) as avg_mag ,avg(case when mag > -9 then mag end) as avg_mag_adjusted FROM earthquakes WHERE place = 'Yellowstone National Park, Wyoming' ; Handling Anomalies | 261 avg_mag avg_mag_adjusted ---------------------- ----------------------", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 183 + }, + { + "text": "filter just to Yellowstone National Park, where many of the –9.99 values occur, the difference is more dramatic: SELECT avg(mag) as avg_mag ,avg(case when mag > -9 then mag end) as avg_mag_adjusted FROM earthquakes WHERE place = 'Yellowstone National Park, Wyoming' ; Handling Anomalies | 261 avg_mag avg_mag_adjusted ---------------------- ---------------------- 0.40639347873981053095 0.92332793709528214616 Although these are still small values, the difference between an average of 0.46 and 0.92 is big enough that we would likely choose to remove the outliers. Notice that there are two options for doing so: either in the WHERE clause, which removes the outliers from all the results, or in a CASE statement, which removes them only from specific calculations. Which option you choose depends on the con‐ text of the analysis, as well as on whether it is important to preserve the rows in order to retain total counts, or useful values in other fields. Replacement with Alternate Values Anomalous values can often be handled by replacing them with other values rather than removing entire records. An alternate value can be a default, a substitute value, the nearest numerical value within a range, or a summary statistic such as the average or median. We’ve seen previously that null values can be replaced with a default using the coa lesce function. When values are not necessarily null but are problematic for some other reason, a CASE statement can be used to substitute a default value. For exam‐ ple, rather than report on all the various seismic events, we might want to group the types that are not earthquakes into a single “Other” value: SELECT case when type = 'earthquake' then type else 'Other' end as event_type ,count(*) FROM earthquakes GROUP BY 1 ; event_type count ---------- ------- earthquake 1461750 Other 34176 This reduces the amount of detail in the data, of course, but it can also be a way to summarize a data set that has a number of outlier values for type, as we saw previ‐ ously. When you know that outlier values are incorrect, and you know the correct value, replacing them with a CASE statement is also a solution that preserves the row in the overall data set. For example, an extra 0 might have been added to the end of a record, or a value might have been recorded in inches instead of miles. Another option for handling numeric outliers is to replace the extreme values with the nearest high or low value that is not extreme. This approach maintains much of 262 | Chapter 6: Anomaly Detection the range of values but prevents misleading averages that can result from extreme outliers. Winsorization is a specific technique for this, where outliers are set to a spe‐ cific percentile of the data. For example, values above the 95th percentile are set to the 95th percentile value, while values below the 5th percentile are set to the 5th percen‐ tile value. To calculate this in SQL, we first calculate the 5th and 95th percentile values:", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 184 + }, + { + "text": "to a spe‐ cific percentile of the data. For example, values above the 95th percentile are set to the 95th percentile value, while values below the 5th percentile are set to the 5th percen‐ tile value. To calculate this in SQL, we first calculate the 5th and 95th percentile values: SELECT percentile_cont(0.95) within group (order by mag) as percentile_95 ,percentile_cont(0.05) within group (order by mag) as percentile_05 FROM earthquakes ; percentile_95 percentile_05 ------------- ------------- 4.5 0.12 We can put this calculation in a subquery and then use a CASE statement to handle setting values for outliers below the 5th percentile and above the 95th. Note the Cartesian JOIN that allows us to compare the percentile values with each individual magnitude: SELECT a.time, a.place, a.mag ,case when a.mag > b.percentile_95 then b.percentile_95 when a.mag < b.percentile_05 then b.percentile_05 else a.mag end as mag_winsorized FROM earthquakes a JOIN ( SELECT percentile_cont(0.95) within group (order by mag) as percentile_95 ,percentile_cont(0.05) within group (order by mag) as percentile_05 FROM earthquakes ) b on 1 = 1 ; time place mag mag_winsorize ------------------- --------------------------- ---- ------------- 2014-01-19 06:31:50 5 km SW of Volcano, Hawaii -9 0.12 2012-06-11 01:59:01 Nevada -2.6 0.12 ... ... ... ... 2020-01-27 21:59:01 31km WNW of Alamo, Nevada 2 2.0 2013-07-07 08:38:59 54km S of Fredonia, Arizona 3.5 3.5 ... ... ... ... 2013-09-25 16:42:43 46km SSE of Acari, Peru 7.1 4.5 2015-04-25 06:11:25 36km E of Khudi, Nepal 7.8 4.5 ... ... ... ... Handling Anomalies | 263 The 5th percentile value is 0.12, while the 95th percentile is 4.5. Values below and above these thresholds are changed to the threshold in the mag_winsorize field. Val‐ ues between these thresholds remain the same. There is no set percentile threshold for winsorizing. The 1st and 99th percentiles or even the 0.01th and 99.9th percentiles can be used depending on the requirements for the analysis and how prevalent and extreme the outliers are. Rescaling Rather than filtering out records or changing the values of outliers, rescaling values provides a path that retains all the values but makes analysis and graphing easier. We discussed the z-score previously, but it’s worth pointing out that this can be used as a way to rescale values. The z-score is useful because it can be used with both posi‐ tive and negative values. Another common transformation is converting to logarithmic (log) scale. The benefit of transforming values into log scale is that they retain the same ordering, but small numbers get spread out more. Log transformations can also be transformed back into the original scale, easing interpretation. A downside is that the log transformation cannot be used on negative numbers. In the earthquakes data set, we learned that the magnitude is already expressed in log scale. The magnitude 9.1 Great Tohoku Earth‐ quake is extreme, but the value would appear even more extreme were it not expressed in log scale! The depth field is measured in kilometers. Here we’ll query both the depth and the depth with", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 185 + }, + { + "text": "that the magnitude is already expressed in log scale. The magnitude 9.1 Great Tohoku Earth‐ quake is extreme, but the value would appear even more extreme were it not expressed in log scale! The depth field is measured in kilometers. Here we’ll query both the depth and the depth with the log function applied and then graph the output in Figures 6-12 and 6-13 in order to demonstrate the difference. The log function uses base 10 as a default. To reduce the result set for easier graphing, the depth is also rounded to one significant digit using the round function. The table is filtered to exclude values less than 0.05, as these would round to zero or less than zero: SELECT round(depth,1) as depth ,log(round(depth,1)) as log_depth ,count(*) as earthquakes FROM earthquakes WHERE depth >= 0.05 GROUP BY 1,2 ; depth log_depth earthquakes ----- ------------------- ----------- 0.1 -1.0000000000000000 6994 0.2 -0.6989700043360188 6876 0.3 -0.5228787452803376 7269 ... ... ... 264 | Chapter 6: Anomaly Detection Figure 6-12. Distribution of earthquakes by depth, with unadjusted depths Figure 6-13. Distribution of earthquakes by depth on a log scale In Figure 6-12, it’s apparent that there are a large number of earthquakes between 0.05 and maybe 20, but beyond that it’s difficult to see the distribution since the x-axis stretches all the way to 700 to capture the range of the data. When the depth is trans‐ formed to a log scale in Figure 6-13, however, the distribution of the smaller values is much easier to see. Notably, the spike at 1.0, which corresponds to a depth of 10 kilo‐ meters, is apparent. Handling Anomalies | 265 Other types of scale transformations, while not necessarily appro‐ priate for removing outliers, can be accomplished with SQL. Some common ones include: • Square root: use the sqrt function • Cube root: use the cbrt function • Reciprocal transformation: 1 / field_name Change the units, such as inches to feet or pounds to kilograms: multiply or divide by the appropriate conversion factor with * or /. Rescaling can be done in SQL code, or often alternatively in the software or coding language used for graphing. The log transformation is particularly useful when there is a large spread of positive values and the patterns that are important to detect exist in the lower values. As with all analysis, deciding how to handle anomalies depends on the purpose and the amount of context or domain knowledge you have about the data set. Removing outliers is the simplest method, but to retain all the records, techniques such as win‐ sorizing and rescaling work well. Conclusion Anomaly detection is a common practice in analysis. The goal may be to detect the outliers, or it may be to manipulate them in order to prepare a data set for further analysis. In either case, the basic tools of sorting, calculating percentiles, and graphing the output of SQL queries can help you find them efficiently. Anomalies come in many varieties, with outlying values, unusual", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 186 + }, + { + "text": "outliers, or it may be to manipulate them in order to prepare a data set for further analysis. In either case, the basic tools of sorting, calculating percentiles, and graphing the output of SQL queries can help you find them efficiently. Anomalies come in many varieties, with outlying values, unusual bursts of activity, and unusual absences being most common. Domain knowledge is almost always helpful as you go through the process of finding and gathering information about the causes of anomalies. Options for dealing with anomalies include investigation, removal, replacement with alternate values, and rescaling the data. The choice depends heavily on the goal, but any of these paths can be accomplished with SQL. In the next chapter, we’ll turn our attention to experimentation, where the goal is to figure out whether a whole group of subjects differs from the norm of the control group. 266 | Chapter 6: Anomaly Detection CHAPTER 7 Experiment Analysis Experimentation, also known as A/B testing or split testing, is considered the gold standard for establishing causality. Much data analysis work involves establishing cor‐ relations: one thing is more likely to happen when another thing also happens, whether that be an action, an attribute, or a seasonal pattern. You’ve probably heard the saying “correlation does not imply causation,” however, and it is exactly this prob‐ lem in data analysis that experimentation attempts to solve. All experiments begin with a hypothesis: a guess about behavioral change that will result from some alteration to a product, process, or message. The change might be to a user interface, a new user onboarding flow, an algorithm that powers recommenda‐ tions, marketing messaging or timing, or any number of other areas. If the organiza‐ tion built it or has control over it, it can be experimented on, at least in theory. Hypotheses are often driven by other data analysis work. For example, we might find that a high percentage of people drop out of the checkout flow, and we could hypothesize that more people might complete the checkout process if the number of steps were reduced. The second element necessary for any experiment is a success metric. The behavioral change we hypothesize might be related to form completion, purchase conversion, click-through, retention, engagement, or any other behavior that is important to the organization’s mission. The success metric should quantify this behavior, be reasona‐ bly easy to measure, and be sensitive enough to detect a change. Click-through, checkout completion, and time to complete a process are often good success metrics. Retention and customer satisfaction are often less suitable success metrics, despite being very important, because they are frequently influenced by many factors beyond what is being tested in any individual experiment and thus are less sensitive to the changes we’d like to test. Good success metrics are often ones that you already track as part of understanding company or organizational health. 267 You may wonder whether an experiment can have multiple success metrics. Certainly with SQL, it is usually possible to", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 187 + }, + { + "text": "and thus are less sensitive to the changes we’d like to test. Good success metrics are often ones that you already track as part of understanding company or organizational health. 267 You may wonder whether an experiment can have multiple success metrics. Certainly with SQL, it is usually possible to generate many different calculations and metrics. You should be aware of the mul‐ tiple comparisons problem, however. I won’t go into a full explana‐ tion here, but the gist is that the more places you look for a significant change, the more likely you are to find one. Check one metric, and you may or may not find a significant change in one of the experiment variants. Check 20 metrics, however, and there’s a pretty good chance that at least one will show significance, whether or not the experiment had anything to do with that metric in the first place. As a rule of thumb, there should be one or maybe two primary success metrics. One to five additional metrics may be used for downside protection. These are sometimes called guardrail metrics. For example, you may want to ensure that an experiment doesn’t hurt page-loading time, even though it’s not the goal of the experiment to improve it. The third element of experimentation is a system that randomly assigns entities to a control or experiment variant group and alters the experience accordingly. This type of system is also sometimes called a cohorting system. A number of software vendors offer experiment-cohorting tools, though some organizations choose to build them internally in order to achieve more flexibility. Either way, to perform experiment analysis with SQL, the entity-level assignment data must flow into a table in the data‐ base that also contains behavioral data. The discussions of experiments in this chapter specifically refer to online experiments, in which variant assignment happens through a computer system and behavior is tracked digitally. There are cer‐ tainly many types of experiments performed across science and social science disciplines. A key difference is that the success met‐ rics and behaviors that are examined in online experiments are usually already tracked for other purposes, whereas in many scien‐ tific studies, the resulting behavior is tracked specifically for the experiment and only during the period of the experiment. With online experiments, we sometimes need to be creative about find‐ ing metrics that are good proxies for an impact when a direct measurement isn’t possible. With a hypothesis, a success metric, and a variant cohorting system in place, you can run experiments, collect the data, and analyze the outcomes using SQL. 268 | Chapter 7: Experiment Analysis Strengths and Limits of Experiment Analysis with SQL SQL is useful for analyzing experiments. In many cases with experiment analysis, the experiment cohort data and behavioral data are already flowing into a database, mak‐ ing SQL a natural choice. Success metrics are often already part of an organization’s reporting and analysis vocabulary, with SQL queries already developed. Joining var‐ iant assignment data to", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 188 + }, + { + "text": "experiments. In many cases with experiment analysis, the experiment cohort data and behavioral data are already flowing into a database, mak‐ ing SQL a natural choice. Success metrics are often already part of an organization’s reporting and analysis vocabulary, with SQL queries already developed. Joining var‐ iant assignment data to existing query logic is often relatively straightforward. SQL is a good choice for automating experiment result reporting. The same query can be run for each experiment, substituting the name or identifier of the experiment in the WHERE clause. Many organizations with high volumes of experiments have created standardized reports to speed up readouts and simplify the interpretation process. While SQL is useful for many of the steps involved with experiment analysis, it does have one major shortcoming: SQL is not able to calculate statistical significance. Many databases allow developers to extend SQL functionality with user-defined func‐ tions (UDFs). UDFs may be able to leverage statistical tests from languages such as Python, but they are beyond the scope of this book. A good option is to calculate summary statistics in SQL and then use an online calculator such as the one provided at Evanmiller.org to determine whether the experiment result is statistically significant. Why Correlation Is Not Causation: How Values Can Relate to Each Other It’s easier to prove that two values are correlated (they rise or fall together, or one exists primarily in the presence of the other) than it is to prove that one causes the other. Why is this the case? Although our brains are wired to detect causality, there are actually five ways in which two values, X and Y, can relate to each other: 1. X causes Y: This is of course what we’re all trying to find. Through some mecha‐ nism, Y is the result of X. 2. Y causes X: The relationship is there, but the direction of causality is reversed. For example, umbrellas don’t cause rain, but rather the presence of rain causes people to use umbrellas. 3. X and Y have a common cause: The values are related because there is some third variable that explains both of them. Sales of ice cream and use of air conditioners both rise in the summer, but neither causes the other. Higher temperatures cause both increases. Strengths and Limits of Experiment Analysis with SQL | 269 4. A feedback loop exists between X and Y: When Y increases, X increases to com‐ pensate, which leads to Y increasing in turn, and so on. This can happen when a customer is in the process of churning from a service. Fewer interactions lead to fewer items to suggest or remind, which leads to fewer interactions, and so on. Did the lack of recommendations cause less engagement, or is it the other way around? 5. There is no relationship; it’s just random: Search long enough and you will find metrics that are correlated even though there is no actual relationship between them. The Data Set For this chapter,", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 189 + }, + { + "text": "on. Did the lack of recommendations cause less engagement, or is it the other way around? 5. There is no relationship; it’s just random: Search long enough and you will find metrics that are correlated even though there is no actual relationship between them. The Data Set For this chapter, we will use a data set for a mobile game from the fictional Tanimura Studios. There are four tables. The game_users table contains records for people who downloaded the mobile game, along with the date and country. A sample of the data is shown in Figure 7-1. Figure 7-1. Sample of the game_users table The game_actions table contains records for things the users did in the game. A sam‐ ple of the data is shown in Figure 7-2. 270 | Chapter 7: Experiment Analysis Figure 7-2. Sample of the game_actions table The game_purchases table tracks purchases of in-game currency in US dollars. A sample of the data is shown in Figure 7-3. Figure 7-3. Sample of the game_purchases table Finally, the exp_assignment table contains records of which variant users were assigned to for a particular experiment. A sample of the data is shown in Figure 7-4. Figure 7-4. Sample of the exp_assignment table The Data Set | 271 1 See https://www.mathsisfun.com/data/chi-square-test.html for a good explanation of this test. All of the data in these tables is fictional, created with random number generators, though the structure is similar to what you might see in the database of a real digital gaming company. Types of Experiments There is a wide range of experiments. If you can change something that a user, cus‐ tomer, constituent, or other entity experiences, you can in theory test that change. From an analysis standpoint, there are two main types of experiments: those with binary outcomes and those with continuous outcomes. Experiments with Binary Outcomes: The Chi-Squared Test As you might expect, a binary outcome experiment has only two outcomes: either an action is taken or it isn’t. Either a user completes a registration flow or they don’t. A consumer clicks on a website ad or they don’t. A student graduates or they don’t. For these types of experiments, we calculate the proportion of each variant that completes the action. The numerator is the number of completers, while the denominator is all units that were exposed. This metric is also described as a rate: completion rate, click- through rate, graduation rate, and so on. To determine whether the rates in the variants are statistically different, we can use the chi-squared test, which is a statistical test for categorical variables.1 Data for a chi- squared test is often shown in the form of a contingency table, which shows the fre‐ quency of observations at the intersection of two attributes. This looks just like a pivot table to those who are familiar with that type of table. Let’s take a look at an example, using our mobile game data set. A product manager has introduced a new version of the", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 190 + }, + { + "text": "fre‐ quency of observations at the intersection of two attributes. This looks just like a pivot table to those who are familiar with that type of table. Let’s take a look at an example, using our mobile game data set. A product manager has introduced a new version of the onboarding flow, a series of screens that teach a new player how the game works. The product manager hopes that the new version will increase the number of players who complete the onboarding and start their first game session. The new version was introduced in an experiment called “Onboarding” that assigned users to either control or variant 1, as tracked in the exp_assignment table. An event called “onboarding complete” in the game_actions table indicates whether a user completed the onboarding flow. The contingency table shows the frequency at the intersection of the variant assign‐ ment (control or variant 1) and whether or not onboarding was completed. We can use a query to find the values for the table. Here we count the number of users with and without an “onboarding complete” action and GROUP BY the variant: 272 | Chapter 7: Experiment Analysis SELECT a.variant ,count(case when b.user_id is not null then a.user_id end) as completed ,count(case when b.user_id is null then a.user_id end) as not_completed FROM exp_assignment a LEFT JOIN game_actions b on a.user_id = b.user_id and b.action = 'onboarding complete' WHERE a.exp_name = 'Onboarding' GROUP BY 1 ; variant completed not_completed --------- --------- ------------- control 36268 13629 variant 1 38280 11995 Adding totals for each row and column turns this output into a contingency table, as in Figure 7-5. Figure 7-5. Contingency table for onboarding completions To make use of one of the online significance calculators, we will need the number of successes, or times when the action was taken, and the total number cohorted for each variant. The SQL to find the required data points is straightforward. The assigned variant and the count of users assigned to that variant are queried from the exp_assignment table. We then LEFT JOIN the game_actions table to find the count of users who completed onboarding. The LEFT JOIN is required since we expect that not all users completed the relevant action. Finally, we find the percent completed in each variant by dividing the number of users who completed by the total number cohorted: SELECT a.variant ,count(a.user_id) as total_cohorted ,count(b.user_id) as completions ,count(b.user_id) / count(a.user_id) as pct_completed FROM exp_assignment a LEFT JOIN game_actions b on a.user_id = b.user_id and b.action = 'onboarding complete' WHERE a.exp_name = 'Onboarding' GROUP BY 1 ; Types of Experiments | 273 variant total_cohorted completions pct_completed --------- -------------- ----------- ------------- control 49897 36268 0.7269 variant 1 50275 38280 0.7614 We can see that variant 1 did indeed have more completions than the control experi‐ ence, with 76.14% completing compared to 72.69%. But is this difference statistically significant, allowing us to reject the hypothesis that there is no difference? For this, we plug our results into an online calculator", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 191 + }, + { + "text": "0.7614 We can see that variant 1 did indeed have more completions than the control experi‐ ence, with 76.14% completing compared to 72.69%. But is this difference statistically significant, allowing us to reject the hypothesis that there is no difference? For this, we plug our results into an online calculator and confirm that the completion rate for variant 1 was significantly higher at a 95% confidence level than the completion rate for the control. Variant 1 can be declared the winner. A 95% confidence level is commonly used, although this is not the only option. There are many online articles and discussions about the meaning of confidence levels, which level to use, and adjust‐ ments in scenarios in which you are comparing multiple variants to a control. Binary outcome experiments follow this basic pattern. Calculate the successes or completions as well as the total members in each variant. The SQL used to derive the success events may be more complicated depending on the tables and how actions are stored in the database, but the output is consistent. Next, we’ll turn to experiments with continuous outcomes. Experiments with Continuous Outcomes: The t-Test Many experiments seek to improve continuous metrics, rather than the binary out‐ comes discussed in the last section. Continuous metrics can take on a range of values. Examples include amount spent by customers, time spent on page, and days an app is used. Ecommerce sites often want to increase sales, and so they might experiment on product pages or checkout flows. Content sites may test layout, navigation, and head‐ lines to try to increase the number of stories read. A company running an app might run a remarketing campaign to remind users to come back to the app. For these and other experiments with continuous success metrics, the goal is to figure out whether the average values in each variant differ from each other in a statistically significant way. The relevant statistical test is the two-sample t-test, which determines whether we can reject the null hypothesis that the averages are equal with a defined confidence interval, usually 95%. The statistical test has three inputs, all of which are straightforward to calculate with SQL: the mean, the standard deviation, and the count of observations. Let’s take a look at an example using our game data. In the last section, we looked at whether a new onboarding flow increased the completion rate. Now we will consider whether that new flow increased user spending on in-game currency. The success 274 | Chapter 7: Experiment Analysis metric is the amount spent, so we need to calculate the mean and standard deviation of this value for each variant. First we need to calculate the amount per user, since users can make multiple purchases. Retrieve the cohort assignment from the exp_assignment table and count the users. Next, LEFT JOIN to the game_purchases table to gather the amount data. The LEFT JOIN is required since not all users make a purchase, but we still need to", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 192 + }, + { + "text": "amount per user, since users can make multiple purchases. Retrieve the cohort assignment from the exp_assignment table and count the users. Next, LEFT JOIN to the game_purchases table to gather the amount data. The LEFT JOIN is required since not all users make a purchase, but we still need to include them in the mean and standard deviation calcu‐ lations. For users without purchases, the amount is set to a default of 0 with coa lesce. Since the avg and stddev functions ignore nulls, the 0 default is required to ensure that these records are included. The outer query summarizes the output val‐ ues by variant: SELECT variant ,count(user_id) as total_cohorted ,avg(amount) as mean_amount ,stddev(amount) as stddev_amount FROM ( SELECT a.variant ,a.user_id ,sum(coalesce(b.amount,0)) as amount FROM exp_assignment a LEFT JOIN game_purchases b on a.user_id = b.user_id WHERE a.exp_name = 'Onboarding' GROUP BY 1,2 ) a GROUP BY 1 ; variant total_cohorted mean_amount stddev_amount --------- -------------- ----------- ------------- control 49897 3.781 18.940 variant 1 50275 3.688 19.220 Next, we plug these values into an online calculator and find that there is no signifi‐ cant difference between the control and variant groups at a 95% confidence interval. The “variant 1” group appears to have increased onboarding completion rates but not the amount spent. Another question we might consider is whether variant 1 affected spending among those users who completed the onboarding. Those who don’t complete the onboard‐ ing never make it into the game and therefore don’t even have the opportunity to make a purchase. To answer this question, we can use a query similar to the previous one, but we’ll add an INNER JOIN to the game_actions table to restrict the users counted to only those who have an action of “onboarding complete”: SELECT variant ,count(user_id) as total_cohorted ,avg(amount) as mean_amount ,stddev(amount) as stddev_amount Types of Experiments | 275 FROM ( SELECT a.variant ,a.user_id ,sum(coalesce(b.amount,0)) as amount FROM exp_assignment a LEFT JOIN game_purchases b on a.user_id = b.user_id JOIN game_actions c on a.user_id = c.user_id and c.action = 'onboarding complete' WHERE a.exp_name = 'Onboarding' GROUP BY 1,2 ) a GROUP BY 1 ; variant total_cohorted mean_amount stddev_amount --------- -------------- ----------- ------------- control 36268 5.202 22.049 variant 1 38280 4.843 21.899 Plugging these values into the calculator reveals that the average for the control group is statistically significantly higher than that for variant 1 at a 95% confidence interval. This result may seem perplexing, but it illustrates why it is so important to agree on the success metric for an experiment up front. The experiment variant 1 had a posi‐ tive effect on onboarding completion and so can be judged a success. It did not have an effect on the overall spending level. This could be due to a mix shift: the additional users who made it through onboarding in variant 1 were less likely to pay. If the underlying hypothesis was that increasing onboarding completion rates would increase revenue, then the experiment should not be judged a success, and the prod‐", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 193 + }, + { + "text": "level. This could be due to a mix shift: the additional users who made it through onboarding in variant 1 were less likely to pay. If the underlying hypothesis was that increasing onboarding completion rates would increase revenue, then the experiment should not be judged a success, and the prod‐ uct managers should come up with some new ideas to test. Challenges with Experiments and Options for Rescuing Flawed Experiments Although experimentation is the gold standard for understanding causality, there are a number of ways experiments can go wrong. If the entire premise is flawed, there won’t be much that SQL can do to save the day. If the flaw is more technical in nature, we may be able to query the data in such a way as to adjust or exclude problematic data points and still interpret some results. Running experiments has a cost in terms of the time spent by engineers, designers, or marketers who create variants. It also has opportunity cost, or the missed benefit that could have been gained by sending cus‐ tomers down an optimal conversion path or product experience. On a practical level, using SQL to help the organization at least learn something from an experiment is often time well spent. 276 | Chapter 7: Experiment Analysis Variant Assignment Random assignment of experiment units (which can be users, sessions, or other enti‐ ties) to control and variant groups is one of the key elements of experimentation. However, sometimes errors in the assignment process happen, whether because of a flaw in the experiment specification, a technical failure, or a limitation in the cohort‐ ing software. As a result, the control and variant groups may be of unequal sizes, fewer overall entities may have been cohorted than expected, or the assignment may not have actually been random. SQL can sometimes help salvage an experiment in which too many units were cohor‐ ted. I have seen this happen when an experiment is meant to target only new users but all users are cohorted instead. Another way this can happen is when an experi‐ ment tests something that only a subset of users will see, either because it is a few clicks into the experience or because certain conditions must be met, such as previous purchase. Due to technical limitations, all users get cohorted, even though a chunk of them would never see the experiment treatment even if they are in that group. The solution is to add a JOIN into the SQL that restricts the users or entities considered to only those that were intended to be eligible. For example, in the case of a new user experiment, we can add an INNER JOIN to a table or subquery that contains user reg‐ istration date and set a WHERE condition to exclude users who registered too far prior to the cohort event to be considered new. The same strategy can be used when a certain condition must be met to even see the experiment. Restrict the entities", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 194 + }, + { + "text": "or subquery that contains user reg‐ istration date and set a WHERE condition to exclude users who registered too far prior to the cohort event to be considered new. The same strategy can be used when a certain condition must be met to even see the experiment. Restrict the entities included by excluding those that shouldn’t be eligible via JOINs and WHERE condi‐ tions. After doing this, you should check to make sure that the resulting population is a large enough sample to produce significant results. If too few users or entities are cohorted, it’s important to check whether the sample is large enough to produce significant results. If not, run the experiment again. If the sample size is large enough, a second consideration is whether there is bias in who or what was cohorted. As an example, I have seen cases in which users on certain brows‐ ers or older app versions were not cohorted due to technical limitations. If popula‐ tions that were excluded aren’t random and represent differences in location, technical savvy, or socioeconomic status, it’s important to consider both how large this population is relative to the rest and whether any adjustments should be made to include them in the final analysis. Another possibility is that the variant assignment system is flawed and entities are not assigned randomly. This is fairly unusual with most modern experiment tools, but if it happens, it invalidates the whole experiment. Results that are “too good to be true” might signal a variant assignment problem. I have seen, for example, cases in which highly engaged users are accidentally assigned to both treatment and control due to a change in experiment configuration. Careful data profiling can check whether entities Challenges with Experiments and Options for Rescuing Flawed Experiments | 277 have been assigned to multiple variants or whether users with high or low engage‐ ment prior to the experiment are clustered in a particular variant. Running an A/A test can help uncover flaws in the variant assign‐ ment software. In this type of test, entities are cohorted and success metrics are compared, just like in any other experiment. However, no changes are made to the experience, and both cohorts receive the control experience. Since the groups receive the same experi‐ ence, we should expect no significant differences in the success metric. If it turns out there is a difference, further investigation should be done to uncover and fix the problem. Outliers Statistical tests for analyzing continuous success metrics rely on averages. As a result, they are sensitive to unusually high or low outlier values. I have seen experiments in which the presence of one or two particularly high-spending customers in a variant gives that variant a statistically significant edge over others. Without those few high spenders, the result may be neutral or even the reverse. In most cases, we are more interested in whether a treatment has an effect across a range of individuals, and thus adjusting for these outliers can make", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 195 + }, + { + "text": "gives that variant a statistically significant edge over others. Without those few high spenders, the result may be neutral or even the reverse. In most cases, we are more interested in whether a treatment has an effect across a range of individuals, and thus adjusting for these outliers can make an experiment result more meaningful. We discussed anomaly detection in Chapter 6, and experiment analysis is another place in which those techniques can be applied. Outlier values can be determined either by analyzing the experiment results or by finding the base rate prior to the experiment. The outliers may be removed via a technique such as winsorizing (also discussed in Chapter 6), which removes values beyond a threshold, such as the 95th or 99th percentile. This can be done in SQL before moving on to the rest of the experiment analysis. Another option for dealing with outliers in continuous success metrics is to turn the success metric into a binary outcome. For example, instead of comparing the average spend across the control and variant groups, which may be distorted due to a few very high spenders, compare the purchase rate between the two groups and then fol‐ low the procedure discussed in the section on experiments with binary outcomes. We could consider the conversion rate to purchaser among users who completed onboarding in the control and variant 1 groups from the “Onboarding” experiment: SELECT a.variant ,count(distinct a.user_id) as total_cohorted ,count(distinct b.user_id) as purchasers ,count(distinct b.user_id) / count(distinct a.user_id) as pct_purchased FROM exp_assignment a LEFT JOIN game_purchases b on a.user_id = b.user_id JOIN game_actions c on a.user_id = c.user_id 278 | Chapter 7: Experiment Analysis and c.action = 'onboarding complete' WHERE a.exp_name = 'Onboarding' GROUP BY 1 ; variant total_cohorted purchasers pct_purchased --------- -------------- ---------- ------------- control 36268 4988 0.1000 variant 1 38280 4981 0.0991 We can look at the numbers and observe that even though there are more users in variant 1, there are fewer purchasers. The percentage of users who purchased in the control group is 10%, compared to 9.91% for variant 1. Next, we plug the data points into an online calculator. The conversion rate is statistically significantly higher for the control group. In this case, even though the rate of purchasing was higher for the control group, on a practical level we may be willing to accept this small decline if we believe that more users completing the onboarding process has other benefits. More players might boost rankings, for example, and players who enjoy the game may spread it to their friends via word of mouth, both of which can help growth and may then lead to attracting other new players who will become purchasers. The success metric can also be set to a threshold, and the share of entities meeting that threshold compared. For example, the success metric could be reading at least three stories or using an app at least two times a week. An infinite number of metrics could be constructed in this way,", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 196 + }, + { + "text": "metric can also be set to a threshold, and the share of entities meeting that threshold compared. For example, the success metric could be reading at least three stories or using an app at least two times a week. An infinite number of metrics could be constructed in this way, so it’s important to understand what is both impor‐ tant and meaningful to the organization. Time Boxing Experiments are often run over the course of several weeks. This means that individ‐ uals who enter the experiment earlier have a longer window in which to complete actions associated with the success metric. To control for this, we can apply time box‐ ing—imposing a fixed length of time relative to the experiment entry date and con‐ sidering actions only during that window. This concept was also covered in Chapter 4. For experiments, the appropriate size of the time box depends on what you are meas‐ uring. The window could be as short as one hour when measuring an action that typ‐ ically has an immediate response, such as clicking on an ad. For purchase conversion, experimenters often allow a window of 1 to 7 days. Shorter windows allow experi‐ ments to be analyzed sooner, since all cohorted entities need to be allowed the full time to complete actions. The best windows balance the need to obtain results with the actual dynamics of the organization. If customers typically convert in a few days, consider a 7-day window; if customers often take 20 or more days, consider a 30-day window. Challenges with Experiments and Options for Rescuing Flawed Experiments | 279 As an example, we can revise our first example from experiments with continuous outcomes by only including purchases within 7 days of the cohorting event. Note that it is important to use the time the entity was assigned to a variant as the starting point of the time box. An additional ON clause is added, restricting the results to purchases that occurred within the interval “7 days”: SELECT variant ,count(user_id) as total_cohorted ,avg(amount) as mean_amount ,stddev(amount) as stddev_amount FROM ( SELECT a.variant ,a.user_id ,sum(coalesce(b.amount,0)) as amount FROM exp_assignment a LEFT JOIN game_purchases b on a.user_id = b.user_id and b.purch_date <= a.exp_date + interval '7 days' WHERE a.exp_name = 'Onboarding' GROUP BY 1,2 ) a GROUP BY 1 ; variant total_cohorted mean_amount stddev_amount --------- -------------- ----------- ------------- control 49897 1.369 5.766 variant 1 50275 1.352 5.613 The means are similar, and in fact statistically they are not significantly different from each other. In this example, the time-boxed conclusion agrees with the conclusion when there was no time box. In this case, purchase events are relatively rare. For metrics that measure common events and those that accumulate quickly, such as pageviews, clicks, likes, and articles read, using a time box can prevent the earliest cohorted users from looking substan‐ tially “better” than those cohorted later. Repeated Exposure Experiments In discussions of online experimentation, most examples are of what I like to call “one-and-done” experiences: the", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 197 + }, + { + "text": "that accumulate quickly, such as pageviews, clicks, likes, and articles read, using a time box can prevent the earliest cohorted users from looking substan‐ tially “better” than those cohorted later. Repeated Exposure Experiments In discussions of online experimentation, most examples are of what I like to call “one-and-done” experiences: the user encounters a treatment once, reacts to it, and does not pass that way again. User registration is a classic example: a consumer signs up for a particular service only once, and therefore any changes to the sign-up pro‐ cess affect only new users. Analyzing tests on these experiences is relatively straightforward. There is another type of experience that I call “repeated exposure,” in which an indi‐ vidual comes into contact with the change many times during the course of using a 280 | Chapter 7: Experiment Analysis product or service. In any experiment involving these changes, we can expect individ‐ uals to encounter them more than once. Changes to an app’s user interface, such as color, text, and placement of important information and links, are experienced by users throughout their app usage. Email marketing programs that send customers reminders or promotions on a regular basis also have this repeated exposure quality. Emails are experienced many times as subject lines in the inbox, and as content if opened. Measuring repeated exposure experiments is trickier than measuring one-and-done experiments due to novelty effects and regression to the mean. A novelty effect is the tendency for behavior to change just because something is new, not because it is nec‐ essarily better. Regression to the mean is the tendency for phenomena to return to an average level over time. As an example, changing any part of a user interface tends to increase the number of people who interact with it, whether it is a new button color, logo, or placement of functionality. Initially the metrics look good, because the click- through rate or engagement goes up. This is the novelty effect. But over time, users get used to the change, and they tend to click or use the functionality at rates that return closer to the baseline. This is the regression to the mean. The important ques‐ tion to answer when running this kind of experiment is whether the new baseline is higher (or lower) than the previous one. One solution is to allow passage of a long enough time period, in which you might expect regression to happen, before evaluat‐ ing the results. In some cases, this will be a few days; in others, it might be a few weeks or months. When there are many changes, or the experiment comes in a series such as email or physical mail campaigns, figuring out whether the entire program makes a difference can be a challenge. It’s easy to claim success when customers receiving a certain email variant purchase a product, but how do we know if they would have made that pur‐ chase anyway? One option is to set up a long-term holdout.", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 198 + }, + { + "text": "out whether the entire program makes a difference can be a challenge. It’s easy to claim success when customers receiving a certain email variant purchase a product, but how do we know if they would have made that pur‐ chase anyway? One option is to set up a long-term holdout. This is a group that is set up to not receive any marketing messages or changes to the product experience. Note that this is different from simply comparing to users who have opted out of market‐ ing messages, since there is usually some bias in who opts out and who doesn’t. Set‐ ting up long-term holdouts can be complicated, but there are few better ways to truly measure the cumulative effects of campaigns and product changes. Another option is to perform cohort analysis (discussed in Chapter 4) on the var‐ iants. The groups can be followed for a longer time period, from weeks to months. Retention or cumulative metrics can be calculated and tested to see whether effects differ between variants over the long term. Even with the various challenges that can be encountered with experiments, they are still the best way to test and prove the causality around changes made to experiences ranging from marketing messaging and creative to in-product experience. We often Challenges with Experiments and Options for Rescuing Flawed Experiments | 281 encounter less than ideal situations in data analysis, however, so next we’ll turn to some analysis options for when A/B testing isn’t possible. When Controlled Experiments Aren’t Possible: Alternative Analyses Randomized experiments are the gold standard for going beyond correlation to establish causality. However, there are a number of reasons why a randomized experi‐ ment may not be possible. It may be unethical to give different treatments to different groups, particularly in medical or educational settings. Regulatory requirements may prevent experiments in other settings, such as financial services. There may be practi‐ cal reasons, such as the difficulty of restricting access to a variant treatment to only a randomized group. It’s always worth considering whether there are pieces that are worth testing or are testable within ethical, regulatory, and practical boundaries. Wording, placement, and other design elements are some examples. A second situation in which experimentation isn’t possible is when a change hap‐ pened in the past and the data has already been collected. Aside from reverting the change, going back and running an experiment isn’t an option. Sometimes a data ana‐ lyst or data scientist wasn’t available to advise on experimentation. More than once, I have joined an organization and been asked to untangle the results of changes that would have been much more straightforward to understand had there been a holdout group. At other times, the change is unintended. Examples include site outages that affect some or all customers, errors in forms, and natural disasters such as storms, earthquakes, and wildfires. Although causal conclusions aren’t as strong in situations in which there was no experiment, there are a few quasi-experimental analysis methods that", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 199 + }, + { + "text": "other times, the change is unintended. Examples include site outages that affect some or all customers, errors in forms, and natural disasters such as storms, earthquakes, and wildfires. Although causal conclusions aren’t as strong in situations in which there was no experiment, there are a few quasi-experimental analysis methods that can be used to draw insights from the data. These rely on constructing groups from the available data that represent “control” and “treatment” conditions as closely as possible. Pre-/Post-Analysis A pre-/post-analysis compares either the same or similar populations before and after a change. The measurement of the population before the change is used as the con‐ trol, while the measurement after the change is used as the variant or treatment. Pre-/post-analysis works best when there is a clearly defined change that happened on a well-known date, so that the before and after groups can be cleanly divided. In this type of analysis, you will need to choose how long to measure before and after the change, but the periods should be equal or close to equal. For example, if two weeks have elapsed since a change, compare that period to the two weeks prior to the change. Consider comparing multiple periods, such as one week, two weeks, three weeks, and four weeks before and after the change. If the results agree across all these 282 | Chapter 7: Experiment Analysis windows, you can have more confidence in the result than you would if the results differed. Let’s walk through an example. Imagine that the onboarding flow for our mobile game includes a step in which the user can check a box to indicate whether they want to receive emails with game news. This had always been checked by default, but a new regulation requires that it now be unchecked by default. On January 27, 2020, the change was released into the game, and we would like to find out if it had a negative effect on email opt-in rates. To do this, we will compare the two weeks before the change to the two weeks after the change and see whether the opt-in rate is statisti‐ cally significantly different. We could use one-week or three-week periods, but two weeks is chosen because it is long enough to allow for some day-of-week variability and also short enough to restrict the number of other factors that could otherwise affect users’ willingness to opt in. The variants are assigned in the SQL query via a CASE statement: the users who were created in the time range prior to the change are labeled “pre,” while those created after the change are labeled “post.” Next, we count the number of users in each group from the game_users table. Then we count the number of users who opted in, which is accomplished with a LEFT JOIN to the game_actions table, restricting to the records with the “email_optin” action. Then we divide the values to find the percent who opted in. I like to include the count", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 200 + }, + { + "text": "the game_users table. Then we count the number of users who opted in, which is accomplished with a LEFT JOIN to the game_actions table, restricting to the records with the “email_optin” action. Then we divide the values to find the percent who opted in. I like to include the count of days in each variant as a quality check, though it is not necessary to perform the rest of the analysis: SELECT case when a.created between '2020-01-13' and '2020-01-26' then 'pre' when a.created between '2020-01-27' and '2020-02-09' then 'post' end as variant ,count(distinct a.user_id) as cohorted ,count(distinct b.user_id) as opted_in ,count(distinct b.user_id) / count(distinct a.user_id) as pct_optin ,count(distinct a.created) as days FROM game_users a LEFT JOIN game_actions b on a.user_id = b.user_id and b.action = 'email_optin' WHERE a.created between '2020-01-13' and '2020-02-09' GROUP BY 1 ; variant cohorted opted_in pct_optin days ------- -------- -------- --------- ---- pre 24662 14489 0.5875 14 post 27617 11220 0.4063 14 When Controlled Experiments Aren’t Possible: Alternative Analyses | 283 Many databases will recognize dates entered as strings, as in '2020-01-13'. If your database does not, cast the string to a date using one of these options: cast('2020-01-13' as date) date('2020-01-13') '2020-01-13'::date In this case, we can see that the users who went through the onboarding flow before the change had a much higher email opt-in rate—58.75%, compared to 40.63% after‐ ward. Plugging the values into an online calculator results in confirmation that the rate for the “pre” group is statistically significantly higher than the rate for the “post” group. In this example, there may not be much the game company can do, since the change is due to a regulation. Further tests could determine whether providing sam‐ ple content or other information about the email program might encourage more new players to opt in, if this is a business goal. When performing a pre-/post-analysis, keep in mind that other factors beyond the change that you’re trying to learn about may cause an increase or a decrease in the metric. External events, seasonality, marketing promotions, and so on can drastically change the environment and customers’ mindsets even within the span of a few weeks. As a result, this type of analysis is not as good as a true randomized experi‐ ment for proving causality. However, sometimes this is one of the few analysis options available, and it can generate working hypotheses that can be tested and refined in future controlled experiments. Natural Experiment Analysis A natural experiment occurs when entities end up with different experiences through some process that approximates randomness. One group receives the normal or con‐ trol experience, and another receives some variation that may have a positive or nega‐ tive effect. Usually these are unintentional, such as when a software bug is introduced, or when an event happens in one location but not in other locations. For this type of analysis to have validity, we must be able to clearly determine which entities were exposed. Additionally, a control group that is as", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 201 + }, + { + "text": "these are unintentional, such as when a software bug is introduced, or when an event happens in one location but not in other locations. For this type of analysis to have validity, we must be able to clearly determine which entities were exposed. Additionally, a control group that is as similar as possible to the exposed group is needed. SQL can be used to construct the variants and to calculate the size of cohorts and suc‐ cess events in the case of a binary outcome event (or the mean, standard deviation, and population sizes in the case of a continuous outcome event). The results can be plugged into an online calculator just like with any other experiment. As an example in the video game data set, imagine that, during the time period of our data, users in Canada were accidentally given a different offer on the virtual currency 284 | Chapter 7: Experiment Analysis purchase page the first time they looked at it: an extra zero was added to the number of virtual coins in each package. So, for example, instead of 10 coins the user would receive 100 game coins, or instead of 100 coins they would receive 1,000 game coins, and so on. The question we would like to answer is whether Canadians converted to buyers at a higher rate than other users. Rather than compare to the entire user base, we will compare only to users in the United States. The countries are close geographi‐ cally, and most users in the two countries speak the same language—and for the sake of the example, we’ll assume that we’ve done other analysis showing that their behav‐ ior is similar, while the behavior of users in other countries differs enough to exclude them. To perform the analysis, we create the “variants” from whatever the distinguishing characteristic is—in this case, the country field from the game_users table—but note that sometimes more complex SQL will be required, depending on the data set. The counts of users cohorted, and those who purchased, are calculated in the same way we saw previously: SELECT a.country ,count(distinct a.user_id) as total_cohorted ,count(distinct b.user_id) as purchasers ,count(distinct b.user_id) / count(distinct a.user_id) as pct_purchased FROM game_users a LEFT JOIN game_purchases b on a.user_id = b.user_id WHERE a.country in ('United States','Canada') GROUP BY 1 ; country total_cohorted purchasers pct_purchased ------------- -------------- ---------- ------------- Canada 20179 5011 0.2483 United States 45012 4958 0.1101 The share of users in Canada who purchased is in fact higher—24.83%, compared to 11.01% of those in the United States. Plugging these values into an online calculator confirms that the conversion rate in Canada is statistically significantly higher at a 95% confidence interval. The hardest part of analyzing a natural experiment tends to be finding a comparable population and showing that the two populations are similar enough to support the conclusions from the statistical test. Although it is virtually impossible to prove that there are no confounding factors, careful comparison of population demographics and behaviors lends credibility", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 202 + }, + { + "text": "analyzing a natural experiment tends to be finding a comparable population and showing that the two populations are similar enough to support the conclusions from the statistical test. Although it is virtually impossible to prove that there are no confounding factors, careful comparison of population demographics and behaviors lends credibility to the results. Since a natural experiment is not a true random experiment, the evidence for causality is weaker, and this should be noted in the presentation of analysis of this type. When Controlled Experiments Aren’t Possible: Alternative Analyses | 285 Analysis of Populations Around a Threshold In some cases, there is a threshold value that results in some people or other subject units getting a treatment, while others do not. For example, a certain grade point average might qualify students for a scholarship, a certain income level might qualify households for subsidized health care, or a high churn risk score might trigger a sales rep to follow up with a customer. In such cases, we can leverage the idea that subjects on either side of the threshold value are likely quite similar to each other. So instead of comparing the entire populations that did and did not receive the reward or inter‐ vention, we can compare only those that were close to the threshold both on the posi‐ tive and the negative side. The formal name for this is regression discontinuity design (RDD). To perform this type of analysis, we can construct “variants” by splitting the data around the threshold value, similar to what we did in the pre-/post-analysis. Unfortu‐ nately, there is no hard-and-fast rule about how wide to make the bands of values on either side of the threshold. The “variants” should be similar in size, and they should be large enough to allow for significance in the results analysis. One option is to per‐ form the analysis several times with a few different ranges. For example, you might analyze the differences between the “treated” group and the control when each group contains subjects that fall within 5%, 7.5%, and 10% of the threshold. If the conclu‐ sions from these analyses agree, there is more support for the conclusions. If they do not agree, however, the data may be considered inconclusive. As with other types of nonexperimental analysis, results from RDD should be taken as proving causality less conclusively. Potential confounding factors should receive careful attention as well. For example, if customers with high churn risk receive inter‐ ventions from multiple teams, or a special discount to encourage them to retain, in addition to a call from a sales rep, the data can potentially be tainted by those other changes. Conclusion Experiment analysis is a rich field that often incorporates different types of analysis seen in other parts of this book, from anomaly detection to cohort analysis. Data profiling can be useful in tracking down issues that occur. When randomized experi‐ ments are not possible, a variety of other techniques are available, and SQL can be used", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 203 + }, + { + "text": "that often incorporates different types of analysis seen in other parts of this book, from anomaly detection to cohort analysis. Data profiling can be useful in tracking down issues that occur. When randomized experi‐ ments are not possible, a variety of other techniques are available, and SQL can be used to create synthetic control and variant groups. In the next chapter, we’ll turn to constructing complex data sets for analysis, an area that brings together various topics that we’ve discussed in the book so far. 286 | Chapter 7: Experiment Analysis CHAPTER 8 Creating Complex Data Sets for Analysis In Chapters 3 through 7, we looked at a number of ways in which SQL can be used to perform analysis on data in databases. In addition to these specific use cases, some‐ times the goal of a query is to assemble a data set that is specific yet general-purpose enough that it can be used to perform a variety of further analyses. The destination might be a database table, a text file, or a business intelligence tool. The SQL that is needed might be simple, requiring only a few filters or aggregations. Often, however, the code or logic needed to achieve the desired data set can become very complex. Additionally, such code is likely to be updated over time, as stakeholders request additional data points or calculations. The organization, performance, and maintain‐ ability of your SQL code become critical in a way that isn’t the case for one-time analyses. In this chapter, I’ll discuss principles for organizing code so that it’s easier to share and update. Then I’ll discuss when to keep query logic in the SQL and when to con‐ sider moving to permanent tables via ETL (extract-transform-load) code. Next, I’ll explain the options for storing intermediate results—subqueries, temp tables, and common table expressions (CTEs)—and considerations for using them in your code. Finally, I’ll wrap up with a look at techniques for reducing data set size and ideas for handling data privacy and removing personally identifiable information (PII). When to Use SQL for Complex Data Sets Almost all data sets prepared for further analysis contain some logic. The logic can range from the relatively simple—such as how tables are JOINed together and how filters are placed in the WHERE clause—to complex calculations that aggregate, cate‐ gorize, parse, or perform window functions over partitions of the data. When creat‐ ing data sets for further analysis, choosing whether to keep the logic within the SQL query or to push it upstream to an ETL job or downstream to another tool is often as 287 much art as science. Convenience, performance, and availability of help from engi‐ neers all factor into the decision. There is often no single right answer, but you will develop intuition and confidence the longer you work with SQL. Advantages of Using SQL SQL is a very flexible language. Hopefully I convinced you in the earlier chapters that a wide variety of data preparation and analysis tasks can", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 204 + }, + { + "text": "decision. There is often no single right answer, but you will develop intuition and confidence the longer you work with SQL. Advantages of Using SQL SQL is a very flexible language. Hopefully I convinced you in the earlier chapters that a wide variety of data preparation and analysis tasks can be accomplished using SQL. This flexibility is the main advantage of using SQL when developing complex data sets. In the initial stages of working with a data set, you may execute many queries. Work often starts with several profiling queries to understand the data. This is followed by building up the query step-by-step, checking transformations and aggregations along the way to be sure that the results returned are correct. This may be interspersed with more profiling, when actual values turn out to differ from our expectations. Complex data sets may be built up by combining several subqueries that answer specific ques‐ tions with JOINs or UNIONs. Running a query and examining the output is fast and allows for rapid iteration. Aside from relying on the quality and timeliness of the data in the tables, SQL has few dependencies. Queries are run on demand and don’t rely on a data engineer or a release process. Queries can often be embedded into business intelligence (BI) tools or into R or Python code by the analyst or data scientist, without requesting technical support. When a stakeholder needs another attribute or aggregation added to the output, changes can be made quickly. Keeping logic in the SQL code itself is ideal when working on a new analysis and when you expect the logic and result set to undergo changes frequently. Additionally, when the query is fast and data is returned to stakeholders quickly, there may never be a need to move the logic anywhere else. When to Build into ETL Instead There are times when moving logic into an ETL process is a better choice than keep‐ ing all of it in a SQL query, especially when working in an organization that has a data warehouse or data lake. The two main reasons to use ETL are performance and visibility. Performance of SQL queries depends on the complexity of the logic, the size of the tables queried, and the computational resources of the underlying database. Although many queries run fast, particularly on the newer databases and hardware, you will inevitably end up writing some queries that have complex calculations, involve JOINs of large tables or Cartesian JOINs, or otherwise cause query time to slow down to minutes or longer. An analyst or a data scientist may be willing to wait for a query to 288 | Chapter 8: Creating Complex Data Sets for Analysis return. However, most consumers of data are used to websites’ rapid response times and will get frustrated if they have to wait more than a few seconds for data. ETL runs behind the scenes at scheduled times and writes the result to a table. Since it is behind the scenes, it", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 205 + }, + { + "text": "However, most consumers of data are used to websites’ rapid response times and will get frustrated if they have to wait more than a few seconds for data. ETL runs behind the scenes at scheduled times and writes the result to a table. Since it is behind the scenes, it can run for 30 seconds, five minutes, or an hour, and end users will not be affected. Schedules are often daily but can be set to shorter intervals. End users can query the resulting table directly, without need for JOINs or other logic, and thus experience fast query times. A good example of when ETL is often a better choice than keeping all of the logic in a SQL query is the daily snapshot table. In many organizations, keeping a daily snap‐ shot of customers, orders, or other entities is useful for answering analytical ques‐ tions. For customers, we might want to calculate total orders or visits to date, current status in a sales pipeline, and other attributes that either change or accumulate. We’ve seen how to create daily series, including for days when an entity was not present, in the discussions of time series analysis in Chapter 3 and cohort analysis in Chapter 4. At the individual entity level, and over long time periods, such queries can become slow. Additionally, attributes such as current status may be overwritten in the source table, so capturing a daily snapshot may be the only way to preserve an accurate pic‐ ture of history. Developing the ETL and storing daily snapshot results in a table are often worth the effort. Visibility is a second reason to move logic into ETL. Often SQL queries exist on an individual’s computer or are buried within report code. It can be difficult for others to even find the logic embedded in the query, let alone understand and check it for mis‐ takes. Moving logic into ETL and storing the ETL code in a repository such as Git‐ Hub makes it easier for others in an organization to find, check, and iterate on it. Most repositories used by development teams also store change history, an additional benefit that allows you to see when a particular line in a query was added or changed. There are good reasons to consider putting logic into ETL, but this approach also has its drawbacks. One is that fresh results are not available until the ETL job has run and refreshed the data, even if new data has arrived in the underlying table. This can be overcome by continuing to run SQL against the raw data for very new records but limiting it to a small time window so that the query runs quickly. This can optionally be combined with a query on the ETL table, using subqueries or UNION. Another drawback to placing logic in ETL is that it becomes harder to change. Updates or bug fixes often need to be handed to a data engineer and code tested, checked into the", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 206 + }, + { + "text": "quickly. This can optionally be combined with a query on the ETL table, using subqueries or UNION. Another drawback to placing logic in ETL is that it becomes harder to change. Updates or bug fixes often need to be handed to a data engineer and code tested, checked into the repository, and released into the production data warehouse. For this reason, I usu‐ ally opt to wait until my SQL queries are past the period of rapid iteration, and the resulting data sets have been reviewed and are in use by the organization, before moving them into ETL. Of course, making code harder to change and enforcing code reviews are excellent ways to ensure consistency and data quality. When to Use SQL for Complex Data Sets | 289 Views as an Alternative to ETL When you want the reusability and code visibility of ETL but without actually storing the results and therefore taking up space permanently, a database view can be a good option. A view is essentially a saved query with a permanent alias that can be refer‐ enced just like any other table in the database. The query can be simple or complex and may involve table joins, filters, and any other elements of SQL. Views can be used to ensure that everyone querying the data uses the same defini‐ tions, as defined in the underlying query—for example, by always filtering out test transactions. They can be used to shield users from the complexity of the underlying logic, helpful for more novice or occasional users of a database. Views can also be used to provide an extra layer of security by restricting access to certain rows or col‐ umns in a database. For example, a view might be created to exclude PII such as email addresses but allow query writers to view other acceptable customer attributes. Views have a few drawbacks, however. They are objects in the database and thus require permissions to create and update their definitions. Views do not store the data, so each time a query is run with a view, the database must go back to the under‐ lying table or tables to fetch the data. As a result, they are not a replacement for ETL that creates a precomputed table of data. Most major databases also have materialized views, which are similar to views but do store the returned data in a table. Planning for the creation and refreshing of materi‐ alized views is usually best done in consultation with an experienced database admin‐ istrator, as there are nuanced performance considerations beyond the scope of this book. When to Put Logic in Other Tools SQL code and the query results output in your query editor are frequently only part of an analysis. Results are often embedded in reports, visualized into tables and graphs, or further manipulated in a range of tools, from spreadsheets and BI software to environments in which statistical or machine learning code is applied. In addition to choosing when to", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 207 + }, + { + "text": "query editor are frequently only part of an analysis. Results are often embedded in reports, visualized into tables and graphs, or further manipulated in a range of tools, from spreadsheets and BI software to environments in which statistical or machine learning code is applied. In addition to choosing when to move logic upstream into ETL, we also have choices about when to move logic downstream into other tools. Both performance and specific use cases are key factors in the decision. Each type of tool has performance strengths and limitations. Spreadsheets are very flexible but are not known for being able to handle large numbers of rows or complex calculations across many rows. Databases definitely have a performance advantage, so it’s often best to perform as much of the calculation as possible in the database and pass the smallest data set possible on to the spreadsheet. 290 | Chapter 8: Creating Complex Data Sets for Analysis BI tools have a range of capabilities, so it’s important to understand both how the software handles calculations and how the data will be used. Some BI tools can cache data (keep a local copy) in an optimized format, speeding up calculations. Others issue a new query each time a field is added to or removed from a report and thus mainly leverage the computational power of the database. Certain calculations such as count distinct and median require detailed, entity-level data. If it’s not possible to anticipate all the variations on these calculations in advance, it may be necessary to pass a larger, more detailed data set than otherwise might be ideal. Additionally, if the goal is to create a data set that allows exploration and slicing in many different ways, more detail is usually better. Figuring out the best combination of SQL, ETL, and BI tool computation can take some iteration. When the goal is to perform statistical or machine learning analysis on the data set using a language such as R or Python, detailed data is usually better. Both of these languages can perform tasks that overlap with SQL, such as aggregation and text manipulation. It’s often best to perform as much of the calculation as possible in SQL, to leverage the computational power of the database, but no more. Flexibility to iter‐ ate is usually an important part of the modeling process. The choice of whether to perform calculations in SQL or another language may also depend on your familiar‐ ity and comfort level with each. Those who are very comfortable in SQL may prefer to do more calculations in the database, while those who are more proficient in R or Python may prefer to do more calculations there. Although there are few rules to deciding where to put logic, I will encourage you to follow one rule in particular: avoid manual steps. It’s easy enough to open a data set in a spreadsheet or text editor, make a small change, save, and move on. But when you need to iterate, or", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 208 + }, + { + "text": "few rules to deciding where to put logic, I will encourage you to follow one rule in particular: avoid manual steps. It’s easy enough to open a data set in a spreadsheet or text editor, make a small change, save, and move on. But when you need to iterate, or when new data arrives, it’s easy to forget that manual step or to perform it inconsistently. In my experience, there’s no such thing as a truly “one-off” request. Put logic into code somewhere, if at all possible. SQL is a great tool and is incredibly flexible. It also sits within the analysis workflow and within an ecosystem of tools. Deciding where to put calculations can take some trial and error as you iterate through what is feasible among SQL, ETL, and down‐ stream tools. The more familiarity and experience you have with all the available options, the better you will be able to estimate trade-offs and continue to improve the performance and flexibility of your work. When to Use SQL for Complex Data Sets | 291 Code Organization SQL has few formatting rules, which can lead to unruly queries. Query clauses must be in the correct order: SELECT is followed by FROM, and GROUP BY cannot pre‐ cede WHERE, for example. A few keywords such as SELECT and FROM are reserved (i.e., they cannot be used as field names, table names, or aliases). However, unlike in some other languages, newlines, whitespace (other than the spaces that separate words), and capitalization are not important and are ignored by the database. Any of the example queries in this book could have been written on a single line, and with or without capital letters, except in quoted strings. As a result, the burden of code orga‐ nization is on the person writing the query. Fortunately, we have some formal and informal tools for keeping code organized, from commenting to “cosmetic” format‐ ting such as indentation and storage options for files of SQL code. Commenting Most coding languages have a way to indicate that a block of text should be treated as a comment and ignored during execution. SQL has two options. The first is to use two dash marks, which turns everything on the line that follows into a comment: -- This is a comment The second option is to use the slash (/) and star (*) characters to start a comment block, which can extend over multiple lines, followed by a star and slash to end the comment block: /* This is a comment block with multiple lines */ Many SQL editors adjust the visual style of code inside comments, by graying them out or otherwise changing the color, to make them easier to spot. Commenting code is a good practice, but admittedly it’s one that many people strug‐ gle to do on a regular basis. SQL is often written quickly, and especially during explo‐ ration or profiling exercises, we don’t expect to keep our code for the long term. Overly commented", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 209 + }, + { + "text": "easier to spot. Commenting code is a good practice, but admittedly it’s one that many people strug‐ gle to do on a regular basis. SQL is often written quickly, and especially during explo‐ ration or profiling exercises, we don’t expect to keep our code for the long term. Overly commented code can be just as difficult to read as code with no comments. And we all suffer from the idea that since we wrote the code, we will always be able to remember why we wrote the code. However, anyone who has inherited a long query written by a colleague or has stepped away from a query for a few months and then come back to update it will know that it can be frustrating and time consuming to decipher the code. To balance the burden and the benefit of commenting, I try to follow a few rules of thumb. First, add a comment anywhere that a value has a nonobvious meaning. Many source systems will encode values as integers, and their meaning is easy to forget. 292 | Chapter 8: Creating Complex Data Sets for Analysis Leaving a note makes the meaning clear and makes the code easier to change if needed: WHERE status in (1,2) -- 1 is Active, 2 is Pending Second, comment on any other nonobvious calculations or transformations. These can be anything that someone who hasn’t spent the time profiling the data set might not know, from data entry errors to the existence of outliers: case when status = 'Live' then 'Active' else status end /* we used to call customers Live but in 2020 we switched it to Active */ The third practice I try to follow around commenting is to leave notes when the query contains multiple subqueries. A quick line about what each subquery calculates makes it easy to skip to the relevant piece when coming back later to quality check or edit a longer query: SELECT... FROM ( -- find the first date for each customer SELECT ... FROM ... ) a JOIN ( -- find all of the products for each customer SELECT ... FROM ... ) b on a.field = b.field ... ; Commenting well takes practice and some discipline, but it’s worth doing for most queries that are longer than a few lines. Commenting can also be used to add useful information to the overall query, such as purpose, author, date created, and so on. Be kind to your colleagues, and to your future self, by placing helpful comments in your code. Capitalization, Indentation, Parentheses, and Other Formatting Tricks Formatting, and consistent formatting especially, is a good way to keep SQL code organized and legible. Databases ignore capitalization and whitespace (spaces, tabs, and newlines) in SQL, so we can use these to our advantage to format code into more legible blocks. Parentheses can both control the order of execution, which we will dis‐ cuss more later, and also visually group calculation elements. Capitalized words stand out from the", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 210 + }, + { + "text": "and whitespace (spaces, tabs, and newlines) in SQL, so we can use these to our advantage to format code into more legible blocks. Parentheses can both control the order of execution, which we will dis‐ cuss more later, and also visually group calculation elements. Capitalized words stand out from the rest, as anyone who has received an email with an all-caps subject line can confirm. I like to use capitalization only for the main Code Organization | 293 clauses: SELECT, FROM, JOIN, WHERE, and so on. Particularly in long or complex queries, being able to spot these quickly and thus to understand where the SELECT clause ends and the FROM clause begins saves me a lot of time. Whitespace is another key way to organize and make parts of the query easier to find, and to understand which parts logically go together. Any SQL query could be written on a single line in the editor, but in most cases this would lead to a lot of scrolling left and right through code. I like to start each clause (SELECT, FROM, etc.) on a new line, which, along with capitalization, helps me keep track of where each one starts and ends. Additionally, I find that putting aggregations on their own lines, as well as functions that take up some space, helps with organization. For CASE statements with more than two WHEN conditions, separating them onto multiple lines is also a good way to easily see and keep track of what is happening in the code. As an exam‐ ple, we can query the type and mag (magnitude), parse place, and then count the records in the earthquakes table, with some filtering in the WHERE clause: SELECT type, mag ,case when place like '%CA%' then 'California' when place like '%AK%' then 'Alaska' else trim(split_part(place,',',2)) end as place ,count(*) FROM earthquakes WHERE date_part('year',time) >= 2019 and mag between 0 and 1 GROUP BY 1,2,3 ; type mag place count ------------------ --- ---------- ----- chemical explosion 0 California 1 earthquake 0 Alaska 160 earthquake 0 Argentina 1 ... ... ... ... Indentation is another trick for keeping code visually organized. Adding spaces or tabs to line up the WHEN items within a CASE statement is one example. You’ve also seen subqueries indented in examples throughout the book. This makes subqueries visually stand apart, and when a query has multiple levels of nested subqueries, it makes it easier to see and understand the order in which they will be evaluated and which subqueries are peers in terms of level: SELECT... FROM ( SELECT... FROM ( SELECT... 294 | Chapter 8: Creating Complex Data Sets for Analysis FROM... ) a JOIN ( SELECT... FROM ) b on... ) a ... ; Any number of other formatting choices can be made, and the query will return the same results. Long-term SQL writers tend to have their own formatting preferences. However, clear and consistent formatting makes creating, maintaining, and sharing SQL code much easier. Many SQL query editors", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 211 + }, + { + "text": "on... ) a ... ; Any number of other formatting choices can be made, and the query will return the same results. Long-term SQL writers tend to have their own formatting preferences. However, clear and consistent formatting makes creating, maintaining, and sharing SQL code much easier. Many SQL query editors provide some form of query formatting and coloration. Usually keywords are colored, making them easier to spot within a query. These vis‐ ual clues make both developing and reviewing SQL queries much easier. If you’ve been writing SQL in a query editor all along, try opening a .sql file in a plain text edi‐ tor to see the difference coloration makes. Figure 8-1 shows an example of a SQL query editor, and the same code is shown in a plain text editor in Figure 8-2 (please note that these may appear in grayscale in some versions of this book). Figure 8-1. Screenshot of keyword coloration in the SQL query editor DBVisualizer Figure 8-2. The same code as plain text in the text editor Atom Code Organization | 295 Formatting is optional from the database perspective, but it’s a good practice. Consis‐ tent use of spacing, capitalization, and other formatting options goes a long way toward keeping your code readable, therefore making it easier to share and maintain. Storing Code After going to the trouble of commenting and formatting code, it’s a good idea to store it somewhere in case you need to use or reference it later. Many data analysts and scientists work with a SQL editor, often a desktop piece of software. SQL editors are useful because they usually include tools for browsing the database schema alongside a code window. They save files with a .sql extension, and these text files can be opened and changed in any text editor. Files can be saved in local directories or in cloud-based file storage services. Since they are text, SQL code files are easy to store in change control repositories such as GitHub. Using a repository provides a nice backup option and makes for easy sharing with others. Repositories also track the change history of files, which is useful when you need to figure out when a particular change was made, or when the change history is required for regulatory reasons. The main drawback of GitHub and other tools is that they are usually not a required step in the analysis workflow. You need to remember to update your code periodically, and as with any manual step, it’s easy to forget to do it. Organizing Computations Two related problems we face when creating complex data sets are getting the logic right and getting good query performance. Logic must be correct, or the results will be meaningless. Query performance for analysis purposes, unlike for transactional systems, usually has a range of “good enough.” Queries that don’t return are problem‐ atic, but the difference between waiting 30 seconds and waiting a minute for results may not matter a great deal. With SQL, there", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 212 + }, + { + "text": "the results will be meaningless. Query performance for analysis purposes, unlike for transactional systems, usually has a range of “good enough.” Queries that don’t return are problem‐ atic, but the difference between waiting 30 seconds and waiting a minute for results may not matter a great deal. With SQL, there is often more than one way to write a query that returns the correct results. We can use this to our advantage to both ensure correct logic and tune performance of long-running queries. There are three main ways to organize the calculation of intermediate results in SQL: the subquery, temp tables, and common table expressions (CTEs). Before we dive into them, we’ll review the order of evaluation in SQL. To wrap up the section, I’ll introduce grouping sets, which can replace the need to UNION queries together in certain cases. Understanding Order of SQL Clause Evaluation Databases translate SQL code into a set of operations that will be carried out in order to return the requested data. While understanding exactly how this happens isn’t nec‐ essary to be good at writing SQL for analysis, understanding the order in which the 296 | Chapter 8: Creating Complex Data Sets for Analysis database will perform its operations is incredibly useful (and is sometimes necessary to debug unexpected results). Many modern databases have sophisticated query optimizers that consider various parts of the query to come up with the most effi‐ cient plan for execution. Although they may consider parts of the query in a different order from that discussed here and therefore may need less query optimization from humans, they won’t calcu‐ late intermediate results in a different order from that discussed here. The general order of evaluation is shown in Table 8-1. SQL queries usually include only a subset of possible clauses, so actual evaluation includes only the steps relevant to the query. Table 8-1. SQL query order of evaluation 1 FROM including JOINs and their ON clauses 2 WHERE 3 GROUP BY including aggregations 4 HAVING 5 Window functions 6 SELECT 7 DISTINCT 8 UNION 9 ORDER BY 10 LIMIT and OFFSET First the tables in the FROM clause are evaluated, along with any JOINs. If the FROM clause includes any subqueries, these are evaluated before proceeding to the rest of the steps. In a JOIN, the ON clause specifies how the tables are to be JOINed, which may also filter the result set. FROM is always evaluated first, with one exception: when the query doesn’t contain a FROM clause. In most databases, it’s possi‐ ble to query using only a SELECT clause, as seen in some of the examples in this book. A SELECT-only query can return system information such as the date and database version. It can also apply mathematical, date, text, and other functions to constants. While there is admittedly little use for such queries in final analyses, they are handy for testing out functions or iterating over tricky calcula‐ tions rapidly. Organizing Computations | 297 Next, the", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 213 + }, + { + "text": "as the date and database version. It can also apply mathematical, date, text, and other functions to constants. While there is admittedly little use for such queries in final analyses, they are handy for testing out functions or iterating over tricky calcula‐ tions rapidly. Organizing Computations | 297 Next, the WHERE clause is evaluated to determine which records should be included in further calculations. Note that WHERE falls early in the order of evaluation and so cannot include the results of calculations that happen in a later step. GROUP BY is calculated next, including the related aggregations such as count, sum, and avg. As you might expect, GROUP BY will include only the values that exist in the FROM tables after any JOINing and filtering in the WHERE clause. HAVING is evaluated next. Since it follows GROUP BY, HAVING can perform filter‐ ing on aggregated values returned by GROUP BY. The only other way to filter by aggregated values is to place the query in a subquery and apply the filters in the main query. For example, we might want to find all the states that have at least one thou‐ sand terms in the legislators_terms table, and we’ll order by terms in descending order for good measure: SELECT state ,count(*) as terms FROM legislators_terms GROUP BY 1 HAVING count(*) >= 1000 ORDER BY 2 desc ; state terms ----- ----- NY 4159 PA 3252 OH 2239 ... ... Window functions, if used, are evaluated next. Interestingly, since aggregates have already been calculated at this point, they can be used in the window function defini‐ tion. For example, in the legislators data set from Chapter 4, we could calculate both the terms served per state and the average terms across all states in a single query: SELECT state ,count(*) as terms ,avg(count(*)) over () as avg_terms FROM legislators_terms GROUP BY 1 ; state terms avg_terms ----- ----- --------- ND 170 746.830 NV 177 746.830 OH 2239 746.830 ... ... ... 298 | Chapter 8: Creating Complex Data Sets for Analysis Aggregates can also be used in the OVER clause, as in the following query that ranks the states in descending order by the total number of terms: SELECT state ,count(*) as terms ,rank() over (order by count(*) desc) FROM legislators_terms GROUP BY 1 ; state terms rank ----- ----- ---- NY 4159 1 PA 3252 2 OH 2239 3 ... ... ... At this point, the SELECT clause is finally evaluated. This is a little counterintuitive since aggregations and window functions are typed in the SELECT section of the query. However, the database has already taken care of the calculations, and the results are then available for further manipulation or for display as is. For example, an aggregation can be placed within a CASE statement and have mathematical, date, or text functions applied if the result of the aggregation is one of those data types. The aggregators sum, count, and avg return numeric values. How‐ ever, min and max functions", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 214 + }, + { + "text": "for display as is. For example, an aggregation can be placed within a CASE statement and have mathematical, date, or text functions applied if the result of the aggregation is one of those data types. The aggregators sum, count, and avg return numeric values. How‐ ever, min and max functions return the same data type as the input and use the inherent ordering of that data type. For example, min and max dates return the earliest and latest calendar dates, while min and max on text fields use alphabetical order to determine the result. Following SELECT is DISTINCT, if present in the query. This means that all the rows are calculated and then deduplication occurs. UNION (or UNION ALL) is performed next. Up to this point, each query that makes up a UNION is evaluated independently. This stage is when the result sets are assem‐ bled together into one. This means that the queries can go about their calculations in very different ways or from different data sets. All UNION looks for is the same num‐ ber of columns and for those columns to have compatible data types. ORDER BY is almost the last step in evaluation. This means that it can access any of the prior calculations to sort the result set. The only caveat is that if DISTINCT is used, ORDER BY cannot include any fields that are not returned in the SELECT clause. Otherwise, it is entirely possible to order a result set by a field that doesn’t otherwise appear in the query. Organizing Computations | 299 LIMIT and OFFSET are evaluated last in the query execution sequence. This ensures that the subset of results returned will have fully calculated results as specified by any of the other clauses that are in the query. This also means that LIMIT has somewhat limited use in controlling the amount of work the database does before the results are returned to you. This is perhaps most noticeable when a query contains a large OFF‐ SET value. In order to OFFSET by, say, three million records, the database still needs to calculate the entire result set, figure out where the three millionth plus one record is, and then return the records specified by the LIMIT. This doesn’t mean LIMIT isn’t useful. Checking a few results can confirm calculations without overwhelming the network or your local machine with data. Also, using LIMIT as early in a query as possible, such as in a subquery, can still dramatically reduce the work required by the database as you develop a more complex query. Now that we have a good understanding of the order in which databases evaluate queries and perform calculations, we’ll turn to some options for controlling these operations in the context of a larger, complex query: subqueries, temporary tables, and CTEs. Subqueries Subqueries are usually the first way we learn how to control the order of evaluation in SQL, or to accomplish calculations that can’t be achieved in a single main query.", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 215 + }, + { + "text": "some options for controlling these operations in the context of a larger, complex query: subqueries, temporary tables, and CTEs. Subqueries Subqueries are usually the first way we learn how to control the order of evaluation in SQL, or to accomplish calculations that can’t be achieved in a single main query. They are versatile and can help organize long queries into smaller chunks with discrete purposes. A subquery is enclosed in parentheses, a notation that should be familiar from math‐ ematics, where parentheses also force evaluation of some part of an equation prior to the rest. Within the parentheses is a standalone query that is evaluated before the main outer query. Assuming the subquery is in the FROM clause, the result set can then be queried by the main code, just like any other table. We’ve already seen many examples with subqueries in this book. An exception to the standalone nature of a subquery is a special type called a lateral subquery, which can access results from previous items in the FROM clause. A comma and the keyword LATERAL are used instead of JOIN, and there is no ON clause. Instead, a prior query is used inside the subquery. As an example, imagine we wanted to analyze previous party membership for currently sitting legislators. We could find the first year they were a member of a different party, and check how com‐ mon that is when grouped by their current party. In the first subquery, we find the currently sitting legislators. In the second, lateral subquery, we use the results from the first subquery to return the earliest term_start where the party is different from the current party: 300 | Chapter 8: Creating Complex Data Sets for Analysis SELECT date_part('year',c.first_term) as first_year ,a.party ,count(a.id_bioguide) as legislators FROM ( SELECT distinct id_bioguide, party FROM legislators_terms WHERE term_end > '2020-06-01' ) a, LATERAL ( SELECT b.id_bioguide ,min(term_start) as first_term FROM legislators_terms b WHERE b.id_bioguide = a.id_bioguide and b.party <> a.party GROUP BY 1 ) c GROUP BY 1,2 ; first_year party legislators ---------- ---------- ----------- 1979.0 Republican 1 2011.0 Libertarian 1 2015.0 Democrat 1 This turns out to be fairly uncommon. Only three current legislators have switched parties, and no party has had more switchers than other parties. There are other ways to return the same result—for example, by changing the query to a JOIN and moving the criteria in the WHERE clause of the second subquery to the ON clause: SELECT date_part('year',c.first_term) as first_year ,a.party ,count(a.id_bioguide) as legislators FROM ( SELECT distinct id_bioguide, party FROM legislators_terms WHERE term_end > '2020-06-01' ) a JOIN ( SELECT id_bioguide, party ,min(term_start) as first_term FROM legislators_terms GROUP BY 1,2 ) c on c.id_bioguide = a.id_bioguide and c.party <> a.party GROUP BY 1,2 ; Organizing Computations | 301 1 Though in the case of Tableau, you can get around this with the Initial SQL option. If the second table is very large, filtering by a value returned in a previous subquery can speed up execution. In my", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 216 + }, + { + "text": "c.party <> a.party GROUP BY 1,2 ; Organizing Computations | 301 1 Though in the case of Tableau, you can get around this with the Initial SQL option. If the second table is very large, filtering by a value returned in a previous subquery can speed up execution. In my experience, use of LATERAL is less common, and therefore less well understood, than other syntax, so it’s good to reserve it for use cases that can’t be solved efficiently another way. Subqueries allow a lot of flexibility and control over the order of calculations. How‐ ever, a complex series of calculations in the middle of a larger query can become diffi‐ cult to understand and maintain. At other times, the performance of subqueries is too slow, or the query won’t return results at all. Fortunately, SQL has some additional options that may help in these situations: temporary tables and common table expressions. Temporary Tables A temporary (temp) table is created in a similar way to any other table in the database, but with a key difference: it persists only for the duration of the current session. Temp tables are useful when you are working with only a small part of a very large table, as small tables are much faster to query. They are also useful when you want to use an intermediate result in multiple queries. Since the temp table is a standalone table, it can be queried many times within the same session. Yet another time they are useful is when you are working in certain databases, such as Redshift or Vertica, that parti‐ tion data across nodes. INSERTing data into a temp table can align the partitioning to other tables that will be JOINed together in a subsequent query. There are two main drawbacks to temp tables. First, they require database privileges to write data, which may not be allowed for security reasons. Second, some BI tools, such as Tableau and Metabase, allow only a single SQL statement to create a data set,1 whereas a temp table requires at least two: the statement to CREATE and INSERT data into the temp table and the query using the temp table. To create a temporary table, use the CREATE command, followed by the keyword TEMPORARY and the name you wish to give it. The table can then be defined and a second statement used to populate it, or you can use CREATE as SELECT to create and populate in one step. For example, you could create a temp table with the distinct states for which there have been legislators: CREATE temporary table temp_states ( state varchar primary key ) ; INSERT into temp_states SELECT distinct state 302 | Chapter 8: Creating Complex Data Sets for Analysis FROM legislators_terms ; The first statement creates the table, while the second statement populates the temp table with values from a query. Note that by defining the table first, I need to specify the data type for all the columns (in this case", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 217 + }, + { + "text": "8: Creating Complex Data Sets for Analysis FROM legislators_terms ; The first statement creates the table, while the second statement populates the temp table with values from a query. Note that by defining the table first, I need to specify the data type for all the columns (in this case varchar for the state column) and can optionally use other elements of table definition, such as setting a primary key. I like to prefix temp table names with “temp_” or “tmp_” to remind myself of the fact that I’m using a temp table in the main query, but this isn’t strictly necessary. The faster and easier way to generate a temp table is the CREATE as SELECT method: CREATE temporary table temp_states as SELECT distinct state FROM legislators_terms ; In this case, the database automatically decides the data type based on the data returned by the SELECT statement, and no primary key is set. Unless you need fine- grained control for performance reasons, this second method will serve you well. Since temp tables are written to disk, if you need to repopulate them during a session, you will have to DROP and re-create the table or TRUNCATE the data. Disconnecting and reconnecting to the database also works. Common Table Expressions CTEs are a relative newcomer to the SQL language, having been introduced into many of the major databases only during the early 2000s. I wrote SQL for years without them, making do with subqueries and temp tables. I have to say that since I became aware of them a few years ago, they have steadily grown on me. You can think of a common table expression as being like a subquery lifted out and placed at the beginning of the query execution. It creates a temporary result set that can then be used anywhere in the subsequent query. A query can have multiple CTEs, and CTEs can use results from previous CTEs to perform additional calculations. CTEs are particularly useful when the result will be used multiple times in the rest of the query. The alternative, defining the same subquery multiple times, is both slow (since the database needs to execute the same query several times) and error-prone. Forgetting to update the logic in each identical subquery introduces error into the final result. Since CTEs are part of a single query, they don’t require any special data‐ base permissions. They can also be a useful way to organize code into discrete chunks and avoid sprawling nested subqueries. The main drawback of CTEs arises from the fact that they are defined at the begin‐ ning, separate from where they are used. This can make a query more difficult to Organizing Computations | 303 decipher for others when the query is very long, as it’s necessary to scroll to the beginning to check the definition and then back to where the CTE is used to under‐ stand what is happening. Good use of comments can help with this. A second chal‐ lenge", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 218 + }, + { + "text": "| 303 decipher for others when the query is very long, as it’s necessary to scroll to the beginning to check the definition and then back to where the CTE is used to under‐ stand what is happening. Good use of comments can help with this. A second chal‐ lenge is that CTEs make execution of sections of long queries more difficult. To check intermediate results in a longer query, it’s fairly easy to select and run just a subquery in a query development tool. If a CTE is involved, however, all of the surrounding code must be commented out first. To create a CTE, we use the WITH keyword at the beginning of the overall query, followed by a name for the CTE and then the query that makes it up enclosed in parentheses. For example, we could create a CTE that calculates the first term for each legislator and then use this result in further calculation, such as the cohort calculation introduced in Chapter 4: WITH first_term as ( SELECT id_bioguide ,min(term_start) as first_term FROM legislators_terms GROUP BY 1 ) SELECT date_part('year',age(b.term_start,a.first_term)) as periods ,count(distinct a.id_bioguide) as cohort_retained FROM first_term a JOIN legislators_terms b on a.id_bioguide = b.id_bioguide GROUP BY 1 ; periods cohort_retained ------- --------------- 0.0 12518 1.0 3600 2.0 3619 ... ... The query result is exactly the same as that returned by the alternate query using sub‐ queries seen in Chapter 4. Multiple CTEs can be used in the same query, separated by commas: WITH first_cte as ( SELECT... ), second_cte as ( SELECT... ) SELECT... ; 304 | Chapter 8: Creating Complex Data Sets for Analysis CTEs are a useful way to control the order of evaluation, improve performance in some instances, and organize your SQL code. They are easy to use once you are familiar with the syntax, and they are available in most major databases. There are often multiple ways to accomplish something in SQL, and although not required, CTEs add useful flexibility to your SQL skills toolbox. grouping sets Although this next topic isn’t strictly about controlling the order of evaluation, it is a handy way to avoid UNIONs and get the database to do all the work in a single query statement. Within the GROUP BY clause, special syntax is available in many major databases that includes grouping sets, cube, and rollup (though Redshift is an exception, and MySQL only has rollup). They are useful when the data set needs to contain subtotals for various combinations of attributes. For examples in this section, we’ll use a data set of video game sales that is available on Kaggle. It contains attributes for the name of each game as well as the platform, year, genre, and game publisher. Sales figures are provided for North America, the EU, Japan, Other (the rest of the world), and the global total. The table name is video game_sales. Figure 8-3 shows a sample of the table. Figure 8-3. Sample of the videogame_sales table So in the", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 219 + }, + { + "text": "platform, year, genre, and game publisher. Sales figures are provided for North America, the EU, Japan, Other (the rest of the world), and the global total. The table name is video game_sales. Figure 8-3 shows a sample of the table. Figure 8-3. Sample of the videogame_sales table So in the video game data set, for example, we might want to aggregate global_sales by platform, genre, and publisher as standalone aggregations (rather than only the combinations of the three fields that exist in the data) but output the results in one query set. This can be accomplished by UNIONing together three queries. Note that each query must contain at least placeholders for all three of the grouping fields: SELECT platform ,null as genre ,null as publisher ,sum(global_sales) as global_sales FROM videogame_sales GROUP BY 1,2,3 Organizing Computations | 305 UNION SELECT null as platform ,genre ,null as publisher ,sum(global_sales) as global_sales FROM videogame_sales GROUP BY 1,2,3 UNION SELECT null as platform ,null as genre ,publisher ,sum(global_sales) as global_sales FROM videogame_sales GROUP BY 1,2,3 ; platform genre publisher global_sales -------- ------ --------- ------------ 2600 (null) (null) 97.08 3DO (null) (null) 0.10 ... ... ... ... (null) Action (null) 1751.18 (null) Adventure (null) 239.04 ... ... ... ... (null) (null) 10TACLE Studios 0.11 (null) (null) 1C Company 0.10 ... ... ... ... This can be achieved in a more compact query using grouping sets. Within the GROUP BY clause, grouping sets is followed by the list of groupings to calculate. The previous query can be replaced by: SELECT platform, genre, publisher ,sum(global_sales) as global_sales FROM videogame_sales GROUP BY grouping sets (platform, genre, publisher) ; platform genre publisher global_sales -------- ------ --------- ------------ 2600 (null) (null) 97.08 3DO (null) (null) 0.10 ... ... ... ... (null) Action (null) 1751.18 (null) Adventure (null) 239.04 ... ... ... ... (null) (null) 10TACLE Studios 0.11 (null) (null) 1C Company 0.10 ... ... ... ... The items inside the grouping sets parentheses can include blanks as well as comma-separated lists of columns. As an example, we can calculate the global sales 306 | Chapter 8: Creating Complex Data Sets for Analysis without any grouping, in addition to the groupings by platform, genre, and pub lisher, by including a list item that is just a pair of parentheses. We’ll also clean up the output by substituting “All” for the null items using coalesce: SELECT coalesce(platform,'All') as platform ,coalesce(genre,'All') as genre ,coalesce(publisher,'All') as publisher ,sum(global_sales) as na_sales FROM videogame_sales GROUP BY grouping sets ((), platform, genre, publisher) ORDER BY 1,2,3 ; platform genre publisher global_sales -------- ------ --------- ------------ All All All 8920.44 2600 All All 97.08 3DO All All 0.10 ... ... ... ... All Action All 1751.18 All Adventure All 239.04 ... ... ... ... All All 10TACLE Studios 0.11 All All 1C Company 0.10 ... ... ... ... If we want to calculate all possible combinations of platform, genre, and publisher, such as the individual subtotals just calculated, plus all combinations of platform and genre, platform and publisher,", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 220 + }, + { + "text": "All 239.04 ... ... ... ... All All 10TACLE Studios 0.11 All All 1C Company 0.10 ... ... ... ... If we want to calculate all possible combinations of platform, genre, and publisher, such as the individual subtotals just calculated, plus all combinations of platform and genre, platform and publisher, and genre and publisher, we could specify all of these combinations in the grouping sets. Or we can use the handy cube syntax, which handles all of this for us: SELECT coalesce(platform,'All') as platform ,coalesce(genre,'All') as genre ,coalesce(publisher,'All') as publisher ,sum(global_sales) as global_sales FROM videogame_sales GROUP BY cube (platform, genre, publisher) ORDER BY 1,2,3 ; platform genre publisher global_sales -------- ------ --------- ------------ All All All 8920.44 PS3 All All 957.84 PS3 Action All 307.88 PS3 Action Atari 0.2 All Action All 1751.18 All Action Atari 26.65 All All Atari 157.22 ... ... ... ... Organizing Computations | 307 A third option is the function rollup, which returns a data set that has combinations determined by the ordering of fields in the parentheses, rather than all possible com‐ binations. So the previous query with the following clause: GROUP BY rollup (platform, genre, publisher) returns aggregations for the combinations of: platform, genre, publisher platform, genre platform But the query does not return aggregations for the combinations of: platform, publisher genre,publisher genre publisher Although it is possible to create the same output using UNION, the grouping sets, cube, and rollup options are big space and time savers when aggregations at multiple levels are needed, because they result in fewer lines of code and fewer scans of the underlying database tables. I once created a query hundreds of lines long using UNIONs to generate output for a dynamic website graphic that needed to have all possible combinations of filters precalculated. Quality checking it was an enormous chore, and updating it was even worse. Leveraging grouping sets, and CTEs for that matter, could have gone a long way toward making the code more compact and easy to write and maintain. Managing Data Set Size and Privacy Concerns After taking care to properly work out the logic in our SQL, organize our code, and make it efficient, we’re often faced with another challenge: the size of the result set. Data storage is ever cheaper, meaning that organizations are storing ever-larger data sets. Computational power is also always increasing, allowing us to crunch this data in the sophisticated ways we’ve seen in previous chapters. However, bottlenecks still occur, either in downstream systems such as BI tools or in the bandwidth available to pass large data sets between systems. Additionally, data privacy is a major concern that impacts how we handle sensitive data. For these reasons, in this section I’ll dis‐ cuss some ways to limit the size of data sets, as well as considerations for data privacy. Sampling with %, mod One way to reduce the size of a result set is to use a sample of the source data. Sam‐ pling means taking only a", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 221 + }, + { + "text": "this section I’ll dis‐ cuss some ways to limit the size of data sets, as well as considerations for data privacy. Sampling with %, mod One way to reduce the size of a result set is to use a sample of the source data. Sam‐ pling means taking only a subset of the data points or observations. This is appropri‐ ate when the data set is large enough and a subset is representative of the entire population. You can often sample website traffic and still retain most of the useful 308 | Chapter 8: Creating Complex Data Sets for Analysis insights, for example. There are two choices to make when sampling. The first is the size of the sample that achieves the right balance between reducing the size of the data set and not losing too much critical detail. The sample might include 10%, 1%, or 0.1% of the data points, depending on the starting volume. The second choice is the entity on which to perform the sampling. We might sample 1% of website visits, but if the goal of the analysis is to understand how users navigate the website, sampling 1% of website visitors would be a better choice in order to preserve all the data points for the users in the sample. The most common way to sample is to filter query results in the WHERE clause by applying a function to an entity-level identifier. Many ID fields are stored as integers. If this is the case, taking a modulo is a quick way to achieve the right result. The mod‐ ulo is the whole number remainder when one number is divided by another. For example, 10 divided by 3 is equal to 3 with a remainder (modulo) of 1. SQL has two equivalent ways to find the modulo—with the % sign and with the mod function: SELECT 123456 % 100 as mod_100; mod_100 ------- 56 SELECT mod(123456,100) as mod_100; mod_100 ------- 56 Both return the same answer, 56, which is also the last two digits of the input value 123456. To generate a 1% sample of the data set, place either syntax in the WHERE clause and set it equal to an integer—in this case, 7: SELECT user_id, ... FROM table WHERE user_id % 100 = 7 ; A mod of 100 creates a 1% sample, while a mod of 1,000 would create a 0.1% sample, and a mod of 10 would create a 10% sample. Although sampling in multiples of 10 is common, it’s not required, and any integer will work. Sampling from alphanumeric identifiers that include both letters and numbers isn’t as straightforward as sampling purely numeric identifiers. String-parsing functions can be used to isolate just the first or last few characters, and filters can be applied to them. For example, we can sample only identifiers ending in the letter “b” by parsing the last character from a string using the right function: Managing Data Set Size and Privacy Concerns | 309 SELECT user_id, ...", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 222 + }, + { + "text": "just the first or last few characters, and filters can be applied to them. For example, we can sample only identifiers ending in the letter “b” by parsing the last character from a string using the right function: Managing Data Set Size and Privacy Concerns | 309 SELECT user_id, ... FROM table WHERE right(user_id,1) = 'b' ; Assuming that any upper- or lowercase letter or number is a possible value, this will result in a sample of approximately 1.6% (1/62). To return a larger sample, adjust the filter to allow multiple values: SELECT user_id, ... FROM table WHERE right(user_id,1) in ('b','f','m') ; To create a smaller sample, include multiple characters: SELECT user_id, ... FROM table WHERE right(user_id,2) = 'c3' ; When sampling, it’s worth validating that the function you use to generate a sample does create a random or close-to-random sam‐ pling of the data. In one of my previous roles, we discovered that certain types of users were more likely to have certain combina‐ tions of the last two digits in their user IDs. In this case, using the mod function to generate a 1% sample resulted in noticeable bias in the results. Alphanumeric identifiers in particular often have com‐ mon patterns at the beginning or end of the string that data profil‐ ing can help identify. Sampling is an easy way to reduce data set size by orders of magnitude. It can both speed up calculations within SQL statements and allow the final result to be more compact, making it faster and easier to transfer to another tool or system. Sometimes the loss of detail from sampling isn’t acceptable, however, and other techniques are needed. Reducing Dimensionality The number of distinct combinations of attributes, or dimensionality, greatly impacts the number of records in a data set. To understand this, we can do a simple thought experiment. Imagine we have a field with 10 distinct values, and we count the num‐ ber of records and GROUP BY that field. The query will return 10 results. Now add in a second field, also with 10 distinct values, count the number of records, and GROUP BY the two fields. The query will return 100 results. Add in a third field with 10 distinct values, and the query result grows to 1,000 results. Even if not all the 310 | Chapter 8: Creating Complex Data Sets for Analysis combinations of the three fields actually exist in the table queried, it’s clear that adding additional fields into a query can increase the size of the results dramatically. When performing analysis, we can often control the number of fields and filter the values included in order to end up with a manageable output. However, when prepar‐ ing data sets for further analysis in other tools, the goal is often to provide flexibility and therefore to include many different attributes and calculations. To retain as much detail as possible while managing the overall size of the data, we can use one or more grouping techniques.", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 223 + }, + { + "text": "when prepar‐ ing data sets for further analysis in other tools, the goal is often to provide flexibility and therefore to include many different attributes and calculations. To retain as much detail as possible while managing the overall size of the data, we can use one or more grouping techniques. Granularity of dates and times is often an obvious place to look to reduce the size of data. Talk to your stakeholders to determine whether daily data is needed, for exam‐ ple, or whether weekly or monthly aggregations would work just as well. Grouping data by month and day of week might be a solution to aggregating data while still providing visibility into patterns that differ on weekdays versus weekends. Restricting the length of time returned is always an option, but that can restrict exploration of longer-term trends. I have seen data teams provide one data set that aggregates to a monthly level and covers several years, while a companion data set includes the same attributes but with daily or even hourly data for a much shorter time window. Text fields are another place to check for possible space savings. Differences in spell‐ ing or capitalization can result in many more distinct values than are useful. Applying text functions discussed in Chapter 5, such as lower, trim, or initcap, standardizes values and usually makes data more useful for stakeholders as well. REPLACE or CASE statements can be used to make more nuanced adjustments, such as adjusting spelling or changing a name that has been updated to a new value. Sometimes only a few values out of a longer list are relevant for analysis, so retaining detail for those while grouping the rest together is effective. I have seen this fre‐ quently when working with geographic locations. There are close to two hundred countries in the world, but often only a handful have enough customers or other data points to make reporting on them individually worthwhile. The legislators data set used in Chapter 4 contains 59 values for state, which includes the 50 states plus US territories that have representatives. We might want to create a data set with detail for the five states with the largest populations (currently California, Texas, Florida, New York, and Pennsylvania), and then group the rest into an “other” category with a CASE statement: SELECT case when state in ('CA','TX','FL','NY','PA') then state else 'Other' end as state_group ,count(*) as terms FROM legislators_terms GROUP BY 1 ORDER BY 2 desc ; Managing Data Set Size and Privacy Concerns | 311 state_group count ----------- ----- Other 31980 NY 4159 PA 3252 CA 2121 TX 1692 FL 859 The query returns only 6 rows, down from 59, which represents a significant decrease. To make the list more dynamic, we can first rank the values in a subquery, in this case by the distinct id_bioguide (legislator ID) values, and then return the state value for the top 5 and “Other” for the rest: SELECT case when b.rank <= 5 then", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 224 + }, + { + "text": "a significant decrease. To make the list more dynamic, we can first rank the values in a subquery, in this case by the distinct id_bioguide (legislator ID) values, and then return the state value for the top 5 and “Other” for the rest: SELECT case when b.rank <= 5 then a.state else 'Other' end as state_group ,count(distinct id_bioguide) as legislators FROM legislators_terms a JOIN ( SELECT state ,count(distinct id_bioguide) ,rank() over (order by count(distinct id_bioguide) desc) FROM legislators_terms GROUP BY 1 ) b on a.state = b.state GROUP BY 1 ORDER BY 2 desc ; state_group legislators ----------- ----------- Other 8317 NY 1494 PA 1075 OH 694 IL 509 VA 451 Several of the states change in this second list. If we continue to update the data set with fresh data points, the dynamic query will ensure that the output always reflects the current top values. Dimensionality can also be reduced by transforming the data into flag values. Flags are usually binary (i.e., they have only two values). BOOLEAN TRUE and FALSE can be used to encode flags, as can 1 and 0, “Yes” and “No,” or any other pair of meaning‐ ful strings. Flags are useful when a threshold value is important, but detail beyond that is less interesting. For example, we might want to know whether or not a website visitor completed a purchase, but detail on the exact number of purchases is less important. 312 | Chapter 8: Creating Complex Data Sets for Analysis In the legislators data set, there are 28 distinct numbers of terms served by the leg‐ islators. Instead of the exact value, however, we might want to include in our output only whether a legislator has served at least two terms, which we can do by turning the detailed values into a flag: SELECT case when terms >= 2 then true else false end as two_terms_flag ,count(*) as legislators FROM ( SELECT id_bioguide ,count(term_id) as terms FROM legislators_terms GROUP BY 1 ) a GROUP BY 1 ; two_terms_flag legislators -------------- ----------- false 4139 true 8379 About twice as many legislators have served at least two terms as compared to those with only one term. When combined with other fields in a data set, this type of trans‐ formation can result in much smaller result sets. Sometimes a simple true/false or presence/absence indicator is not quite enough to capture the needed nuance. In this case, numeric data can be transformed into several levels to maintain some additional detail. This is accomplished with a CASE state‐ ment, and the return value can be a number or string. We might want to include not only whether a legislator served a second term but also another indicator for those who served 10 or more terms: SELECT case when terms >= 10 then '10+' when terms >= 2 then '2 - 9' else '1' end as terms_level ,count(*) as legislators FROM ( SELECT id_bioguide ,count(term_id) as terms FROM legislators_terms GROUP BY 1 ) a GROUP BY 1 ;", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 225 + }, + { + "text": "for those who served 10 or more terms: SELECT case when terms >= 10 then '10+' when terms >= 2 then '2 - 9' else '1' end as terms_level ,count(*) as legislators FROM ( SELECT id_bioguide ,count(term_id) as terms FROM legislators_terms GROUP BY 1 ) a GROUP BY 1 ; Managing Data Set Size and Privacy Concerns | 313 terms_level legislators ----------- ----------- 1 4139 2 - 9 7496 10+ 883 Here we have reduced 28 distinct values down to 3, while retaining the notion of single-term legislators, those who were reelected, and the ones who are exceptionally good at staying in office. Such groupings or distinctions occur in many domains. As with all the transformations discussed here, it may take some trial and error to find the exact thresholds that are most meaningful for stakeholders. Finding the right bal‐ ance of detail and aggregation can greatly decrease data set size and therefore often speeds delivery time and performance of the downstream application. PII and Data Privacy Data privacy is one of the most important issues facing data professionals today. Large data sets with many attributes allow for more robust analysis with detailed insights and recommendations. However, when the data set is about individuals, we need to be mindful of both the ethical and the regulatory dimensions of the data col‐ lected and used. Regulations around the privacy of patients, students, and financial services customers have existed for many years. Laws regulating the data privacy rights of consumers have also come into force in recent years. The General Data Pro‐ tection Regulation (GDPR) passed by the EU is probably the most widely known. Other regulations include the California Consumer Privacy Act (CCPA), the Austral‐ ian Privacy Principles, and Brazil’s General Data Protection Law (LGPD). These and other regulations cover the handling, storage, and (in some cases) deletion of personally identifiable information (PII). Some categories of PII are obvious: name, address, email, date of birth, and Social Security number. PII also includes health indicators such as heart rate, blood pressure, and medical diagnoses. Location infor‐ mation, such as GPS coordinates, is also considered PII, since a small number of GPS locations can uniquely identify an individual. For example, GPS readings at my house and at my children’s school could uniquely identify someone in my household. A third GPS point at my office could uniquely identify me. As a data practitioner, it’s worthwhile to become familiar with what these regulations cover and to discuss how they affect your work with the privacy lawyers at your organization, who will have the most up-to-date information. A best practice when analyzing data that includes PII is to avoid including the PII itself in the outputs. This can be accomplished by aggregating data, substituting val‐ ues, or hashing values. 314 | Chapter 8: Creating Complex Data Sets for Analysis For most analyses, the goal is to find trends and patterns. Counting customers and averaging their behavior, rather than including individual detail in the output, is often", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 226 + }, + { + "text": "This can be accomplished by aggregating data, substituting val‐ ues, or hashing values. 314 | Chapter 8: Creating Complex Data Sets for Analysis For most analyses, the goal is to find trends and patterns. Counting customers and averaging their behavior, rather than including individual detail in the output, is often the purpose. Aggregations generally remove PII; however, be aware that a com‐ bination of attributes that have a user count of 1 could potentially be tied back to an individual. These can be treated as outliers and removed from the result in order to maintain a higher degree of privacy. If individual data is needed for some reason—to be able to calculate distinct users in a downstream tool, for instance—we can replace problematic values with random alternate values that maintain uniqueness. The row_number window function can be used to assign a new value to each individual in a table: SELECT email ,row_number() over (order by ...) FROM users ; The challenge in this case is to find a field to put in the ORDER BY that makes the ordering sufficiently random such that we can consider the resulting user identifier anonymized. Hashing values is another option. Hashing takes an input value and uses an algorithm to create a new output value. A particular input value will always result in the same output, making this a good option for maintaining uniqueness while obscuring sensi‐ tive values. The md5 function can be used to generate a hashed value: SELECT md5('my info'); md5 -------------------------------- 0fb1d3f29f5dd1b7cabbad56cf043d1a The md5 function hashes input values but does not encrypt them, and therefore it can be reversed to obtain the original value. For highly sensitive data, you should work with a database administra‐ tor to truly encrypt the data. Avoiding PII in the output of your SQL queries is always the best option if possible, since you avoid proliferating it into other systems or files. Replacing or masking the values is a second-best option. You can also explore secure methods to share data, such as developing a secured data pipeline directly between a database and an email system to avoid writing email addresses out to files, for example. With care and part‐ nership with technical and legal colleagues, it is possible to achieve high-quality anal‐ ysis while also preserving individuals’ privacy. Managing Data Set Size and Privacy Concerns | 315 Conclusion Surrounding every analysis, there are a number of decisions to be made around organizing the code, managing complexity, optimizing query performance, and safe‐ guarding privacy in the output. In this chapter, we’ve discussed a number of options and strategies and special SQL syntax that can help with these tasks. Try not to get overwhelmed by all of these options or to become concerned that, without mastery of these topics, you can’t be an efficient data analyst or data scientist. Not all of the tech‐ niques are required in every analysis, and there are often other ways to get the job done. The longer you spend analyzing data", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 227 + }, + { + "text": "these options or to become concerned that, without mastery of these topics, you can’t be an efficient data analyst or data scientist. Not all of the tech‐ niques are required in every analysis, and there are often other ways to get the job done. The longer you spend analyzing data with SQL, the more likely you are to come across situations in which one or more of these techniques come in handy. 316 | Chapter 8: Creating Complex Data Sets for Analysis CHAPTER 9 Conclusion Throughout the book, we’ve seen how SQL is a flexible and powerful language for a range of data analysis tasks. From data profiling to time series, text analysis, and anomaly detection, SQL can tackle a number of common requirements. Techniques and functions can also be combined in any given SQL statement to perform experi‐ ment analysis and build complex data sets. While SQL can’t accomplish all analysis goals, it fits well into the ecosystem of analysis tools. In this final chapter, I’ll discuss a few additional types of analysis and point out how various SQL techniques covered in the book can be combined to accomplish them. Then I’ll wrap up with some resources that you can use to continue your journey of mastering data analysis or to dig deeper into specific topics. Funnel Analysis A funnel consists of a series of steps that must be completed to reach a defined goal. The goal might be registering for a service, completing a purchase, or obtaining a course completion certificate. Steps in a website purchase funnel, for example, might include clicking the “Add to Cart” button, filling out shipping information, entering a credit card, and finally clicking the “Place Order” button. Funnel analysis combines elements of time series analysis, discussed in Chapter 3, and cohort analysis, discussed in Chapter 4. The data for funnel analysis comes from a time series of events, although in this case those events correspond to distinct real- world actions rather than being repetitions of the same event. Measuring retention from step to step is a key goal of funnel analysis, although in this context we often use the term conversion. Typically, entities drop out along the steps of the process, and a graph of their number at each stage ends up looking like a household funnel—hence the name. 317 This type of analysis is used to identify areas of friction, difficulty, or confusion. Steps at which large numbers of users drop out, or that many fail to complete, provide insight into opportunities for optimization. For example, a checkout process that asks for credit card information before showing the total amount including shipping might turn off some would-be purchasers. Showing the total before this step may encourage more purchase completions. Such changes are often subjects of experi‐ ments, discussed in Chapter 7. Funnels can also be monitored in order to detect unexpected external events. For example, changes in completion rates might corre‐ spond to good (or bad) PR or to a change in", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 228 + }, + { + "text": "this step may encourage more purchase completions. Such changes are often subjects of experi‐ ments, discussed in Chapter 7. Funnels can also be monitored in order to detect unexpected external events. For example, changes in completion rates might corre‐ spond to good (or bad) PR or to a change in the pricing or tactics of a competitor. The first step in a funnel analysis is to figure out the base population of all users, cus‐ tomers, or other entities that were eligible to enter the process. Next, assemble the data set of completion for each step of interest, including the final goal. Often this includes one or more LEFT JOINs in order to include all of the base population, along with those who completed each step. Then count the users in each step and divide these step-wise counts by the total count. There are two ways to set up the queries, depending on whether all steps are required. When all steps in the funnel are required—or if you only want to include users who have completed all steps—LEFT JOIN each table to the previous table: SELECT count(a.user_id) as all_users ,count(b.user_id) as step_one_users ,count(b.user_id) / count(a.user_id) as pct_step_one ,count(c.user_id) as step_two_users ,count(c.user_id) / count(b.user_id) as pct_one_to_two FROM users a LEFT JOIN step_one b on a.user_id = b.user_id LEFT JOIN step_two c on b.user_id = c.user_id ; When users can skip a step, or if you want to allow for this possibility, LEFT JOIN each table to the one containing the full population and calculate the share of that starting group: SELECT count(a.user_id) as all_users ,count(b.user_id) as step_one_users ,count(b.user_id) / count(a.user_id) as pct_step_one ,count(c.user_id) as step_two_users ,count(c.user_id) / count(b.user_id) as pct_step_two FROM users a LEFT JOIN step_one b on a.user_id = b.user_id LEFT JOIN step_two c on a.user_id = c.user_id ; It’s a subtle difference, but it’s one worth paying attention to and tailoring to the spe‐ cific context. Consider including time boxes, to only include users who complete an action within a specific time frame, if users can reenter the funnel after a lengthy absence. Funnel analyses can also include additional dimensions, such as cohort or 318 | Chapter 9: Conclusion other entity attributes, to facilitate comparisons and generate additional hypotheses about why a funnel is or is not performing well. Churn, Lapse, and Other Definitions of Departure The topic of churn came up in Chapter 4, since churn is essentially the opposite of retention. Often organizations want or need to come up with a specific definition of churn in order to measure it directly. In some cases, there is a contractually defined end date, such as with B2B software. But often churn is a fuzzier concept, and a time- based definition is more appropriate. Even when there is a contractual end date, measuring when a customer stops using a product can be an early warning sign of an imminent contract cancellation. Churn definitions can also be applied to certain products or features, even when the customer doesn’t churn from the", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 229 + }, + { + "text": "definition is more appropriate. Even when there is a contractual end date, measuring when a customer stops using a product can be an early warning sign of an imminent contract cancellation. Churn definitions can also be applied to certain products or features, even when the customer doesn’t churn from the organization entirely. A time-based churn metric counts customers as churned when they haven’t pur‐ chased or interacted with a product for a period of time, usually ranging from 30 days to as much as a year. The exact length depends a lot on the type of business and on typical usage patterns. To arrive at a good churn definition, you can use gap analysis to find typical periods between purchases or usage. To do gap analysis, you will need a time series of actions or events, the lag window function, and some date math. As an example, we can calculate the typical gaps between representatives’ terms, using the legislators data set introduced in Chapter 4. We’ll ignore the fact that politicians are often voted out of office rather than choosing to leave, since otherwise this data set has the right structure for this type of analysis. First we’ll find the average gap. To do this, we create a subquery that calculates the gap between the start_date and the previous start_date for each legislator for each term, and then we find the average value in the outer query. The previous start_date can be found using the lag func‐ tion, and the gap as a time interval is calculated with the age function: SELECT avg(gap_interval) as avg_gap FROM ( SELECT id_bioguide, term_start ,lag(term_start) over (partition by id_bioguide order by term_start) as prev ,age(term_start, lag(term_start) over (partition by id_bioguide order by term_start) ) as gap_interval FROM legislators_terms WHERE term_type = 'rep' ) a WHERE gap_interval is not null ; Churn, Lapse, and Other Definitions of Departure | 319 avg_gap ------------------------------------- 2 years 2 mons 17 days 15:41:54.83805 As we might expect, the average is close to two years, which makes sense since the term length for this office is two years. We can also create a distribution of gaps in order to pick a realistic churn threshold. In this case, we’ll transform the gap to months: SELECT gap_months, count(*) as instances FROM ( SELECT id_bioguide, term_start ,lag(term_start) over (partition by id_bioguide order by term_start) as prev ,age(term_start, lag(term_start) over (partition by id_bioguide order by term_start) ) as gap_interval ,date_part('year', age(term_start, lag(term_start) over (partition by id_bioguide order by term_start) ) ) * 12 + date_part('month', age(term_start, lag(term_start) over (partition by id_bioguide order by term_start) ) ) as gap_months FROM legislators_terms WHERE term_type = 'rep' ) a GROUP BY 1 ; gap_months instances ---------- --------- 1.0 25 2.0 4 3.0 2 ... ... If date_part is not supported in your database, extract can be used as an alternative. (Refer to Chapter 3 for an explanation and examples.) The output can be plotted, as in Figure 9-1. Since there is a long tail of months, this", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 230 + }, + { + "text": "1.0 25 2.0 4 3.0 2 ... ... If date_part is not supported in your database, extract can be used as an alternative. (Refer to Chapter 3 for an explanation and examples.) The output can be plotted, as in Figure 9-1. Since there is a long tail of months, this plot is zoomed in to show the range in which most gaps fall. The most common gap is 24 months, but there are also 320 | Chapter 9: Conclusion several hundred instances per month out to 32 months. There is another small bump to over 100 at 47 and 48 months. With the average and distribution in hand, I would likely set a threshold of either 36 or 48 months and say that any representative who hasn’t been reelected within this window has “churned.” Figure 9-1. Distribution of length of gap between representative term start dates, show‐ ing range from 10 to 59 months Once you have a defined threshold for churn, you can monitor the customer base with a “time since last” analysis. This can refer to last purchase, last payment, last time an app was opened, or whatever time-based metric is relevant for the organiza‐ tion. For this calculation, you need a data set that has the most recent date or time‐ stamp for each customer. If starting with a time series, first find the most recent timestamp for each customer in a subquery. Then apply date math to find the time elapsed between that date and the current date, or the latest date in the data set if some time has elapsed since the data was assembled. For example, we could find the distribution of years since the last election from the legislators_terms table. In the subquery, calculate the latest starting date using the max function and then find the time elapsed since then using the age function. In this case, the maximum data in the data set, May 5, 2019, is used. In a data set with up-to- date data, substitute current_date or an equivalent expression. The outer query finds the years from the interval using date_part and counts the number of legislators: SELECT date_part('year',interval_since_last) as years_since_last ,count(*) as reps FROM ( SELECT id_bioguide Churn, Lapse, and Other Definitions of Departure | 321 ,max(term_start) as max_date ,age('2020-05-19',max(term_start)) as interval_since_last FROM legislators_terms WHERE term_type = 'rep' GROUP BY 1 ) a GROUP BY 1 ; years_since_last reps ---------------- ----- 0.0 6 1.0 440 2.0 1 ... ... A related concept is “lapsed,” which is often used as an intermediate stage between fully active customers and churned customers and might alternatively be called “dor‐ mant.” A lapsed customer may be at higher risk of churning because we haven’t seen them for a while but still have a decent likelihood of returning based on our past experience. In consumer services, I’ve seen “lapsed” cover periods from 7 to 30 days, with “churned” being defined as a customer not using the service for more than 30 days. Companies often experiment", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 231 + }, + { + "text": "seen them for a while but still have a decent likelihood of returning based on our past experience. In consumer services, I’ve seen “lapsed” cover periods from 7 to 30 days, with “churned” being defined as a customer not using the service for more than 30 days. Companies often experiment with reactivating lapsed users, using tactics rang‐ ing from email to support team outreach. Customers in each state can be defined by first finding their “time since last” as above and then tagging them with a CASE state‐ ment using the appropriate number of days or months. For example, we can group the representatives according to how long ago they were elected: SELECT case when months_since_last <= 23 then 'Current' when months_since_last <= 48 then 'Lapsed' else 'Churned' end as status ,sum(reps) as total_reps FROM ( SELECT date_part('year',interval_since_last) * 12 + date_part('month',interval_since_last) as months_since_last ,count(*) as reps FROM ( SELECT id_bioguide ,max(term_start) as max_date ,age('2020-05-19',max(term_start)) as interval_since_last FROM legislators_terms WHERE term_type = 'rep' GROUP BY 1 ) a GROUP BY 1 322 | Chapter 9: Conclusion ) a GROUP BY 1 ; status total_reps ------- ---------- Churned 10685 Current 446 Lapsed 105 This data set contains more than two hundred years of legislator terms, so of course many of the people included have died, and some are still living but are retired. In the context of a business, we would hope that our churned customers didn’t outnumber our current customers by such a wide margin, and we would want to know more about the lapsed customers. Most organizations are very concerned about churn, since customers are generally more expensive to acquire than to retain. To learn more about the customers in any status or about the range of time since last seen, these analyses can be further sliced by any of the customer attributes available in the data set. Basket Analysis I have three kids, and when I go to the grocery store, my basket (or more often my shopping cart) fills up quickly with grocery items to feed them for the week. Milk, eggs, and bread are usually in there, but other items might change depending on what produce is in season, whether the kids are in school or on break, or if we’re planning to cook a special meal. Basket analysis takes its name from the practice of analyzing the products consumers buy together to find patterns that can be used for marketing, store placement, or other strategic decisions. The goal of a basket analysis may be to find groups of items purchased together. It can also be framed around a particular product: when someone buys ice cream, what else do they buy? Although basket analysis was originally framed around items purchased together in a single transaction, the concept can be extended in several ways. A retailer or an ecommerce store might be interested in the basket of items a customer purchases across their lifetime. Services and product feature usage can also be analyzed in this fashion. Services", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 232 + }, + { + "text": "framed around items purchased together in a single transaction, the concept can be extended in several ways. A retailer or an ecommerce store might be interested in the basket of items a customer purchases across their lifetime. Services and product feature usage can also be analyzed in this fashion. Services that are commonly purchased together might be bundled into a new offering, such as when travel sites offer deals if a flight, hotel, and rental car are booked together. Product features that are used together might be placed in the same navigation window or used to make suggestions for where to go next in an applica‐ tion. Basket analysis can also be used to identify stakeholder personas, or segments, which are then used in other types of analysis. To find the most common baskets, using all items in a basket, we can use the string_agg function (or an analogous one, depending on the type of database—see Basket Analysis | 323 Chapter 5). For example, imagine we have a purchases table that has one row for each product bought by a customer_id. First, use the string_agg function to find the list of products purchased by each customer in a subquery. Then GROUP BY this list and count the number of customers: SELECT products ,count(customer_id) as customers FROM ( SELECT customer_id ,string_agg(product,', ') as products FROM purchases GROUP BY 1 ) a GROUP BY 1 ORDER BY 2 desc ; This technique works well when there is a relatively small number of possible items. Another option is to find pairs of products purchased together. To do this, self-JOIN the purchases table to itself, JOINing on the customer_id. The second JOIN condi‐ tion solves the problem of duplicate entries that differ only in their order. For exam‐ ple, imagine a customer who purchased apples and bananas—without this clause, the result set would include “apples, bananas” and “bananas, apples.” The clause b.prod uct > a.product ensures only one of these variations is included and also filters out results in which a product is matched with itself: SELECT product1, product2 ,count(customer_id) as customers FROM ( SELECT a.customer_id ,a.product as product1 ,b.product as product2 FROM purchases a JOIN purchases b on a.customer_id = b.customer_id and b.product > a.product ) a GROUP BY 1,2 ORDER BY 3 desc ; This can be extended to include three or more products by adding additional JOINs. To include baskets that contain only one item, change the JOIN to a LEFT JOIN. There are a few common challenges when running a basket analysis. The first is per‐ formance, particularly when there is a large catalog of products, services, or features. The resultant calculations can become slow on the database, particularly when the goal is to find groups of three or more items, and thus the SQL contains three or more self-JOINs. Consider filtering the tables with WHERE clauses to remove 324 | Chapter 9: Conclusion infrequently purchased items before performing the JOINs. Another challenge occurs when a few items are so", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 233 + }, + { + "text": "the goal is to find groups of three or more items, and thus the SQL contains three or more self-JOINs. Consider filtering the tables with WHERE clauses to remove 324 | Chapter 9: Conclusion infrequently purchased items before performing the JOINs. Another challenge occurs when a few items are so common that they swamp all other combinations. For exam‐ ple, milk is so frequently purchased that groups with it and any other item top the list of combinations. The query results, while accurate, may still be meaningless in a practical sense. In this case, consider removing the most common items entirely, again with a WHERE clause, before performing the JOINs. This should have the added benefit of improving query performance by making the data set smaller. A final challenge with basket analysis is the self-fulfilling prophecy. Items that show up together in a basket analysis may then be marketed together, increasing the fre‐ quency with which they are purchased together. This may strengthen the case to mar‐ ket them together further, leading to more copurchasing, and so on. Products that are even better matches may never have a chance, simply because they didn’t appear in the original analysis and become candidates for promotion. The famous beer and dia‐ pers correlation is only one example of this. Various machine learning techniques and large online companies have tried to tackle this problem, and there are plenty of interesting directions for analysis in this area still to be developed. Resources Data analysis as a profession (or even as a hobby!) requires a mix of technical profi‐ ciency, domain knowledge, curiosity, and communication skills. I thought I would share some of my favorite resources so that you might draw on them as you continue your journey, both to learn more and to practice your new skills on real data sets. Books and Blogs Although this book assumes a working knowledge of SQL, good resources for the basics or for a refresher are: • Forta, Ben. Sams Teach Yourself SQL in 10 Minutes a Day. 5th ed. Hoboken, NJ: Sams, 2020. • The software company Mode offers a SQL tutorial with an interactive query interface, useful for practicing your skills. There is no single universally accepted SQL style, but you may find the SQL Style Guide and the Modern SQL Style Guide useful. Note that their styles don’t exactly match those used in this book, or each other. I believe that using a style that is both consistent with itself and readable is the most important consideration. Resources | 325 Your approach to analysis and to communicating the results can often matter just as much as the code you write. Two good books for sharpening both aspects are: • Hubbard, Douglas W. How to Measure Anything: Finding the Value of “Intangi‐ bles” in Business. 2nd ed. Hoboken, NJ: Wiley, 2010. • Kahneman, Daniel. Thinking, Fast and Slow. New York: Farrar, Straus and Giroux, 2011. The Towards Data Science blog is a great source for articles", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 234 + }, + { + "text": "both aspects are: • Hubbard, Douglas W. How to Measure Anything: Finding the Value of “Intangi‐ bles” in Business. 2nd ed. Hoboken, NJ: Wiley, 2010. • Kahneman, Daniel. Thinking, Fast and Slow. New York: Farrar, Straus and Giroux, 2011. The Towards Data Science blog is a great source for articles about many analysis top‐ ics. Although many of the posts there focus on Python as a programming language, approaches and techniques can often be adapted to SQL. For an amusing take on correlation versus causation, see Tyler Vigen’s Spurious Correlations. Regular expressions can be tricky. If you’re looking to increase your understanding or to solve complex cases not covered in this book, a good resource is: • Forta, Ben. Learning Regular Expressions. Boston: Addison-Wesley, 2018. Randomized testing has a long history and touches many fields across the natural and social sciences. Compared to statistics, however, analysis of online experiments is still relatively new. Many classic statistics texts give a good introduction but discuss prob‐ lems in which the sample size is very small, so they fail to address many of the unique opportunities and challenges of online testing. A couple of good books that discuss online experiments are: • Georgiev, Georgi Z. Statistical Methods in Online A/B Testing. Sofia, Bulgaria: self- published, 2019. • Kohavi, Ron, Diane Tang, and Ya Xu. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge, UK: Cambridge University Press, 2020. Evan Miller’s Awesome A/B Tools has calculators for both binary and continuous outcome experiments, as well as several other tests that may be useful for experiment designs beyond the scope of this book. Data Sets The best way to learn and improve your SQL skills is to put them to use on real data. If you are employed and have access to a database within your organization, that’s a good place to start since you probably already have context on how the data is pro‐ duced and what it means. There are plenty of interesting public data sets that you can 326 | Chapter 9: Conclusion analyze instead, however, and these range across a wide variety of topics. Listed below are a few good places to start when looking for interesting data sets: • Data Is Plural is a newsletter of new and interesting data sets, and the Data Is Plu‐ ral archive is a searchable treasure trove of data sets. • FiveThirtyEight is a journalism site that covers politics, sports, and science through a data lens. The data sets behind the stories are on the FiveThirtyEight GitHub site. • Gapminder is a Swedish foundation that publishes yearly data for many human and economic development indicators, including many sourced from the World Bank. • The United Nations publishes a number of statistics. The UN’s Department of Economic and Social Affairs produces data on population dynamics in a rela‐ tively easy-to-use format. • Kaggle hosts data analysis competitions and has a library of data sets that can be downloaded and analyzed even outside of", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 235 + }, + { + "text": "• The United Nations publishes a number of statistics. The UN’s Department of Economic and Social Affairs produces data on population dynamics in a rela‐ tively easy-to-use format. • Kaggle hosts data analysis competitions and has a library of data sets that can be downloaded and analyzed even outside of the formal competitions. • Many governments at all levels, from national to local, have adopted the open data movement and publish various statistics. Data.gov maintains a list of sites both in the United States and around the world that is a good starting point. Final Thoughts I hope you’ve found the techniques and code in this book useful. I believe that it’s important to have a good foundation in the tools you’re using, and there are many useful SQL functions and expressions that can make your analyses faster and more accurate. Developing great analysis skills isn’t just about learning the latest fancy tech‐ niques or language, however. Great analysis comes from asking good questions; tak‐ ing the time to understand the data and the domain; applying appropriate analysis techniques to come up with high-quality, reliable answers; and finally, communicat‐ ing the results to your audience in a way that is relevant and supports decision mak‐ ing. Even after almost 20 years of working with SQL, I still get excited about finding new ways to apply it, new data sets to apply it to, and all of the insights in the world patiently waiting to be discovered. Final Thoughts | 327 Index Symbols ! (exclamation point) negation operator in reg‐ ular expressions, 204 % (percent sign) modulo operator, 309 wildcard matching with LIKE, 195 () (parentheses) enclosing expressions in regular expres‐ sions, 209 in SQL code, 293 using to control order of operations, 197 * (asterisk) matching multiple characters in regular expressions, 205 matching zero or more times in regular expressions, 207 + (plus sign) addition operator, 70 concatenation operator, 68 matching one or more times in regular expressions, 207 - (dash), indicating range of characters in regu‐ lar expressions, 206 -- (double dash) single line comments in SQL, 292 . (period), matching any single character in reg‐ ular expressions, 204 /* */ comment for multiple lines of code, 292 :: (double colon) operator, 190 ? (question mark), matching zero or more times in regular expressions, 207 [] (brackets), enclosing character class in regu‐ lar expressions, 205 \\ (backslash) escape character in pattern matching, 195 escape character in regular expressions, 208 \\A, matching beginning of string in regular expressions, 211 \\n line feed or newline, 209 \\r carriage return, 209 \\s space in regular expressions, 209 \\y at beginning and end of patterns in regular expressions, 210 \\Z, matching end of string in regular expres‐ sions, 211 _ (underscore), wildcard matching character, 195 {} (curly braces), matching a character set in regular expressions, 207 || (concatenation) operator, 44, 219 ~ (tilde) alternatives to, functions, 212 comparator in regular expressions, 204 ~* making comparator case insensitive, 204 ʌ (caret), negating pattern matches", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 236 + }, + { + "text": "string in regular expres‐ sions, 211 _ (underscore), wildcard matching character, 195 {} (curly braces), matching a character set in regular expressions, 207 || (concatenation) operator, 44, 219 ~ (tilde) alternatives to, functions, 212 comparator in regular expressions, 204 ~* making comparator case insensitive, 204 ʌ (caret), negating pattern matches in regular expressions, 208 – (minus sign) subtraction operator, 69 A A/A tests, 278 A/B testing, 122, 267 absence of data, anomalies from, 258-260 active user calculations, 96 age function, 128, 321 aggregate functions aggregation returning one row per entity, 39 in cohort analysis, 123 followed by frequency count, 30 329 HAVING clause filtering aggregation value, 37 pivoting data with, 114 in retention analysis, 127 in time series calculations, 95 use in creating histograms, 31 use in OVER clause, 299 use with CASE to pivot or unpivot, 54 window functions, 34 aggregate metric, 123 aggregations, 25 in GROUP BY clause, order of evaluation, 298 in indexing of time series data, 91 performing on records containing string pattern, 199 PII and, 315 aliases, 25 ALTER commands, 6 Amazon Redshift, 15 AND operator, 196 using with OR, controlling order of opera‐ tions, 197 AND/OR logic in CASE statements, 40 anomaly detection, 227-266 applying to experiment analysis, 278 capabilities and limits of SQL for, 228 detecting outliers, 230-249 calculating percentiles and standard deviations, 234-241 graphing to find anomalies, 241-249 sorting to find anomalies, 231-234 forms of anomalies, 250-260 absence of data, 258-260 anomalous counts or frequencies, 254-258 anomalous values, 250 handling anomalies, 260-266 investing cause of, 260 removing anomalies, 260 replacement with alternate values, 262 rescaling, 264-266 sources of anomalies, 227 USGS earthquakes data set, 229 anomaly, defined, 227 approximate_percentile function, 238 ASC (ascending) or DESC (descending) sort order, 236 at time zone, 63 Atom text editor, 296 avg function, 31, 103, 118, 261 B bar graphs, 241 basket analysis, 323-325 BETWEEN clause, 73 BI (business intelligence) tools, 52 allowing only one SQL statement to create a data set, 302 capabilities of, 291 SQL queries embedded into, 288 BIGINT type, 20 binary outcome events, 284 binary outcome experiments, 272-274 changing continuous success metric to binary outcome, 278 bins (or buckets), 31-33 arbitrary- and fixed-size bins, 32 using n-tiles for, 33-35 BLOB type, 20 BOOLEAN type, 21 LIKE and ILIKE giving TRUE/FALSE results, 198 TRUE/FALSE values for flags, 312 using to create flags, 42 box plots, 245 C California Consumer Privacy Act (CCPA), 3 capitalization case insensitivity in SQL code, 292 case sensitivity in pattern matching, 196 differences in capitalization in data, 252 in SQL code, 293 cardinality, 252 Cartesian JOINs, 25, 93 calculating rolling time windows with sparse or missing data, 104 forcing when subqueries don't have fields in common, 148 self-JOIN leveraging, 106 CASE statements, 25 cleaning data with CASE transformations, 39-42 combined with aggregate functions, pivot‐ ing with, 81 combined with IN list, 253 creating bins with, 31 330 | Index pivoting data with, 53-55 replacing nulls with alternative values, 46 setting nonconforming records to null, 191 transforming numeric data into", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 237 + }, + { + "text": "106 CASE statements, 25 cleaning data with CASE transformations, 39-42 combined with aggregate functions, pivot‐ ing with, 81 combined with IN list, 253 creating bins with, 31 330 | Index pivoting data with, 53-55 replacing nulls with alternative values, 46 setting nonconforming records to null, 191 transforming numeric data into several lev‐ els, 313 using LIKE operator, 197 using to avoid division by zero, 93 using to exclude extreme values, 261 using to return current month's value without date dimension, 103 using to substitute a default value, 262, 263 variants assigned via, 283 cast function changing data type with, 43 field as type syntax, 190 casting, 190 date or timestamp values to date format, 68 causality, 267 nonexperimental analyses and, 286 causation versus correlation, 269, 326 CCPA (California Consumer Privacy Act), 3 CHAR type, 20 chi-squared test, 272-274 churn, 319-323 clauses capitalization of main clauses, 294 order of evaluation, 296-300 cleaning data, 39-52 alternatives to CASE statement, 41 cleaning text fields, 177 dealing with nulls, 45-47 missing data, 47-52 using CASE transformations, 39-42 using casting and type conversions, 42-45 CLOB type, 21 CLTV (customer lifetime value), 124, 163 coalesce function, 46, 133, 222, 262, 275 code organization, 292-296 commenting code, 292 formatting tricks for, 293 storing code, 296 cohort analysis, 121-174, 317 about, 122 cohorts, 122-124 cohort grouping, 122 segments versus, 123 cross-section analysis, through cohort lens, 166-174 cumulative calculations, 163-166 ensuring that absences in data are noticed, 258 performing on variants in experiment anal‐ ysis, 281 retention, 127-153 returnship or repeat purchase behavior, 158-163 survivorship, 154-158 time boxing, 279 cohort grouping, 122 cohorted variant system, 273 too many/too few cohorts, 277 cohorting system, 268 cohorts defining from a separate table, 142-146 defining from dates other than first date, 151-153 deriving from time series itself, 137-142 column-store databases, 15 comments, 292 rules of thumb for commenting, 292 common table expressions (see CTEs) comparators in regular expressions, 204 composite key, 14 computations, organizing, 296-308 order of SQL clause evaluation, 296-300 subqueries, 300-302 temporary tables, 302 using CTEs, 303-305 concat function, 219, 221 concatenation assembling date or timestamp from parts, 68 concatenating strings, 44 creating new text with, 218-222 concat_ws function, 219 confidence levels, 274, 276 contingency tables, 272 example for onboarding completions, 273 continuous metrics, 274 outliers in, 278 continuous outcome events, 284 control group, 268, 277, 284 convert_timezone or convert_tz function, 64 Coordinated Universal Time (see UTC) correlations, 267 causation versus, 269, 326 costs of experiments, 276 count function Index | 331 avoiding nulls in results, 55 count distinct, 29 in frequency queries, 28 using other aggregate functions instead of, 31 count of observations, 274 counts or frequencies (anomalous), 254-258 CREATE statements, 6 creating temporary table, 302 cross-section analysis, through cohort lens, 166-174 CTEs (common table expressions), 303-305 cube root (cbrt) function, 266 cube syntax (in GROUP BY), 305, 307 cumulative calculations, 124, 163-166 calculating cumulative values in rolling time windows, 104-107 cumulative lifetime value, 163 (see also CLTV) current date or time, getting, 64 CURRENT ROW, 100 current_timestamp function,", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 238 + }, + { + "text": "cohort lens, 166-174 CTEs (common table expressions), 303-305 cube root (cbrt) function, 266 cube syntax (in GROUP BY), 305, 307 cumulative calculations, 124, 163-166 calculating cumulative values in rolling time windows, 104-107 cumulative lifetime value, 163 (see also CLTV) current date or time, getting, 64 CURRENT ROW, 100 current_timestamp function, 259 customer lifetime value (see CLTV) cyclicality, day of week, 107 D daily snapshot tables, 289 data analysis, 17 about, 1-3 benefits of SQL for, 7 database types and, 12 industries engaging in, 2 reasons for not doing, 3 resources for further learning, 325-327 SQL in workflow, 9-12 SQL versus other tools, 291 data binning (see bins) Data Control Language (DCL), 6 Data Definition Language (DDL), 5 data dictionaries, 19 data infrastructure, 13 taking advantage of increases in computing power, 7 types other than databases, 16 data lakes, 10, 16 data marts, 10 (see also data warehouses) data preparation (see preparing data) data privacy, 314-315 data quality, 35-39 deduplication with GROUP BY and DIS‐ TINCT, 38 detecting duplicates, 36-38 Data Query Language (DQL), 5 data sets, complex, creating for analysis, 287 managing data set size and privacy con‐ cerns, 308-315 PII and data privacy, 314-315 reducing dimensionality, 310-314 sampling with %, mod, 308-310 organizing computations, 296-308 putting logic in other tools, 290 using an ETL instead of SQL, 288 when to use SQL for, 287 data sets, resources for, 326 data stores, 10 NoSQL, 17 data types, 20-25 casting and type conversions, using in data cleaning, 42-45 converting, 190 database types, 20-21 first-, second-, and third-party data, 23 quantitative versus qualitative data, 22 sparse data, 24 specifying for columns in temporary table, 303 structured versus unstructured data, 22 data warehouses, 10 databases, 12-16 automatic conversions of data types with type coercion, 44 column-store, 15 data types, 20 interacting with, SQL as de facto standard, 7 listing GROUP BY fields, 26 order of operations, 296 overview of organization and database objects, 5 performance benefits of, 290 queries on, controlling size with LIMIT and sampling, 26 query optimizers, 297 regular expression implementations, 212 row-store, 13 SQL harnessing power of, 176 text in, 175 time zone information system tables, 64 332 | Index using temporary tables with, 302 views and materialized views, 290 DataFrames (Python), 8 date dimension, 102 date function, converting dates to/from strings, 44 DATE type casting to a TIMESTAMP, 44 datediff function, 69 replacing age and date_part in retention analysis, 129 dates and times, 62-73 casting fields into timestamps and dates, 190 converting in/out of date or datetime for‐ mats with to_datatype functions, 44 date and datetime type conversions, 43 date dimension table, 50 date, datetime, and time manipulations date and timestamp format conversions, 64-68 date math, 68-71 joining data from different sources, 72-73 time math, 71-72 time zone conversions, 62 dates entered as strings, 284 granularity of, adjusting to reduce data size, 311 time-based churn metric, 319 datetime types, 20, 21 date_add or dateadd function, 71 date_format function, 65 date_from_parts or datefromparts function, 68 date_part function, 66, 70, 114,", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 239 + }, + { + "text": "from different sources, 72-73 time math, 71-72 time zone conversions, 62 dates entered as strings, 284 granularity of, adjusting to reduce data size, 311 time-based churn metric, 319 datetime types, 20, 21 date_add or dateadd function, 71 date_format function, 65 date_from_parts or datefromparts function, 68 date_part function, 66, 70, 114, 320 in PARTITION BY clause, 112, 118 in retention analysis, 128 date_trunc function, 65, 254 DAU (daily active users), 96 day of week cyclicality, 107 day-time intervals, 69 dbplyr (R), 9 DBVisualizer editor, 295 DCL (Data Control Language), 6 DDL (Data Definition Language), 5 decimals DECIMAL type, 20 rounding to various decimal places, 32 DELETE commands, 6 deletes, 6 in column-store databases, 15 delimiters, 223 denormalization, 14 dialects of SQL, 6 difference, ratio, and percent difference between time series in data set, 82 dimensionality, reducing, 310-314 DISTINCT keyword order of evaluation, 299 using in removing duplicates, 38 using to select dates needed, 104 distributions of data, 27-35 binning, 31-33 histograms and frequencies, 28 n-tiles, 33-35 division by zero, 93 double colon (::) operator, 190 DOUBLE type, 20 DQL (Data Query Language), 5 DROP statements, 6 deleting a table, 303 duplicates deduplication with GROUP BY and DIS‐ TINCT, 38 detecting, 36-38 E earthquakes data set (USGS), 229 EDA (exploratory data analysis), 28 editors (SQL), 296 ELSE statements, 31, 55 ELT (extract, load, transform), 11 empty strings, 46 nulls versus, 191 escape character, backslash (\\) in regular expressions, 208 ethical considerations with data, 3 ETL (extract, transform, load), 6, 11 drawbacks to, 289 using instead of SQL for complex data sets, 288 views as alternative to, 290 experiment analysis, 267-286 alternatives to controlled experiments, 282 analysis of populations around a thres‐ hold, 286 natural experiment analysis, 284 pre-/post-analysis, 282 Index | 333 challenges with experiments and rescues for flawed experiments, 276-282 outliers, 278 repeated exposure experiments, 280 time boxing, 279 variant assignment, 277-278 elements of experiments hypotheses, 267 random assignments to control or var‐ iant group, 268 success metrics, 267 SQL's strengths and limits in, 269 types of experiments, 272-276 experiments with binary outcomes, 272-274 experiments with continuous outcomes, 274-276 why correlation is not causation, 269 exploratory data analysis (EDA), 28 extract function, 66, 254, 259, 320 F field name notation, GROUP BY fields, 26 file storage systems, 16 HDFS, 16 fill forward and fill backward, 49 filling in missing data, 48 first-, second-, and third-party data, 23 first_value window function, 129, 140 flagging data, using CASE statements, 41 flags, 312 summarizing presence or absence of prop‐ erties, 21 FLOAT type, 20 forecasting, 61 frame clauses, 99 example specification, 99 and rows they include, 101, 118 frame_end, 100 frame_exclusion option, 100 frame_start, 100 frequencies, 28 anomalous, 254-258 frequency plots, 29 FROM clause, 25 order of evaluation, 297 FULL OUTER JOIN, 25 UNION ALL as alternative to, 57 functions, aggregate and special, in window functions, 34 funnel analysis, 317-319 future time periods, calculating moving aver‐ ages for, 101 G gap analysis, 319-322 General Data Protection Regulation (GDPR), 3 generate_series function, 50 returning desired", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 240 + }, + { + "text": "25 order of evaluation, 297 FULL OUTER JOIN, 25 UNION ALL as alternative to, 57 functions, aggregate and special, in window functions, 34 funnel analysis, 317-319 future time periods, calculating moving aver‐ ages for, 101 G gap analysis, 319-322 General Data Protection Regulation (GDPR), 3 generate_series function, 50 returning desired dates, 133 geographical types, 21 GPS location information, 314 GRANT commands, 6 graphs of retention curves, using to compare cohorts, 127 of time series data, 75 tools for, 241 using to find anomalies, 241-249 bar graph, 241 box plot, 245 scatter plot, 243 Greenwich Mean Time (GMT), 63 ground truth, comparing data against, 36 GROUP BY clause, 26, 87 creating histogram by age, 29 in frequency queries, 28 grouping sets, 306 order of evaluation, 298 special syntax, grouping sets, cube, and rollup, 305 using in deduplication, 38 using to find anomalies in subsets of data, 233 using with string_agg, 223 grouping sets, 305-308 groups (frame type), 100 guardrail metrics, 268 H Hadoop distributed filesystem (HDFS), 16 hashing values, 315 HAVING clause alternative to subquery, 37 order of evaluation, 298 HDFS (Hadoop distributed filesystem), 16 histograms, 29, 241 hypotheses, 267 334 | Index I identifiers, 14, 30 sampling on, 309-310 ILIKE operator, 196 imputation techniques, 48 IN operator, 40, 200-203 CASE statement combined with, 253 NOT IN, 200 indentation in code, 294 indexes for database tables, 14 revealing percent of change over time, 90-94 initcap function, 187 INNER JOIN, 25, 275, 277 INSERT commands, 6 integers converting to strings, 43 INT, SMALLINT, and BIGINT types, 20 interquartile range, 245 intervals addition with dates, 70 in date math, 69 finding number of months component, 70 multiplying, 72 requested date and time parts, 67 subtracting from dates, 70 subtracting from times, 72 subtracting times to result in, 72 investigation into cause of anomalies, 260 IoT (Internet of Things), 12 ISO (International Organization for Stand‐ ards) , 4 J JavaScript, 53 JOINs in basket analysis, 324 Cartesian JOIN in standard deviation calcu‐ lation, 240 combining date and time data from differ‐ ent sources, 72-73 date math in JOIN conditions, 71 to date series or data dimension, 258 duplicates created by hidden many-to-many JOIN, 36 JOIN conditions and JOIN types, 25 LEFT JOIN, 224 LIKE operator in JOIN...ON clauses, 197 ON clause, 297 restricting entities included for experiment, 277 sales months table joined to date dimension, 102 self-JOIN, 86, 88 leveraging a Cartesian JOIN, 106 unusual JOIN clause in indexing time series data, 93 using on date dimensions, 51 values derived through text transformations as criteria, 198 JSON databases' support of, 21 using to deal with sparse data, 24 K key-value stores, 17 keywords, 292 coloration in SQL query editor, 295 L lag function, 116, 319 using in YoY and MoM comparisons, 109-111 using to fill in missing data, 49 using with partitioning to compare same month versus last year, 112-115 lapse, 322 lateral subqueries, 300 laws regulating data privacy, 314 lead function, 109, 259 calculating term_end data, 136 using to fill in missing data,", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 241 + }, + { + "text": "319 using in YoY and MoM comparisons, 109-111 using to fill in missing data, 49 using with partitioning to compare same month versus last year, 112-115 lapse, 322 lateral subqueries, 300 laws regulating data privacy, 314 lead function, 109, 259 calculating term_end data, 136 using to fill in missing data, 49 left function, 182-183 LEFT JOIN, 25, 273, 275 in funnel analysis, 318 legislators data set, 125-126 reducing dimensionality of, 311-314 length function, 179 LIKE operator, 195-200 NOT LIKE, 196 using in CASE statement, 197 using in SELECT clause, 197 LIMIT clause, 26 order of evaluation, 300 listagg function, 223 location information, 314 logarithms conversions to log scale, 264 Index | 335 log function, 33, 264 using to create bins, 33 long-term holdout, 281 lower function, 187 LTM (last twelve months), 95 LTV (lifetime value), 124, 163 M machine learning (ML), 177, 178 feeding SQL into ML algorithms, 12 outputting data for ML models, 53 Python and R, benefits for, 291 magnitude, 230 make_date or makedate function, 68, 133 manual steps, avoiding, 291 Mash Until No Good (Mung), 19 materialized views, 290 Matplotlib, 241 MAU (monthly active users), 96 max function, 31, 35, 114, 259 md5 function, 315 mean, 274, 284 median function, 34, 234, 238 min function, 31, 35 turning into window function, 140 missing data, 47-52 calculating rolling time windows with, 104 filling in for time series to increase retention accuracy, 131-137 misspellings, detecting, 253 mix shifts, 166 ML (see machine learning) mobile devices, 12 mod (modulus) function, 27, 309 modulo operator (%), 309 MoM (month-over-month) comparisons, 109-111 moving calculations, 95 (see also rolling time windows) moving average, 101 in time series calculations, 95 MSSQL, 6 multiple comparisons problem, 268 N n-tiles, 33-35, 235-238 natural experiment analysis, 284 natural language processing (NLP), 176 machine learning approaches best for, 178 newlines, 292 matching in regular expressions, 209 NLP (natural language processing), 176 nonstandard values, standardizing, 39 normalization, 14 NoSQL, 17 NOT IN operator, 200 NOT LIKE operator, 196 NOT operator, 196 novelty effects, 281 nullif function, 47 nulls, 275 avoiding in sum aggregation results using else 0, 54 dealing with in data cleaning, 45-47 empty strings versus, 191 replacing with coalesce or CASE statements, 219 substituting a value for, 307 WHERE clause filter to remove, 82 numeric types, 20 nvl function, 46 O OFFSET clause, 300 offset FOLLOWING, 100 offset PRECEDING, 100 offsets, 100, 104, 109 in lag function, 109 (see also lag) no offset provided to lag, 116 UTC, 63 ON clause, 297 “one-and-done experiences”, 280 online experiments, 268, 326 opportunity cost, 276 OR operator, 47, 201 using with AND, controlling order of opera‐ tions, 197 Oracle SQL, 6 ORDER BY clause, 91 changing from ascending to descending sort order, 109 controlling sorting in percent_rank, 234 in lag function, 109 not required in sum window function, 87 order of evaluation, 299 in percentile calculations, 236 sorting data to find anomalies, 231-234 336 | Index using GROUP BY with, 233 in string_agg function, 222 in window functions, 34 order", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 242 + }, + { + "text": "sort order, 109 controlling sorting in percent_rank, 234 in lag function, 109 not required in sum window function, 87 order of evaluation, 299 in percentile calculations, 236 sorting data to find anomalies, 231-234 336 | Index using GROUP BY with, 233 in string_agg function, 222 in window functions, 34 order of evaluation controlling with subqueries, 300-302 SQL clauses in queries, 296-300 OUTER JOIN, 253 outliers, 227 (see also anomaly detection) in continuous outcome experiments, 278 OVER clause, 234 aggregate functions in, 299 OVER...PARTITION BY... syntax, 237 in window functions, 34 overstuffed fields, 181 parsing into several new fields, 181 P pandas package (Python), 8 parsing data with SQL, 182-186 PARTITION BY clause, 87, 91 date_part function in, 112, 118 first_value function returning first record in, 129 in lag function, 109 percent_rank function and, 234 in window functions, 34 partitioning or grouping of data in window, 96 parts of dates or times, 66 creating date from parts from different sources, 67 full list of date parts, 67 pattern matching, 195-203 (see also regular expressions) exact matches with IN and NOT IN opera‐ tors, 200-203 using LIKE, ILIKE and NOT LIKE, 195-200 percentages indexing to show percent of change over time, 90-94 percent of total, calculating for time series data, 86 in retention analysis, 127 finding percent retained, 129 percentiles, 245 calculating, 234-238 outliers set to specific percentile of the data, 263 winsorizing and, 264 percentile_cont function, 236 median function and, 238 percentile_disc function, 236 percent_rank function, 35, 234 performance database performance issues with JOIN using text parsing as criteria, 198 factors for SQL queries, 288 percentile and median functions, 238 regex_split_to_table and similar functions, 225 period-over-period comparisons for multiple prior periods, 116-119 same month versus last year, 112-115 YoY and MoM, 109-111 PII (personally identifiable information), 314-315 pivoting and unpivoting data for graphing with SQL, 241 pivot and unpivot functions, 57-59 pivoting data with aggregate functions, 114, 115 pivoting using CASE statements with aggre‐ gate functions, 81 pivoting with CASE statements, 53-55 unpivoting with UNION statements, 55-57 population sizes, 284 populations, analysis around a threshold, 286 position notation, GROUP BY fields, 26 POSIX syntax, regular expressions comparators, 204 databases and, 212 PostgreSQL (or Postgres), 6 dividing integers, 21 generate_series function in Postgres, 50 unnest function in Postgres, 58 pre-/post-analysis, 282 preparing data, 19-59 cleaning data, 39-52 profiling distributions, 27-35 shaping data, 52-59 primary keys, 14 privacy concerns with data, 3, 314-315 profiling creation of trend as step in, 75 data quality, 35-39 distributions, 27-35 Python, 53, 229, 291 SQL or R versus, 8-9 Index | 337 SQL queries embedded in, 288 Q qualitative analysis, 176 quantitative analysis, 176 quantitative versus qualitative data, 22 queries (SQL) limiting size of results with LIMIT and sam‐ pling, 26 structure of, 25-27 query optimizers, 297 R R language, 53, 229, 291 SQL or Python versus, 8-9 SQL queries embedded in, 288 range (frame type), 100 range, rows, and groups (frame type), 100 ranges in regular expressions, 206 ratios, 84 reciprocal transformation, 266 regexp_like function, 212", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 243 + }, + { + "text": "and sam‐ pling, 26 structure of, 25-27 query optimizers, 297 R R language, 53, 229, 291 SQL or Python versus, 8-9 SQL queries embedded in, 288 range (frame type), 100 range, rows, and groups (frame type), 100 ranges in regular expressions, 206 ratios, 84 reciprocal transformation, 266 regexp_like function, 212 regexp_matches function, 213, 215 regexp_replace function, 215, 217-218 regexp_split_to_table function, 223 regexp_substr function, 213 regression discontinuity design (RDD), 286 regression to the mean, 281 regular expressions, 203-218 finding and replacing with, 212-218 matching character set multiple times, sym‐ bols for, 207 matching whitespace characters, 209 range patterns, 206 using parentheses to enclose patterns, 209 relational databases, 4 removal of anomalies, 260 repeat purchase behavior, 158 (see also returnship) repeated exposure experiments, 280 replace function, 192, 253 using with regexp_replace, 218 replacement of anomalous values, 262 replacing text in strings, 216 repository, storing ETL code in, 289 rescaling values, 264-266 reserved words, 292 reshaping text, 222-225 Retail Sales data set, 74 retention, 124, 127-153 adjusting time series to increase accuracy of, 131 analyzing, main question in, 127 basic retention curve, SQL for, 128-131 cohorts derived from time series itself, 137-142 dealing with sparse cohorts, 146-150 defining cohort from separate table, 142-146 defining cohorts from dates other than first date, 151-153 as success metric, 267 returnship, 124 analysis of, 158-163 REVOKE commands, 6 right function, 27, 309 RIGHT JOIN, 25 rlike function, 212 rolling time windows, 95-107 calculating, 97-102 calculating cumulative values, 104-107 important pieces in calculations, 95 with sparse data, 102-104 rollup function, 305, 308 round function, 32, 264 rounding, 32 row-store databases, 13 rows (frame type), 100 row_number function, 315 rule-based systems, 177 S SaaS (see software as a service) sampling, 27, 308-310 scatter plots, 243 seasonality analyzing data with, 107-119 comparing to multiple prior periods, 116-119 period-over-period comparisons, same month versus last year, 112-115 period-over-period comparisons, YoY and MoM, 109-111 time scales, 107 second-party data, 23 segments versus cohorts, 123 SELECT clause, 5, 25 date math in, 71 LIKE operator in, 197 338 | Index order of evaluation, 299 SELECT-only queries, 297 self-JOIN, 86 in indexing of time series data, 91 moving average calculation using, 101 series of, using to index without window functions, 92 window functions versus in calculating moving aggregations, 101 semistructured data, 22, 175, 181 sentiment analysis, 178 separators, 219 shaping data, 52-59, 222 for different outputs, 52 pivot and unpivot functions, 57-59 pivoting with CASE statements, 53-55 unpivoting with UNION statements, 55-57 significant digits, 250 SMALLINT type, 20 Snowflake (database), 15 snowflake schema, 15 software as a service (SaaS) data sourced from vendors, 23, 73 retention curves measuring subscription revenue, 127 sparse data, 24 dealing with sparse cohorts, 146-150 detection with frequency queries, 28 rolling time windows with, 102-104 split testing, 267 split_part function, 183-186, 257 split_to_table function, 223 spreadsheets, 290 SQL, 4-12 benefits of, 7 capabilities and limits for anomaly detec‐ tion, 228 capabilities for text analysis, 176 code organization, 292-296 in data analysis workflow, 9-12 finding key values for box plot with, 247", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 244 + }, + { + "text": "time windows with, 102-104 split testing, 267 split_part function, 183-186, 257 split_to_table function, 223 spreadsheets, 290 SQL, 4-12 benefits of, 7 capabilities and limits for anomaly detec‐ tion, 228 capabilities for text analysis, 176 code organization, 292-296 in data analysis workflow, 9-12 finding key values for box plot with, 247 limitations for text analysis, 177 many benefits for working with data, 17 query editors, query formatting, 295 query structure, 25 versus R or Python, 8-9 resources for further learning, 325 standards, 4 strengths and limits for experiment analysis, 269 strengths in using for complex data sets, 287 sublanguages, 5 variations or dialects, 6 SQL editors, 296 SQLAlchemy (Python), 9 square root (sqrt) function, 266 standard deviations, 238-241, 274, 284 and z-scores for normal distribution, 239 star schema, 14 statistical analysis, 12, 291 statistical significance, SQL unable to calculate, 269 statistics packages, outputting data for, 53 stddev function, 239 stddev_pop function, 239 stddev_samp function, 239 stop words, 224 storing code, 296 string types, 20 strings, 175 (see also text analysis) capitalizing initial character, 187 changing case with upper and lower func‐ tions, 187 converting integers to, 43 dates entered as, databases recognizing, 284 empty, 191 removing blank spaces at beginning and end, 189 removing characters from beginning or end with trim, 189 replacing text in, 192 string_agg function, 222-223 structured data, 175 versus unstructured data, 22 structuring text fields, 177 subqueries, 300-302 HAVING clause versus in duplicate detec‐ tion, 37 multiple in a query, commenting, 293 placing intermediate calculation in, 82 self-JOIN in, 88 subscription-based contexts, start and end dates, 137 success metrics, 267 continuous, 274 importance of agreeing on up front, 276 Index | 339 multiple, for an experiment, 268 in online experiments, 268 survivorship, 124 analysis of, 154-158 survivorship bias, 123, 167 system time, 64 T t-test, 274-276 table aliases, 25 joining, 106 TEMPORARY keyword, 302 temporary tables, 302 term-based contexts, start and end dates, 136 text analysis, 23, 175-226 about, 176 constructing and reshaping text, 218-225 finding elements within larger blocks of text, 195-218 exact matches using IN and NOT IN, 200-203 using regular expressions, 203-218 wildcard matches with LIKE and ILIKE, 195-200 goals or strategies, 176 parsing text, 182-186 text characteristics, 179-181 text transformations, 187-195 UFO sightings data set, 178 use cases for SQL, 176 use cases SQL is not good for, 177 text fields, data size reduction in, 311 TEXT type, 20 THEN condition, 31 third normal form, 14 third-party data, 24 threshold, analysis of populations around, 286 tidy data, 53 time, 43 (see also dates and times) math with, 71-72 standard time period arguments, 65 time boxes, 158, 279, 318 time series analysis, 61-119, 317 about, 61 analyzing with seasonality, 107-119 date, datetime, and time manipulations, 62-73 finding trends in the data, 75-94 Retail Sales data set, 74 rolling time windows, 95-107 showing anomalous counts or frequencies, 254-258 time series data adjusting to increase retention accuracy, 131-137 for cohort analysis basic retention curve, 128-131 in cohort analysis, 122 cohorts derived from, 137-142 TIME type, 21", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 245 + }, + { + "text": "manipulations, 62-73 finding trends in the data, 75-94 Retail Sales data set, 74 rolling time windows, 95-107 showing anomalous counts or frequencies, 254-258 time series data adjusting to increase retention accuracy, 131-137 for cohort analysis basic retention curve, 128-131 in cohort analysis, 122 cohorts derived from, 137-142 TIME type, 21 time zones caution with when combining data from different sources, 73 conversions, 62 UTC offset for local time zones, 63 information system tables in databases, 64 TIMESTAMP type, 21 casting to/from DATE, 44 timestamps, 259 creating from separate date and time com‐ ponents, 68 functions returning only that portion of current system time, 65 interchangeablility with dates, 66 reducing granularity with date_trunc, 65 slightly out of sync in data from different sources, 73 to_char function, 67, 114 to_datatype functions, 44 to_date function, 68 transactional databases, 13 (see also row-store databases) transformations nonobvious, commenting on, 293 rescaling values, 264 scale, other types of, 266 text, 187-195 transforming data into flag values, 312 trending data, 75-94 indexing to see percent change over time, 90-94 percent of total, calculating, 86 simple trends, 75-77 trends analysis of, 174 correlation between cohort characteristics and, 122 340 | Index trim function, 253 removing blank spaces at beginning and end of strings, 189 removing characters from beginning or end of strings, 189 trunc function, 254 truncating dates and times, 65, 254 TTM (trailing twelve months), 95 two-sample t-test, 274 (see also t-test) type coercion, 44 type conversions, 43 (see also data types) U UFO sightings data set, 178 UNBOUNDED FOLLOWING, 100 UNBOUNDED keyword, 100 UNBOUNDED PRECEDING, 100 UNION clauses combining three queries, 305 creating same output as grouping sets, cube, and rollup, 308 UNION and UNION ALL, order of evalua‐ tion, 299 UNION statements unpivoting with, 55-57 UNION versus UNION ALL, 57 uniqueness, enforcing in tables with primary key, 14 Unix epochs, 67 unnest function (Postgres), 58 unpivot function, 58 unstructured data, 175 structured data versus, 22 updates in column-store databases, 15 UPDATE command, 6 upper function, 187 US Congress, legislators data set, 125 US Geological Survey (USGS), earthquakes data set, 229 US retail sales data set, 74 (see also Retail Sales data set) user-defined functions (UDFs), 269 UTC (Coordinated Universal Time), 63 considerations when joining data from dif‐ ferent sources, 73 drawback to, 63 offset for local time zones, 63 V values, anomalous, 250 VARCHAR type, 20, 179 converting integers to, 43 variant assignment, 268, 277-278 in natural experiment analysis, 285 in pre-/post-analysis, 283 variant cohorting system, 268 Vertica (database), 15 views as alternative to ETL, 290 visualizations, 12, 53 (see also graphs) W WAU (weekly active users), 96 WHEN condition, 31 WHERE clause, 26 date math in, 71 filtering data to remove null values, 82 filtering query results for sampling, 309 filtering records with LIKE operator, 195 nulls in, 47 OR and AND operators in, 197 order of evaluation, 298 removing anomalous records, 261 restricting users cohorted for experiment, 277 whitespace matching in regular expressions, 209 splitting text on, 224 in SQL, 292 using in", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 246 + }, + { + "text": "filtering query results for sampling, 309 filtering records with LIKE operator, 195 nulls in, 47 OR and AND operators in, 197 order of evaluation, 298 removing anomalous records, 261 restricting users cohorted for experiment, 277 whitespace matching in regular expressions, 209 splitting text on, 224 in SQL, 292 using in SQL code formatting, 294 wildcard matches using LIKE and ILIKE opera‐ tors, 195-200 window functions, 34, 86, 88 calculating moving average, 101 calculating rolling time windows with advantages over JOINs, 107 in indexing of time series data, 91 n-tile functions, 34 order of evaluation in SQL code, 298 PARTITION BY statement, 96 percentile_cont and percentile_disc, 236 use for moving calculations self-JOINs versus, 101 windows, 95 (see also rolling time windows) Index | 341 size of in rolling time series, 95 winsorization, 263, 278 WITHIN GROUP clause, 236 Y year-month intervals, 69 YoY (year-over-year) comparisons, 109-111 YTD (year-to-date), 95 Z z-scores, 239, 264 zero (0), division by, 93 342 | Index About the Author Cathy Tanimura has a passion for connecting people and organizations to the data they need to make an impact. She has been analyzing data for over 20 years across a wide range of industries, from finance to B2B software to consumer services. She has experience analyzing data with SQL across most of the major proprietary and open source databases. She has built and managed data teams and data infrastructure at a number of leading tech companies. Cathy is also a frequent speaker at top conferen‐ ces, on topics including building data cultures, data-driven product development, and inclusive data analysis. Colophon The animal on the cover of SQL for Data Analysis is a green magpie (Cissa chinensis). Usually referred to as the common green magpie, this jewel-toned bird is a member of the crow family. Found throughout the lowland evergreen and bamboo forests of northeastern India, central Thailand, Malaysia, Sumatra, and northwestern Borneo, this species of bird is noisy and highly social. In the wild, they can be identified by their jade-colored plumage, which contrasts elegantly with their red beak and a black band running along the eyes. They also have a white-tipped tail and reddish wings. Highly social and noisy, the green magpie can be identified by its piercing shrieks fol‐ lowed by a hollow and decisive-sounding “chup” note. They are also often difficult to spot because they glide from tree to tree in the middle-upper levels of the forest. They build their nests in trees, large shrubs, and tangles of various climbing vines. Some‐ times referred to as the hunting cissas, they are primarily carnivorous—consuming a variety of invertebrates, as well as young birds and eggs, small reptiles, and mammals. The green magpie is fascinating because of its ability to change colors. Although they are jade green in the wild, they have been observed to turn distinctly turquoise in captivity. They get their green coloration from a combination of two sources: a special feather structure that produces blue coloring due to the feather refracting light, and", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 247 + }, + { + "text": "of its ability to change colors. Although they are jade green in the wild, they have been observed to turn distinctly turquoise in captivity. They get their green coloration from a combination of two sources: a special feather structure that produces blue coloring due to the feather refracting light, and carotenoids—yellow, orange, and red pigments that come from the bird’s diet. Pro‐ longed exposure to harsh sunlight destroys the carotenoids, hence making the bird appear turquoise. The green magpie species has an extremely large range and although the population trend seems to be decreasing, the decline is not rapid enough to push the species into the Vulnerable category. As such, their current conservation status is “Least Concern.” The cover illustration is by Karen Montgomery, based on a black-and-white engrav‐ ing from English Cyclopedia. The cover fonts are Gilroy Semibold and Guardian Sans. The text font is Adobe Minion Pro; the heading font is Adobe Myriad Condensed; and the code font is Dalton Maag’s Ubuntu Mono.", + "source": "SQL-for-Data-Analysis-Advanced-Techniques-for-Transforming-Data-into-Insights-2021.pdf", + "chunk_id": 248 + }, + { + "text": "www.it-ebooks.info www.it-ebooks.info The Art of SQL While heeding the profit of my counsel, avail yourself also of any helpful circumstances over and beyond the ordinary rules. —Sun Tzu, The Art of War www.it-ebooks.info Other resources from O’Reilly Related titles SQL in a Nutshell SQL Tuning SQL Pocket Guide SQL Cookbook™ oreilly.com oreilly.com is more than a complete catalog of O’Reilly books. You’ll also find links to news, events, articles, weblogs, sample chapters, and code examples. oreillynet.com is the essential portal for developers interested in open and emerging technologies, including new plat- forms, programming languages, and operating systems. Conferences O’Reilly brings diverse innovators together to nurture the ideas that spark revolutionary industries. We specialize in documenting the latest tools and systems, translating the innovator’s knowledge into useful skills for those in the trenches. Visit conferences.oreilly.com for our upcoming events. Safari Bookshelf (safari.oreilly.com) is the premier online reference library for programmers and IT professionals. Conduct searches across more than 1,000 books. Sub- scribers can zero in on answers to time-critical questions in a matter of seconds. Read the books on your Bookshelf from cover to cover or simply flip to the page you need. Try it today for free. www.it-ebooks.info Beijing • Cambridge • Farnham • Köln • Paris • Sebastopol • Taipei • Tokyo The Art of SQL Stéphane Faroult with Peter Robson www.it-ebooks.info The Art of SQL by Stéphane Faroult with Peter Robson Copyright © 2006 O’Reilly Media, Inc. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc. 1005 Gravenstein Highway North, Sebastopol, CA 95472 O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (safari.oreilly.com). For more information, contact our corporate/institutional sales department: (800) 998-9938 or corporate@oreilly.com. Editor: Jonathan Gennick Production Editors: Jamie Peppard and Marlowe Shaeffer Copyeditor: Nancy Reinhardt Indexer: Ellen Troutman Zaig Cover Designer: Mike Kohnke Interior Designer: Marcia Friedman Illustrators: Robert Romano, Jessamyn Read, and Lesley Borash Printing History: March 2006: First Edition. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. The Art of SQL and related trade dress are trademarks of O’Reilly Media, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O’Reilly Media, Inc. was aware of a trademark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this book, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein. This book uses RepKover™, a durable and flexible lay-flat binding. ISBN-10: 0-596-00894-5 ISBN-13: 978-0-596-00894-9 [M] [10/06] www.it-ebooks.info The French humorist Alphonse Allais (1854–1905), once dedicated one of his short stories as follows: To the only woman I love and who knows it well. . . . with the following footnote: This is a very convenient dedication that I cannot recommend too warmly to", + "source": "The Art of SQL.pdf", + "chunk_id": 0 + }, + { + "text": "978-0-596-00894-9 [M] [10/06] www.it-ebooks.info The French humorist Alphonse Allais (1854–1905), once dedicated one of his short stories as follows: To the only woman I love and who knows it well. . . . with the following footnote: This is a very convenient dedication that I cannot recommend too warmly to my fellow writers. It costs nothing, and can, all at once, please five or six persons. I can take a piece of wise advice when I meet one. STÉPHANE FAROULT www.it-ebooks.info www.it-ebooks.info C O N T E N T S Preface ix 1 Laying Plans 1 Designing Databases for Performance 2 Waging War 27 Accessing Databases Efficiently 3 Tactical Dispositions 55 Indexing 4 Maneuvering 75 Thinking SQL Statements 5 Terrain 105 Understanding Physical Implementation 6 The Nine Situations 127 Recognizing Classic SQL Patterns 7 Variations in Tactics 167 Dealing with Hierarchical Data 8 Weaknesses and Strengths 199 Recognizing and Handling Difficult Cases 9 Multiple Fronts 225 Tackling Concurrency 10 Assembly of Forces 247 Coping with Large Volumes of Data 11 Stratagems 279 Trying to Salvage Response Times 12 Employment of Spies 307 Monitoring Performance Photo Credits 333 Index 335 www.it-ebooks.info www.it-ebooks.info ix C H A P T E R P R E F A C E T here used to be a time when what is known today as “Information Technology” or IT was less glamorously known as “Electronic Data Processing.” And the truth is that for all the buzz about trendy techniques, the processing of data is still at the core of our sys- tems—and all the more as the volume of data under management seems to be increasing even faster than the speed of processors. The most vital corporate data is today stored in databases and accessed through the imperfect, but widely known, SQL language—a com- bination that had begun to gain acceptance in the pinstriped circles at the beginning of the 1980s and has since wiped out the competition. You can hardly interview a young developer today who doesn’t claim a good working knowledge of SQL, the lingua franca of database access, a standard part of any basic IT course. This claim is usually reasonably true, if you define knowledge as the ability to obtain, after some effort, functionally correct results. However, enterprises all over the world are today confronted with exploding volumes of data. As a result, “functionally correct” results are no longer enough: they also have to be fast. Database performance has become a major headache in many companies. Interestingly, although everyone agrees that the source of performance issues lies in the code, it seems accepted everywhere that the first concern of developers should be to provide code that works— which seems to be a reasonable expectation. The thought seems to be that the database www.it-ebooks.info x P R E F A C E access part of their code should be as simple as possible, for maintenance reasons, and that “bad SQL” should be given to senior database administrators (DBAs) to tweak and make run", + "source": "The Art of SQL.pdf", + "chunk_id": 1 + }, + { + "text": "reasonable expectation. The thought seems to be that the database www.it-ebooks.info x P R E F A C E access part of their code should be as simple as possible, for maintenance reasons, and that “bad SQL” should be given to senior database administrators (DBAs) to tweak and make run faster, with the help of a few magic database parameters. And if such tweaking isn’t enough, then it seems that upgrading the hardware is the proper course to take. It is quite often that what appears to be the common-sense and safe approach ends up being extremely harmful. Writing inefficient code and relying on experts for tuning the “bad SQL” is actually sweeping the dirt under the carpet. In my view, the first ones to be concerned with performance should be developers, and I see SQL issues as something encompassing much more than the proper writing of a few queries. Performance seen from a developer’s perspective is something profoundly different from “tuning,” as practiced by DBAs. A database administrator tries to get the most out of a system—a given hardware, processors and storage subsystem, or a given version of the database. A database administrator may have some SQL skills and be able to tune an especially poorly performing statement. But developers are writing code that may well run for 5 to 10 years, surviving several major releases (Internet-enabled, ready-for-the-grid, you name it) of the Database Management System (DBMS) it was written for—and on several generations of hardware. Your code must be fast and sound from the start. It is a sorry assessment to make but if many developers “know” SQL, very few have a sound understanding of this language and of the relational theory. Why Another SQL Book? There are three main types of SQL books: books that teach the logic and the syntax of a particular SQL dialect, books that teach advanced techniques and take a problem-solving approach, and performance and tuning books that target experts and senior DBAs. On one hand, books show how to write SQL code. On the other hand, they show how to diagnose and fix SQL code that has been badly written. I have tried, in this book, to teach people who are no longer novices how to write good SQL code from the start and, most importantly, to have a view of SQL code that goes beyond individual SQL statements. Teaching how to use a language is difficult enough; but how can one teach how to efficiently use a language? SQL is a language that can look deceivingly simple once you have been initiated. And yet it allows for an almost infinite number of cases and combinations. The first comparison that occurred to me was the game of chess, but it suddenly dawned on me that chess was invented to teach war. I have a natural tendency to consider every new performance challenge as a battle to be fought against an army of rows, and I realized that the problem of teaching developers", + "source": "The Art of SQL.pdf", + "chunk_id": 2 + }, + { + "text": "me was the game of chess, but it suddenly dawned on me that chess was invented to teach war. I have a natural tendency to consider every new performance challenge as a battle to be fought against an army of rows, and I realized that the problem of teaching developers how to use databases efficiently was similar to the problem of teaching officers how to conduct a war. You need knowledge, you need skills, and you need talent. Talent cannot be taught, but it can be nurtured. This is what most strategists, from Sun Tzu, who wrote his Art of War 25 www.it-ebooks.info P R E F A C E xi centuries ago, to modern-day generals, have believed—so they tried to pass on the experience acquired on the field through simple maxims and rules that they hoped would serve as guiding stars among the sound and fury of battles. I have tried to apply this method to more peaceful aims, and I have mostly followed the same plan as Sun Tzu—and I’ve borrowed his title. Many respected IT specialists claim the status of scientists; “Art” seems to me more appropriate than “Science” when it comes to defining an activity that requires flair, experience, and creativity, as much as rigor and understanding.* It is quite likely that my fondness for Art will be frowned upon by some partisans of Science, who claim that for each SQL problem, there is one optimal solution, which can be attained by rigorous analysis and a good knowledge of data. However, I don’t see the two positions at odds. Rigor and a scientific approach will help you out of one problem at one given moment. In SQL development, if you don’t have the uncertainties linked to the next move of the adversary, the big uncertainties lie in future evolutions. What if, rather unexpectedly, the volume of this or that table increases? What if, following a merger, the number of users doubles? What if we want to keep several years of data online? How will a program behave on hardware totally different from what we have now? Some architectural choices are gambles on the future. You will certainly need rigor and a very sound theoretical knowledge—but those qualities are prerequisites of any art. Ferdinand Foch, the future Supreme Commander of the Allied armies of WWI, remarked at a lecture at the French Ecole Supérieure de Guerre in 1900 that: The art of war, like all other arts, has its theory, its principles—otherwise, it wouldn’t be an art. This book is not a cookbook, listing problems and giving “recipes.” The aim is much more to help developers—and their managers—to raise good questions. You may well still write awful, costly queries after having read and digested this book. One sometimes has to. But, hopefully, it will be knowingly and with good reason. Audience This book is targeted at: • Developers with significant (one year or, preferably, more) experience of develop- ment with an SQL database • Their managers •", + "source": "The Art of SQL.pdf", + "chunk_id": 3 + }, + { + "text": "costly queries after having read and digested this book. One sometimes has to. But, hopefully, it will be knowingly and with good reason. Audience This book is targeted at: • Developers with significant (one year or, preferably, more) experience of develop- ment with an SQL database • Their managers • Software architects who design programs with significant database components * One of my favorite computer books happens to be D.E. Knuth’s classic Art of Computer Programming (Addison Wesley). www.it-ebooks.info xii P R E F A C E Although I hope that some DBAs, and particularly those that support development databases, will enjoy reading this book, I am sorry to tell them I had somebody else in mind while writing. Assumptions This Book Makes I assume in this book that you have already mastered the SQL language. By mastering I don’t mean that you took SQL 101 at the university and got an A+, nor, at the other end of the spectrum, that you are an internationally acknowledged SQL guru. I mean that you have already developed database applications using the SQL language, that you have had to think about indexing, and that you don’t consider a 5,000-row table to be a big table. It is not the purpose of this book to tell you what a “join” is—not even an outer one—nor what indexes are meant to be used for. Although you don’t need to feel totally comfortable with arcane SQL constructs, if, when given a set of tables and a question to answer, you are unable to come up with a functionally correct piece of code, there are probably a couple of books you had better read before this one. I also assume that you are at least familiar with one computer language and with the principles of computer programming. I assume that you have already been down in the trenches and that you have already heard users complain about slow and poorly performing systems. Contents of This Book I found the parallel between war and SQL so strong that I mostly followed Sun Tzu’s outline—and kept most of his titles.* This book is divided into twelve chapters, each containing a number of principles or maxims. I have tried to explain and illustrate these principles through examples, preferably from real-life cases. Chapter 1, Laying Plans Examines how to design databases for performance Chapter 2, Waging War Explains how programs must be designed to access databases efficiently Chapter 3, Tactical Dispositions Tells why and how to index Chapter 4, Maneuvering Explains how to envision SQL statements Chapter 5, Terrain Shows how physical implementation impacts performance * A few titles were borrowed from Clausewitz’s On War. www.it-ebooks.info P R E F A C E xiii Chapter 6, The Nine Situations Covers classic SQL patterns and how to approach them Chapter 7, Variations in Tactics Explains how to deal with hierarchical data Chapter 8, Weaknesses and Strengths Provides indications about how to recognize and handle some difficult cases Chapter 9, Multiple Fronts", + "source": "The Art of SQL.pdf", + "chunk_id": 4 + }, + { + "text": "A C E xiii Chapter 6, The Nine Situations Covers classic SQL patterns and how to approach them Chapter 7, Variations in Tactics Explains how to deal with hierarchical data Chapter 8, Weaknesses and Strengths Provides indications about how to recognize and handle some difficult cases Chapter 9, Multiple Fronts Describes how to face concurrency Chapter 10, Assembly of Forces Addresses how to cope with large volumes of data Chapter 11, Stratagems Offers a few tricks that will help you survive rotten database designs Chapter 12, Employment of Spies Concludes the book by explaining how to define and monitor performance Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates emphasis and new terms, as well as book titles. Constant width Indicates SQL and, generally speaking, programming languages’ keywords; table, index and column names; functions; code; or the output from commands. Constant width bold Shows commands or other text that should be typed literally by the user. This style is used only in code examples that mix both input and output. Constant width italic Shows text that should be replaced with user-supplied values. This icon signifies a maxim and summarizes an important principle in SQL. NOTE This is a tip, suggestion, or general note. It contains useful supplementary information about the topic at hand. www.it-ebooks.info xiv P R E F A C E Using Code Examples This book is here to help you get your job done. In general, you may use the code in this book in your programs and documentation. You do not need to contact O’Reilly for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. O’Reilly, Media Inc. appreciates, but does not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “The Art of SQL by Stéphane Faroult with Peter Robson. Copyright © 2006 O’Reilly Media, 0-596-00894-5.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact the publisher at permissions@oreilly.com. Comments and Questions Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 (800) 998-9938 (in the U.S. or Canada) (707) 829-0515 (international or local) (707) 829-0104 (fax) The publisher has a web page for this book, where we list errata, examples, and any additional information. You can access this page at: http://www.oreilly.com/catalog/artofsql To comment or ask technical questions about this book, send email to: bookquestions@oreilly.com For more information about our books, conferences, Resource Centers, and the O’Reilly Network, see O’Reilly’s web site at: http://www.oreilly.com You can also", + "source": "The Art of SQL.pdf", + "chunk_id": 5 + }, + { + "text": "we list errata, examples, and any additional information. You can access this page at: http://www.oreilly.com/catalog/artofsql To comment or ask technical questions about this book, send email to: bookquestions@oreilly.com For more information about our books, conferences, Resource Centers, and the O’Reilly Network, see O’Reilly’s web site at: http://www.oreilly.com You can also visit the author’s company web site at: http://www.roughsea.com www.it-ebooks.info P R E F A C E xv Safari® Enabled When you see a Safari® Enabled icon on the cover of your favorite technology book, that means the book is available online through the O’Reilly Network Safari Bookshelf. Safari offers a solution that’s better than e-books. It’s a virtual library that lets you easily search thousands of top tech books, cut and paste code samples, download chapters, and find quick answers when you need the most accurate, current information. Try it for free at http://safari.oreilly.com. Acknowledgments Writing a book in a language that is neither your native language nor the language of the country where you live requires an optimism that (in retrospect) borders on insanity. Fortunately, Peter Robson, whom I had met at several conferences as a fellow speaker, brought to this book not only his knowledge of the SQL language and database design issues, but an unabated enthusiasm for mercilessly chopping my long sentences, placing adverbs where they belong, or suggesting an alternative to replace a word that was last heard in Merry England under the Plantagenets.* Being edited by Jonathan Gennick, the best-selling author of the O’Reilly SQL Pocket Guide and several other noted books, was a slightly scary honor. I discovered in Jonathan an editor extremely respectful of authors. His professionalism, attention to detail, and challenging views made this book a much better book than Peter and I would have written on our own. Jonathan also contributed to give a more mid-Atlantic flavor to this book (as Peter and I discovered, setting the spelling checker to “English (US)” is a prerequisite, but not quite enough). I would like to express my gratitude to the various people, from three continents, who took the time to read parts or the whole of the drafts of this book and to give me frank opinions: Philippe Bertolino, Rachel Carmichael, Sunil CS, Larry Elkins, Tim Gorman, Jean-Paul Martin, Sanjay Mishra, Anthony Molinaro, and Tiong Soo Hua. I feel a particular debt towards Larry, because the concept of this book probably finds its origin in some of our email discussions. I would also like to thank the numerous people at O’Reilly who made this book a reality. These include Marcia Friedman, Rob Romano, Jamie Peppard, Mike Kohnke, Ron Bilodeau, Jessamyn Read, and Andrew Savikas. Thanks, too, to Nancy Reinhardt for her most excellent copyedit of the manuscript. * For readers unfamiliar with British history, the Plantagenet dynasty ruled England between 1154 and 1485. www.it-ebooks.info xvi P R E F A C E Special thanks to Yann-Arzel Durelle-Marc for kindly providing a suitable scan of the picture used to illustrate Chapter 12. Thanks too,", + "source": "The Art of SQL.pdf", + "chunk_id": 6 + }, + { + "text": "copyedit of the manuscript. * For readers unfamiliar with British history, the Plantagenet dynasty ruled England between 1154 and 1485. www.it-ebooks.info xvi P R E F A C E Special thanks to Yann-Arzel Durelle-Marc for kindly providing a suitable scan of the picture used to illustrate Chapter 12. Thanks too, to Paul McWhorter for permission to use his battle map as the basis for the Chapter 6 figure. Finally, I would like to thank Roger Manser and the staff at Steel Business Briefing for supplying Peter and me with an office and much-needed coffee for work sessions in London, halfway between our respective bases, and Qian Lena (Ashley) for providing me with the Chinese text of the Sun Tzu quote at the beginning of this book. www.it-ebooks.info Chapter 1. C H A P T E R O N E Laying Plans Designing Databases for Performance C’est le premier pas qui, dans toutes les guerres, décèle le génie. It is the first step that reveals genius in all wars. —Joseph de Maistre (1754–1821) Lettre du 27 Juillet 1812 à Monsieur le Comte de Front www.it-ebooks.info 2 C H A P T E R O N E T he great nineteenth century German strategist, Clausewitz, famously remarked that war is the continuation of politics by other means. Likewise, any computer program is, in one way or another, the continuation of the general activity within an organization, allowing it to do more, faster, better, or cheaper. The main purpose of a computer program is not simply to extract data from a database and then to process it, but to extract and process data for some particular goal. The means are not the end. A reminder that the goal of a given computer program is first of all to meet some business requirement* may come across as a platitude. In practice, the excitement of technological challenges often slowly causes attention to drift from the end to the means, from upholding the quality of the data that records business activity to writing programs that perform as intended and in an acceptable amount of time. Like a general in command of his army at the beginning of a campaign, we must know clearly what our objectives are— and we must stick to them, even if unexpected difficulties or opportunities make us alter the original plan. Whenever the SQL language is involved, we are fighting to keep a faithful and consistent record of business activity over time. Both faithfulness and consistency are primarily associated with the quality of the database model. The database model that SQL was initially designed to support is the relational model. One cannot overemphasize the importance of having a good model and a proper database design, because this is the very foundation of any information system. The Relational View of Data A database is nothing but a model of a small part of a real-life situation. As any representation, a database is always an imperfect model, and a very narrow depiction of a", + "source": "The Art of SQL.pdf", + "chunk_id": 7 + }, + { + "text": "proper database design, because this is the very foundation of any information system. The Relational View of Data A database is nothing but a model of a small part of a real-life situation. As any representation, a database is always an imperfect model, and a very narrow depiction of a rich and complex reality. There is rarely a single way to represent some business activity, but rather several variants that in a technical sense will be semantically correct. However, for a given set of processes to apply, there is usually one representation that best meets the business requirement. The relational model is thus named, not because you can relate tables to one another (a popular misconception), but as a reference to the relationships between the columns in a table. These are the relationships that give the model its name; in other words, relational means that if several values belong to the same row in a table, they are related. The way columns are related to each other defines a relation, and a relation is a table (more exactly, a table represents one relation). The business requirements determine the scope of the real-world situation that is to be modeled. Once you have defined the scope, you can proceed to identify the data that you * The expression business requirement is meant to encompass non-commercial as well as commercial activities. www.it-ebooks.info L A Y I N G P L A N S 3 need to properly record business activity. If we say that you are a used car dealer and want to model the cars you have for sale (for instance to advertise them on a web site), items such as make, model, version, style (sedan, coupe, convertible...), year, mileage, and price may be the very first pieces of information that come to mind. But potential buyers may want to learn about many more characteristics to be able to make an informed choice before settling for one particular car. For instance: • General state of the vehicle (even if we don’t expect anything but “excellent”) • Safety equipment • Manual or automatic transmission • Color (body and interiors), metallic paintwork or not, upholstery, hard or soft top, perhaps a picture of the car • Seating capacity, trunk capacity, number of doors • Power steering, air conditioning, audio equipment • Engine capacity, cylinders, horsepower and top speed, brakes (everyone isn’t a car enthusiast who would know technical specifications from the car description) • Fuel, consumption, tank capacity • Current location of the car (may matter to buyers if the site lists cars available from a number of physical places) • And so on... If we decide to model the available cars into a database, then each row in a table summarizes a particular statement of fact—for instance, that there is for sale a 1964 pink Cadillac Coupe DeVille that has already been driven twenty times around the Earth. Through relational operations, such as joins, and also by filtering, selection of particular attributes, or", + "source": "The Art of SQL.pdf", + "chunk_id": 8 + }, + { + "text": "database, then each row in a table summarizes a particular statement of fact—for instance, that there is for sale a 1964 pink Cadillac Coupe DeVille that has already been driven twenty times around the Earth. Through relational operations, such as joins, and also by filtering, selection of particular attributes, or computations applied to attributes (say computing from consumption and tank capacity how many miles we can drive without refueling), we can derive new factual statements. If the original statements are true, the derived statements will be true. Whenever we are dealing with knowledge, we start with facts that we accept as truths that need no proof (in mathematics these are known as axioms, but this argument is by no means restricted to mathematics and you could call those unproved true facts principles in other disciplines). It is possible to build upon these true facts (proving theorems in mathematics) to derive new truths. These truths themselves may form the foundations from which further new truths emerge. Relational databases work in exactly the same way. It is absolutely no accident that the relational model is mathematically based. The relations we define (which once again means, for an SQL database, the tables we create) represent facts that we accept, a priori, as true. The views we define, and the queries we write, are new truths that we prove. www.it-ebooks.info 4 C H A P T E R O N E NOTE The coherence of the relational model is a critically important concept to grasp. Because of the inherent mathematical stability of the principles that underlie relational data modeling, we can be totally confident that the result of any query of our original database will indeed generate equally valid facts—if we respect the relational principles. Some of the key principles of the relational theory are that a relation, by definition, contains no duplicate, and that row ordering isn’t significant. As you shall see in Chapter 4, SQL allows developers to take a number of liberties with the relational theory, liberties that may be the reasons for either surprising results or the failure of a database optimizer to perform efficiently. There is, however, considerable freedom in the choice of our basic truths. Sometimes the exercise of this freedom can be done very badly. For example, wouldn’t it be a little tedious if every time someone went to buy some apples, the grocer felt compelled to prove all Newtonian physics before weighing them? What must be thought of a program where the most basic operation requires a 25-way join? We may use much data in common with our suppliers and customers. However, it is likely that, if we are not direct competitors, our view of the same data will be different, reflecting our particular perspective on our real-life situation. For example, our business requirements will differ from those of our suppliers and customers, even though we are all using the same data. One size doesn’t fit all. A good design is a design that doesn’t require", + "source": "The Art of SQL.pdf", + "chunk_id": 9 + }, + { + "text": "same data will be different, reflecting our particular perspective on our real-life situation. For example, our business requirements will differ from those of our suppliers and customers, even though we are all using the same data. One size doesn’t fit all. A good design is a design that doesn’t require crazy queries. Modeling is the projection of business requirements. The Importance of Being Normal Normalization, and especially that which progresses to the third normal form (3NF), is a part of relational theory that most students in computer science have been told about. It is like so many things learned at school (classical literature springs to mind), often remembered as dusty, boring, and totally disconnected from today’s reality. Many years later, it is rediscovered with fresh eyes and in light of experience, with an understanding that the essence of both principles and classicism is timelessness. The principle of normalization is the application of logical rigor to the assemblage of items of data—which may then become structured information. This rigor is expressed in the definition of various normal forms, most typically three, although purists argue that one www.it-ebooks.info L A Y I N G P L A N S 5 should analyze data beyond 3NF to what is known in the trade as Boyce-Codd normal form (BCNF), or even to fifth normal form (5NF). Don’t panic. We will discuss only the first three forms. In the vast majority of cases, a database modeled in 3NF will also be in BCNF* and 5NF. You may wonder why normalization matters. Normalization is applying order to chaos. After the battle, mistakes may appear obvious, and successful moves sometimes look like nothing other than common sense. Likewise, after normalization the structures of the various tables in the database may look natural, and the normalization rules are sometimes dismissively considered as glorified common sense. We all want to believe we have an ample supply of common sense; but it’s easy to get confused when dealing with complex data. The three first normal forms are based on the application of strict logic and are a useful sanity checklist. The odds that our creating un-normalized tables will increase our risk of being struck by divine lightning and reduced to a little mound of ashes are indeed very low (or so I believe; it’s an untested theory). Data inconsistency, the difficulty of coding data-entry controls, and error management in what become bloated application programs are real risks, as well as poor performance and the inability to make the model evolve. These risks have a very high probability of occurring if we don’t adhere to normal form, and I will soon show why. How is data moved from a heterogeneous collection of unstructured bits of information into a usable data model? The method itself isn’t complicated. We must follow a few steps, which are illustrated with examples in the following subsections. Step 1: Ensure Atomicity First of all, we must ensure that the characteristics, or attributes, we are dealing with are", + "source": "The Art of SQL.pdf", + "chunk_id": 10 + }, + { + "text": "of unstructured bits of information into a usable data model? The method itself isn’t complicated. We must follow a few steps, which are illustrated with examples in the following subsections. Step 1: Ensure Atomicity First of all, we must ensure that the characteristics, or attributes, we are dealing with are atomic. The whole idea of atomicity is rather elusive, in spite of its apparent simplicity. The word atom comes from ideas first advanced by Leucippus, a Greek philosopher who lived in the fifth century B.C., and means “that cannot be split.” (Atomic fission is a contradiction in terms.) Deciding whether data can be considered atomic or not is chiefly a question of scale. For example, a regiment may be an atomic fighting unit to a general- in-chief, but it will be very far from atomic to the colonel in command of that regiment, who deals at the more granular level of battalions or squadrons. In the same way, a car may be an atomic item of information to a car dealer, but to a garage mechanic, it is very far from atomic and consists of a whole host of further components that form the mechanic’s perception of atomic data items. * You can have 3NF but not BCNF if your table contains several sets of columns that are unique (can- didate keys, which are possible unique identifiers of a row) and share one column. Such situations are not very common. www.it-ebooks.info 6 C H A P T E R O N E From a purely practical point of view, we shall define an atomic attribute as an attribute that, in a where clause, can always be referred to in full. You can split and chop an attribute as much as you want in the select list (where it is returned); but if you need to refer to parts of the attribute inside the where clause, the attribute lacks the level of atomicity you need. Let me give an example. In the previous list of attributes for used cars, you’ll find “safety equipment,” which is a generic name for several pieces of information, such as the presence of an antilock braking system (ABS), or airbags (passenger-only, passenger and driver, frontal, lateral, and so on), or possibly other features, such as the centralized locking of doors. We can, of course, define a column named safety_equipment that is just a description of available safety features. But we must be aware that by using a description we forfeit at least two major benefits: The ability to perform an efficient search If some users consider ABS critical because they often drive on wet, slippery roads, a search that specifies “ABS” as the main criterion will be very slow if we must search column safety_equipment in every row for the “ABS” substring. As I’ll show in Chapter 3, regular indexes require atomic (in the sense just defined) values as keys. One can sometimes use query accelerators other than regular indexes (full-text indexing, for instance), but such", + "source": "The Art of SQL.pdf", + "chunk_id": 11 + }, + { + "text": "be very slow if we must search column safety_equipment in every row for the “ABS” substring. As I’ll show in Chapter 3, regular indexes require atomic (in the sense just defined) values as keys. One can sometimes use query accelerators other than regular indexes (full-text indexing, for instance), but such accelerators usually have drawbacks, such as not being maintained in real time. Also take note that full- text search may produce awkward results at times. Let’s take the example of a color column that contains a description of both body and interior colors. If you search for “blue” because you’d prefer to buy a blue car, gray cars with a blue interior will also be returned. We have all experienced irrelevant full-text search results through web searches. Database-guaranteed data correctness Data-entry is prone to error. More importantly than dissuasive search times, if “ASB” is entered instead of “ABS” into a descriptive string, the database man- agement system will have no way to check whether the string “ASB” is mean- ingful. As a result, the row will never be returned when a user specifies “ABS” in a search, whether as the main or as a secondary criterion. In other words, some of our queries will return wrong results (either incomplete, or even plain wrong if we want to count how many cars feature ABS). If we want to ensure data correctness, our only means (other than double-checking what we have typed) is to write some complicated function to parse and analyze the safety equipment string when it is entered or updated. It is hard to decide what will be worse: the hell that the maintenance of such a function would be, or the perfor- mance penalty that it will inflict on loads. By contrast, a mandatory Y/N has_ABS column would not guarantee that the information is correct, but at least declara- tive check constraints can make the DBMS reject any value other than Y or N. Partially updating a complex string of data requires first-rate mastery of string functions. Thus, you want to avoid cramming multiple values into a single string. www.it-ebooks.info L A Y I N G P L A N S 7 Defining data atoms isn’t always a simple exercise. For example, the handling of addresses frequently raises difficult questions about atomicity. Must we consider the address as some big, opaque string? Or must we break it into its components? And if we decompose the address, to what level should we split it up? Remember the points made earlier about atomicity and business requirements. How we represent an address actually depends on what we want to do with the address. For example, if we want to compute statistics or search by postal code and town, then it is desirable to break the address up into sufficient attribute components to uniquely identify those important data items. The question then arises as to how far this decomposition of the address should be taken. The guiding principle in determining the extent to", + "source": "The Art of SQL.pdf", + "chunk_id": 12 + }, + { + "text": "search by postal code and town, then it is desirable to break the address up into sufficient attribute components to uniquely identify those important data items. The question then arises as to how far this decomposition of the address should be taken. The guiding principle in determining the extent to which an address should be broken into components is to test each component against the business requirements, and from those requirements derive the atomic address attributes. What these various address attributes will be cannot be predicted (although the variation is not great), but we must be aware of the danger of adopting an address format just because some other organization may have chosen it, before we have tested it critically against our own business needs. Note that sometimes, the devil is in the details. By trying to be too precise, we may open the door to many distracting and potentially irrelevant issues. If we settle for a level of detail that includes building number and street as atomic items, what of ACME Corp, the address of which is simply “ACME Building”? We should not create design problems for information we don’t need to process. Properly defining the level of information that is needed can be particularly important when transferring data from an operational to a decision-support system. Once all atomic data items have been identified, and their mutual interrelationships resolved, distinct relations emerge. The next step is to identify what uniquely characterizes a row—the primary key. At this stage, it is very likely that this key will be a compound one, consisting of two or more individual attributes. To go on with our used car example, for a customer it’s the combination of make, model, version, style, year, and mileage that will identify a particular vehicle—not the current registration number. It isn’t always easy to correctly define a key. A good, classic example of attribute analysis is the business definition of “customer.” A customer may be identified by a name. However, a name may not be the best identifier. If our customers are companies, the way we identify them may be the source of ambiguities—is it “RSI,” “Relational Software,” “Relational Software Inc” (with or without a dot following “Inc,” with or without a comma after “Relational Software”) that identifies this given company? Uppercase? Lowercase? Capitalized initials? We have here all the conditions for storing information inside a database and never seeing it again. The choice of the customer name as identifier is a challenging one, because it demands the strict application of naming standards to avoid possible ambiguities. It may be preferable to identify a customer on the basis of either a standard short name, or possibly by use of a www.it-ebooks.info 8 C H A P T E R O N E unique code. And one should always keep in mind the impact on related data of Relational Software Inc. changing its name to, say, Oracle Corporation. If we need to keep a history of our relationship, then we", + "source": "The Art of SQL.pdf", + "chunk_id": 13 + }, + { + "text": "a www.it-ebooks.info 8 C H A P T E R O N E unique code. And one should always keep in mind the impact on related data of Relational Software Inc. changing its name to, say, Oracle Corporation. If we need to keep a history of our relationship, then we must be able to identify both names as representing the same company at different points in time. As a general rule, you should, whenever possible, use a unique identifier that has meaning rather than some obscure sequential integer. I must stress that the primary key is what characterizes the data—which is not the case with some sequential identifier associated with each new row. You may choose to add such an identifier later, for instance because you find your own company_id easier to handle than the place of incorporation and registration number that truly identify a company. You can even promote the sequential identifier to the envied status of primary key, as a technical substitute (or shorthand) for the true key, in exactly the same way that you’d use table aliases in a query in order to be able to write: where a.id = b.id instead of: where table_with_a_long_name.id = table_even_worse_than_the_other.id But a technical, numerical identifier doesn’t constitute a real primary key by the mere virtue of its existence and mustn’t be mistaken for the real thing. Once all the attributes are atomic and keys are identified, our data is in first normal form (1NF). Step 2: Check Dependence on the Whole Key I have pointed out that some of the information that we should store to help used car buyers make an informed choice would already be known by a car enthusiast. In fact, many used car characteristics are not specific to one particular car. For example, all the cars sharing make, model, version, and style will have the same seating and cargo capacity, regardless of year and mileage. In other words, we have attributes that depend on only a part of the key. What are the implications of keeping them inside a used_cars table? Data redundancy If we happen to have for sale many cars of the same make, model, version, and style (a set of characteristics that we can generically call the car model), all the attributes that are not specific to one particular car will be stored as many times as we have cars of the same model. There are two issues with the storage of redundant data. First, redundant data increases the odds of encountering contra- dictory information because of input errors (and it makes correction more time- consuming). Second, redundant data is an obvious storage waste. It is custom- ary to hear that nowadays storage is so cheap that one no longer needs to be obsessed with space. True enough, except that such an argument overlooks the fact that there is also more and more data to store in today’s world. It also over- looks the fact that data is often mirrored, possibly backed up", + "source": "The Art of SQL.pdf", + "chunk_id": 14 + }, + { + "text": "is so cheap that one no longer needs to be obsessed with space. True enough, except that such an argument overlooks the fact that there is also more and more data to store in today’s world. It also over- looks the fact that data is often mirrored, possibly backed up to other disks on a www.it-ebooks.info L A Y I N G P L A N S 9 disaster recovery site where it is mirrored again, and that many development databases are mere copies of production databases. As a result, every wasted byte isn’t wasted once, but four or five times in the very best of cases. When you add up all the wasted bytes, you sometimes get surprisingly high figures. Besides the mere cost of storage, sometimes—more importantly—there is also the issue of recovery. There are cases when one experiences “unplanned downtime,” a very severe crash for which the only solution is to restore the database from a backup. All other things being equal, a database that is twice as big as necessary will take twice the time to restore than would otherwise be needed. There are environments in which a long time to restore can cost a lot of money. In a hos- pital, it can even cost lives. Query performance A table that contains a lot of information (with a large number of columns) takes much longer to scan than a table with a reduced set of columns. As we shall see in other chapters, a full table scan is not necessarily the scary situation that many beginners believe it to be; there are many cases where it is by far the best solution. However, the more bytes in the average row, the more pages will be required to store the table, and the longer it takes to scan the table. If you want to display a selectable list of the available car models, an un-normalized table will require a select distinct applied to all the available cars. Running a select distinct doesn’t mean only scanning many more rows than we would with a separate car_model table, but it also means having to sort those rows to eliminate duplicates. If the data is split in such a way that the DBMS engine can operate against only a subset of the data to resolve the query, performance will be significantly better than when it operates against the whole. To remove dependencies on a part of the key, we must create tables (such as car_model). The keys of those new tables will each be a part of the key for our original table (in our example, make, model, version, and style). Then we must move all the attributes that depend on those new keys to the new tables, and retain only make, model, version, and style in the original table. We may have to repeat this process, since the engine and its characteristics will not depend on the style. Once we have completed the removal of attributes that depend", + "source": "The Art of SQL.pdf", + "chunk_id": 15 + }, + { + "text": "depend on those new keys to the new tables, and retain only make, model, version, and style in the original table. We may have to repeat this process, since the engine and its characteristics will not depend on the style. Once we have completed the removal of attributes that depend on only a part of the key, our tables are in second normal form (2NF). Step 3: Check Attribute Independence When all data has been correctly moved into 2NF, we can commence the process of identifying the third normal form (3NF). Very often, a data set in 2NF will already be in 3NF, but nevertheless, we should check the 2NF set. We now know that each attribute in the current set is fully dependent on the unique key. 3NF is reached when we cannot infer the value of an attribute from any attribute other than those in the unique key. For example, the question must be asked: “Given the value of attribute A, can the value of attribute B be determined?” www.it-ebooks.info 10 C H A P T E R O N E International contact information provides an excellent example of when you can have an attribute dependent on another non-key attribute: if you know the country, you need not record the international dialing code with the phone number (the reverse is not true, since the United States and Canada share the same code). If you need both bits of information, you ought to associate each contact with, say, an ISO country code (for instance IT for Italy), and have a separate country_info table that uses the country code as primary key and that holds useful country information that your business requires. For instance, a country_info table may record that the international dialing code for Italy is 39, but also that the Italian currency is the euro, and so on. Every pair of attributes in our 2NF data set should be examined in turn to check whether one depends on the other. Such checking is a slow process, but essential if the data is to be truly modeled in 3NF. What are the risks associated with not having the data modeled in 3NF? Basically you have the same risks as from not respecting 2NF. There are various reasons that modeling to the third normal form is important. (Note that there are cases in which designers deliberately choose not to model in third normal form; dimensional modeling, which will be briefly introduced in Chapter 10, is such a case. But before you stray from the rule, you must know the rule and weigh the risks involved.) Here are some reasons: A properly normalized model protects against the evolution of requirements. As Chapter 10 will show, a non-normalized model such as the dimensional one finds its justification in assumptions about how the data is maintained and que- ried (the same can be said of the physical data structures that you’ll see in Chapter 5; but a physical implementation change will not jeopardize", + "source": "The Art of SQL.pdf", + "chunk_id": 16 + }, + { + "text": "Chapter 10 will show, a non-normalized model such as the dimensional one finds its justification in assumptions about how the data is maintained and que- ried (the same can be said of the physical data structures that you’ll see in Chapter 5; but a physical implementation change will not jeopardize the logic of programs, even if it can seriously impact their performance). If the assumptions prove wrong one day, all you can do is throw everything away and rebuild from scratch. By contrast, a 3NF model may require some query adjustments, but it will be flexible enough to accommodate changes. Normalization minimizes data duplication. As I have already pointed out, duplicate data is costly, both in terms of disk space and processing power, but it also introduces a much-increased possibility of data becoming corrupt. Corruption happens when one instance of a data value is modified, but the same data held in another part of the database fails to be simul- taneously (and identically) modified. Losing information doesn’t only mean data erasure: if one part of the database says “white” while another part says “black,” you have lost information. Data inconsistency can be prevented by the DBMS if the modeling allows it—if your atomic attributes let you define column con- straints, or if you can declare referential integrity constraints. Otherwise, it has to be prevented by additional programming traps. You then have the choice between using triggers and stored procedures that can grow very complex and add significant overhead, or making programs unnecessarily complicated and www.it-ebooks.info L A Y I N G P L A N S 11 therefore costlier to maintain. Triggers and stored procedures must be extremely well documented. Data consistency ensured in programs moves the protection of data integrity out of the database and into the application layer. Any other pro- gram that needs to access the same data has the choice between duplicating the data integrity protection effort, or happily corrupting the data painfully main- tained in a consistent state by other programs. The normalization process is fundamentally based on the application of atomicity to the world you are modeling. To Be or Not to Be, or to Be Null A very common modeling mistake is to associate large numbers of possible characteristics within a relation, which may result in a table with a large number of columns. Some scientific disciplines may require a very detailed characterization of objects under study, and thus require a large number of attributes, but this is rarely the case in business applications. In any case, a sure sign that a database design is flawed is when columns of some prominent tables mostly contain null values, and especially when two columns cannot possibly contain a value at the same time; if one is defined, the other must be null, and vice versa. This condition would undoubtedly indicate a violation of either 2NF or 3NF. If we admit that a row in a table represents a statement about the characteristics of a given", + "source": "The Art of SQL.pdf", + "chunk_id": 17 + }, + { + "text": "possibly contain a value at the same time; if one is defined, the other must be null, and vice versa. This condition would undoubtedly indicate a violation of either 2NF or 3NF. If we admit that a row in a table represents a statement about the characteristics of a given “thing,” indicating that “we don’t know” for most characteristics seriously downgrades the table as a source of reliable information. This may be a minor inconvenience if the data is stored for informative purpose only. It becomes a major issue if the unknown values are supposed to help us define a result set, and this state of affairs is indicative of a flawed model. All columns in a row should ultimately contain a value, even if business processes are such that various pieces of information are entered from more than one source and/or at different points in time. A stamp collector might likewise keep some room in an album for a series temporarily absent from the collection. But even so, there is a risk of wasting storage if it is actually reserved because one always tailors for the maximum size. There is also a risk of very serious performance problems if only placeholders are used and data goes to some remote overflow area when it is entered at last. The existence of null values also raises an important point with regard to relational modeling, which is the main foundation for the query optimizer. The completeness of a relational model is founded on the application of two-valued logic; in which things are or they aren’t. Any in-between case, a null value, is indeterminate; but in a where clause, conditions cannot be indeterminate. They are true or they are false, because you return a row or you don’t; you cannot return a row with a “maybe this one answers the question www.it-ebooks.info 12 C H A P T E R O N E but I’m not really sure” qualifier. The transition from the three-valued logic implied by nulls (true, false, or indeterminate) to the two-valued logic of the result set is perilous. This is why all SQL practitioners can recall cases when what looked like a good SQL query failed to return the proper result set because of an encounter with null values. For instance, if a column named color contains the values RED, GREEN, and BLACK, this condition: where color not in ('BLUE', 'BLACK', null) will result in no row being returned, because we don’t know what null is and the SQL engine will consider that there is a possibility that it might be RED or GREEN, whereas: where color in ('BLUE', 'BLACK', null) will return all rows for which color is BLACK, and nothing else (remember, we have no BLUE in our table), since there is a possibility that null would be neither RED nor GREEN. As you can see, an SQL engine is even more risk-averse than a banker. Finding an explicit null inside an in ( ) list is,", + "source": "The Art of SQL.pdf", + "chunk_id": 18 + }, + { + "text": "BLACK, and nothing else (remember, we have no BLUE in our table), since there is a possibility that null would be neither RED nor GREEN. As you can see, an SQL engine is even more risk-averse than a banker. Finding an explicit null inside an in ( ) list is, of course, unusual; but such a situation may occur if, instead of an explicit list, we have a subquery and fail to ensure that no null value is returned by that subquery. A representation of customers can provide a very good example of the difficulties inherent to dealing with missing information. Each customer has an address, which is normally the address that will appear on an invoice to that customer. But what if the address to which we must ship our goods is different? Must we consider the shipping address to be a characteristic of the order? It can make sense if we sell once, only to never see customers again. If we are not a funeral parlor, however, and especially if we repeatedly ship goods to the same address, it makes no sense at all from a business point of view. Entering the same data over and over again, besides being a waste of time, also increases the risk of a mistake—hence goods get sent to the wrong address, creating a dissatisfied customer or perhaps an ex-customer. The shipping address is, obviously, a characteristic of the customer, not of the order. This situation ought to have been resolved in the analysis of dependencies during the original design of the model. It is also possible to have the accounting department at a location different from the official, customer delivery address if the customer is a company. So, for one customer, we may have one “official” address, a billing address, and also a shipping address. It is quite common to see customer tables with three sets of columns (each set describing one address) for this purpose. However, if we can have all these addresses, what is likely to be the most common case? Well, it is quite possible that in 90% of the cases we shall have only one useful address, the official address. So, what must we do with all our other columns? Two possibilities come to mind: www.it-ebooks.info L A Y I N G P L A N S 13 Set billing and shipping addresses to null. This is not a very sound strategy, because this will require our programs to use implicit rules, such as “if the billing address is undefined, then send the invoice to the corporate address.” The logic of such programs will become much more complicated, with an increased risk of bugs entering the code. Replicate the information, copying the corporate address to the billing address columns where there is no special billing address. This approach will require special processing during data entry, by a trigger per- haps. In such a case the overhead may not matter much, but in another case the overhead might", + "source": "The Art of SQL.pdf", + "chunk_id": 19 + }, + { + "text": "Replicate the information, copying the corporate address to the billing address columns where there is no special billing address. This approach will require special processing during data entry, by a trigger per- haps. In such a case the overhead may not matter much, but in another case the overhead might matter a lot. Moreover, we must also take care of replicating changes—each update of the corporate address must be replicated to those of the other addresses that are identical, for fear of inconsistency. Both of these scenarios betray a critical lack of understanding on the part of the original modelers. Using null values and implicit rules is a classic fudge to accommodate three- valued logic. The use of nulls inevitably introduces three-valued logic, which immediately introduces semantic inconsistency; no amount of clever programming can remove semantic issues. Replicating data illustrates what happens when dependencies have not been properly analyzed. One solution to our address conundrum might be to get the address information out of the customer table. One design we may contemplate is to store each address in an address table, together with a customer identifier and some column (a bit mask, perhaps) indicating the role of the address. But this is not necessarily the best solution, because issues such as the true meaning of addresses often appear after programs have been rushed into production and an attempt to remodel the original data as part of a later release can introduce insuperable problems. We have so far assumed that we have one shipping address for each customer, which may or may not be identical to the corporate, registered address. What if we send our invoices to a single place but must ship our goods to many different branches, with several distinct shipments belonging to the same invoice? This is not necessarily unusual! It is no longer workable for our design to have a single (mostly null) “shipping address” (represented by several columns) in the customer table. We are, ironically, back to the “shipping address is a characteristic of the order” situation. This means that if we want to refer (especially repeatedly) to addresses in orders, we must associate some kind of purpose-built identifier to our addresses, which will spare us repeating the whole shipping address in each order (normalization in action). Or perhaps we should begin to contemplate the introduction of a shipments table. There is no such thing as the totally perfect design for the customers/addresses conundrum. I have just wandered through likely problems and tried to sketch some of www.it-ebooks.info 14 C H A P T E R O N E the possible solutions. But there will be one solution that works best in your case, and many other solutions that will lead to the risks of inconsistencies. With an inappropriate solution, code will be at best more complicated than necessary with very high odds of being underperforming as well. The question of null values is probably the thorniest issue of the relational theory. Dr. E.F. Codd, the", + "source": "The Art of SQL.pdf", + "chunk_id": 20 + }, + { + "text": "other solutions that will lead to the risks of inconsistencies. With an inappropriate solution, code will be at best more complicated than necessary with very high odds of being underperforming as well. The question of null values is probably the thorniest issue of the relational theory. Dr. E.F. Codd, the father of the relational model, introduced null values early, and explicitly asked in the 3rd of the 12 rules that he published in 1985 for a systematic treatment of null values. (The 12 rules were a concise definition of the required properties of a relational database.) However, the battle is still raging among theorists. The problem is that “not known” may encompass quite a number of different cases. Let’s consider a list of famous writers, each with a birth date and a death date. A null birth date would unambiguously mean “unknown.” But what does a null death date mean? Alive? We don’t know when this author died? We don’t know whether this author is alive or not? I cannot resist the pleasure of quoting the immortal words of the then–U.S. Secretary of Defense, Mr. Donald Rumsfeld, at a February 2002 news briefing of his department: As we know, there are known knowns. There are things we know we know. We also know there are known unknowns. That is to say we know there are some things we do not know. But there are also unknown unknowns, the ones we don’t know we don’t know. I don’t find it unusual to have null values for, to put it in Rumsfeldese, “known unknowns,” attributes that are known to exist and have some value we don’t know at one point in time, for various reasons. For the rest, speculating leads nowhere. Strangely, some of the most interesting usages of null values may perfectly involve nothing but tables where all columns of all rows contain values: null values can be generated through outer joins. Some efficient techniques for checking the absence of particular values that I discuss in Chapter 6 are precisely based on outer joins and tests on null values. Nulls can be hazardous to your logic; if you must use them, be very sure you understand the consequences of doing so in your particular situation. Qualifying Boolean Columns Even though the Boolean type doesn’t exist in SQL, many people feel a need to implement flags to indicate a Boolean true/false status (for instance order_completed). You should aim for increasing the density of your data—order_completed may be useful information to know, but then perhaps other information would be nice to store too: www.it-ebooks.info L A Y I N G P L A N S 15 when was it completed? Who completed it? So that means that instead of having a single “Y/N” column, we can have a completion_date column, and perhaps a completed_by column, both of which will tell us more (although we may not necessarily want to see a null value as long as the order isn’t completed; a solution may", + "source": "The Art of SQL.pdf", + "chunk_id": 21 + }, + { + "text": "So that means that instead of having a single “Y/N” column, we can have a completion_date column, and perhaps a completed_by column, both of which will tell us more (although we may not necessarily want to see a null value as long as the order isn’t completed; a solution may be to use a distinct table to track the various stages of every order from creation to completion). As before, examine the dependencies in the context of your business requirements, and only include those additional columns where the successful operation of the business requires it. Alternatively, a series of essentially Boolean attributes can sometimes be advantageously combined into a unique status attribute. For instance, if you have four attributes that can be either true or false, you can assign a numerical value between 0 and 15 to each of the possible combinations and define the “status” as being represented by this value. But beware—this technique may offend the basic rule of atomicity, so if you must use this approach, do so with considerable caution. Data for data’s sake is a path to disaster. Understanding Subtypes Another reason for the appearance of unnecessarily wide tables (as in having too many attributes) is a lack of understanding of the true relationship between data items. Consider the example of subtypes. A company may have a mix of employees, some of whom are permanent, others who are contractors. They all have several properties in common (name, year of birth, department, room, phone number, and so forth), but there are also properties that are unique to each type of employee (for instance, hire date and salary for permanent employees, rate and contract reference for contractors). The manner in which the common attributes can be shared, while ensuring that the distinctive features are kept separate, introduces the topic of subtypes. We can model this situation by defining three tables. First, the employee table contains all information that is common to every employee, regardless of their status. However, an attribute tells the status of each employee. It has as many distinct values as there are distinct employee types, for example “P” (for permanent employee), and “C” (for contract employee). This table uses an employee number as the primary key. Next, we create additional tables, one for each employee type. In this case, there are two tables. Tables permanent and contract represent subtypes of the table employee, for example. Each permanent or contract employee inherits certain characteristics from the employee table, in addition to possessing unique characteristics, as defined in their own tables. www.it-ebooks.info 16 C H A P T E R O N E Now let’s examine the creation of the primary keys between these two types of tables, as it’s the primary key construct that implements the subtype relationships. The unique key for all tables is the unique identifier for each member of staff—the employee number. The set of primary keys of employee is the union of the primary keys of the various subtype tables, and", + "source": "The Art of SQL.pdf", + "chunk_id": 22 + }, + { + "text": "of tables, as it’s the primary key construct that implements the subtype relationships. The unique key for all tables is the unique identifier for each member of staff—the employee number. The set of primary keys of employee is the union of the primary keys of the various subtype tables, and the intersection of the primary keys of all subtype tables is by construction empty, because each employee belongs to just one, in this case, of the two categories. The primary keys of subtype tables are also foreign keys, referencing the primary key of employee. Please note that assigning totally independent primary keys to the subtype tables would, of course, be a disastrous mistake. In the real world however, you will certainly find examples in which this disastrous mistake has been perpetrated. Note also that entity sub-types are not the same as master-detail relationships. They can quickly be distinguished on examination of their respective primary keys. For those who would think that this type of discussion is a bit academic (associating with the word “academic” some vague, slightly pejorative connotation), I’ll just say that whenever different subtypes use a primary key that is not a subset of the primary key of the parent table, the result is almost invariably pathetic performance, from many points of view. One of the main principles to follow in order to achieve efficient database access is a principle attributed to Philip II of Macedonia, father of Alexander the Great, and that principle is: Divide and Rule. It is quite likely that the vast majority of the queries executed by the HR department will belong to either of two categories: they will be either generic queries about all the people working in an organization or specific queries about one category of person. In both cases, by using subtypes correctly,* we will only need to examine that data which is most likely to provide the result that we require, and no time will be wasted examining irrelevant information. If we were to put everything into a single table, the most modest query would have to plow through a much greater quantity of data, most of which is useless in the context of that query. Tables in which specific columns appear as null indicate the need for subtypes. * You can use subtypes incorrectly. As one of the reviewers remarked, having a kind of super-generic parent table that is referred to several times in the most innocuous query isn’t a model for efficiency. Such a super-generic parent table is hammered by all queries if it stores vital information. Subtypes must be born of logical distinction, not of an ill-conceived desire to implement with tables a strong inheritance scheme inspired from object-oriented techniques. www.it-ebooks.info L A Y I N G P L A N S 17 Stating the Obvious It is always an unsound situation in which there are implicit constraints on your data— for instance “if the business line is such, then the identifier is numeric (although defined as", + "source": "The Art of SQL.pdf", + "chunk_id": 23 + }, + { + "text": "from object-oriented techniques. www.it-ebooks.info L A Y I N G P L A N S 17 Stating the Obvious It is always an unsound situation in which there are implicit constraints on your data— for instance “if the business line is such, then the identifier is numeric (although defined as a string of characters to accommodate other business lines),” or “if the model is T, then the color is necessarily black.” Sometimes, such general knowledge information can prove extremely efficient when filtering data. However, if it remains human knowledge, the DBMS engine, unaware of it, will be unable to take advantage of it, and the optimizer will not possess the necessary information to affect the most efficient database access. In the worst case, implicit constraints can even lead to a runtime failure. For instance, you might inadvertently require the database engine to apply an arithmetic process to a character string. This can happen when a character-defined column is used only for storing numeric data, and a non-numeric character slips in. As an aside, the example of a string identifier that sometimes contains character data and sometimes numerical data illustrates a confusion over domain definitions in the initial database design. It is quite clear that the nature of such a field varies according to circumstances—which is totally unacceptable in a properly designed database. If we need to store, for instance, configuration parameters of various natures (numerical, Boolean, character, and so on), we should not store them in a single table configuration(parameter_ name, parameter_value), but rather use a generic table configuration(parameter_id, parameter_name, parameter_type) and have as many subtypes as we have parameter types. If we use, for instance, configuration_numeric(parameter_id, parameter_value), where parameter_value is a numeric column, any mistyping of the letter “O” instead of zero will be detected by the DBMS when the configuration is changed, instead of resulting in a runtime error when the parameter is used. Define all the constraints you can. Primary keys are, of course, a sine qua non in a relational database. Use alternate key, when they characterize the data and any type of unique constraints. Foreign keys, which ensure that your data is consistent by mapping to master tables, are vital as part of the comprehensive expression of the meaning of the data model. Constraints that control the range of values that can be entered are also valuable. Constraints have two major impacts: • They contribute to ensuring the integrity of your data, guaranteeing that everything, as far as defined rules are concerned, is consistent with those rules. • They provide valuable information about your data to the DBMS kernel, and more specifically to the optimizer. Even if today the optimizer does not make full use of all available constraint data, it is likely that in future releases of the database system, that constraint data will become used for more sophisticated processing by the kernel. The earlier example of the confusion over multiple shipping and billing addresses is a further example of the way semantic information", + "source": "The Art of SQL.pdf", + "chunk_id": 24 + }, + { + "text": "of all available constraint data, it is likely that in future releases of the database system, that constraint data will become used for more sophisticated processing by the kernel. The earlier example of the confusion over multiple shipping and billing addresses is a further example of the way semantic information is lost to the database by a www.it-ebooks.info 18 C H A P T E R O N E fundamentally weak design. This essential information must therefore be placed into an unpredictable number of application programs. “If the billing address is null, then the headquarters address applies” is a rule that is unknown to the database and must therefore be handled in the programs—note the use of the plural programs here! Once again, everything that is defined in the database is defined only once, thus guaranteeing that no program will use the data inconsistently. Implicit rules about, for example, address precedence must be coded into every program accessing the data. Because these implicit rules are totally arbitrary, it is not impossible at all that in some cases the billing address will be the shipping address, and not the headquarters address. Data semantics belong in the DBMS, not in the application programs. The Dangers of Excess Flexibility As always, pushing a line of reasoning to the limits (and often past them) can result in a monument to human madness. A great favorite with third-party software editors is the “more-flexible-than-thou” construct, in which most data of interest is stored in some general purpose table, with equally general purpose attributes such as: entity_id, attribute_id, attribute_value. In this “design,” everything is stored as a character string into attribute_value. The design certainly avoids the risk of having null values. However, the proponents of this type of design usually store the mandatory attributes in attribute_ value as well. Their mantra, by the way, is usually that this design approach makes it easy to add new attributes whenever they are needed. Without commenting on the quality of a design that makes it necessary to anticipate the necessarily haphazard addition of attributes, let’s just remark that it’s all very nice to store data, but usually, somehow, one day you will have to retrieve and process that same data (if data retrieval is not being planned, there is something seriously wrong somewhere). Adding a column to a table really pales into insignificance when compared to writing a program to do something useful with the new bits of information that you are given to manage (as enthusiasts that praise the flexibility of the Extensible Markup Language [XML] are bound to understand). The database cost of such pseudoflexibility rockets sky-high. Your database integrity is totally sacrificed, because you can hardly have a weaker way of typing your data. You cannot have any referential integrity. You cannot, in fact, have any type of declarative constraints. The simplest query becomes a monstrous join, in which the “value table” is joined 10, 15, or 20 times to the very same entity, depending on the", + "source": "The Art of SQL.pdf", + "chunk_id": 25 + }, + { + "text": "a weaker way of typing your data. You cannot have any referential integrity. You cannot, in fact, have any type of declarative constraints. The simplest query becomes a monstrous join, in which the “value table” is joined 10, 15, or 20 times to the very same entity, depending on the number of attributes one wants to select. Needless to say, even the cleverest optimizer is at a loss on such a www.it-ebooks.info L A Y I N G P L A N S 19 query, and performance is what one should expect—dismal. (You can try to improve the performance of such a query as described in Chapter 11, but the SQL code is not a pretty sight.) By comparison, the most inept campaign of military history looks like a masterpiece of strategic planning. True design flexibility is born of sound data-modeling practices. The Difficulties of Historical Data Working with historical data is an extremely common condition—the process of valuation, or specifying the price of goods or a service at a particular point in time, is based on historical data—but one of the really difficult issues of relational design is the handling of data that is associated with some period (as opposed to point) of time. There are several ways to model historical data. Let’s assume that we want to record the successive prices of goods identified by some article_id. An obvious way to do so is to store the following items: (article_id, effective_from_date, price) where effective_from_date is the date when the new price takes effect, and the primary key of the historical table is (article_id, effective_from_date). Logically correct, this type of model is rather clumsy to use when working with current data, which in many cases will be our main concern. How are we going to identify the current value? It’s the one associated with the highest effective_from_date, and it will be retrieved by running a query looking like: select a.article_name, h.price from articles a, price_history h where a.article_name = some_name and h.article_id = a.article_id and h.effective_from_date = (select max(b.effective_from_date) from price_history b where b.article_id = h.article_id) Executing this query requires two passes over the same data: one in the inner query to identify which is the most recent date we have for a given article, and one in the outer query to return the price from a row that we have necessarily hit in the inner query (Chapter 6 talks about special functions implemented by some DBMS systems that can avoid, to some extent, multiple passes). Executing repeated queries following this pattern can prove very costly. www.it-ebooks.info 20 C H A P T E R O N E However, the choice of how to register the validity period for a price is arbitrary. Instead of storing the effective date from which the price applies, why not store the “end date” (e.g., the last date on which the current price prevails), identifying the time intervals by their upper bound instead of by their lower bound? This new approach may look", + "source": "The Art of SQL.pdf", + "chunk_id": 26 + }, + { + "text": "a price is arbitrary. Instead of storing the effective date from which the price applies, why not store the “end date” (e.g., the last date on which the current price prevails), identifying the time intervals by their upper bound instead of by their lower bound? This new approach may look like an attractive solution. You have two ways to define current values—either that the end date is undefined, which looks neat but isn’t necessarily a good idea, or that the end date is something like December 31, 3000. It’s quite obvious that looking for the price of an article as of December 31, 3000 will take you directly to the row you want, in a single pass. Definitely attractive. Is this the perfect solution? Not quite. There may be some practical worries with the optimizer, which I discuss in Chapter 6, but there is also a major logical issue: prices, as any consumer knows, rarely stay constant, and price increases are not usually decided instantly (financial environments may be something different). What happens when, for example, in October, new prices are decided for the next year and duly recorded in the database? What we get in our valuation table are two records for each item: one stating the current price, valid until December 31, and one giving the price that will be applied from January 1. If we store the first date when the price applies we will have one row with an effective_from_date in the past (for instance January 1 of the current year) and another one in the future (say, the next January 1). In effect, what will define the current price is not the highest date, but the highest date before today (returned in Oracle by the system function sysdate). The preceding query needs to be modified only slightly: select a.article_name, h.price from articles a, price_history h where a.article_name = some_name and h.article_id = a.article_id and h.effective_from_date = (select max(b.effective_from_date) from price_history b where b.article_id = h.article_id and b.effective_from_date <= sysdate) If we store the last day when the price applies, we will have one row with an end_date set to December 31 and another with end_date set either to null or doomsday. Expressing that we want the price for which the end_date is the smallest date after the current date is no obvious improvement on the query just shown. Denormalization is of course a possible solution—one can imagine storing both the date when a price becomes effective and the date when it ceases to be, or one could also argue for storing the effective_from_date and the number of days for which the effective_from_ date price applies. This could allow using either the start or the end of the period, as best suits the query. www.it-ebooks.info L A Y I N G P L A N S 21 Denormalization always implies taking a risk with data integrity—a minor date entry error can leave black holes when no price is defined. You can of course minimize the risk", + "source": "The Art of SQL.pdf", + "chunk_id": 27 + }, + { + "text": "of the period, as best suits the query. www.it-ebooks.info L A Y I N G P L A N S 21 Denormalization always implies taking a risk with data integrity—a minor date entry error can leave black holes when no price is defined. You can of course minimize the risk by adding more checks when data is inserted or updated, but there is always a performance penalty associated with such checks. Another possible solution is to have a current table and a historical table and plan a migration of rows from current to historical when prices change. This approach can suit some kinds of applications, but may be complicated to maintain. Moreover, the “pre-recording” of future prices fits rather badly into the picture. In practice, particular storage techniques such as partitioning, which I discuss in Chapter 5, will come to the rescue, making constructs such as the one using the effective_from_date less painful than they might otherwise have been, especially for mass processing. But before settling for one solution, we must acknowledge that valuation tables come in all shapes and sizes. For instance, those of telecom companies, which handle tremendous amounts of data, have a relatively short price list that doesn’t change very often. By contrast, an investment bank stores new prices for all the securities, derivatives, and any type of financial product it may be dealing with almost continuously. A good solution in one case will not necessarily be a good solution in another. Handling data that both accumulates and changes requires very careful design and tactics that vary according to the rate of change. Design and Performance It is flattering (and a bit frightening too) to performance specialists to see the faith in their talents devotedly manifested by some developers. But, at the risk of repeating myself, I must once again stress what I said in the introduction to this book: tuning is about getting the best possible performance, now. When we develop, we must have a different mindset and not think “let’s code it, and then have a specialist tune it later in production.” The impact of tuning on the structure of programs is usually nil, and on queries, often minimal once the big mistakes have been corrected. There are indeed two aspects to this matter: • One aspect of tuning is the improvement of the overall condition of the system, by set- ting some parameters in accordance with the current resources in terms of CPU power, memory available, and I/O subsystems, and sometimes taking advantage of the physical implementation of the DBMS. This is a highly technical task, which may indeed improve the performance of some processes by a significant factor, but rarely by more than 20 or 30 percent unless big mistakes were made. www.it-ebooks.info 22 C H A P T E R O N E • The other aspect of tuning is the modification of specific queries, a practice that may, unfortunately, expose the limitations of the query optimizer and changes of behavior between", + "source": "The Art of SQL.pdf", + "chunk_id": 28 + }, + { + "text": "20 or 30 percent unless big mistakes were made. www.it-ebooks.info 22 C H A P T E R O N E • The other aspect of tuning is the modification of specific queries, a practice that may, unfortunately, expose the limitations of the query optimizer and changes of behavior between successive DBMS releases. That is all there is to it. In my view, adding indexes doesn’t really belong to the tuning of production databases (even if some tuning engagements are sometimes a matter of reviewing and correcting the indexing scheme for a database). Most indexes can and must be correctly defined from the outset as part of the designing process, and performance tests should resolve any ambiguous cases. Performance is no more a question of making a couple of queries faster than war is a question of winning a couple of battles. You can win a battle and lose the war. You can tune your queries and nevertheless have an application with dismal performance that nobody will want to use, except at gunpoint. Your database and programs, as well as your SQL queries, must all be properly designed. A functionally correct design is not enough. Performance must be incorporated into the design—and down-stream tuning provides for that little surplus of power that can provide peace of mind. The single largest contributory factor to poor performance is a design that is wrong. Processing Flow Besides all the questions addressed earlier in this chapter, the operating mode is also a matter that may have significant impact on the working system. What I mean by operating mode is whether data should be processed asynchronously (as is the case with batch programs) or synchronously (as in a typical transactional program). Batch programs are the historical ancestors of all data processing and are still very much in use today even if no longer very fashionable; synchronous processing is rarely as necessary as one might think. However, the improvement of networks and the increase in bandwidth has led to the “global reach” of an increasing number of applications. As a result, shutting down your online transaction processing (OLTP) application running in the American Midwest may become difficult because of East Asian users connected during one part of the Midwestern night and European users connected during the other part. Batch programs can no longer assume that they are running on empty machines. Moreover, ever- increasing volumes of data may require that incoming data is processed immediately rather www.it-ebooks.info L A Y I N G P L A N S 23 than being allowed to accumulate into unmanageably large data sets. Processing streams of data may simply be the most efficient way to manage such quantities. The way you process data is not without influence on the way you “think” of your system, especially in terms of physical structures—which I talk about more in Chapter 5. When you have massive batch programs, you are mostly interested in throughput—raw efficiency, using as much of the hardware resources as possible.", + "source": "The Art of SQL.pdf", + "chunk_id": 29 + }, + { + "text": "you process data is not without influence on the way you “think” of your system, especially in terms of physical structures—which I talk about more in Chapter 5. When you have massive batch programs, you are mostly interested in throughput—raw efficiency, using as much of the hardware resources as possible. In terms of data processing, a batch program is in the realm of brute force. When you are processing data on the fly, most activity will be small queries that are going to be repeatedly executed a tremendous number of times. For such queries, performing moderately well is not good enough—they have to perform at the maximum possible efficiency. With an asynchronous program, it is easy to notice that something is wrong (if not always easy to fix): it just takes too long to complete. With synchronous processing, the situation is much more subtle, because performance problems usually show up at the worst moment, when there are surges of activity. If you are not able to spot weaknesses early enough, your system is likely to let you down when your business reaches maximum demand levels—the very worst time to fail. A data model is not complete until consideration has also been taken of data flow. Centralizing Your Data For all the talk about grids, clustered servers, and the like, spreading data across many servers means adding a considerable amount of complexity to a system. The more complicated a structure—any type of structure—the less robust it is. Technological advance does indeed slowly push up the threshold of acceptability. In the eighteenth century, clocks indicating the minutes were considered much less reliable than those indicating only the hour, and much more reliable than those showing the day in the month or the phases of the moon. But nevertheless, try to keep the theater of operations limited to that which is strictly required. Transparent references to remote data are performance killers, for two reasons. First, however “transparent” it may look, crossing more software layers and a network has a heavy cost. To convince yourself, just run a procedure that inserts a few thousands rows into a local table, and another one doing the very same thing across—for instance, an Oracle database link, even on the same database—you can expect performance to be in the neighborhood of five times slower, if not worse, as you see demonstrated in Chapter 8. Second, combining data from several sources is extremely difficult. When comparing data from source A to data from source B, you have no choice other than literally copying the www.it-ebooks.info 24 C H A P T E R O N E data from A to B or the reverse. Transfer is one significant overhead. Data drawn from its own carefully constructed environment no longer benefits from the planning which went into establishing that environment (carefully thought-out physical layout, indexes, and so forth). Instead, that data lands in some temporary storage—in memory if the amount of data transferred is modest, otherwise on disk. The management", + "source": "The Art of SQL.pdf", + "chunk_id": 30 + }, + { + "text": "Data drawn from its own carefully constructed environment no longer benefits from the planning which went into establishing that environment (carefully thought-out physical layout, indexes, and so forth). Instead, that data lands in some temporary storage—in memory if the amount of data transferred is modest, otherwise on disk. The management of temporary storage is another major overhead. In a case where nested loops would be, in theory, the most efficient way to proceed when querying local data, an optimizer is left with two unattractive possibilities when some of the data is remotely located: • Using nested loops and incurring high overhead with each iteration • Sucking the remote data in, and then operating against the local copy, which has left all indexes behind Optimizers can be forgiven for not performing at their best under these circumstances. When it comes to the placement of major data repositories, some of the art is simply keeping a balance. If your company operates worldwide, keeping all the data at one location is unlikely to be a popular solution with people who live and work at the antipodes. Hitting a remote server is certainly no problem when surfing the Internet—it is quite another matter when using an application intensely. It’s not a question of bandwidth, it’s a question of light speed, for which, unfortunately, not much improvement can be expected from technological progress. Whatever you do, issuing a query against a server located on another continent adds another quarter or half second to response times, depending on the continent—and this at the best of times. If you need everyone to have the global picture, replication solutions and products (as opposed to remote access) should be contemplated. For each group of players, keep their own chessboard right at hand—don’t make players reach. The nearer you are to your data, the faster you can get at it! System Complexity Other points to keep in mind when designing are what will happen if some piece of hardware breaks (for example, a disk controller) or if some mistake is made (for instance, the same batch program is applied twice). Even if your administrators are wizards who are doing night shifts to bring everything back on course by dawn, transfer rates are limited; the recovery of a huge database always takes a lot of time. “Spare” backup databases maintained in synch (or with some slight delay) may help. But backup databases will not be of any use in the case of a program inadvertently run twice, www.it-ebooks.info especially if the synchronization delay is shorter than the execution time of the program. What is already complicated with one database becomes a nightmare with several related databases, because you must be perfectly certain that all the databases are correctly synchronized after any recovery, to avoid any risk of data corruption. This particular point of recovery is often a bone of contention between developers and database administrators, because developers tend to consider, not unreasonably, that backups and recoveries belong to administrators, while administrators", + "source": "The Art of SQL.pdf", + "chunk_id": 31 + }, + { + "text": "certain that all the databases are correctly synchronized after any recovery, to avoid any risk of data corruption. This particular point of recovery is often a bone of contention between developers and database administrators, because developers tend to consider, not unreasonably, that backups and recoveries belong to administrators, while administrators point out, logically, that if they can guarantee that the container is in working order, they have no idea about the status of the contents. Indeed, any functional check in case of recovery should not be forgotten by developers. The more complicated the overall design, the more important it is for developers to keep in mind the constraints of operations. Database systems are joint ventures; they need the active and cooperative participation of users, administrators, and developers. The Completed Plans We have reviewed the basic foundations for laying plans in constructing a database system. We have reviewed the fundamentals of data modeling, and in particular the broad steps involved in normalizing data to third normal form. We have then proceeded to review a number of scenarios, in which a faulty design can be identified as the road to disaster. Most examples in this chapter come directly from or are inspired by cases I have encountered in some big companies. And it is always striking to consider how much energy and intelligence can be wasted trying to solve performance problems that are born from the ignorance of elementary design principles. Such performance issues need not be present, yet they are quite common and often made worse by further denormalization of what is already a questionable design, on the unassailable grounds of “performance improvement.” One query may, in fact, run much faster, but unfortunately, the nightly batch program now takes twice as long. In this way, and almost without being noticed, a full information system is built on a foundation of sand. Successful data modeling is the disciplined application of what are, fundamentally, simple design principles. www.it-ebooks.info www.it-ebooks.info Chapter 2. C H A P T E R T W O Waging War Accessing Databases Efficiently Il existe un petit nombre de principes fondamentaux de la guerre, dont on ne saurait s’écarter sans danger, et dont l’application au contraire a été presque en tous temps couronnée par le succès. There exist a small number of fundamental principles of war, which it is dangerous to ignore: indeed, following these principles has almost invariably led to success. —Général Antoine-Henri de Jomini (1779–1869) Précis de l’Art de la Guerre www.it-ebooks.info 28 C H A P T E R T W O A nybody who has ever been involved in the switch from development to production of a critical system knows how much it can feel like the noise and tumult of battle. Very often, a few weeks before D-Day, performance tests will show that the new system is going to fall short of expectations. Experts are brought in, SQL statements are fine-tuned, and database and system administrators are called to contribute to a succession of crisis", + "source": "The Art of SQL.pdf", + "chunk_id": 32 + }, + { + "text": "the noise and tumult of battle. Very often, a few weeks before D-Day, performance tests will show that the new system is going to fall short of expectations. Experts are brought in, SQL statements are fine-tuned, and database and system administrators are called to contribute to a succession of crisis meetings. Finally, performance vaguely comparable to the previous system is obtained on hardware that is now twice as expensive as the original installation. Tactics are often used as a substitute for a strategic approach. The latter demands the adoption of a sound overall architecture and design. As in war, the basic principles here are also few, but too often ignored. Architectural mistakes can prove extremely costly, and the SQL programmer must enter the battle fully prepared, knowing where to go and how to get there. In this chapter, we are going to review the key goals that will increase our chances of success in writing programs that access databases efficiently. Query Identification For centuries, the only means that a general had to check the progress of his troops during the heat of battle was to observe the position of his units as indicated by the color of the soldiers’ uniforms and the flags they were carrying. When some process in the database environment is consuming an inordinate amount of CPU, it is often possible to identify which piece of SQL code is actually running. But it is very often much more difficult, especially in a large and complicated system that includes dynamically built queries, to identify which precise part of a given application issued that statement and needs reviewing. Despite the fact that many products have good monitoring facilities, it is sometimes surprisingly difficult to relate an SQL statement to its broader environment. Therefore, you should adopt the habit of identifying your programs and critical modules whenever possible by inserting comments into your SQL to help identify where in the programs a given query is used. For instance: /* CUSTOMER REGISTRATION */ select blah ... These identifying comments can be important and helpful in subsequently tracking down any erroneous code. They can also be helpful when trying to determine how much load is put on a server by a single application, especially when some localized increase in activity is expected and when you are trying to assess whether the current hardware can absorb the surge. Some products have special registration facilities that can spare you the admittedly tedious step of commenting each and every statement. Oracle’s dbms_application_info package allows you to register a program using a 48-character module name, a 32-character action www.it-ebooks.info W A G I N G W A R 29 name, and a 64-character client information field. The content of those fields is left to your discretion. In an Oracle environment, you can use this package to keep track not only of which application is running, but also what that application is doing at any given time. This is because you can easily query the information that", + "source": "The Art of SQL.pdf", + "chunk_id": 33 + }, + { + "text": "The content of those fields is left to your discretion. In an Oracle environment, you can use this package to keep track not only of which application is running, but also what that application is doing at any given time. This is because you can easily query the information that your application passes to the package through the Oracle V$ dynamic views that show what is currently happening in memory. Identifiable statements make the identification of performance issues easier. Stable Database Connections A new database connection can be created quickly and easily, but this ease can disguise the high cost of making repeated connections. You must manage the use of database connections with great care. The consequences of allowing multiple connections to occur, perhaps hidden within an application, can be substantial, as the next example illustrates. Some time ago I came across an application in which numerous small files of up to an arbitrary maximum of 100 lines were being processed. Each line in these small text files contained both data and the identification of the database instance into which that data had to be loaded. In this particular case, there was a single server, but the principle being illustrated is exactly the same as if there were a hundred database instances. The process for each file was coded as follows: Open the file Until the end of file is reached Read a row Connect to the server specified by the row Insert the data Disconnect Close the file This process worked quite satisfactorily, except for the occasional circumstance in which a large number of small files would arrive in a very short space of time, and at a rate greater than the ability of the application to process them. This resulted in a substantial backlog, which took considerable time to clear. I explained the problem of performance degradation as a consequence of frequent connection and disconnection to the customer with the help of a simple program (written in C) emulating the current application. Table 2-1 gives the results from that demonstration. www.it-ebooks.info 30 C H A P T E R T W O NOTE The program generating the results in Table 2-1 used a conventional insert statement. I mentioned in passing to the customer the existence of direct-loading techniques that are even faster. The demonstration showed the importance of trying to minimize the number of separate database connections that had to be made. Thus, there was an obvious and enormous advantage in applying a simple check to determine whether the “next” insert was into the same database as the previous one. The rationalization could go further, as the number of database instances was of course finite. You could likely achieve further performance gain by setting up an array of handlers, one for each specific database connection, opening a new connection each time a new database is referenced, and thus connecting at most once to each database. As Table 2-1 shows, the simple technique of connecting only once (or a", + "source": "The Art of SQL.pdf", + "chunk_id": 34 + }, + { + "text": "achieve further performance gain by setting up an array of handlers, one for each specific database connection, opening a new connection each time a new database is referenced, and thus connecting at most once to each database. As Table 2-1 shows, the simple technique of connecting only once (or a very few times) improved performance by a factor of more than 200 with very little additional effort. Of course, this was an excellent opportunity to show that minimizing the number of round-trips between a program and the database kernel, using arrays and populating them with incoming data, can also lead to spectacular improvements in performance. By inserting several rows at once, the throughput could be radically improved—by another factor of five. The results in Table 2-1 demonstrate that improvements in the process could reach a modest factor of 1,200. Why such dramatic improvement? The reason for the first and biggest improvement is that a database connection is fundamentally a “heavy,” or high-resource operation. In the familiar client/server environment (which is still very widely used), the simple connection routine hides the fact that the client program first has to establish contact with a listener program on a remote machine; and then, depending on whether shared servers are being used on this machine, the lis- tener must either spawn another process or thread and make it run some data- base kernel program, or hand the request, directly or indirectly, to an existing server process. Whatever the number of system operations (process spawning or thread creation and the start of executions) your database system will need to create a new environment Test Results Connect/disconnect for each line in turn 7.4 lines loaded per second Connect once, all candidate lines individually inserted 1,681 lines loaded per second Connect once, all candidate lines inserted in arrays of 10 lines 5,914 lines loaded per second Connect once, all candidate lines inserted in arrays of 100 lines 9,190 lines loaded per second TABLE 2-1. Result of connect/disconnect performance tests www.it-ebooks.info W A G I N G W A R 31 for each session, to keep track of what it does. Your DBMS will need to check the password provided against the encrypted password of the account for which a new session is to be created. Your DBMS may also have to execute the code for some logon trigger. It may have to execute some initialization code for stored procedures or packages the first time they are called. This does not include the base machine handshaking protocols between client and server processes. This is the reason tech- niques that allow the upkeep of permanent connections to the database, such as connection pooling, are so important to performance. The reason for the second improvement is that a round-trip between your program (and even a stored procedure) and the database also has its costs. Even when you are connected and maintain a connection, context switches between your program and the DBMS kernel take their toll. Therefore if your DBMS allows", + "source": "The Art of SQL.pdf", + "chunk_id": 35 + }, + { + "text": "reason for the second improvement is that a round-trip between your program (and even a stored procedure) and the database also has its costs. Even when you are connected and maintain a connection, context switches between your program and the DBMS kernel take their toll. Therefore if your DBMS allows you to communicate through an array interface of some kind, you should not hesitate to use it. If, as sometimes happens, the array interface is implicit (the application program interface [API] uses arrays when you use only scalar values), it is wise to check the default array size that is used and perhaps tailor it to your particular needs. And of course, any row-by-row logic suffers the same context-switch mechanisms and is a cardinal sin—as you shall have several opportunities to see throughout this chapter. Database connections and round-trips are like Chinese Walls— the more you have, the longer it takes to receive the correct message. Strategy Before Tactics Strategy defines the tactics, not the other way round. A skillful developer doesn’t think of a process in terms of little steps, but in terms of the final result. The most efficient way to obtain that result may not be to proceed in the order specified in the business rules, but rather to follow a less obvious approach. The following example will show how paying too much attention to the procedural processes within a business can distract ones’ attention from the most efficient solution. Some years ago I was given a stored procedure to try to optimize; “try” is the operative word here. Two attempts at optimization had already been made, once by the original authors, and secondly by a self-styled Oracle expert. Despite these efforts, this procedure was still taking 20 minutes to run, which was unacceptable to the users. The purpose of the procedure was to compute quantities of raw materials to be ordered by a central factory unit, based on existing stocks and on orders that were coming from a number of different sources. Basically, the data from several identical tables for each data source had to be aggregated inside one master table. The procedure consisted of a www.it-ebooks.info 32 C H A P T E R T W O succession of similar statements simplified as follows. First, all data from each distinct source table were inserted into the single master table. Second, an aggregate/update was applied to each instance of raw material in that master table. Finally, the spurious data not relevant to the aggregate result was deleted from the table. These stages were repeated in sequence inside the procedure for every distinct source table. None of the SQL statements were particularly complex, and none of them could be described as being particularly inefficient. It took the better half of a day to understand the process, which eventually prompted the question: why was this process being done in multiple steps? A subquery in a from clause with a union operator would allow the aggregation of all the", + "source": "The Art of SQL.pdf", + "chunk_id": 36 + }, + { + "text": "could be described as being particularly inefficient. It took the better half of a day to understand the process, which eventually prompted the question: why was this process being done in multiple steps? A subquery in a from clause with a union operator would allow the aggregation of all the various sources. A single select statement could provide in one step the result set that had to be inserted into the target table. The difference in performance was so impressive—from 20 minutes down to 20 seconds—that it took some time to verify that the final result was indeed identical to that previously obtained. Extraordinary skills were not required to achieve the tremendous performance improvement just described, but merely an ability to think outside the box. Previous attempts to improve this process had really been hindered by the participants allowing themselves to get too close to the problem. One needed to take a fresh look, to stand back, and try to see the bigger picture. The key questions to ask were “What do we have when we enter this procedure?” and “Which result do we want when we return from it?” Together with some fresh thinking, the answers to those questions led to a dramatically improved process. Stand back from your problem to get the wider picture before plunging into the details of the solution. Problem Definition Before Solution A little knowledge can be a dangerous thing. Frequently, people may have read or heard about new or unusual techniques—which in some cases can indeed be quite interesting— and then they will try to fit their problem to one of these new solutions. Ordinary developers and architects often jump quickly on to such “solutions,” which often turn out to be at the root of many subsequent problems. At the top of the list of ready-made solutions, we usually meet denormalization. Blissfully unaware of the update nightmare that it turns out to be in practice, denormalization advocates often suggest it at an early stage in the hunt for “performance”—and in fact often at a point in the development cycle when better design (or learning how to use www.it-ebooks.info W A G I N G W A R 33 joins) is still an option. A particular type of denormalization, the materialized view, is also often seen as being something of a panacea. (Materialized views are sometimes referred to as snapshots, a less impressive term, but one that is closer to the sad reality: copies of data at one point in time.) This is not to say that sometimes, as a last resort option, theoretically questionable techniques cannot be used. To quote Franz Kafka: “Logic is doubtless unshakable, but it cannot withstand a man who wants to go on living.” But the immense majority of problems can be solved using fairly traditional techniques in an intelligent manner. Learn first how to get the best of simple, traditional techniques. It’s only when you can fully master them that you will be able to appreciate their limitations,", + "source": "The Art of SQL.pdf", + "chunk_id": 37 + }, + { + "text": "wants to go on living.” But the immense majority of problems can be solved using fairly traditional techniques in an intelligent manner. Learn first how to get the best of simple, traditional techniques. It’s only when you can fully master them that you will be able to appreciate their limitations, and then to truly be able to judge the potential advantage (if any) of new technical solutions. All technological solutions are merely means to an end; the great danger for the inexperienced developer is that the attractions of the latest technology become an end in themselves. And the danger is all the greater for enthusiastic, curious, and technically minded individuals! Foundations before Fashion: learn your craft before playing with the latest tools. Stable Database Schema The use of data definition language (DDL) to create, alter, or drop database objects inside an application is a very bad practice that in most cases should be banned. There is no reason to dynamically create, alter, or drop objects, with the possible exception of partitions—which I describe in Chapter 5—and temporary tables that are known to the DBMS to be temporary tables. (We shall also meet another major exception to this rule in Chapter 10.) The use of DDL is fundamentally based on the core database data dictionary. Since this dictionary is also central to all database operations, any activity on it introduces global locks that can have massive performance consequences. The only acceptable DDL operation is truncate table, which is a very fast way of emptying a table of all rows (without the protection of rollback recovery, remember!). Creating, altering, or dropping database objects belong to application design, not to regular operations. www.it-ebooks.info 34 C H A P T E R T W O Operations Against Actual Data Many developers like to create temporary work tables into which they extract lists of data for subsequent processing, before they begin with the serious stuff. This approach is often questionable and may reflect an inability to think beyond the details of the business processes. You must remember that temporary tables cannot offer storage options of the same degree of sophistication as permanent tables (you see some of these options in Chapter 5). Their indexing, if they are indexed, may be less than optimal. As a result, queries that use temporary tables may perform less efficiently than well-written statements against permanent tables, with the additional overhead of having to fill temporary tables as a prerequisite to any query. Even when the use of temporary tables is justified, they should never be implemented as permanent tables masquerading as work tables if the number of rows to be stored in them is or can be large. One of the problems lies in the automated collection of statistics: when statistics are not collected in real time, they are typically gathered by the DBMS at a time of zero or low activity. The nature of work tables is that they will probably be empty at such slack times, thus giving", + "source": "The Art of SQL.pdf", + "chunk_id": 38 + }, + { + "text": "the problems lies in the automated collection of statistics: when statistics are not collected in real time, they are typically gathered by the DBMS at a time of zero or low activity. The nature of work tables is that they will probably be empty at such slack times, thus giving a wholly erroneous indicator to the optimizer. The result of this incorrect, and biased, statistical data can be totally inappropriate execution plans that not surprisingly lead to dismal performance. If you really have to use temporary storage, use tables that the database can recognize as being temporary. Temporary work tables mean more byte-pushing to less suitable storage. Set Processing in SQL SQL processes data in complete sets. For most update or delete operations against a database—and assuming one is not operating against the entire table contents—one has to define precisely the set of rows in that table that will be affected by the process. This defines the granularity of the impending process, which may be described as coarse if a large number of rows will be affected or as fine if only few rows will be involved. Any attempt to process a large amount of data in small chunks is usually a very bad idea and can be massively inefficient. This approach can be defended only where very extensive changes will be made to the database which can, first, consume an enormous amount of space for storing prior values in case of a transaction rollback, and second, take a very long time to rollback if any attempted change should fail. Many people would argue that where very considerable changes are to be made, regular commit statements should be scattered throughout the data manipulation language (DML) code. However, www.it-ebooks.info W A G I N G W A R 35 regular commit statements may not help when resuming a file upload that has failed. From a strictly practical standpoint, it is often much easier, simpler, and faster to resume a process from the start rather than try to locate where and when the failure occurred and then to skip over what has already been committed. Concerning the size of the log required to rollback transactions in case of failure, it can also be argued that the physical database layout has to accommodate processes, and not that processes have to make do with a given physical implementation. If the amount of undo storage that is required is really enormous, perhaps the question should be raised as to the frequency with which changes are applied. It may be that switching from massive monthly updates to not-so-massive weekly ones or even smaller daily ones may provide an effective solution. Thousands of statements in a cursor loop for endless batch processing, multiple statements applied to the same data for users doomed to wait, one swoop statement to outperform them all. Action-Packed SQL Statements SQL is not a procedural language. Although procedural logic can be applied to SQL, such approaches should be used with caution. The confusion", + "source": "The Art of SQL.pdf", + "chunk_id": 39 + }, + { + "text": "loop for endless batch processing, multiple statements applied to the same data for users doomed to wait, one swoop statement to outperform them all. Action-Packed SQL Statements SQL is not a procedural language. Although procedural logic can be applied to SQL, such approaches should be used with caution. The confusion between procedural and declarative processing is most frequently seen when data is required to be extracted from the database, processed, and then re-inserted back into the database. When a program— or a function within a program—is provided with some input value, it is all too common to see that input value used to retrieve one or several other values from the database, followed by a loop or some conditional logic (usually if...then...else) being applied to yet other statements applied to the database. In most cases, this behavior is the result of deeply ingrained bad habits or a poor knowledge of SQL, combined with a slavish obsession with functional specifications. Many relatively complex operations can be accomplished in a single SQL statement. If the user provides some value, try to get the result set that is of interest without decomposing the process into multiple statements fetching intermediate results of only minimal relevance to the final output. There are two main reasons for shunning procedural logic in SQL: Any access to the database means crossing quite a number of software layers, some of which may include network accesses. Even when no network is involved, there will be interprocess communications; more accesses mean more function calls, more bandwidth, and more time wait- ing for the answer. As soon as those calls are repeated a fair number of times, the impact on process performance can become distinctly perceptible. www.it-ebooks.info 36 C H A P T E R T W O Procedural means that performance and future maintenance burdens fall to your program. Most database systems incorporate sophisticated algorithms for executing opera- tions such as joins, and for transforming queries so as to execute them in a more efficient way. Cost-based optimizers (CBOs) are complex pieces of software that have sometimes grown from being totally unusable when originally introduced to becoming mature products, capable of giving excellent results in most cases. A good CBO can be extremely efficient in choosing the most suitable execution plan. However, the scope of operation of the CBO is the SQL statement, noth- ing more. By doing as much as possible in a single statement, you shift the bur- den of achieving the best possible performance from your program to the DBMS kernel. You enable your program to take advantage of any improvement to the DBMS code, and therefore you are indirectly shifting a large part of the future maintenance of your program to the DBMS vendor. As ever, there will be exceptions to the general rule that you should shun procedural logic, where in some cases procedural logic may indeed help make things faster. The monstrous all-singing-and-dancing SQL statement is not always a model for efficiency. However, the procedural", + "source": "The Art of SQL.pdf", + "chunk_id": 40 + }, + { + "text": "of your program to the DBMS vendor. As ever, there will be exceptions to the general rule that you should shun procedural logic, where in some cases procedural logic may indeed help make things faster. The monstrous all-singing-and-dancing SQL statement is not always a model for efficiency. However, the procedural logic that glues together successive statements that work on the same data and hit the same rows can often be pushed into one SQL statement. The CBO can consider a single statement that stays close to the sound rules of the relational model as a whole and can execute it in the most efficient way. Leave as much as you possibly can to the database optimizer to sort out. Profitable Database Accesses When you plan a visit to several shops, the first step is to decide what purchases have to be made at each shop. From this point, a trip is planned that will ensure minimum repetitive walking backward and forward between different shops. The first shop is then visited, the purchase completed, and then the next closest shop is visited. This is only common sense, and yet the principle underlying this obvious approach is not seen in the practical implementation of many database programs. When several pieces of information are required from a single table—even if it appears as if they are “unrelated” (which in fact is unlikely to be the case)—it is highly inefficient to retrieve this data in several separate visits to the database. For example, do not fetch row values column by column if multiple columns are required: do the work in one operation. www.it-ebooks.info W A G I N G W A R 37 Unfortunately, good object-oriented (OO) practice makes a virtue out of defining one method for returning each attribute. But do not confuse OO methods with relational database processing. It is a fatal mistake to mix relational and object-oriented concepts and to consider tables to be classes with columns as the attributes. Maximize each visit to the database to complete as much work as can reasonably be achieved for every visit. Closeness to the DBMS Kernel The nearer to the DBMS kernel your code can execute, the faster it will run. This is where the true strength of the database lies. For example, several database management products allow you to extend them by adding new functions, which can sometimes be written in comparatively low-level languages such as C. The snag with a low-level language that manipulates pointers is that if you mishandle a pointer, you can end up corrupting memory. It would be bad enough if you were the only user affected. But the trouble with a database server is that, as the name implies, it can serve a large number of users: if you corrupt the server memory, you can corrupt the data handled by another, totally innocent program. As a consequence, responsible DBMS kernels run code in a kind of sandbox, where it can crash without taking everything with it in", + "source": "The Art of SQL.pdf", + "chunk_id": 41 + }, + { + "text": "name implies, it can serve a large number of users: if you corrupt the server memory, you can corrupt the data handled by another, totally innocent program. As a consequence, responsible DBMS kernels run code in a kind of sandbox, where it can crash without taking everything with it in its downfall. For instance, Oracle implements a complicated communication mechanism between external functions and itself. In some ways, this process is similar to that which controls database links, by which communication between two (or more) database instances on separate servers is managed. If the overall gain achieved by running tightly tailored C functions rather than stored PL/SQL procedures is greater than the costs of setting up an external environment and context-switching, use external functions. But do not use them if you intend to call a function for every row of a very large table. It is a question of balance, of knowing the full implications of the alternative strategies available to solve any given problem. If functions are to be used, try to always use those that are provided by the DBMS. It is not merely a matter of not reinventing the wheel: built-in functions always execute much closer to the database kernel than any code a third-party programmer can construct, and are accordingly far more efficient. Here is a simple example using Oracle’s SQL that will demonstrate the efficiencies to be gained by using Oracle functions. Let’s assume we have some text data that has been manually input and that contains multiple instances of adjacent “space” characters. We require a function that will replace any sequence of two or more spaces by a single space. www.it-ebooks.info 38 C H A P T E R T W O Ignoring the regular expressions available since Oracle Database 10g, our function might be written as follows: create or replace function squeeze1(p_string in varchar2) return varchar2 is v_string varchar2(512) := ''; c_char char(1); n_len number := length(p_string); i binary_integer := 1; j binary_integer; begin while (i <= n_len) loop c_char := substr(p_string, i, 1); v_string := v_string || c_char; if (c_char = ' ') then j := i + 1; while (substr(p_string || 'X', j, 1) = ' ') loop j := j + 1; end loop; i := j; else i := i + 1; end if; end loop; return v_string; end; / As a side note, 'X' is concatenated to the string in the inner loop to avoid testing j against the length of the string. There are alternate ways of writing a function to eliminate multiple spaces, which can make use of some of the string functions provided by Oracle. Here’s one alternative: create or replace function squeeze2(p_string in varchar2) return varchar2 is v_string varchar2(512) := p_string; i binary_integer := 1; begin i := instr(v_string, ' '); while (i > 0) loop v_string := substr(v_string, 1, i) || ltrim(substr(v_string, i + 1)); i := instr(v_string, ' '); end loop; return v_string; end; / www.it-ebooks.info W A G I N G", + "source": "The Art of SQL.pdf", + "chunk_id": 42 + }, + { + "text": "return varchar2 is v_string varchar2(512) := p_string; i binary_integer := 1; begin i := instr(v_string, ' '); while (i > 0) loop v_string := substr(v_string, 1, i) || ltrim(substr(v_string, i + 1)); i := instr(v_string, ' '); end loop; return v_string; end; / www.it-ebooks.info W A G I N G W A R 39 And here’s a third way to do it: create or replace function squeeze3(p_string in varchar2) return varchar2 is v_string varchar2(512) := p_string; len1 number; len2 number; begin len1 := length(p_string); v_string := replace(p_string, ' ', ' '); len2 := length(v_string); while (len2 < len1) loop len1 := len2; v_string := replace(v_string, ' ', ' '); len2 := length(v_string); end loop; return v_string; end; / When these three alternative methods are tested on a simple example, each behaves exactly as specified, and there is no visible performance difference: SQL> select squeeze1('azeryt hgfrdt r') 2 from dual 3 / azeryt hgfrdt r Elapsed: 00:00:00.00 SQL> select squeeze2('azeryt hgfrdt r') 2 from dual 3 / azeryt hgfrdt r Elapsed: 00:00:00.01 SQL> select squeeze3('azeryt hgfrdt r') 2 from dual 3 / azeryt hgfrdt r Elapsed: 00:00:00.00 Assume now that this operation of stripping out multiple spaces is to be called many thousands of times each day. You can use the following code to create and populate a test table with random data, by which you can examine whether there are differences in performance among these three space-stripping functions under a more realistic load: create table squeezable(random_text varchar2(50)) / declare i binary_integer; www.it-ebooks.info 40 C H A P T E R T W O j binary_integer; k binary_integer; v_string varchar2(50); begin for i in 1 .. 10000 loop j := dbms_random.value(1, 100); v_string := dbms_random.string('U', 50); while (j < length(v_string)) loop k := dbms_random.value(1, 3); v_string := substr(substr(v_string, 1, j) || rpad(' ', k) || substr(v_string, j + 1), 1, 50); j := dbms_random.value(1, 100); end loop; insert into squeezable values(v_string); end loop; commit; end; / This script creates a total of 10,000 rows in the test table (a fairly modest total when it is considered how many times some SQL statements are executed). The test can now be run as follows: select squeeze_func(random_text) from squeezable; When I ran this test, headers and screen display were all switched off. Getting rid of output operations ensured that the results reflected the space-reduction algorithm and not the time needed to display the results. The statements were executed several times to ensure that there was no caching effect. Table 2-2 shows the results on the test machine. Even though all functions can be called 10,000 times in under one second, squeeze3 is 1. 8 times as fast as squeeze1, and squeeze2 almost 2.2 times as fast. Why? Simply because PL/SQL is not “as close to the kernel” as is a SQL function. The performance difference may look like a tiny thing when functions are executed once in a while, but it can make quite a difference in a batch program—or on a heavily loaded", + "source": "The Art of SQL.pdf", + "chunk_id": 43 + }, + { + "text": "as fast. Why? Simply because PL/SQL is not “as close to the kernel” as is a SQL function. The performance difference may look like a tiny thing when functions are executed once in a while, but it can make quite a difference in a batch program—or on a heavily loaded OLTP server. Function Mechanism Time squeeze1 PL/SQL loop on chars 0.86 seconds squeeze2 instr() + ltrim( ) 0.48 seconds squeeze3 replace( ) called in a loop 0.39 seconds TABLE 2-2. Time to trim spaces from 10,000 rows www.it-ebooks.info W A G I N G W A R 41 Code loves the SQL kernel—the closer they get, the hotter the code. Doing Only What Is Required Developers often use count(*) for no purpose other than to implement an existence test. This usually happens as a result of a specification such as: If there are rows meeting a certain condition Then do something to them which immediately becomes: select count(*) into counter from table_name where if (counter > 0) then Of course in 90% of the cases the count(*) is totally unnecessary and superfluous, as in the above example. If an action is required to operate on a number of rows, just do it. If no row is affected, so what? No harm is done. Moreover, if the process to be applied to those hypothetical rows is complex, the very first operation will tell you how many of them were affected, either in a system variable (@@ROWCOUNT with Transact-SQL, SQL%ROWCOUNT with PL/SQL, and so forth), in a special field of the SQL Communication Area (SQLCA) when using embedded SQL, or through special APIs such as mysql_ affected_rows( ) in PHP. The number of processed rows is also sometimes directly returned by the function, which interacts with the database, such as the JDBC executeUpdate( ) method. Counting rows very often achieves nothing other than doubling your total search effort, because it applies a process twice to the same data. Further, do not forget that if your purpose is to update or insert rows (a frequent case when rows are counted first to check whether the key already exists), some database systems provide dedicated statements (for instance, Oracle 9i Database’s MERGE statement) that operate far more efficiently than you can ever achieve by executing redundant counts. There is no need to code explicitly what the database performs implicitly. www.it-ebooks.info 42 C H A P T E R T W O SQL Statements Mirror Business Logic Most database systems provide monitoring facilities that allow you to check statements currently being executed, as well as to monitor how many times they are executed. At the same time, you should have an idea of how many “business units” are being processed— activities such as orders or claims to be processed, customers to be billed, or anything else that makes sense to the business managers. You should review whether there is a reasonable (not absolutely precise) correlation between the two classes of activities. In other words, for", + "source": "The Art of SQL.pdf", + "chunk_id": 44 + }, + { + "text": "“business units” are being processed— activities such as orders or claims to be processed, customers to be billed, or anything else that makes sense to the business managers. You should review whether there is a reasonable (not absolutely precise) correlation between the two classes of activities. In other words, for a given number of customers, is the same number of activities being initiated against the database? If a query against the customers table is executed 20 times more than the number of customers being processed at the same time, it is a certainty that there is a problem somewhere. This situation would suggest that instead of going once to the table to find required information, repeated (and superfluous) visits are being made to the same rows in the same table. Check that your database activity is reasonably consistent with the business requirements currently being addressed. Program Logic into Queries There are several ways to achieve procedural logic in a database application. It’s possible to put some degree of procedurality inside an SQL statement (even if a statement should say what, and not how). Even when using a well-integrated host language within which SQL statements are embedded, it is still preferable to embed as much procedural logic as possible within an actual SQL statement, rather than in the host language. Of the two alternatives, embedding logic in the SQL statement will yield higher performance than embedding it in the application. Procedural languages are characterized by the ability to iterate (loops) and to perform conditional logic (if...then...else constructs). SQL doesn’t need looping, since by essence it operates on sets; all it requires is the ability to test logically for some conditions. Obtaining conditional logic breaks down into two components—IF and ELSE. Achieving IF is easy enough—the where condition provides the capability. What is difficult is to obtain the ELSE logic. For example, we may need to retrieve a set of rows, and then apply different transformations to different subsets. The case expression (Oracle has also long provided a functionally equivalent operator in decode()*) makes it easy to simulate some logic: it allows us to change on the fly the values that are returned to the result set by testing on row values. In pseudocode, the case construct operates like this:† * decode( ) is a bit more rudimentary than case and may require the use of additional functions such as sign( ) to obtain the same results. † There are two variants of the case construct; the example shown is the most sophisticated variant. www.it-ebooks.info W A G I N G W A R 43 CASE WHEN condition THEN WHEN condition THEN ... WHEN condition THEN ELSE END Comparing numerical values or dates is straightforward. With strings, functions such as Oracle’s greatest( ) or least( ) or MySQL’s strcmp( ) can be useful. It is also sometimes possible to add some logic to insert", + "source": "The Art of SQL.pdf", + "chunk_id": 45 + }, + { + "text": "WHEN condition THEN ELSE END Comparing numerical values or dates is straightforward. With strings, functions such as Oracle’s greatest( ) or least( ) or MySQL’s strcmp( ) can be useful. It is also sometimes possible to add some logic to insert statements, through multiple table inserts and conditional inserts,* and by using the merge statement. Don’t hesitate to use such statements if they are available with your DBMS. In other words, a lot of logic can be pushed into SQL statements; although the benefit may be small when executing only one of several statements, the gain can be much greater if you can manage to use case or merge or similar functionality to combine several statements into one. Wherever possible, try to embed your conditional logic within your SQL statements rather than in an associated host language. Multiple Updates at Once My basic assertion here is that successive updates to a single table are acceptable if they affect disjoint sets of rows; otherwise they should be combined. For example, here is some code from an actual application:† update tbo_invoice_extractor set pga_status = 0 where pga_status in (1,3) and inv_type = 0; update tbo_invoice_extractor set rd_status = 0 where rd_status in (1,3) and inv_type = 0; Two successive updates are being applied to the same table. Will the same rows be hit twice? There is no way to tell. The question is, how efficient are the search criteria? Any attribute with a name like type or status is typically a column with a totally skewed distribution. It is quite possible that both updates may result in two successive full scans of the same table. One update may use an index efficiently, and the second update may result in an unavoidable full table scan. Or, fortuitously, both may be able to make * Available, for instance, in Oracle since release 9.2. † Table names have been changed. www.it-ebooks.info 44 C H A P T E R T W O efficient use of an index. In any case, there is almost nothing to lose and everything to win by trying to combine both updates into a single statement: update tbo_invoice_extractor set pga_status = (case pga_status when 1 then 0 when 3 then 0 else pga_status end), rd_status = (case rd_status when 1 then 0 when 3 then 0 else rd_status end) where (pga_status in (1,3) or rd_status in (1, 3)) and inv_type = 0; There is indeed the possibility of some slight overhead due to the update of some columns with exactly the same contents they already have. But in most cases, one update is a lot faster than several separate ones. Notice that in regard to the previous section on logic, how we have used implicit conditional logic, by virtue of the case statement, to process only those rows that meet the update criteria, irrespective of how many different update criteria there may be. Apply updates in one fell swoop if possible; try", + "source": "The Art of SQL.pdf", + "chunk_id": 46 + }, + { + "text": "in regard to the previous section on logic, how we have used implicit conditional logic, by virtue of the case statement, to process only those rows that meet the update criteria, irrespective of how many different update criteria there may be. Apply updates in one fell swoop if possible; try to minimize repeated visits to the same table. Careful Use of User-Written Functions When a user-written function is embedded in a statement, the function may be called a large number of times. If the function appears within the select list, it is called for each returned row. If it appears within the where clause, it is called for each and every row that has successfully passed the filtering criteria previously evaluated. This may be a considerable number of times if the other criteria are not very selective. Consider what happens if that same function executes a query. The query is executed each time the function is called; in practice, the result is exactly the same as a correlated subquery, except that the function is an excellent way to prevent the cost-based optimizer from executing the main query more intelligently! Precisely because the subquery is hidden within the function, the database optimizer cannot take any account of this query. Moreover, the stored procedure is not as close to the SQL execution engine as is a correlated subquery, and it will consequently be even less efficient. www.it-ebooks.info W A G I N G W A R 45 Now I shall present an example demonstrating the dangers of hiding SQL code away inside a user-written function. Consider a table flights that describes commercial flights, with columns for flight number, departure time, arrival time, and the usual three-letter IATA* codes for airports. The translation of those codes (over 9,000 of them) is stored in a reference table that contains the name of the city (or of the particular airport when there are several located in one city), and of course the name of the country, and so on. Quite obviously any display of flight information should include the name of the destination city airport rather than the rather austere IATA code. Here we come to one of the contradictions in modern software engineering. What is often regarded as “good practice” in programming is modularity, with many insulated software layers. That principle is fine in the general case, but in the context of database programming, in which code is a shared activity between the developer and the database engine itself, the desirability of code modularity is less clear. For example, we can follow the principle of modularity by building a small function to look up IATA codes and present the full airport name whenever the function is cited in a query: create or replace function airport_city(iata_code in char) return varchar2 is city_name varchar2(50); begin select city into city_name from iata_airport_codes where code = iata_code; return(city_name); end; / For readers unfamiliar with Oracle syntax, trunc(sysdate) in the following query refers to today at 00:00 a.m., and", + "source": "The Art of SQL.pdf", + "chunk_id": 47 + }, + { + "text": "function is cited in a query: create or replace function airport_city(iata_code in char) return varchar2 is city_name varchar2(50); begin select city into city_name from iata_airport_codes where code = iata_code; return(city_name); end; / For readers unfamiliar with Oracle syntax, trunc(sysdate) in the following query refers to today at 00:00 a.m., and date arithmetic is based on days; the condition on departure times therefore refers to times between 8:30 a.m. and 4:00 p.m. today. Queries using the airport_city function might be very simple. For example: select flight_number, to_char(departure_time, 'HH24:MI') DEPARTURE, airport_city(arrival) \"TO\" from flights where departure_time between trunc(sysdate) + 17/48 and trunc(sysdate) + 16/24 order by departure_time / This query executes with satisfactory speed; on a random sample on my machine, 77 rows were returned in 0.18 seconds (the average of several runs), the kind of time that * International Air Transport Association. www.it-ebooks.info 46 C H A P T E R T W O leaves users happy (statistics indicate that 303 database blocks were accessed, 53 read from disk—and there is one recursive call per row). As an alternative to using a look-up function we could simply write a join, which of course looks slightly more complicated: select f.flight_number, to_char(f.departure_time, 'HH24:MI') DEPARTURE, a.city \"TO\" from flights f, iata_airport_codes a where a.code = f.arrival and departure_time between trunc(sysdate) + 17/48 and trunc(sysdate) + 16/24 order by departure_time / This query runs in only 0.05 seconds (the same statistics, but there are no recursive calls). It may seem petty and futile to be more than three times as fast for a query that runs for less than a fifth of a second. However, it is quite common in large systems (particularly in the airline world) to have extremely fast queries running several hundred thousand times in one day. Let’s say that a query such as the one above runs only 50,000 times per day. Using the query with the lookup function, the query time will amount to a total of 2:30 hours. Without the lookup function, it will be under 42 minutes. This maintains an improvement ratio of well over 300%, which in a high traffic environment represents real and tangible savings that may ultimately translate into a financial saving. Very often, the use of lookup functions makes the performance of batch programs dreadful. Moreover, they increase the “service time” of queries for no benefit—which means that fewer concurrent users can use the same box, as you shall see in Chapter 9. The code of user-written functions is beyond the examination of the optimizer. Succinct SQL The skillful developer will attempt to do as much as possible with as few SQL statements as possible. By contrast, the ordinary developer tends to closely follow the different functional stages that have been specified; here is an actual example: -- Get the start of the accounting period select closure_date into dtPerSta from tperrslt where fiscal_year=to_char(Param_dtAcc,'YYYY') and rslt_period='1' || to_char(Param_dtAcc,'MM'); www.it-ebooks.info W A G I N G W A R 47 -- Get the end of the", + "source": "The Art of SQL.pdf", + "chunk_id": 48 + }, + { + "text": "the different functional stages that have been specified; here is an actual example: -- Get the start of the accounting period select closure_date into dtPerSta from tperrslt where fiscal_year=to_char(Param_dtAcc,'YYYY') and rslt_period='1' || to_char(Param_dtAcc,'MM'); www.it-ebooks.info W A G I N G W A R 47 -- Get the end of the period out of closure select closure_date into dtPerClosure from tperrslt where fiscal_year=to_char(Param_dtAcc,'YYYY') and rslt_period='9' || to_char(Param_dtAcc,'MM'); This is an example of very poor code, even if in terms of raw speed it is probably acceptable. Unfortunately, this quality of code is typical of much of the coding that performance specialists encounter. Two values are being collected from the very same table. Why are they being collected through two different, successive statements? This particular example uses Oracle, and a bulk collect of the two values into an array can easily be implemented. The key to doing that is to add an order by clause on rslt_period, as follows: select closure_date bulk collect into dtPerStaArray from tperrslt where fiscal_year=to_char(Param_dtAcc,'YYYY') and rslt_period in ('1' || to_char(Param_dtAcc,'MM'), '9' || to_char(Param_dtAcc,'MM')) order by rslt_period; The two dates are stored respectively into the first and second positions of the array. bulk collect is specific to the PL/SQL language but the same reasoning applies to any language allowing an explicit or implicit array fetch. Note that an array is not even required, and the two values can be retrieved into two distinct scalar variables using the following little trick:* select max(decode(substr(rslt_period, 1, 1), -- Check the first character '1', closure_date, -- If it's '1' return the date we want to_date('14/10/1066', 'DD/MM/YYYY'))), -- Otherwise something old max(decode(substr(rslt_period, 1, 1), '9', closure_date, -- The date we want to_date('14/10/1066', 'DD/MM/YYYY'))), into dtPerSta, dtPerClosure from tperrslt where fiscal_year=to_char(Param_dtAcc,'YYYY') and rslt_period in ('1' || to_char(Param_dtAcc,'MM'), '9' || to_char(Param_dtAcc,'MM')); In this example, since we expect two rows to be returned, the problem is to retrieve in one row and two columns what would naturally arrive as two rows of a single column each (as in the array fetch example). We do that by checking each time the column that allows * The Oracle function decode( ) works like case. What is compared is the first argument. If it is equal to the second argument, then the third one is returned; if there is no fifth parameter, then the fourth one corresponds to else; otherwise, if the first argument is equal to the fourth one, the fifth one is returned and so on as long as we have pairs of values. www.it-ebooks.info 48 C H A P T E R T W O distinction between the two rows, rslt_period. If the row is the required one, the date of interest is returned. Otherwise, we return a date (here the arbitrary date is that of the battle of Hastings), which we know to be in all cases much older (smaller in terms of date comparison) than the one we want. By taking the maximum each time, we can be ensured that the correct date is", + "source": "The Art of SQL.pdf", + "chunk_id": 49 + }, + { + "text": "return a date (here the arbitrary date is that of the battle of Hastings), which we know to be in all cases much older (smaller in terms of date comparison) than the one we want. By taking the maximum each time, we can be ensured that the correct date is obtained. This is a very practical trick that can be applied equally well to character or numerical data; we shall study it in more detail in Chapter 11. SQL is a declarative language, so try to distance your code from the procedurality of business specifications. Offensive Coding with SQL Programmers are often advised to code defensively, checking the validity of all parameters before proceeding. In reality, when accessing a database, there is a real advantage in coding offensively, trying to do several things simultaneously. A good example is a succession of various checks, designed to flag up an exception whenever the criterion required by any of these checks fails to be met. Let’s assume that some kind of payment by a credit card has to be processed. There are a number of steps involved. It may be necessary to check that the customer id and card number that have been submitted are valid, and that they are correctly associated one with the other. The card expiration date must also be validated. Finally, the current purchase must not exceed the credit limit for the card. If everything is correct, the debit operation may proceed. An unskilled developer may write as follows: select count(*) from customers where customer_id = provided_id and will check the result. Then the next stage will be something like this: select card_num, expiry_date, credit_limit from accounts where customer_id = provided_id These returns will be checked against appropriate error codes. The financial transaction will then proceed. www.it-ebooks.info W A G I N G W A R 49 A skillful developer will do something more like the following (assuming that today( ) is the function that returns the current date): update accounts set balance = balance - purchased_amount where balance >= purchased_amount and credit_limit >= purchased_amount and expiry_date > today( ) and customer_id = provided_id and card_num = provided_cardnum Then the number of rows updated will be checked. If the result is 0, the reason can be determined in a single operation, by executing: select c.customer_id, a.card_num, a.expiry_date, a.credit_limit, a.balance from customers c left outer join accounts a on a.customer_id = c.customer_id and a.card_num = provided_cardnum where c.customer_id = provided_id If the query returns no row, the inference is that the value of customer_id is wrong, if card_num is null the card number is wrong, and so on. But in most cases this query will not even be executed. NOTE Did you notice the use of count(*) in the first piece of novice code? This is a perfect illustration of the misuse of count(*) to perform an existence test. The essential characteristic of “aggressive coding” is to proceed on the basis of reasonable probabilities. For example, there is little point", + "source": "The Art of SQL.pdf", + "chunk_id": 50 + }, + { + "text": "Did you notice the use of count(*) in the first piece of novice code? This is a perfect illustration of the misuse of count(*) to perform an existence test. The essential characteristic of “aggressive coding” is to proceed on the basis of reasonable probabilities. For example, there is little point in checking whether the customer exists— if they don’t, they won’t be in the database in the first place! Assume nothing will fail, and if it does, have mechanisms in place that will address the problem at that point and only that point. Interestingly, this approach is analogous to the “optimistic concurrency control” method adopted in some database systems. Here update conflicts are assumed not to occur, and it is only when they do that control strictures are brought into play. The result is much higher throughput than for systems using pessimistic methods. Code on a probabilistic basis. Assume the most likely outcome and fall back on exception traps only when strictly necessary. www.it-ebooks.info 50 C H A P T E R T W O Discerning Use of Exceptions There is a thin line between courage and rashness; when I recommend coding aggressively, my model is not the charge of the Light Brigade at Balaclava.* Programming by exception can also be the consequence of an almost foolhardy bravado, in which our proud developers determine to “go for it.” They have an overriding confidence that testing and the ability to handle exceptions will see them through. Ah, the brave die young! As their name implies, exceptions should be exceptional occurrences. In the particular case of database programming, all exceptions do not require the same computer resources—and this is probably the key point to understand if they are to be used intelligently. There are good exceptions, conditions that are raised before anything has been done, and bad exceptions, which are raised only when the full extent of the disaster has actually happened. For instance, a query against a primary key that finds no row will take minimal resources—the situation is detected while searching the index. However, if the query cannot use an index, then you have to carry out a full table scan before being able to tell positively that no data has been found. For a very large table, a total sequential read can represent a disaster on a machine near maximum capacity. Some exceptions are extremely costly, even in the best-case scenario; take the detection of duplicate keys. How is uniqueness enforced? Almost always by creating a unique index, and it is when a key is submitted for entry into that index that any constraint violation of that unique index will be revealed. However, when an index entry is created, the physical address of the row must be provided, which means that the insertion into the table takes place prior to the insertion into the index. The constraint violation requires that the partial insert must be undone, together with the identification of the exact constraint violated being returned as an", + "source": "The Art of SQL.pdf", + "chunk_id": 51 + }, + { + "text": "the physical address of the row must be provided, which means that the insertion into the table takes place prior to the insertion into the index. The constraint violation requires that the partial insert must be undone, together with the identification of the exact constraint violated being returned as an error message. All of these activities carry some significant processing cost. But the greatest sin is trying to fight at the individual exception level. Here, one is forced to think about individual rows rather than data sets— the very antithesis of relational database processing. The consequence of repeated constraint violations can be a serious deterioration in performance. Let’s look at an Oracle example of the previous points. Assume that following the merger of two companies, email addresses are standardized on the pattern, on 12 characters at most, with all spaces or quotes replaced by an underscore character. * During the Crimean War of 1854 that saw England, France, and Turkey fight against Russia, a poorly specified order and personal enmity between some of the commanders led more than 600 British cavalry men to charge down a valley in full line of fire of the Russian guns. Around 120 men and half the horses were killed, for no result. The bravery of the men, celebrated in a poem by Ten- nyson and (later) several Hollywood movies, helped turn a stupid military action into a myth. www.it-ebooks.info W A G I N G W A R 51 Let’s assume that a new employee table is created with the new email addresses obtained from a 3,000-row employee_old table. We want each employee to have a unique email address. We must therefore assign, for instance, flopez to Fernando Lopez, and flopez2 to Francisco Lopez (no relation). In fact, in our test data, a total of 33 potential duplicate entries exist, which is the reason for the following result: SQL> insert into employees(emp_num, emp_name, emp_firstname, emp_email) 2 select emp_num, 3 emp_name, 4 emp_firstname, 5 substr(substr(EMP_FIRSTNAME, 1, 1) 6 ||translate(EMP_NAME, ' ''', '_ _'), 1, 12) 7 from employees_old; insert into employees(emp_num, emp_name, emp_firstname, emp_email) * ERROR at line 1: ORA-00001: unique constraint (EMP_EMAIL_UQ) violated Elapsed: 00:00:00.85 Thirty-three duplicates out of 3,000 is about 1%, so perhaps it would be possible to quietly process the conformant 99% and handle the rest through exceptions? After all, it would seem that a 1% load could be accommodated with some additional exception processing which should not be too significant. Following is the code for this optimistic approach: SQL> declare 2 v_counter varchar2(12); 3 b_ok boolean; 4 n_counter number; 5 cursor c is select emp_num, 6 emp_name, 7 emp_firstname 8 from employees_old; 9 begin 10 for rec in c 11 loop 12 begin 13 insert into employees(emp_num, emp_name, 14 emp_firstname, emp_email) 15 values (rec.emp_num, 16 rec.emp_name, 17 rec.emp_firstname, 18 substr(substr(rec.emp_firstname, 1, 1) 19 ||translate(rec.emp_name, ' ''', '_ _'), 1, 12)); 20 exception 21 when dup_val_on_index then 22 b_ok := FALSE; 23 n_counter := 1; 24 begin 25 v_counter :=", + "source": "The Art of SQL.pdf", + "chunk_id": 52 + }, + { + "text": "11 loop 12 begin 13 insert into employees(emp_num, emp_name, 14 emp_firstname, emp_email) 15 values (rec.emp_num, 16 rec.emp_name, 17 rec.emp_firstname, 18 substr(substr(rec.emp_firstname, 1, 1) 19 ||translate(rec.emp_name, ' ''', '_ _'), 1, 12)); 20 exception 21 when dup_val_on_index then 22 b_ok := FALSE; 23 n_counter := 1; 24 begin 25 v_counter := ltrim(to_char(n_counter)); www.it-ebooks.info 52 C H A P T E R T W O 26 insert into employees(emp_num, emp_name, 27 emp_firstname, emp_email) 28 values (rec.emp_num, 29 rec.emp_name, 30 rec.emp_firstname, 31 substr(substr(rec.emp_firstname, 1, 1) 32 ||translate(rec.emp_name, ' ''', '_ _'), 1, 33 12 - length(v_counter)) || v_counter); 34 b_ok := TRUE; 35 exception 36 when dup_val_on_index then 37 n_counter := n_counter + 1; 38 end; 39 end; 40 end loop; 41 end; 40 / PL/SQL procedure successfully completed. Elapsed: 00:00:18.41 But what exactly is the cost of this exception handling? If the same exercise is attempted after removing the “problem” rows, the comparison between the loop with duplicates and the loop without duplicates shows that the cost of processing exceptions in the loop is fairly negligible—with duplicates the procedure also takes about 18 seconds to run. However, when we run the insert...select of our first attempt without duplicates it is considerably faster than the loop: we discover that the switch to the one-row-at-a-time logic adds close to 50% to processing time. But in such a case, is it possible to avoid the row-at-a-time process? Yes, but only by avoiding exceptions. It’s the decision of dealing with problem rows through exception handling that forced our adoption of sequential row processing. Alternatively, there might be value in attempting to identify those rows that contain email addresses subject to contention, and assigning those addresses some arbitrary number to achieve uniqueness. It is easy to determine how many rows are involved in this contention by adding a group by clause to the SQL statement. However, assigning numbers might be a difficult thing to do without using the analytical functions available in the major database systems. (Oracle calls them analytical functions, DB2 knows them as online analytical processing, or OLAP, functions, SQL Server as ranking functions.) It is worthwhile to explore the solution to this problem in terms of pure SQL. Each email address can be assigned a unique number: 1 for the oldest employee whose first name initial and surname result in the given email address, 2 to the second oldest and so on. By pushing this result into a subquery, it is possible to check and concatenate nothing to the first email address in each group, and the sequence numbers (not in the Oracle sense of the word) to the following ones. The following code shows how our logic can be applied: www.it-ebooks.info W A G I N G W A R 53 SQL> insert into employees(emp_num, emp_firstname, 2 emp_name, emp_email) 3 select emp_num, 4 emp_firstname, 5 emp_name, 6 decode(rn, 1, emp_email, 7 substr(emp_email, 8 1, 12 - length(ltrim(to_char(rn)))) 9 || ltrim(to_char(rn))) 10 from (select emp_num, 11 emp_firstname, 12 emp_name, 13 substr(substr(emp_firstname, 1, 1) 14", + "source": "The Art of SQL.pdf", + "chunk_id": 53 + }, + { + "text": "G I N G W A R 53 SQL> insert into employees(emp_num, emp_firstname, 2 emp_name, emp_email) 3 select emp_num, 4 emp_firstname, 5 emp_name, 6 decode(rn, 1, emp_email, 7 substr(emp_email, 8 1, 12 - length(ltrim(to_char(rn)))) 9 || ltrim(to_char(rn))) 10 from (select emp_num, 11 emp_firstname, 12 emp_name, 13 substr(substr(emp_firstname, 1, 1) 14 ||translate(emp_name, ' ''', '_ _'), 1, 12) 15 emp_email, 16 row_number( ) 17 over (partition by 18 substr(substr(emp_firstname, 1, 1) 19 ||translate(emp_name,' ''','_ _'),1,12) 20 order by emp_num) rn 21 from employees_old) 22 / 3000 rows created. Elapsed: 00:00:11.68 We avoid the costs of row-at-a-time processing, and this solution requires only 60% of the original time. Exception handling forces the adoption of procedural logic. Always try to anticipate possible exceptions by remaining within declarative SQL. www.it-ebooks.info www.it-ebooks.info Chapter 3. C H A P T E R T H R E E Tactical Dispositions Indexing Chi vuole fare tutte queste cose, conviene che tenga lo stile e modo romano: il quale fu in prima di fare le guerre, come dicano i Franciosi, corte e grosse. Whoever wants to do all these things must hold to the Roman conduct and method, which was first to make the war, as the French say, short and sharp. —Niccolò Machiavelli (1469–1527) Discorsi sopra la prima Deca di Tito Livio, II, 6 www.it-ebooks.info 56 C H A P T E R T H R E E O nce the layout of the battlefield is determined, the general should be able to precisely identify which are the key parts of the enemy possessions that must be captured. It is exactly the same with information systems. The crucial data to be retrieved will determine the most efficient access paths into the data system. Here, the fundamental tactic is indexing. It is a complex area, and one in which competing priorities must be resolved. In this chapter, we discuss various aspects of indexes and indexing strategy, which, taken together, provide general guidelines for database access strategies. The Identification of “Entry Points” Even before starting to write the very first SQL statement in a program, you should have an idea about the search criteria that will be of importance to users. Values that are fed into a program and the size of the data subset defined lay the foundations for indexing. Indexes are, above all, a technique for achieving the fastest possible access to specific data. Note that I say “specific data,” as indexes must be carefully deployed. They are not a panacea: they will not enable fast access to all data. In fact, sometimes the very opposite is the result, if there is a serious mismatch between the original index strategy and the new data-retrieval requirements. Indexes can be considered to be shortcuts to data, but they are not shortcuts in the same sense as a shortcut in a graphical desktop environment. Indexes come with some heavy costs, both in terms of disk space and, possibly more importantly, in terms of processing costs. For example, it is not uncommon", + "source": "The Art of SQL.pdf", + "chunk_id": 54 + }, + { + "text": "considered to be shortcuts to data, but they are not shortcuts in the same sense as a shortcut in a graphical desktop environment. Indexes come with some heavy costs, both in terms of disk space and, possibly more importantly, in terms of processing costs. For example, it is not uncommon to encounter tables in which the volume of index data is much larger than the volume of the actual data being indexed. I can say the same of index data as I said of redundant table data in Chapter 1: indexes are usually mirrored, backed up to other disks, and so on, and the very large volumes involved cost a lot, not only in terms of storage, but also in terms of downtime when you have to restore from a backup. Figure 3-1 shows a real-life case, the main accounting table of a major bank; out of 33 GB total for all indexes and the table, indexes take more than 75%. Let’s forget about storage for a moment and consider processing. Whenever we insert or delete a row, all the indexes on the table have to be adjusted to reflect the new data. This adjustment, or “maintenance,” also applies whenever we update an indexed column; for example, if we change the value of an attribute in a column that is either itself indexed, or is part of a compound index in which more than one column is indexed together. In practice this maintenance activity means a lot of CPU resources are used to scan data blocks in memory, I/O activity is needed to record the changes to logfiles, together with possibly more I/O work against the database files. Finally, recursive operations may be required on the database system to maintain storage allocations. www.it-ebooks.info T A C T I C A L D I S P O S I T I O N S 57 Tests have quantified the real cost of maintaining indexes on a table. For example, if the unit time required to insert data into a non-indexed table is 100 (seconds, minutes, or hours—it does not really matter for this illustration), each additional index on that table will add an additional unit time of anything from 100 to 250. Maintenance costs for one index may exceed those for one table. Although index implementation varies from DBMS to DBMS, the high cost of index maintenance is true for all products, as Figures 3-2 and 3-3 show with Oracle and MySQL. FIGURE 3-1. A real-life case: Data versus Index out of a 33 GB total FIGURE 3-2. The impact of indexes on insertion with Oracle www.it-ebooks.info 58 C H A P T E R T H R E E Interestingly, this index maintenance overhead is of the same magnitude as a simple trigger. I have created a simple trigger to record into a log table the key of each row inserted together with the name of the user and a timestamp—a typical audit trail. As one might expect, performance suffers—but", + "source": "The Art of SQL.pdf", + "chunk_id": 55 + }, + { + "text": "this index maintenance overhead is of the same magnitude as a simple trigger. I have created a simple trigger to record into a log table the key of each row inserted together with the name of the user and a timestamp—a typical audit trail. As one might expect, performance suffers—but in the same order of magnitude as the addition of two indexes, as shown in Figure 3-4. Recall how often one is urged to avoid triggers for performance reasons! People are usually more reluctant to use triggers than they are to use indexes, yet the impact may well be very similar. Generating more work isn’t the only way for indexes to hinder performance. In an environment with heavy concurrent accesses, more indexes will mean aggrieved contention and locking. By nature, an index is usually a more compact structure than a table—just compare the number of index pages in this book to the number of pages in the book itself. Remember that updating an indexed table requires two data activities: updating the data itself and updating the index data. As a result, concurrent updates, which may affect relatively scattered areas of a huge table, and therefore not suffer from FIGURE 3-3. The impact of indexes on insertion with MySQL FIGURE 3-4. Comparing the performance impact of indexes and triggers www.it-ebooks.info T A C T I C A L D I S P O S I T I O N S 59 any serialization during the changes to the actual data, may easily find themselves with much less elbow room when updating the indexes. As explained above, these indexes are by definition much “tighter” data assemblages. It must be stressed that, whatever the cost in terms of storage and processing power, indexes are vital components of databases. Their importance is nowhere greater, as I discuss in Chapter 6, than in transactional databases where most SQL statements must either return or operate on few rows in large tables. Chapter 10 shows that decision support systems are also heavily dependent for performance on indexing. However, if the data tables we are dealing with have been properly normalized (and once again I make no apologies for referring to the importance of design), those columns deserving some a priori indexing will be very few in a transactional database. They will of course include the primary key (the tuple, or row, identifier). This column (or columns in the case of a compound key) will be automatically indexed simply by virtue of its declaration as the primary key. Unique columns are similar and will, in all probability, be indexed simply as a by-product of the implementation of integrity constraints. Consideration should also be given to indexing columns that, although not unique, approach uniqueness—in other words, columns with a high variability of values. As a general rule, experience would suggest that very few indexes are required for most tables in a general purpose or transactional database, because many tables are searched with a very limited set of criteria. The", + "source": "The Art of SQL.pdf", + "chunk_id": 56 + }, + { + "text": "although not unique, approach uniqueness—in other words, columns with a high variability of values. As a general rule, experience would suggest that very few indexes are required for most tables in a general purpose or transactional database, because many tables are searched with a very limited set of criteria. The rationale may be very different in decision support systems, as you shall see in Chapter 10. I tend to grow suspicious of tables with many indexes, especially when these tables are very large and much updated. A high number of indexes may exceptionally be justified, but one should revisit the original design to validate the case for heavily indexed tables. In a transactional database, “too many indexes” is often the mark of an uncertain design. Indexes and Content Lists The book metaphor can be helpful in another respect—as a means of better understanding the role of the index in the DBMS. It is important to recognize the distinction between the two mechanisms of the table of contents and the book index. Both provide a means of fast access into the data, but at two very different levels of granularity. The table of contents provides a structured overview of the whole book. As such, it is regarded as complementary to the index device in books, which is often compared to the index of a database. www.it-ebooks.info 60 C H A P T E R T H R E E When you look for a very precise bit of information in a book, you turn to the index. You are ready to check 2 or 3 entries, but not 20—flipping pages between the index and the book itself to check so many entries would be both tedious and inefficient. Like a book index, a database index will direct you to specific values in one or more records (I overlook the use of indexes in range searching for the moment). If you look for substantial information in a book, you either turn to the index, get the first index entry about the topic you want to study, and then read on, or you turn to the table of contents and identify the chapter that is most relevant to your topic. The distinction between the table of contents and the index is crucial: an entry in a table of contents directs the reader to a block of text, perhaps a chapter, or a section. Similarly, Chapter 5 shows mechanisms by which you can organize a table and enable data retrieval in a manner similar to a table of contents’ access. An index must primarily be regarded as a means of accessing data at an atomic level of granularity, as defined by the original data design, and not as a means of retrieving large quantities of undifferentiated data. When an indexing strategy is used to pull in large quantities of data, the role of indexes is being seriously misunderstood. Indexing is being used as a desperate measure to recover from an already untenable situation. The", + "source": "The Art of SQL.pdf", + "chunk_id": 57 + }, + { + "text": "design, and not as a means of retrieving large quantities of undifferentiated data. When an indexing strategy is used to pull in large quantities of data, the role of indexes is being seriously misunderstood. Indexing is being used as a desperate measure to recover from an already untenable situation. The commander is beginning to panic and is sending off sorties in all directions, hoping that sheer numbers will compensate for the lack of a coherent strategy. It never does, of course. Be very sure you understand what you are indexing, and why you are indexing it. Making Indexes Work To justify the use of an index, it must provide benefit. Just as in our metaphor of the book, you may use an index if you simply require very particular information on one item of data. But if you want to review an entire subject area, you will turn not to the index, but to the table of contents of the book. There will always be times when the decision between using an index or a broader categorization is a difficult one. This is an area where the use of retrieval ratios makes its persuasive appearance. Such ratios have a hypnotic attraction to many IT and data practitioners because they are so neat, so easy, so very scientific! The applicability of an index has long been judged on the percentage of the total data retrieved by a query that uses a key value as only search criterion, and conventionally that percentage has often been set at 10% (the percentage of rows that match, on www.it-ebooks.info T A C T I C A L D I S P O S I T I O N S 61 average, an index key defines the selectivity of the index; the lower the percentage, the more selective the index). You will often find this kind of rule in the literature. This ratio, and others like it, is based on old assumptions regarding such things as the relative performance of disk access and memory access. Even if we forget that these performance ratios, which have been around since at least the mid-1980s, were based on what is today outdated technology (ideal percentages are grossly simplistic views), far more factors need to be taken into account. When magical ratios such as our 10% ratio were designed, a 500,000–row table was considered a very big table; 10% of such a table usually meant a few tens of thousand rows. When you have tables with hundreds of millions or even billions of rows, the number of rows returned by using an index with a similar selectivity of about 10% may easily be greater than the number of rows in those mega-tables of yore against which the original ratios were estimated. Consider the part played by modern hard disk systems, equipped as they are with large cache storage. What the DBMS sees as “physical I/O” may well be memory access; moreover, since the kernel usually shifts different amounts of data", + "source": "The Art of SQL.pdf", + "chunk_id": 58 + }, + { + "text": "those mega-tables of yore against which the original ratios were estimated. Consider the part played by modern hard disk systems, equipped as they are with large cache storage. What the DBMS sees as “physical I/O” may well be memory access; moreover, since the kernel usually shifts different amounts of data into memory depending on the type of access (table or index), you may be in for a surprise when comparing the relative performance of retrievals with and without using an index. But these are not the only factors to consider. You also need to watch the number of operations, which can truly be performed in parallel. Take note of whether the rows associated with an index key value are likely to be physically close. For instance, when you have an index on the insertion date, barring any quirk such as the special storage options I describe in Chapter 5, any query on a range of insertion dates will probably find the corresponding rows grouped together by construction. Any block or page pointed to by the very first key in the range will probably contain as well the rows pointed to by the immediately following key values. Therefore, any chunk of table we return through use of the index will be rich in data of interest to our query, and any data block found through the index will be of considerable value to the query’s performance. When the indexed rows associated with an index key are spread all over the table (for example, the references to an article in a table of orders), it is quite another matter. Even though the number of relevant rows is small as a proportion of the whole, because they are scattered all over the disk, the value of the index diminishes. This is illustrated by Figure 3-5: we can have two unique indexes that are strictly equivalent for fetching a single row, and yet one will perform significantly better than the other if we look for a range of values, a frequent occurrence when working with dates. Factors such as these blur the picture, and make it difficult to give a prescriptive statement on the use of indexes. www.it-ebooks.info 62 C H A P T E R T H R E E Rows ordered as index keys lead to a faster range scan. Indexes with Functions and Conversions Indexes are usually implemented as tree structures—mostly complex trees—to avoid a fast decay of indexes on heavily inserted, updated, and deleted tables. To find the physical location of a row, the address of which is stored in the index, one must compare the key value to the value stored in the current node of the tree to be able to determine which sub-tree must be recursively searched. Let’s now suppose that the value that drives our search doesn’t exactly match an actual column value but can be compared to the result of a function f( ) applied to the column value. In that case we may be", + "source": "The Art of SQL.pdf", + "chunk_id": 59 + }, + { + "text": "be able to determine which sub-tree must be recursively searched. Let’s now suppose that the value that drives our search doesn’t exactly match an actual column value but can be compared to the result of a function f( ) applied to the column value. In that case we may be tempted to express a condition as follows: where f(indexed_column) = 'some value' This kind of condition will typically torpedo the index, making it useless. The problem is that nothing guarantees that the function f( ) will keep the same order as the index data; in fact, in most cases it will not. For instance, let’s suppose that our tree-index looks like Figure 3-6. FIGURE 3-5. When two highly selective indexes may perform differently www.it-ebooks.info T A C T I C A L D I S P O S I T I O N S 63 (If the names look familiar, it is because they are those of some of Napoleon’s marshals.) Figure 3-6 is of course an outrageously simplified representation, just for the purpose of explaining a particular point; the actual indexes do not look exactly like the binary tree shown in Figure 3-6. If we look for the MASSENA key, with this search condition: where name = 'MASSENA' then the index search is simple enough. We hit LANNES at the root of the tree and compare MASSENA to LANNES. We find MASSENA to be greater, based on the alphabetical order. We therefore recursively search the right-hand sub-tree, the root of which is MORTIER. Our search key is smaller than MORTIER, so we search the left-hand sub-tree and immediately hit MASSENA. Bingo—success. Now, let’s say that we have a condition such as: where substr(name, 3, 1) = 'R' The third letter is an uppercase R—which should return BERNADOTTE, MORTIER, and MURAT. When we make the first visit to the index, we hit LANNES, which doesn’t satisfy the condition. Not only that, the value that is associated with the current tree node gives us no indication whatsoever as to which branch we should continue our search into. We are at a loss : the fact that the third letter isR is of no help in deciding whether we should search the left sub-tree or the right sub-tree (in fact, we find elements belonging to our result set in both sub-trees), and we are unable to descend the tree in the usual way, by selecting a branch thanks to the comparison of the search key to the value stored in the current node. Given the index represented in Figure 3-6, selecting names with an R in the third position is going to require a sequential data scan, but here another question arises. If the optimizer is sufficiently sophisticated, it may be able to judge whether the most efficient execution path is a scan of the actual data table or inspecting, in turn, each and every node in the index on the column in question. In the latter case, the search would lead", + "source": "The Art of SQL.pdf", + "chunk_id": 60 + }, + { + "text": "If the optimizer is sufficiently sophisticated, it may be able to judge whether the most efficient execution path is a scan of the actual data table or inspecting, in turn, each and every node in the index on the column in question. In the latter case, the search would lead to an index-based retrieval, but not as envisaged in the original model design since we would be using the index in a rather inefficient way. FIGURE 3-6. A simplistic representation of how names might be stored in an index www.it-ebooks.info 64 C H A P T E R T H R E E Recall the discussion on atomicity in Chapter 1. Our performance issue stems from a very simple fact: if we need to apply a function to a column, it means that the atomicity of data in the table isn’t suitable for our business requirements. We are not in 1NF! Atomicity, though, isn’t a simple notion. The ultra-classic example is a search condition on dates. Oracle, for instance, uses the date type to store not only the date information, but also the time information, down to the second (this type is actually known as datetime to most other database systems). However, to test the unwary, the default date format doesn’t display the time information. If you enter something such as: where date_entered = to_date('18-JUN-1815', 'DD-MON-YYYY') then only the rows for which the date (and time!) happens to be exactly the 18th of June 1815 at 00:00 (i.e., at midnight) are returned. Everyone gets caught out by this issue the very first time that they query datetime data. Quite naturally, the first impulse is to suppress the time information from date_entered, which the junior practitioners usually do in the following way: where trunc(date_entered) = to_date('18-JUN-1815', 'DD-MON-YYYY') Despite the joy of seeing the query “work,” many people fail to realize (before the first performance issues begin to arise) that by writing their query in such a way they have waved goodbye to using the index on date_entered, assuming there was one. Does all this mean that you cannot be in 1NF if you are using datetime columns? Fortunately, no. In Chapter 1, I defined an atomic attribute as an attribute in which a where clause can always be referred to in full. You can refer in full to a date if you are using a range condition. An index on date_entered is usable if the preceding condition is written as: where date_entered >= to_date('18-JUN-1815', 'DD-MON-YYYY') and date_entered < to_date('19-JUN-1815', 'DD-MON-YYYY') Finding rows with a given date in this way makes an index on date_entered usable, because the very first condition allows us to descend the tree and reach a sorted list of all keys at the bottom of the index hierarchy (we may envision the index as a sorted list of keys and associated addresses, above which is plugged a tree allowing us to get direct access to every item in the list). Therefore, once the first condition has taken us", + "source": "The Art of SQL.pdf", + "chunk_id": 61 + }, + { + "text": "of all keys at the bottom of the index hierarchy (we may envision the index as a sorted list of keys and associated addresses, above which is plugged a tree allowing us to get direct access to every item in the list). Therefore, once the first condition has taken us to the bottom layer of the index and to the very first item of interest in the list, all we have to do is scan the list as long as the second condition is true. This type of access is known as an index range scan. The trap of functions preventing the use of indexes is often even worse if the DBMS engine is able to perform implicit conversions when a column of a given type is equated or www.it-ebooks.info T A C T I C A L D I S P O S I T I O N S 65 compared to a constant of another type in a where condition—a logical error and yet one that is allowed by SQL. Once again Oracle provides an excellent example of such behavior. For instance, dangers arise when a character column is compared to a number. Instead of immediately generating a run-time error, Oracle implicitly converts the column to a number to enable the comparison to take place. The conversion may indeed generate a run-time error if there is an alpha character in that numerical string, but in many cases when a string of digits without any true numerical meaning is stored as characters (social security numbers or a date of birth shown as mmddyy, or ddmmyy, both meaning the same, but having very different numerical values), the conversion and subsequent comparison will “work”—except that the conversion will have rendered any index on the character column almost useless. In the light of the neutralization of indexes by functions, Oracle’s design choice to apply the conversion to the column rather than to the constant may at first look surprising. However, that decision does make some sense. First of all, comparing potatoes to carrots is a logical error. By applying the conversion to the column, the DBMS is more likely (depending on the execution path) to encounter a value to which the conversion does not apply, and therefore the DBMS is more likely to generate a runtime error. An error at this stage of the process will prove a healthy reminder to the developer, doubtless prompting for a correction in the actual data field and raising agonizing questions about the quality of the data. Second, assuming that no error is generated, the very last thing we want is to return incorrect information. If we encounter: where account_number = 12345 it is quite possible, and in fact most likely, that the person who wrote the query was expecting the account 0000012345 to be returned—which will be the case if account_number (the alpha string) is converted to number, but not if the query 12345 is converted to a string without any special format specification. One", + "source": "The Art of SQL.pdf", + "chunk_id": 62 + }, + { + "text": "and in fact most likely, that the person who wrote the query was expecting the account 0000012345 to be returned—which will be the case if account_number (the alpha string) is converted to number, but not if the query 12345 is converted to a string without any special format specification. One may think that implicit conversions are a rare occurrence, akin to bugs. There is much truth in the latter point, but implicit conversions are in fact pretty common, especially when such things come into play as a parameters table holding in a column named parameter_value string representations of numbers and dates, as well as filenames or any other regular character string. Always make conversions explicit by using conversion functions. It is sometimes possible to index the result of a function applied to one or more columns. This facility is available with most products under various names (functional index, function-based index, index extension, and so on, or, more simply, index on a computed column). In my view, one should be careful with this type of feature and use it only as a standby for those cases in which the code cannot be modified. www.it-ebooks.info 66 C H A P T E R T H R E E I have already mentioned the heavy overhead added to data modifications as a result of the presence of indexes. Calling a function in addition to the normal index load each time an index needs to be modified cannot improve the situation: indeed it only adds to the total index maintenance cost. As the date_entered example given earlier demonstrates, creating a function-based index may be the lazy solution to something that can easily be remedied by writing the query in a different way. Furthermore, nothing guarantees that a function applied to a column retains the same degree of precision that a query against the raw column will achieve. Suppose that we store five years of sales online and that the sales_date column is indexed. On the face of it, such an index looks like an efficient one. But indexing with a function that is the month part of the date is not necessarily very selective, especially if every year the bulk of sales occurs in the run up to Christmas. Evaluating whether the resulting functional index will really bring any benefit is not necessarily easy without very careful study. From a purely design point of view, one can argue that a function is an implicit recognition that the column in question may be storing two or more discrete items of data. Use of a functional index is, in most cases, a way to extract some part of the data from a column. As pointed out earlier, we are violating the famous first normal form, which requires data to be “atomic.” Not using strictly “atomic” data in the select list is a forgivable sin. Repeatedly using “subatomic” search criteria is a deadly one. There are some cases, though, when a function-based index may be justified. Case-", + "source": "The Art of SQL.pdf", + "chunk_id": 63 + }, + { + "text": "earlier, we are violating the famous first normal form, which requires data to be “atomic.” Not using strictly “atomic” data in the select list is a forgivable sin. Repeatedly using “subatomic” search criteria is a deadly one. There are some cases, though, when a function-based index may be justified. Case- insensitive searches are probably the best example; indexing a column converted to upper- or lowercase will allow us to perform case-insensitive searches on that column efficiently. That said, forcing the case during inserts and updates is not a bad solution either. In any event, if data is stored in lowercase, then required in uppercase, one has to question the thoroughness with which the original data design was carried out. Another tricky conundrum is the matter of duration in the absence of a dedicated interval data type. Given three time fields, a start date, a completion date, and a duration, one value can be determined from any existing two—but only by either building a functional index or by storing redundant data. Whichever solution is followed, redundancy will be the inevitable consequence: in the final analysis, you must weigh the benefits and disadvantages of the issues surrounding function-based indexes so that you can make informed decisions about using them. Use of functional indexes is often implicit recognition that your data analysis has not even resolved basic data item atomicity. www.it-ebooks.info T A C T I C A L D I S P O S I T I O N S 67 Indexes and Foreign Keys It is quite customary to systematically index the foreign keys of a table; and it is widely acknowledged to be common wisdom to do so. In fact, some design tools automatically generate indexes on these keys, and so do some DBMS. However, I urge caution in this respect. Given the overall cost of indexes, unnecessarily indexing foreign keys may prove a mistake, especially for a table that has many foreign keys. NOTE Of course, if your DBMS automatically indexes foreign keys, then you have no choice in the matter. You will have to resign yourself to potentially incurring unnecessary index overhead. The rule of indexing the foreign keys comes from what happens when (for example) a foreign key in table A references the primary key in table B, and then both tables are concurrently modified. The simple model in Figure 3-7 illustrates this point. Imagine that table A is very large. If user U1 wants to remove a row from table B, since the primary key for B is referenced by a foreign key in A, the DBMS must check that removal of the row will not lead to inconsistencies in the intertable dependencies, and must therefore see whether there is any child row in A referencing the row about to be deleted from B. If there does happen to be a row in A that references our row in B, then the deletion must fail, because otherwise we would end up with an orphaned row and", + "source": "The Art of SQL.pdf", + "chunk_id": 64 + }, + { + "text": "see whether there is any child row in A referencing the row about to be deleted from B. If there does happen to be a row in A that references our row in B, then the deletion must fail, because otherwise we would end up with an orphaned row and inconsistent data. If the foreign key in A is indexed, it can be checked very quickly. If it is not indexed, it will take a significant period of time since the session of user U1 will have to scan all of table A. Another problem is that we are not supposed to be alone on this database, and lots of things can happen while we scan A. For instance, just after user U1 has started the hunt in table A for an hypothetical child row, somebody else, say user U2, may want to insert a new row into table A which references that very same row we want to delete from table B. FIGURE 3-7. The simple, Master-Detail example www.it-ebooks.info 68 C H A P T E R T H R E E This situation is described in Figure 3-8, with user U1 first accessing table B to check the identifier of the row it wants to delete (1), and then searching for a child in table A (2). Meanwhile, U2 will have to check that the parent row exists in table B. But we have a primary key index on B, which means that unlike user U1, who is condemned to a slow sequential scan of the foreign key values of table A, user U2 will get the answer immediately from table B. If U2 quietly inserts the new row in table A (3), U2 may commit the change at such a point that user U1 finishes checking and wrongly concludes, having found no row, that the path is clear for the delete. Locking is required to prevent such a case, which would otherwise irremediably lead to inconsistent data. Data integrity is, as it should be, one of the prime concerns of an enterprise-grade DBMS. It will take no chance. Whenever we want to delete a row from table B, we must prevent insertion into any table that references B of a row referencing that particular one while we look for child rows. We have two ways to prevent insertions into referencing tables (there may be several ones) such as table A: • We lock all referencing tables (the heavy-handed approach). • We apply a lock to table B and make another process, such as U2, wait for this lock to be released before inserting a new row into a referencing table (the approach taken by most DBMS). The lock will apply to the table, a page, or the row, depending on the granularity allowed by the DBMS. In any case, if foreign keys are not indexed, checking for child rows will be slow, and we will hold locks for a very long time, potentially blocking many changes. In", + "source": "The Art of SQL.pdf", + "chunk_id": 65 + }, + { + "text": "lock will apply to the table, a page, or the row, depending on the granularity allowed by the DBMS. In any case, if foreign keys are not indexed, checking for child rows will be slow, and we will hold locks for a very long time, potentially blocking many changes. In the worst case of the heavy-handed approach we can even encounter deadlocks, with two processes holding locks and stubbornly refusing to release them as long as the other process doesn’t release its lock first. In such a case, the DBMS usually solves the dispute by killing one of the processes (hasta la vista, baby...) to let the other one proceed. FIGURE 3-8. Fight for the primary key www.it-ebooks.info T A C T I C A L D I S P O S I T I O N S 69 The case of concurrent updates therefore truly requires indexing foreign keys to prevent sessions from locking objects much longer than necessary. Hence the oft heard rule that “foreign keys should always be indexed.” The benefit of indexing the foreign key is that the elapsed time for each process can be drastically reduced, and in turn locking is reduced to the minimum level required for ensuring data integrity. What people often forget is that “always index foreign keys” is a rule associated with a special case. Interestingly, that special case often arises from design quirks, such as the maintenance of summary or aggregate denormalized columns in the master table of a master/detail relationship. There may be excellent reasons for updating concurrently two tables linked by referential integrity constraints. But there are also many cases with transactional databases where the referenced table is a “true” reference table (e.g., a dictionary, or “look-up” table that is very rarely updated, or it’s updated in the middle of the night when there is no other activity). In such a case, the only justification for the creation of an index on the foreign key columns should be whether such an index would be of any benefit from a strictly performance standpoint. We mustn’t forget the heavy penalty performance imposed by index maintenance. There are many cases when an index on a foreign key is not required. There must be a reason behind indexing; this is as true of foreign keys as of other columns. Multiple Indexing of the Same Columns The systematic indexing of foreign keys can often lead to situations in which columns belong to several indexes. Let’s consider once again a classic example. This consists of an ordering system in which some order_details table contains, for each order (identified by an order_id, a foreign key referencing the orders table) articles (identified by article_id, a foreign key referencing the articles table) that have been purchased, and in what quantity. What we have here is an associative table (order_details) resolving a many-to- many relationship between the tables orders and articles. Figure 3-9 illustrates the relationships among the three tables. FIGURE 3-9. The Orders/Articles example www.it-ebooks.info 70 C", + "source": "The Art of SQL.pdf", + "chunk_id": 66 + }, + { + "text": "key referencing the articles table) that have been purchased, and in what quantity. What we have here is an associative table (order_details) resolving a many-to- many relationship between the tables orders and articles. Figure 3-9 illustrates the relationships among the three tables. FIGURE 3-9. The Orders/Articles example www.it-ebooks.info 70 C H A P T E R T H R E E Typically, the primary key of order_details will be a composite key, made of the two foreign keys. Order entry is the very case when the referenced table and the referencing table are likely to be concurrently modified, and therefore we must index the order_id foreign key. However, the column that is defined here as a foreign key is already indexed as part of the composite primary key, and (this is the important point) as the very first column in the primary key. Since this column is the first column of the composite primary key, it can for all intents and purposes provide all the benefits as if it were an indexed foreign key. A composite index is perfectly usable even if not all columns in the key are specified, as long as those at the beginning of the key are. When descending an index tree such as the one described earlier in this chapter, it is quite sufficient to be able to compare the leading characters of the key to the index nodes to determine which branch of the index the search should continue down. There is therefore no reason to index order_id alone, since the DBMS will be able to use the index on (order_id, article_id) to check for child rows when somebody is working on the orders table. Locks will therefore not be required for both tables. Note, once again, that this reasoning applies only because order_id happens to be the very first column in the composite primary key. Had the primary key been defined as (article_id, order_id), then we would have had to create an index on order_id alone, while not building an index on the other foreign key, article_id. Indexing every foreign key may result in redundant indexing. System-Generated Keys System-generated keys (whether through a special number column defined as self- incrementing or through the use of system-generated counters such as Oracle’s sequences) require special care. Some inexperienced designers just love system-generated keys even when they have perfectly valid natural identifiers at their disposal. System- generated sequential numbers are certainly a far better solution than looking for the greatest current value and incrementing it by one (a certain recipe for generating duplicates in an environment with some degree of concurrency), or storing a “next value” that has to be locked and updated into a dedicated table (a mechanism that serializes and dramatically slows down accesses). Nevertheless, when many concurrent insertions are running against the same table in which these automatic keys are being generated, some very serious contention can occur at the creation point of the primary key index level. The purpose of the primary", + "source": "The Art of SQL.pdf", + "chunk_id": 67 + }, + { + "text": "table (a mechanism that serializes and dramatically slows down accesses). Nevertheless, when many concurrent insertions are running against the same table in which these automatic keys are being generated, some very serious contention can occur at the creation point of the primary key index level. The purpose of the primary key index is primarily to ensure the uniqueness of the primary key columns. www.it-ebooks.info T A C T I C A L D I S P O S I T I O N S 71 The problem is usually that if there is one unique generator (as opposed to as many generators as there are concurrent processes, hitting totally disjoint ranges of values) we are going to rapidly generate numbers that are in close proximity to each other. As a result, when trying to insert key values into the primary key index, all processes are going to converge on the same index page, and the DBMS engine will have to serialize— through locks, latches, semaphores, or whichever locking mechanism is at its disposal— the various processes so that each one does not try to overwrite the bytes that another one is writing to. This is a typical example of contention that leads to some severe underuse of the hardware. Processes that could and should work in parallel have to wait in order, one behind the other. This bottleneck can be particularly severe on multi- processor machines, the very environment in which parallelism should be operating. Some database systems provide some means to reduce the impact of system-generated keys; for instance, Oracle allows you to define reverse indexes, indexes in which the sequence of bits making up the key is inversed before being stored into the index. To indicate a very approximate idea of what such an index looks like, let’s simply take the same names of marshals as we did in Figure 3-5 and reverse the letters instead of bits. We get something looking like Figure 3-10. It is easy to understand that even when we insert names that are alphabetically very close to one another, they are spread all over the various branches of the index tree: look for instance at the respective positions of MASSENA (a.k.a. ANESSAM), MORTIER (REITROM) and MURAT (TARUM). Therefore, we hit different places in the index and have much less contention than with a normally organized index. Close grouping of the index values would lead to a very high write activity that is very localized within the index. Before searching, Oracle simply applies the same reversing to the value we want to search against, and then proceeds as usual to traverse the index tree. Of course, every silver lining has its cloud: when our search condition attempts to use a leading string search like this: where name like 'M%' which is a typical range search, the reverse index is no help at all. By contrast, a regular index can be used to quickly identify the range of values beginning with a certain string that", + "source": "The Art of SQL.pdf", + "chunk_id": 68 + }, + { + "text": "search condition attempts to use a leading string search like this: where name like 'M%' which is a typical range search, the reverse index is no help at all. By contrast, a regular index can be used to quickly identify the range of values beginning with a certain string that we are interested in. The inability of reverse indexes to be used for range searches is, FIGURE 3-10. A simplified representation of reverse indexing www.it-ebooks.info 72 C H A P T E R T H R E E of course, a very minor inconvenience with system-generated keys, which are often unknown to end users and therefore unlikely to be the object of range scans. But it becomes a major hindrance when rows contain timestamps. Timestamped rows might arrive in close succession, making the timestamp column a potentially good candidate for reverse indexing, but then a timestamp is also the type of column against which we are quite likely to be looking for ranges of values. The hash index used in some database systems represents a different way to avoid bottlenecking index updates all on one index page. With hash indexing, the actual key is transformed into a meaningless, randomly distributed numeric key generated by the system, which is based on the column value being indexed. Although it is not impossible for two values to be transformed into similar, meaningless keys, two originally “close” keys will normally hash into two totally disconnected values. Once again, hash indexes represent a trick to avoid having a hot spot inside an index tree, but that benefit too, comes with certain restrictions. The use of a hash index is on an “equality or nothing” basis; in other words, range searching, or indeed any query against a part of the index key, is out of the question. Nevertheless, direct access based on one particular value of the key can be very fast. Even when there are solutions to alleviate contention risks, you should not create too many system-generated identifiers. I have sometimes found system-generated keys in each and every table (for instance a special, single detail_id for the type of order_details table mentioned above, instead of the more natural combination of order_id and some sequential number within the order). This blunderbuss approach is something that, by any standard, simply cannot be justified—especially for tables that are referenced by no other table. System-generated keys can provide benefit in the right circumstances, but beware of their indiscriminate use! Variability of Index Accesses It is very common to believe that if indexes are used in a query, then everything is fine. This is a gross misconception: there are many different characteristics of index access. Obviously, the most efficient type of index access is through a unique index, in which, at most, one row matches a given search value. Typically, such a search operation might be based on the primary key. However, as you saw in Chapter 2, accessing a table through its primary key may be very bad—if you", + "source": "The Art of SQL.pdf", + "chunk_id": 69 + }, + { + "text": "of index access is through a unique index, in which, at most, one row matches a given search value. Typically, such a search operation might be based on the primary key. However, as you saw in Chapter 2, accessing a table through its primary key may be very bad—if you are looping on all key values. Such an approach would be like using a teaspoon to move a big heap of sand instead of the big shovel of a full scan. So, at the tactical level, the most efficient index access is through a unique index, but the wider picture may reveal that this could be a costly mistake. www.it-ebooks.info T A C T I C A L D I S P O S I T I O N S 73 When several rows may match a single key value in a non-unique index (or when we search on a range of distinct values against a unique index), then we enter the world of range scanning. In this situation, we may retrieve a series of row addresses from the index, all containing the key values we are looking for. It may be a near-unique index, in which all key values match one row with the exception of a handful of values that match very few rows. Or it may be the other extreme of the non-unique indexed column for which all rows contain the same value. Indexed columns in which all rows contain the same value are in fact something you occasionally find with off-the-shelf software packages in which most columns are indexed, just in case. Never forget that finding the row in the index is all the work that is required only if: 1. You need no other information than data that is part of the index key. 2. The index is not compressed; otherwise, finding a match in the index is nothing more than a presumption that must be corroborated by the actual value found in the table. In all other cases, we are only halfway to meeting the query requirement, and we must now access each data block (or page) by the address that is provided by the search of the index. Once again, all other things being equal, we may have widely different performance, depending on whether we shall find the rows matching our search value lumped together in the same area of the disk, or scattered all over the place. The preceding description applies to the “regular” index accesses. However, a clever query optimizer may decide to use indexes in another way. It could operate on several indexes, combining them and doing some kind of pre-filtering before fetching the rows. It may decide to execute a full scan of a particular index, a strategy based on the judgment that this is the most efficient of all available methods for this particular query (we won’t go into the subtleties of what “most efficient” means here). The query optimizer may decide to systematically collect row addresses from", + "source": "The Art of SQL.pdf", + "chunk_id": 70 + }, + { + "text": "a full scan of a particular index, a strategy based on the judgment that this is the most efficient of all available methods for this particular query (we won’t go into the subtleties of what “most efficient” means here). The query optimizer may decide to systematically collect row addresses from an index, without taking the trouble to descend the index tree. So, any reference to an index in an execution plan is far from meaning that “all’s well that runs well.” Some index accesses may indeed be very fast—and some desperately slow. Even a fast access in a query is no guarantee that by combining the query with another one, we could have got the result even faster. In addition, if the optimizer is indeed smart enough to ignore a useless index in queries, that same useless index will nevertheless require to be maintained whenever the table contents are modified. This index maintenance is something that may be especially significant in the massive uploads or purges routinely performed by a batch program. Useful or useless, an index has to be maintained. Indexing is not a panacea: effective deployment rests on your complete understanding of the data you are dealing with and making the appropriate judgments. www.it-ebooks.info www.it-ebooks.info Chapter 4. C H A P T E R F O U R Maneuvering Thinking SQL Statements There is only one principle of war, and that’s this. Hit the other fellow, as quickly as you can, as hard as you can, where it hurts him most, when he ain’t lookin’. —Field Marshal Sir William Slim (1891–1970) quoting an anonymous Sergeant-Major www.it-ebooks.info 76 C H A P T E R F O U R I n this chapter, we are going to take a close look at the SQL query and examine how its construct can vary according to the tactical demands of particular situations. This will involve examining complex queries and reviewing how they can be decomposed into a succession of smaller components, all interdependent, and all contributing to a final, complete query. The Nature of SQL Before we begin examining query constructs in detail, we need to review some of the general characteristics of SQL itself: how it relates to the database engine and the associated optimizer, and what may limit the efficiency of the optimizer. SQL and Databases Relational databases owe their existence to pioneering work by E.F. Codd on the relational theory. From the outset, Codd’s work provided a very strong mathematical basis to what had so far been a mostly empirical discipline. To make an analogy, for thousands of years mankind has built bridges to span rivers, but frequently these structures were grossly overengineered simply because the master builders of the time didn’t fully understand the true relationships between the materials they used to build their bridges, and the consequent strengths of these bridges. Once the science of civil engineering developed a solid theoretical knowledge of material strengths, bridges of a far greater sophistication and safety began to emerge,", + "source": "The Art of SQL.pdf", + "chunk_id": 71 + }, + { + "text": "of the time didn’t fully understand the true relationships between the materials they used to build their bridges, and the consequent strengths of these bridges. Once the science of civil engineering developed a solid theoretical knowledge of material strengths, bridges of a far greater sophistication and safety began to emerge, demonstrating the full exploitation of the various construction materials being used. Indeed, the extraordinary dimensions of some modern bridges reflect the similarly huge increase in the data volumes that modern DBMS software is able to address. Relational theory has done for databases what civil engineering has done for bridges. It is very common to find confusion between the SQL language, databases, and the relational model. The function of a database is primarily to store data according to a model of the part of the real world from which that data has been obtained. Accordingly, a database must provide a solid infrastructure that will allow multiple users to make use of that same data, without, at any time, prejudicing the integrity of that data when they change it. This will require the database to handle contention between users and, in the extreme case, to keep the data consistent if the machine were to fail in mid-transaction. The database must also perform many other functions outside the scope of this book. As its name says, Structured Query Language, or SQL for short, is nothing other than a language, though admittedly with a very tight coupling to databases. Equating the SQL language with relational databases—or even worse with the relational theory—is as misguided as assuming that familiarity with a spreadsheet program or a word processor is www.it-ebooks.info M A N E U V E R I N G 77 indicative of having mastered “information technology.” In fact, some products that are not databases support SQL,* and before becoming a standard SQL had to compete against other languages such as RDO or QUEL, which were considered by many theorists to be superior to SQL. Whenever you have to solve what I shall generically call an SQL problem, you must realize that there are two components in action: the SQL expression of the query and the database optimizer. These two components interact within three distinct zones, as shown in Figure 4-1. At the center lies the relational theory, where mathematicians freely roam. If we simplify excessively, we can say that (amongst other useful things) the theory informs us that we can retrieve data that satisfies some criteria by using a handful of relational operators, and that these operators will allow us to answer basically any question. Most importantly, because the relational theory is so firmly grounded in mathematics, we can be totally confident that relational expressions can be written in different ways and yet return the same result. In exactly the same way, arithmetic teaches us that 246/369 is exactly the same as 2/3. However, despite the crucial theoretical importance of relational theory, there are aspects of great practical relevance that the relational theory has", + "source": "The Art of SQL.pdf", + "chunk_id": 72 + }, + { + "text": "expressions can be written in different ways and yet return the same result. In exactly the same way, arithmetic teaches us that 246/369 is exactly the same as 2/3. However, despite the crucial theoretical importance of relational theory, there are aspects of great practical relevance that the relational theory has nothing to say about. These fall into an area I call “reporting requirements.” The most obvious example in this area is the ordering of result sets. Relational theory is concerned only with the retrieval of a correct data set, as defined by a query. As we are practitioners and not theorists, for us the relational phase consists in correctly identifying the rows that will belong to our final result set. The matter of how some attributes (columns) of one row relate to similar attributes in another row doesn’t belong to this phase, and yet this is what ordering is all about. Further, relational theory has nothing to say about the numerous statistical functions (such as percentiles and the like) that often appear in various dialects of the * A good example would be sqlite, a remarkable storage engine that allows the management of data inside a file using SQL, but that is not a database server. FIGURE 4-1. DBMS Protagonists www.it-ebooks.info 78 C H A P T E R F O U R SQL language. The relational theory operates on set, and knows nothing of the imposition of ordering on these sets. Despite the fact that there are many mathematical theories built around ordering, none have any relevance to the relational theory. At this stage I must point out that what distinguishes relational operations from what I have called reporting requirements is that relational operations apply to mathematical sets of theoretically infinite extent. Irrespective of whether we are operating on tables of 10, one million, or one billion rows, we can apply any filtering criterion in an identical fashion. Once again, we are concerned only with identifying and returning the data that matches our criteria. Here, we are in the environment where the relational theory is fully applicable. Now, when we want to order rows (or perform an operation such as group by that most people would consider a relational operation) we are no longer working on a potentially infinite data set, but on a necessarily finite set. The consequent data set thus ceases to be a relation in the mathematical sense of the word. We are outside the bounds of the relational theory. Of course, this doesn’t mean that we cannot still do clever and useful things against this data using SQL. So we may, as a first approximation, represent an SQL query as a double-layered operation as shown in Figure 4-2; first, a relational core identifying the set of data we are going to operate on, second, a non-relational layer which works on this now finite set to give the polishing touch and produce the final result that the user expects. Despite Figure 4-2’s appealingly simple representation of", + "source": "The Art of SQL.pdf", + "chunk_id": 73 + }, + { + "text": "in Figure 4-2; first, a relational core identifying the set of data we are going to operate on, second, a non-relational layer which works on this now finite set to give the polishing touch and produce the final result that the user expects. Despite Figure 4-2’s appealingly simple representation of the place of SQL within the data environment, an SQL query will in most cases be considerably more complex than Figure 4-2 may suggest; Figure 4-2 only represents the overall pattern. The relational filter may be a generic name for several independent filters combined, for instance, through a union construct or by the means of subqueries, and the complexity of some SQL constructs can be considerable. I shall come back to the topic of SQL code a little later. But first I must talk about the relationship between the physical implementation of data and the database optimizer. FIGURE 4-2. The various layers of an SQL query www.it-ebooks.info M A N E U V E R I N G 79 Do not confuse the true relational functionality of the SQL query execution with the additional presentation layer. SQL and the Optimizer An SQL engine that receives a query to process will have to use the optimizer to find out how to execute that query in the most efficient way possible. Here the relational theory strikes again, because that theory informs the optimizer of transformations that are valid equivalents of the semantically correct query initially provided by the developer—even if that original query was clumsily written. Optimization is when the physical implementation of data comes into play. Depending on the existence of indexes and their usability in relation to a query, some transformations may result in much faster execution than other semantically equivalent transformations. Various storage models that I introduce in Chapter 5 may also make one particular way to execute a query irresistibly attractive. The optimizer examines the disposition of the indexes that are available, the physical layout of data, how much memory is available, and how many processors are available to be applied to the task of executing the query. The optimizer will also take into account information concerning the volume of the various tables and indexes that may be involved, directly or indirectly, through views used by the query. By weighing the alternatives that theory says are valid equivalents against the possibilities allowed by the implementation of the database, the optimizer will generate what is, hopefully, the best execution plan for the query. However, the key point to remember is that, although the optimizer may not always be totally weaponless in the non-relational layer of an SQL query, it is mainly in the relational core that it will be able to deploy its full power—precisely because of the mathematical underpinnings of the relational theory. The transformation from one SQL query to another raises an important point: it reminds us that SQL is supposed to be a declarative language. In other words, one should use SQL to express what is", + "source": "The Art of SQL.pdf", + "chunk_id": 74 + }, + { + "text": "able to deploy its full power—precisely because of the mathematical underpinnings of the relational theory. The transformation from one SQL query to another raises an important point: it reminds us that SQL is supposed to be a declarative language. In other words, one should use SQL to express what is required, rather than how that requirement is to be met. Going from what to how, should, in theory, be the work of the optimizer. You saw in Chapters 1 and 2 that SQL queries are only some of the variables in the equation; but even at the tactical query level, a poorly written query may prevent the optimizer from working efficiently. Remember, the mathematical basis of the relational theory provides an unassailable logic to the proceedings. Therefore, part of the art of SQL is to minimize the thickness, so to speak, of the non-relational layer—outside this layer, there is not much that the optimizer can safely do that guarantees returning exactly the same rows as the original query. www.it-ebooks.info 80 C H A P T E R F O U R Another part of the art of SQL is that when performing non-relational operations— loosely defined as operations for which the whole (at least at this stage) resulting dataset is known—we must be extremely careful to operate on only the data that is strictly required to answer the original question, and nothing more. Somehow, a finite data set, as opposed to the current row, has to be stored somewhere, and storing anything in temporary storage (memory or disk) requires significant overhead due to byte-pushing. This overhead may dramatically increase as the result set data volumes themselves increase, particularly if main memory becomes unavailable. A shortage of main memory would initiate the high-resource activity of swapping to disk, with all its attendant overheads. Moreover, always remember that indexes refer to disk addresses, not temporary storage—as soon as the data is in temporary storage, we must wave farewell to most fast access methods (with the possible exception of hashing). Some SQL dialects mislead users into believing that they are still in the relational world when they have long since left it. Take as a simple example the query “Who are the five top earners among employees who are not executives?”—a reasonable real-life question, although one that includes a distinctly non-relational twist. Identifying employees who are not executives is the relational part of the query, from which we obtain a finite set of employees that we can order. Several SQL dialects allow one to limit the number of rows returned by adding a special clause to the select statement. It is then fairly obvious that both the ordering and the limitation criteria are outside the relational layer. However, other dialects, the Oracle version figuring prominently here, use other mechanisms. What Oracle has is a dummy column named rownum that applies a sequential numbering to the rows in the order in which they are returned—which means the numbering is applied during the relational phase.", + "source": "The Art of SQL.pdf", + "chunk_id": 75 + }, + { + "text": "outside the relational layer. However, other dialects, the Oracle version figuring prominently here, use other mechanisms. What Oracle has is a dummy column named rownum that applies a sequential numbering to the rows in the order in which they are returned—which means the numbering is applied during the relational phase. If we write something such as: select empname, salary from employees where status != 'EXECUTIVE' and rownum <= 5 order by salary desc we get an incorrect result, at least in the sense that we are not getting the top five most highly paid nonexecutives, as the query might suggest at first glance. Instead, we get back the first five nonexecutives found—they could be the five lowest paid!—ordered in descending order of salary. (This query illustrates a well-known trap among Oracle practitioners, who have all been burnt at least once.) Let’s just be very clear about what is happening with the preceding query. The relational component of the query simply retrieves the first five rows (attributes empname and salary only) from the table employees where the employee is not an executive in a totally unpredictable order. Remember that relational theory tells us that a relation (and therefore the table that represents it) is not defined in any way by the order in which www.it-ebooks.info M A N E U V E R I N G 81 tuples (and therefore the rows in that table) are either stored or retrieved. As a consequence the nonexecutive employee with the highest salary may or may not be included in this result set—and there is no way we will ever know whether this result set actually meets our search criteria correctly. What we really want is to get all nonexecutives, order them by decreasing salary, and only then get the top five in the set. We can achieve this objective as follows: select * from (select empname, salary from employees where status != 'EXECUTIVE' order by salary desc) where rownum <= 5 So, how is our query layered in this case? Many would be tempted to say that by applying a filtering condition to an ordered result, we end up with something looking more or less like Figure 4-3. The truth, however, is more like Figure 4-4. Using constructs that look relational doesn’t take us back to the relational world, because to be in the relational world we must apply relational operators to relations. Our subquery uses an order by to sort the results. Once we’ve imposed ordering, we no longer have, strictly speaking, a relation (a relation is a set, and a set has no order). We end up with an outer select that looks relational on the surface but is applied to the output of an inline view in which a significant component (the order by clause) is not a relational process. FIGURE 4-3. A misleading view of what the \"top five nonexecutives\" query looks like www.it-ebooks.info 82 C H A P T E R F O U R My example of", + "source": "The Art of SQL.pdf", + "chunk_id": 76 + }, + { + "text": "the output of an inline view in which a significant component (the order by clause) is not a relational process. FIGURE 4-3. A misleading view of what the \"top five nonexecutives\" query looks like www.it-ebooks.info 82 C H A P T E R F O U R My example of the top five nonexecutives is, of course, a simple example, but do understand that once we have left the relational sphere in the execution of a query, we can no longer return to it. The best we can possibly do is to use the output of such a query to feed into the relational phase of an outer query. For instance, “in which departments are our five top nonexecutive earners working?” What is extremely important to understand, though, is that at this stage no matter how clever the optimizer is, it will be absolutely unable to combine the queries, and will more or less have to execute them in a given sequence. Further, any resulting set from an intermediate query is likely to be held in temporary storage, whether in memory or on disk, where the choice of access methods may be reduced. Once outside the pure relational layer, the way we write a query is of paramount importance for performance because it will inevitably impose onto the query some execution path from which the SQL engine will not be able to stray. To summarize, we can say that the safest approach we can adopt is to try to do as much of the job as possible inside the relational layer, where the optimizer can operate to maximum efficiency. When the situation is such that a given SQL task is no longer a purely relational problem, then we must be particularly careful about the construct, or the writing of the query itself. Understanding that SQL has, like Dr. Jekyll, a double nature is the key to mastering the language. If you see SQL as a single-edged sword, then you are condemned to remain in the world of tips and tricks for dummies, smarties, and mere mortals, possibly useful for impressing the opposite sex—although in my experience it doesn’t work much—but an approach that will never provide you with a deep understanding of how to cope with a difficult SQL problem. FIGURE 4-4. What the \"top five nonexecutives\" query is really like www.it-ebooks.info M A N E U V E R I N G 83 The optimizer rewards those who do the most work in the relational layer. Limits of the Optimizer Any decent SQL engine relies heavily on its query optimizer, which very often performs an excellent job. However, there are many aspects of the way optimizers work that you must keep in mind: Optimizers rely on the information they find in the database. This information is of two types: general statistical data (which must be verified as being fitting), and the essential declarative information held in the data defi- nitions. Where important semantic information relating to the data relations", + "source": "The Art of SQL.pdf", + "chunk_id": 77 + }, + { + "text": "must keep in mind: Optimizers rely on the information they find in the database. This information is of two types: general statistical data (which must be verified as being fitting), and the essential declarative information held in the data defi- nitions. Where important semantic information relating to the data relations is embedded in triggers or, worse, in application program code, that vital informa- tion will be totally unavailable to the optimizer. Such circumstances will inevita- bly impact the potential performance of the optimizer. Optimizers can perform to their best advantage where they can apply transformations that are mathematically proven to be equivalent. When they are required to assess components of a query that are non-relational in character, they are on less certain grounds and the execution path will stick more closely to what was voluntarily or involuntarily suggested by the original writing. The work of the optimizer contributes to the overall response time. Comparing a large number of alternative execution paths may take time. The end user sees only the total elapsed time and is unaware of how much was spent on optimization and how much on execution. A clever optimizer might allow itself more time to try to improve a query that it expects to take a lot of time to run, but there is always a self-imposed limit on its work. The trouble is that when you have a 20-way join (which is by no means unusual in some applica- tions), the number of combinations the optimizer could examine can become unmanageably large even when adequate indexing make some links obvious. Compound this with the inclusion of a combination of complex views and sub- queries, and at some point, the optimizer will have to give in. It is quite possible to find a situation in which a query running in isolation of any others may be very well optimized, while the same query deeply nested inside a much more complex outer query may take a completely wrong path. The optimizer improves individual queries. It is unable to relate independent queries one to another, however. Whatever its efforts, if the bulk of your program is fetching data inside procedural code just to feed into subsequent queries, the optimizer will not be able to do anything for you. www.it-ebooks.info 84 C H A P T E R F O U R Feed the optimizer with little chunks, and it will optimize little pieces. Feed it with a big chunk, and it will optimize a task. Five Factors Governing the Art of SQL You have seen in the first part of this chapter exactly how SQL includes both relational and non-relational characteristics. You have also seen how this affects the efficient (and not-so-efficient) workings of the database optimizer. From this point forward, and bearing in mind the lessons of the first part of this chapter, we can concentrate on the key factors that must be considered when using SQL. In my view, there are five main factors: • The total quantity", + "source": "The Art of SQL.pdf", + "chunk_id": 78 + }, + { + "text": "(and not-so-efficient) workings of the database optimizer. From this point forward, and bearing in mind the lessons of the first part of this chapter, we can concentrate on the key factors that must be considered when using SQL. In my view, there are five main factors: • The total quantity of data from which a result set has to be obtained • The criteria required to define the result set • The size of the result set • The number of tables to be processed in order to obtain the desired result set • The number of other users also modifying this same data Total Quantity of Data The volume of data we need to read is probably the most important factor to take into account; an execution plan that is perfectly suitable for a fourteen-row emp table and a four-row dept table may be entirely inappropriate for dealing with a 15 million–row financial_flows table against which we have to join a 5 million–row products table. Note that even a 15 million–row table will not be considered particularly large by the standards of many companies. As a matter of consequence, it is hard to pronounce on the efficiency of a query before having run it against the target volume of data. Criteria Defining the Result Set When we write an SQL statement, in most cases it will involve filtering conditions located in where clauses, and we may have several where clauses—a major one as well as minor ones—in subqueries or views (regular views or in-line views). A filtering condition may be efficient or inefficient. However, the significance of efficient or inefficient is strongly affected by other factors, such as physical implementation (as discussed in Chapter 5) and once again, by how much data we have to wade through. We need to approach the subject of defining the result in several parts, by considering filtering, the central SQL statements, and the impact of large data volumes on our queries. But this is a particularly complex area that needs to be treated in some depth, so I’ll reserve this discussion until later in this chapter, in the major section entitled “Filtering.” www.it-ebooks.info M A N E U V E R I N G 85 Size of the Result Set An important and often overlooked factor is how much data a query returns (or how much data a statement changes). This is often dependent on the size of the tables and the details of the filtering, but not in every case. Typically, the combination of several selection criteria which of themselves are of little value in selecting data may result in highly efficient filtering when used in combination with one another. For example, one could cite that retrieving students’ names based on whether they received a science or an arts degree will give a large result set, but if both criteria are used (e.g., students who studied under both disciplines) the consequent result set will collapse to a tiny number. In the case", + "source": "The Art of SQL.pdf", + "chunk_id": 79 + }, + { + "text": "one could cite that retrieving students’ names based on whether they received a science or an arts degree will give a large result set, but if both criteria are used (e.g., students who studied under both disciplines) the consequent result set will collapse to a tiny number. In the case of queries in particular, the size of the result set matters not so much from a technical standpoint, but mostly because of the end user’s perception. To a very large extent, end users adjust their patience to the number of rows they expect: when they ask for one needle, they pay little attention to the size of the haystack. The extreme case is a query that returns nothing, and a good developer should always try to write queries that return few or no rows as fast as possible. There are few experiences more frustrating than waiting for several minutes before finally seeing a “no data found” message. This is especially annoying if you have mistyped something, realized your error just after hitting Enter, and then have been unable to abort the query. End users are willing to wait to get a lot of data, but not to get an empty result. If we consider that each of our filtering criteria defines a particular result set and the final result set is either the intersection (when conditions are anded) or the union (when conditions are ored together) of all the intermediate result sets, a zero result is most likely to result from the intersection of small, intermediate result sets. In other words, the (relatively) most precise criteria are usually the primary reason for a zero result set. Whenever there is the slightest possibility that a query might return no data, the most likely condition that would result in a null return should be checked first—especially if it can be done quickly. Needless to say, the order of evaluation of criteria is extremely context-sensitive as you shall see later under “Filtering.” A skillful developer should aim for response times proportional to the number of rows returned. Number of Tables The number of tables involved in a query will naturally have some influence on performance. This is not because a DBMS engine performs joins badly—on the contrary, modern systems are able to join large numbers of tables very efficiently. www.it-ebooks.info 86 C H A P T E R F O U R Joins The perception of poor join performance is another enduring myth associated with relational databases. Folklore has it that one should not join too many tables, with five often suggested as the limit. In fact, you can quite easily have 15 table joins perform extremely well. But there are additional problems associated with joining a large number of tables, of which the following are examples: • When you routinely need to join, say, 15 tables, you can legitimately question the cor- rectness of the design; keep in mind what I said in Chapter 1—that a row in a table states some", + "source": "The Art of SQL.pdf", + "chunk_id": 80 + }, + { + "text": "associated with joining a large number of tables, of which the following are examples: • When you routinely need to join, say, 15 tables, you can legitimately question the cor- rectness of the design; keep in mind what I said in Chapter 1—that a row in a table states some kind of truth and can be compared to a mathematical axiom. By joining tables, we derive other truths. But there is a point at which we must decide whether something is an obvious truth that we can call an axiom, or whether it is a less obvi- ous truth that we must derive. If we spend much of our time deriving our truths, per- haps our axioms are poorly chosen in the first place. • For the optimizer, the complexity increases exponentially as the number of tables increases. Once again, the excellent work usually performed by a statistical optimizer may comprise a significant part of the total response time for a query, particularly when the query is run for the first time. With large numbers of tables, it is quite impractical for the optimizer to explore all possible query paths. Unless a query is writ- ten in a way that eases the work of the optimizer, the more complex the query, the greater the chance that the optimizer will bet on the wrong horse. • When we write a complex query involving many tables, and when joins can be written in several fairly distinct ways, the odds are high that we’ll pick the wrong construct. If we join tables A to B to C to D, the optimizer may not have all the information present to know that A can be very efficiently joined directly to D, particularly if that join hap- pens to be a special case. A sloppy developer trying to fix duplicate rows with a distinct can also easily overlook a missing join condition. Complex queries and complex views Be aware that the apparent number of tables involved in a query can be deceptive; some of the tables may actually be views, and sometimes pretty complex ones, too. Just as with queries, views can also have varying degrees of complexity. They can be used to mask columns, rows, or even a combination of rows and columns to all but a few privileged users. They can also be used as an alternate perspective on the data, building relations that are derived from the existing relations stored as tables. In cases such as these, a view can be considered shorthand for a query, and this is probably one of the most common usages of views. With increasingly complex queries, there is a temptation to break a query down into a succession of individual views, each representing a component of the greater query. www.it-ebooks.info M A N E U V E R I N G 87 The simplicity of a given query may hide the complexity of participating views. Like most extreme positions, it would be absurd to banish views", + "source": "The Art of SQL.pdf", + "chunk_id": 81 + }, + { + "text": "a succession of individual views, each representing a component of the greater query. www.it-ebooks.info M A N E U V E R I N G 87 The simplicity of a given query may hide the complexity of participating views. Like most extreme positions, it would be absurd to banish views altogether. Many of them are rather harmless animals. However, when a view is itself used in a rather complex query, in most cases we are only interested in a small fraction of the data returned by the view—possibly in a couple of columns, out of a score or more. The optimizer may attempt to recombine a simple view into a larger query statement. However, once a query reaches a relatively modest level of complexity, this approach may become too complex in itself to enable efficient processing. In some cases a view may be written in a way that effectively prevents the optimizer from combining it into the larger statement. I have already mentioned rownums, those virtual columns used in Oracle to indicate the order in which rows are initially found. When rownums are used inside a view, a further level of complexity is introduced. Any attempt to combine a view that references a rownum into a larger statement would be almost guaranteed to change the subsequent rownum order, and therefore the optimizer doesn’t permit a query rewrite in those circumstances. In a complicated query, such a view will necessarily be executed in isolation. In quite a number of cases then, the DBMS optimizer will push a view as is into a statement,* running it as a step in the statement execution, and using only those elements that are required from the result of the view execution. Frequently, many of the operations executed in a view (typically joins to return a description associated with codes) will be irrelevant in the context of a larger query, or a query may have special search criteria that would have been particularly selective when applied to the tables underlying the view. For instance, a subsequent union may prove to be totally unnecessary because the view is the union of several tables representing subtypes, and the larger query filters on only one of the subtypes. There is also the danger of joining a view with a table that itself appears in the same view, thus forcing multiple passes over this table and probably hitting the same rows several times when one pass would have been quite sufficient. When a view returns much more data than required in the context of a query that references that view, dramatic performance gains can often be obtained by eliminating the view (or using a simpler version of the view). Begin by replacing the view reference in the main query with the underlying SQL query used to define the view. With the components of the view in full sight, it becomes easy to remove everything that is not * The optimizer may also sometimes push criteria down into the view. www.it-ebooks.info", + "source": "The Art of SQL.pdf", + "chunk_id": 82 + }, + { + "text": "by replacing the view reference in the main query with the underlying SQL query used to define the view. With the components of the view in full sight, it becomes easy to remove everything that is not * The optimizer may also sometimes push criteria down into the view. www.it-ebooks.info 88 C H A P T E R F O U R strictly necessary. More often than not, it’s precisely what isn’t necessary that prevents the view from being merged by the optimizer, and a simpler, cut-down view may give excellent results. When the query is correctly reduced to its most basic components, it runs much faster. Many developers may hesitate to push the code for a very complex view into an already complex query, not least because it can make a complex situation even more complicated. The exercise of developing and factoring a complex SQL expression may indeed appear to be daunting. It is, however, an exercise quite similar to the development of mathematical expressions, as practiced in high school. It is, in my view, a very formative exercise and well worth the effort of mastering. It is a discipline that provides a very sound understanding of the inner workings of a query for developers anxious to improve their skills, and in most cases the results can be highly rewarding. Rather than embedding a view inside a query when that view returns unnecessary elements, try to decompose the view components into the main query body. Number of Other Users Finally, concurrency is a factor that you must carefully take into account when designing your SQL code. Concurrency is usually a concern while writing to the database where block-access contention, locking, latching (which means locking of internal DBMS resources), and others are the more obvious problem areas; even read consistency can in some cases lead to some degree of contention. Any server, no matter how impressive its specification, will always have a finite capacity. The ideal plan for a query running on a machine with little to no concurrency is not necessarily the same as the ideal plan for the same query running on the same machine with a high level of concurrency. Sorts may no longer find the memory they need and may instead resort to writing to disk, thus creating a new source of contention. Some CPU-intensive operations—for example, the computation of complicated functions, repetitive scanning of index blocks, and so forth— may cause the computer to overload. I have seen cases in which more physical I/Os resulted in a significantly better time to perform a given task. In those cases, there was a high level of concurrency for CPU-intensive operations, and when some processes had to wait for I/Os, the overworked CPUs were relieved and could run other processes, thus ensuring a better overlap. We must often think in terms of global throughput of one particular business task, rather than in terms of individual user response-time. NOTE Chapter 9 examines concurrency in greater detail. www.it-ebooks.info M A", + "source": "The Art of SQL.pdf", + "chunk_id": 83 + }, + { + "text": "for I/Os, the overworked CPUs were relieved and could run other processes, thus ensuring a better overlap. We must often think in terms of global throughput of one particular business task, rather than in terms of individual user response-time. NOTE Chapter 9 examines concurrency in greater detail. www.it-ebooks.info M A N E U V E R I N G 89 Filtering How you restrict your result set is one of the most critical factors that helps you determine which tactics to apply when writing an SQL statement. The collective criteria that filters the data are often seen as a motley assortment of conditions associated in the where clause. However, you should very closely examine the various where-clause (and having-clause, too) conditions when writing SQL code. Meaning of Filtering Conditions Given the syntax of the SQL language, it is quite natural to consider that all filtering conditions, as expressed in the where clause, are similar in nature. This is absolutely not the case. Some filtering conditions apply directly to the select operator of relational theory, where checking that a column in a row (purists would say an attribute in a relation variable) matches (or doesn’t match) a given condition. However, historically the where clause also contains conditions that implement another operator—the join operator. There is, since the advent of the SQL92 join syntax, an attempt to differentiate join filtering conditions, located between the (main) from clause and the where clause, from the select filtering conditions listed in the where clause. Joining two (or more) tables logically creates a new relation. Consider this general example of a join: select ..... from t1 inner join t2 on t1.join1 = t2.joind2 where ... Should a condition on column c2 belonging to t2 come as an additional condition on the inner join, expressing that in fact you join on a subset of t2? Or should a condition inside the where clause, along with conditions on columns of t1, express that the filtering applies to the result of joining t1 to t2? Wherever you choose to place your join condition ought not to make much of a difference; however, it has been known to lead to variations in performance with some optimizers. We may also have conditions other than joins and the simple filtering of values. For instance, we may have conditions restricting the returned set of rows to some subtype; we may also have conditions that are just required to check the existence of something inside another table. All these conditions are not necessarily semantically identical, although the SQL syntax makes all of them look equivalent. In some cases, the order of evaluation of the conditions is of no consequence; in other cases, it is significant. Here’s an example that you can actually find in more than one commercial software package to illustrate the importance of the order of the evaluation of conditions. Suppose that we have a parameters table, which holds: parameter_name, parameter_type, and www.it-ebooks.info 90 C H A P T E R F", + "source": "The Art of SQL.pdf", + "chunk_id": 84 + }, + { + "text": "significant. Here’s an example that you can actually find in more than one commercial software package to illustrate the importance of the order of the evaluation of conditions. Suppose that we have a parameters table, which holds: parameter_name, parameter_type, and www.it-ebooks.info 90 C H A P T E R F O U R parameter_value, with parameter_value being the string representation of whatever type of parameter we have, as defined by the attribute parameter_type. (To the logical mind this is indeed a story of more woe than that of Juliet and her Romeo, since the domain type of attribute parameter_value is a variable feast and thus offends a primary rule of relational theory.) Say that we issue a query such as: select * from parameters where parameter_name like '%size' and parameter_type = 'NUMBER' With this query, it does not matter whether the first condition is evaluated before or after the second one. However, if we add the following condition, where int( ) is a function to convert from char to integer value, then the order of evaluation becomes very significant: and int(parameter_value) > 1000 Now, the condition on parameter_type must be evaluated before the condition on the value, because otherwise we risk a run-time error consequent upon attempting to convert a character string (if for example parameter_type for that row is defined as char) to an integer. The optimizer may not be able to figure out that the poor design demands that one condition should have higher priority—and you may have trouble specifying it to the database. All search criteria are not equal; some are more equal than others. Evaluation of Filtering Conditions The very first questions to consider when writing a SQL statement are: • What data is required, and from which tables? • What input values will we pass to the DBMS engine? • What are the filtering conditions that allow us to discard unwanted rows? Be aware, however, that some data (principally data used for joining tables) may be stored redundantly in several tables. A requirement to return values known to be held in the primary key of a given table doesn’t necessarily mean that this table must appear in the from clause, since this primary key may well appear as the foreign key of another table from which we also need the data. Even before writing a query, we should rank the filtering conditions. The really efficient ones (of which there may be several, and which may apply to different tables) will drive the query, and the inefficient ones will come as icing on the cake. What is the criterion that defines an efficient filter? Primarily, one that allows us to cut down the volume of www.it-ebooks.info M A N E U V E R I N G 91 the data we have to deal with as fast as possible. And here we must pay a lot of attention to the way we write; the following subsections work through a simple example to illustrate my point. Buyers", + "source": "The Art of SQL.pdf", + "chunk_id": 85 + }, + { + "text": "M A N E U V E R I N G 91 the data we have to deal with as fast as possible. And here we must pay a lot of attention to the way we write; the following subsections work through a simple example to illustrate my point. Buyers of Batmobiles Assume that we have four tables, namely customers, orders, orderdetail, and a table of articles, as shown in Figure 4-5. Please note that in the figure the sizes of the boxes representing each table are more or less proportional to the volume of data in each table, not simply to the number of columns. Primary key columns are underlined. Let’s now suppose that our SQL problem is to find the names of all the customers living in the city named “Gotham” who have ordered the article called “Batmobile” during the last six months. We have, of course, several ways to formulate this query; the following is probably what an ANSI SQL fan would write: select distinct c.custname from customers c join orders o on o.custid = c.custid join orderdetail od on od.ordid = o.ordid join articles a on a.artid = od.artid where c.city = 'GOTHAM' and a.artname = 'BATMOBILE' and o.ordered >= somefunc somefunc is supposed to be a function that returns the date six months prior to the current date. Notice too, the presence of distinct, which may be required if one of our customers is an especially heavy consumer of Batmobiles and has recently ordered several of them. Let’s forget for a while that the optimizer may rewrite the query, and look at the execution plan such a statement suggests. First, we walk the customers table, keeping only rows for which the city happens to be Gotham. Then we search the orders table, which means that the custid column there had better be indexed, because otherwise the only hope the SQL engine has of executing the query reasonably fast is to perform some FIGURE 4-5. A classical order schema www.it-ebooks.info 92 C H A P T E R F O U R sorting and merging or to scan the orders table to build a hash table and then operate against that. We are going to apply another filter at this level, against the order date. A clever optimizer will not mind finding the filtering condition in the where clause and will understand that in order to minimize the amount of data to join it must filter on the date before performing the join. A not so clever optimizer might be tempted to join first, and then filter, and may therefore be grateful to you for specifying the filtering condition with the join condition, as follows: join orders o on o.custid = c.custid and a.ordered >= somefunc Even if the filtering condition really has nothing to do with the join, it is sometimes difficult for the optimizer to understand when that is the case. If the primary key of orderdetail is defined as (ordid, artid) then, because", + "source": "The Art of SQL.pdf", + "chunk_id": 86 + }, + { + "text": "orders o on o.custid = c.custid and a.ordered >= somefunc Even if the filtering condition really has nothing to do with the join, it is sometimes difficult for the optimizer to understand when that is the case. If the primary key of orderdetail is defined as (ordid, artid) then, because ordid is the first attribute of the index, we can make use of that index to find the rows associated with an order as in Chapter 3. But if the primary key happens to be (artid, ordid) (and note, either version is exactly the same as far as relational theory is concerned), then tough luck. Some products may be able to make some use of the index* in that case, but it will not provide the efficient access that (ordid, artid) would have allowed. Other products will be totally unable to use the index. The only circumstance that may save us is the existence of a separate index on ordid. Once we have linked orderdetails to orders, we can proceed to articles—without any problem this time since we found artid, the primary key, in orderdetail. Finally, we can check whether the value in articles is or is not a Batmobile. Is this the end of the story? Not quite. As instructed by distinct, we must now sort the resulting set of customer names that have passed across all the filtering layers so as to eliminate duplicates. It turns out that there are several alternative ways of expressing the query that I’ve just described. One example is to use the older join syntax, as follows: select distinct c.custname from customers c, orders o, orderdetail od, articles a where c.city = 'GOTHAM' and c.custid = o.custid and o.ordid = od.ordid and od.artid = a.artid and a.artname = 'BATMOBILE' and o.ordered >= somefunc * A feature known as skip-scan may allow for searching the index. www.it-ebooks.info M A N E U V E R I N G 93 It may just be old habits dying hard, but I prefer this older way, if only for one simple reason: it makes it slightly more obvious that from a logical point of view the order in which we process data is arbitrary, because the same data will be returned irrespective of the order in which we inspect tables. Certainly the customers table is particularly important, since that is the source from which we obtain the data that is ultimately required, while in this very specific context, all the other tables are used purely to support the remaining selection processes. One really has to understand that there is no one recipe that works for all cases. The pattern of table joins will vary for each situation you encounter. The deciding factor is the nature of the data you are dealing with. A given approach in SQL may solve one problem, but make another situation worse. The way queries are written is a bit like a drug that may heal one patient but kill another. More Batmobile", + "source": "The Art of SQL.pdf", + "chunk_id": 87 + }, + { + "text": "you encounter. The deciding factor is the nature of the data you are dealing with. A given approach in SQL may solve one problem, but make another situation worse. The way queries are written is a bit like a drug that may heal one patient but kill another. More Batmobile purchases Let’s explore alternative ways to list our buyers of Batmobiles. In my view, as a general rule, distinct at the top level should be avoided whenever possible. The reason is that if we have overlooked a join condition, a distinct will hide the problem. Admittedly this is a greater risk when building queries with the older syntax, but nevertheless still a risk when using the ANSI/SQL92 syntax if tables are joined through several columns. It is usually much easier to spot duplicate rows than it is to identify incorrect data. It’s easy to give a proof of the assertion that incorrect results may be difficult to spot: the two previous queries that use distinct to return the names of the customers may actually return a wrong result. If we happen to have several customers named “Wayne,” we won’t get that information because distinct will not only remove duplicates resulting from multiple orders by the same customer, but also remove duplicates resulting from homonyms. In fact, we should return both the unique customer id and the customer name to be certain that we have the full list of Batmobile buyers. We can only guess at how long it might take to identify such an issue in production. How can we get rid of distinct then? By acknowledging that we are looking for customers in Gotham that satisfy an existence test, namely a purchase order for a Batmobile in the past six months. Note that most, but not all, SQL dialects support the following syntax: select c.custname from customers c where c.city = 'GOTHAM' and exists (select null from orders o, orderdetail od, articles a where a.artname = 'BATMOBILE' and a.artid = od.artid and od.ordid = o.ordid and o.custid = c.custid and o.ordered >= somefunc) www.it-ebooks.info 94 C H A P T E R F O U R If we use an existence test such as this query uses, a name may appear more than once if it is common to several customers, but each individual customer will appear only once, irrespective of the number of orders they placed. You might think that my criticism of the ANSI SQL syntax was a little harsh, since customers figure as prominently, if not more prominently than before. However, it now features as the source for the data we want the query to return. And another query, nested this time, appears as a major phase in the identification of the subset of customers. The inner query in the preceding example is strongly linked to the outer select. As you can see on line 11 (in bold), the inner query refers to the current row of the outer query. Thus, the inner query is what", + "source": "The Art of SQL.pdf", + "chunk_id": 88 + }, + { + "text": "phase in the identification of the subset of customers. The inner query in the preceding example is strongly linked to the outer select. As you can see on line 11 (in bold), the inner query refers to the current row of the outer query. Thus, the inner query is what is called a correlated subquery. The snag with this type of subquery is that we cannot execute it before we know the current customer. Once again, we are assuming that the optimizer doesn’t rewrite the query. Therefore we must first find each customer and then check for each one whether the existence test is satisfied. Our query as a whole may perform excellently if we have very few customers in Gotham. It may be dreadful if Gotham is the place where most of our customers are located (a case in which a sophisticated optimizer might well try to execute the query in a different way). We have still another way to write our query, which is as follows: select custname from customers where city = 'GOTHAM' and custid in (select o.custid from orders o, orderdetail od, articles a where a.artname = 'BATMOBILE' and a.artid = od.artid and od.ordid = o.ordid and o.ordered >= somefunc) In this case, the inner query no longer depends on the outer query: it has become an uncorrelated subquery. It needs to be executed only once. It should be obvious that we have now reverted the flow of execution. In the previous case, we had to search first for customers in the right location (e.g., where city is Gotham), and then check each order in turn. In this latest version of the query, the identifiers of customers who have ordered what we are looking for are obtained via a join that takes place in the inner query. If you have a closer look, however, there are more subtle differences as well between the current and preceding examples. In the case of the correlated subquery, it is of paramount importance to have the orders table indexed on custid; in the second case, it no longer matters, since then the index (if any) that will be used is the index associated with the primary key of customers. www.it-ebooks.info M A N E U V E R I N G 95 You might notice that the most recent version of the query performs an implicit distinct. Indeed, the subquery, because of its join, might return many rows for a single customer. That duplication doesn’t matter, because the in condition checks only to see whether a value is in the list returned by the subquery, and in doesn’t care whether a given value is in that list one time or a hundred times. Perhaps though, for the sake of consistency we should apply the same rules to the subquery that we have applied to the query as a whole, namely to acknowledge that we have an existence test within the subquery as well: select custname from customers where city =", + "source": "The Art of SQL.pdf", + "chunk_id": 89 + }, + { + "text": "a hundred times. Perhaps though, for the sake of consistency we should apply the same rules to the subquery that we have applied to the query as a whole, namely to acknowledge that we have an existence test within the subquery as well: select custname from customers where city = 'GOTHAM' and custid in (select o.custid from orders o where o.ordered >= somefunc and exists (select null from orderdetail od, articles a where a.artname = 'BATMOBILE' and a.artid = od.artid and od.ordid = o.ordid)) or: select custname from customers where city = 'GOTHAM' and custid in (select custid from orders where ordered >= somefunc and ordid in (select od.ordid from orderdetail od, articles a where a.artname = 'BATMOBILE' and a.artid = od.artid) Irrespective of the fact that our nesting is getting deeper and becoming less legible, choosing which query is the best between the exists and the in follows the very same rule inside the subquery as before: the choice depends on the effectiveness of the condition on the date versus the condition on the article. Unless business has been very, very slow for the past six months, one might reasonably expect that the most efficient condition on which to filter the data will be the one on the article name. Therefore, in the particular case of the subquery, in is better than exists because it will be faster to find all the order lines that refer to a Batmobile and then to check whether the sale occurred in the last six months rather than the other way round. This approach will be faster assuming that the table orderdetail is indexed on artid; otherwise, our bright, tactical move will fail dismally. www.it-ebooks.info 96 C H A P T E R F O U R NOTE It may be a good idea to check in against exists whenever an existence test is applied to a significant number of rows. Most SQL dialects allow you to rewrite uncorrelated subqueries as inline views in the from clause. However, you must always remember that an in performs an implicit removal of duplicate values, which must become explicit when the subquery is moved to become an in-line view in the from clause. For example: select custname from customers where city = 'GOTHAM' and custid in (select o.custid from orders o, (select distinct od.ordid from orderdetail od, articles a where a.artname = 'BATMOBILE' and a.artid = od.artid) x where o.ordered >= somefunc and x.ordid = o.ordid) The different ways you have to write functionally equivalent queries (and variants other than those given in this section are possible) are comparable to words that are synonyms. In written and spoken language, synonyms have roughly the same meaning, but each one introduces a subtle difference that makes one particular word more suitable to a particular situation or expression (and in some cases another synonym is totally inappropriate). In the same way, both data and implementation details may dictate the choice of one query variant over others. Lessons to be", + "source": "The Art of SQL.pdf", + "chunk_id": 90 + }, + { + "text": "each one introduces a subtle difference that makes one particular word more suitable to a particular situation or expression (and in some cases another synonym is totally inappropriate). In the same way, both data and implementation details may dictate the choice of one query variant over others. Lessons to be learned from the Batmobile trade The various examples of SQL that you saw in the preceding section may look like an idle exercise in programming dexterity, but they are more than that. The key point is that there are many different ways in which we can attack the data, and that we don’t necessarily have to go first through customers, then orders, then orderdetail, and then articles as some of the ways of writing the query might suggest. If we represent the strength of our search criteria with arrows—the more discriminant the criterion, the larger the arrow—we can assume that we have very few customers in Gotham, but that we sell quite a number of Batmobiles and business has been brisk for the past six months, in which case our battle map may look like Figure 4-6. Although we have a condition on the article name, the medium arrow points to orderdetail because that is what truly matters. We may have very few articles for sale, which may represent similar percentages of our revenue, or we may have a huge number of articles, of which one of the best sellers is the Batmobile. www.it-ebooks.info M A N E U V E R I N G 97 Alternatively, we can assume that most of our customers are indeed based in Gotham, but that very few actually buy Batmobiles, in which case our battle plan will look more like Figure 4-7. It is quite obvious then, that we really have to cut to pieces the orderdetail table, which is the largest one. The faster we slash this table, the faster our query will run. Note also—and this is a very important point—that the criterion “during the last six months” is not a very precise one. But what if we change the criterion to specify the last two months and happen to have 10 years of sales history online? In that case, it may be more efficient to get to those recent orders first—which, thanks to some techniques described in Chapter 5, may be clustered together—and then start from there, selecting customers from Gotham, on the one hand, and orders for Batmobiles on the other. To put it another way, the best execution plan does not only depend on the data values, it may also evolve over time. FIGURE 4-6. When query discrimination is based on location FIGURE 4-7. When query discrimination is based on purchase www.it-ebooks.info 98 C H A P T E R F O U R What then can we conclude from all this? First, that there is more than one way to skin a cat...and that an expression of a query is usually associated with implicit assumptions about the", + "source": "The Art of SQL.pdf", + "chunk_id": 91 + }, + { + "text": "based on purchase www.it-ebooks.info 98 C H A P T E R F O U R What then can we conclude from all this? First, that there is more than one way to skin a cat...and that an expression of a query is usually associated with implicit assumptions about the data. With each different expression of a query we will obtain the same result set, but it may be at significantly different speeds. The way we write the query may influence the execution path, especially when we have to apply criteria that cannot be expressed within the truly relational part of the environment. If the optimizer is to be allowed to function at its best, we must try to maximize the amount of true relational processing and ensure the non-relational component has minimum impact on the final result. We have assumed all along in this chapter that statements will be run as suggested by the way they are written. Be aware though, that an optimizer may rewrite queries— sometimes pretty aggressively. You could argue that rewrites by the optimizer don’t matter, because SQL is supposed to be a declarative language in which you state what you want and let the DBMS provide it. However, you have seen that each time we have rewritten a query in a different way, we have had to change assumptions about the distribution of data and about existing indexes. It is highly important, therefore, to anticipate the work of the optimizer to be certain that it will find what it needs, whether in terms of indexes or in terms of detailed-enough statistical information about the data. The correct result from an SQL statement is only the first step in building the best SQL. Querying Large Quantities of Data It may sound obvious, but the sooner we get rid of unwanted data, the less we have to process at later stages of a query—and the more efficiently the query will run. An excellent application of this principle can be found with set operators, of which union is probably the most widely used. It is quite common to find in a moderately complex union a number of tables appearing in several of the queries “glued” together with the union operator. One often sees the union of fairly complex joins, with most of the joined tables occurring in both select statements of the union—for example, on both sides of the union, something like the following: select ... from A, B, C, D, E1 where (condition on E1) and (joins and other conditions) www.it-ebooks.info M A N E U V E R I N G 99 union select ... from A, B, C, D, E2 where (condition on E2) and (joins and other conditions) This type of query is typical of the cut-and-paste school of programming. In many cases it may be more efficient to use a union of those tables that are not common, complete with the screening conditions, and to then push that union into an inline", + "source": "The Art of SQL.pdf", + "chunk_id": 92 + }, + { + "text": "and (joins and other conditions) This type of query is typical of the cut-and-paste school of programming. In many cases it may be more efficient to use a union of those tables that are not common, complete with the screening conditions, and to then push that union into an inline view and join the result, writing something similar to: select ... from A, B, C, D, (select ... from E1 where (condition on E1) union select ... from E2 where (condition on E2)) E where (joins and other conditions) Another classic example of conditions applied at the wrong place is a danger associated with filtering when a statement contains a group by clause. You can filter on the columns that define the grouping, or the result of the aggregate (for instance when you want to check whether the result of a count( ) is smaller than a threshold) or both. SQL allows you to specify all such conditions inside the having clause that filters after the group by (in practice, a sort followed by an aggregation) has been completed. Any condition bearing on the result of an aggregate function must be inside the having clause, since the result of such a function is unknown before the group by. Any condition that is independent on the aggregate should go to the where clause and contribute to decrease the number of rows that we shall have to sort to perform the group by. Let’s return to our customers and orders example, admitting that the way we process orders is rather complicated. Before an order is considered complete, we have to go through several phases that are recorded in the table orderstatus, of which the main columns are ordid, the identifier of the order; status; and statusdate, which is a timestamp. The primary key is compound, consisting of ordid, and statusdate. Our requirement is to list, for all orders for which the status is not flagged as complete www.it-ebooks.info 100 C H A P T E R F O U R (assumed to be final), the identifier of the order, the customer name, the last known order status, and when this status was set. To that end, we might build the following query, filtering out completed orders and identifying the current status as the latest status assigned: select c.custname, o.ordid, os.status, os.statusdate from customers c, orders o, orderstatus os where o.ordid = os.ordid and not exists (select null from orderstatus os2 where os2.status = 'COMPLETE' and os2.ordid = o.ordid) and os.statusdate = (select max(statusdate) from orderstatus os3 where os3.ordid = o.ordid) and o.custid = c.custid At first sight this query looks reasonable, but in fact it contains a number of deeply disturbing features. First, notice that we have two subqueries, and notice too that they are not nested as in the previous examples, but are only indirectly related to each other. Most worrying of all, both subqueries hit the very same table, already referenced at the outer level. What kind of filtering condition", + "source": "The Art of SQL.pdf", + "chunk_id": 93 + }, + { + "text": "First, notice that we have two subqueries, and notice too that they are not nested as in the previous examples, but are only indirectly related to each other. Most worrying of all, both subqueries hit the very same table, already referenced at the outer level. What kind of filtering condition are we providing? Not a very precise one, as it only checks for the fact that orders are not yet complete. How can such a query be executed? An obvious approach is to scan the orders table, for each row checking whether each order is or is not complete. (Note that we might have been happy to find this information in the orders table itself, but this is not the case.) Then, and only then, can we check the date of the most recent status, executing the subqueries in the order in which they are written. The unpleasant fact is that both subqueries are correlated. Since we have to scan the orders table, it means that for every row from orders we shall have to check whether we encounter the status set to COMPLETE for that order. The subquery to check for that status will be fast to execute, but not so fast when repeated a large number of times. When there is no COMPLETE status to be found, then a second subquery must be executed. What about trying to un-correlate queries? The easiest query to un-correlate happens to be the second one. In fact, we can write, at least with some SQL dialects: and (o.ordid, os.statusdate) = (select ordid, max(statusdate) from orderstatus group by ordid) www.it-ebooks.info M A N E U V E R I N G 101 The subquery that we have now will require a full scan of orderstatus; but that’s not necessarily bad, and we’ll discuss our reasoning in a moment. There is something quite awkward in the condition of the pair of columns on the left- hand side of the rewritten subquery condition. These columns come from different tables, and they need not do so. In fact, we want the order identifier to be the same in orders and orderstatus; will the optimizer understand the subtlety of this situation? That is rather uncertain. If the optimizer doesn’t understand, then it will be able to execute the subquery first, but will have to join the two other tables together before being able to exploit the result of the subquery. If the query were written slightly differently, the optimizer would have greater freedom to decide whether it actually wants to do what I’ve just described or exploit the result of the subquery and then join orders to orderstatus: and (os.ordid, os.statusdate) = (select ordid, max(statusdate) from orderstatus group by ordid) The reference on the left side to two columns from the same table removes the dependence of identification of the most recent status for the order on a preliminary join between orderstatus and orders. A very clever optimizer might have performed the modification for us, but it", + "source": "The Art of SQL.pdf", + "chunk_id": 94 + }, + { + "text": "group by ordid) The reference on the left side to two columns from the same table removes the dependence of identification of the most recent status for the order on a preliminary join between orderstatus and orders. A very clever optimizer might have performed the modification for us, but it is wiser to take no risk and specify both columns from the same table to begin with. It is always much better to leave the optimizer with as much freedom as we can. You have seen previously that an uncorrelated subquery can become a join in an inline view without much effort. We can indeed rewrite the entire query to list pending orders as follows: select c.custname, o.ordid, os.status, os.statusdate from customers c, orders o, orderstatus os, (select ordid, max(statusdate) laststatusdate from orderstatus group by ordid) x where o.ordid = os.ordid and not exists (select null from orderstatus os2 where os2.status = 'COMPLETE' and os2.ordid = o.ordid) and os.statusdate = x.laststatusdate and os.ordid = x.ordid and o.custid = c.custid www.it-ebooks.info 102 C H A P T E R F O U R But then, if COMPLETE is indeed the final status, do we need the subquery to check the nonexistence of the last stage? The inline view helps us to identify which is the last status, whether it is COMPLETE or anything else. We can apply a perfectly satisfactory filter by checking the latest known status: select c.custname, o.ordid, os.status, os.statusdate from customers c, orders o, orderstatus os, (select ordid, max(statusdate) laststatusdate from orderstatus group by ordid) x where o.ordid = os.ordid and os.statusdate = x.laststatusdate and os.ordid = x.ordid and os.status != 'COMPLETE' and o.custid = c.custid The duplicate reference to orderstatus can be further avoided by using OLAP or analytical functions available with some SQL engines. But let’s pause here and consider how we have modified the query and, more importantly, the execution path. Basically, our natural path was initially to scan the orders table, and then access through what may reasonably be expected to be an efficient index on the table orderstatus. In the last version of our query, we will attack through a full scan of orderstatus, to perform a group by. In terms of the number of rows, orderstatus will necessarily be several times bigger than orders. However, in terms of mere volume of data to scan, we can expect it to be smaller, possibly significantly smaller, depending on how much information is stored for each order. We cannot say with certainty which approach will be better, it depends on the data. Let me add that seeing a full scan on a table that is expected to grow is not a good idea (restricting the search to the last month’s, or last few months’ worth of data can help). But there are significant chances that this last version of our query will perform better than the first attempt with the subquery in the where clause. We cannot leave the subject of large data volumes without", + "source": "The Art of SQL.pdf", + "chunk_id": 95 + }, + { + "text": "search to the last month’s, or last few months’ worth of data can help). But there are significant chances that this last version of our query will perform better than the first attempt with the subquery in the where clause. We cannot leave the subject of large data volumes without mentioning a slightly special case. When a query returns a very large amount of data, you have reasonable grounds for suspecting that it’s not an individual sitting at a terminal that executed the query. The likelihood is that such a query is part of a batch process. Even if there is a longish “preparatory phase,” nobody will complain so long as the whole process performs to a satisfactory standard. Do not, of course, forget that a phase, preparatory or not, requires resources—CPU, memory, and possibly temporary disk space. It helps to understand that the optimizer, when returning a lot of data, may choose a path which has nothing in common with the path it would adopt when returning few rows, even if the fundamental query is identical. www.it-ebooks.info M A N E U V E R I N G 103 Filter out unneeded data as early as possible. The Proportions of Retrieved Data A typical and frequently quoted saying is the famous “don’t use indexes when your query returns more than 10% of the rows of a table.” This states implicitly that (regular) indexes are efficient when an index key points to 10% or less of the rows in a table. As I have already pointed out in Chapter 3, this rule of thumb dates back to a time when relational databases were still regarded with suspicion in many companies. In those days, their use was mostly confined to that of departmental databases. This was a time when a 100,000–row table was considered a really big one. Compared to 10% of a 500 million– row table, 10% of 100,000 rows is a trifle. Can we seriously hope that the best execution plan in one case will still be the best execution plan in the other case? Such is wishful thinking. Independently from the evolution of table sizes since the time when the “10% of rows” rule of thumb was first coined, be aware that the number of rows returned means nothing in itself, except in terms of response time expectations by end users. If you compute an average value over 1 billion rows, you return a single row, and yet the DBMS performs a lot of work. Even without any aggregation, what matters is the number of data pages that the DBMS is going to hit when performing the query. Data page hits don’t only depend on the existence of indexes: as you saw in Chapter 3, the relation of indexes to the physical order of rows in the table can make a significant difference in the number of pages to visit. Other implementation issues that I am going to discuss in Chapter 5 play an important part, too: depending", + "source": "The Art of SQL.pdf", + "chunk_id": 96 + }, + { + "text": "indexes: as you saw in Chapter 3, the relation of indexes to the physical order of rows in the table can make a significant difference in the number of pages to visit. Other implementation issues that I am going to discuss in Chapter 5 play an important part, too: depending on how data is physically stored, the same number of rows returned may mean that you have to visit massively different numbers of data pages. Furthermore, operations that would execute sequentially with one access path may be massively parallelized with another one. Don’t fall into the row percentage trap. When we want a lot of data, we don’t necessarily want an index. www.it-ebooks.info www.it-ebooks.info Chapter 5 . C H A P T E R F I V E Terrain Understanding Physical Implementation [...] haben Gegend und Boden eine sehr nahe [...] Beziehung zur kriegerischen Tätigkeit, nämlich einen sehr entscheidenden Einfluß auf das Gefecht. [...] Country and ground bear a most intimate [...] relation to the business of war, which is their decisive influence on the battle. —Carl von Clausewitz (1780–1831) Vom Kriege, V, 17 www.it-ebooks.info 106 C H A P T E R F I V E W hat a program sees as a table is not always the plain table it may look like. Sometimes it’s a view, and sometimes it really is a table, but with storage parameters that have been very carefully established to optimize certain types of operations. In this chapter, I explore different ways to arrange the data in a table and the operations that those arrangements facilitate. I should emphasize from the start that the topic of this chapter is not disk layout, nor even the relative placement of journal and data files. These are the kinds of subjects that usually send system engineers and database administrators into mouth-watering paroxysms of delight—but no one else. There is much more to database organization than the physical dispersion of bytes on permanent storage. It is the actual nature of the data that dictates the most important choices. Both system engineers and database administrators know how much storage is used, and they know the various possibilities available in terms of data containers, whether very low-level data containers such as disk stripes or high-level data containers such as tables. But frequently, even database administrators have only a scant knowledge of what lies inside those containers. It can sometimes be helpful to choose the terrain on which to fight. Just as a general may discuss tactics with the engineering corps, so the architect of an application can study with the database administrators how best to structure data at the physical level. Nevertheless, you may be required to fight your battle on terrain over which you have no control or, worse, to use structures that were optimized for totally different purposes. Structural Types Even though matters of physical database structure are not directly related to the SQL language, the underlying structures of your database will certainly influence your tactical use", + "source": "The Art of SQL.pdf", + "chunk_id": 97 + }, + { + "text": "battle on terrain over which you have no control or, worse, to use structures that were optimized for totally different purposes. Structural Types Even though matters of physical database structure are not directly related to the SQL language, the underlying structures of your database will certainly influence your tactical use of SQL. The chances are that any well-established and working database will fall into one of the following structural types: The fixed, inflexible model There are times when you will have absolutely no choice in the matter. You will have to work with the existing database structures, no matter how obvi- ous it may be to you that they are contributing to the performance difficul- ties, if they are not their actual cause. Whether you are developing new applications, or simply trying to improve existing ones, the underlying struc- tures are going to control the choices you can make in the deployment of your SQL armory. You must try to understand the reasoning behind the system and work with it. www.it-ebooks.info T E R R A I N 107 The evolutionary model Everything is not always cast in stone, and altering the physical layout of data (without modifying the logical model) is sometimes an option. Be very aware that there are dangers here and that the reluctance of database administrators to make such modifications doesn’t stem from laziness. In spite of the risks and potential for service interruption attached to such operations, many people cling to database reorganization as their last hope when facing performance issues. Physical reorganization is not in itself the panacea for correcting poor perfor- mance. It may be quite helpful in some cases, irrelevant elsewhere, and even harmful in other cases. It is important to know both what you can and cannot expect from such drastic action. In a sense, if you have to work with a flawed design, neither scenario is a particularly attractive option. “Abandon hope, all ye that have an incorrect design” might just possibly be overstating the situation, but nevertheless I am stressing once again the crucial importance of getting the design right at the earliest opportunity. In more than one way, implementation choices are comparable to the choice of tires in Formula One motor racing: you have to take a bet on the race conditions that you are expecting. The wrong tire choice may prove costly, the right one may help you win, but even the best choice will not, of itself, assure you of victory. I won’t discuss SQL constructs in this chapter, nor will I delve into the intricacies of specific implementations, which in any case are all very much product dependent. However, it is difficult in practice to design a reliable architecture without an understanding of all the various conditions, good and bad, with or against which the design will have to function. Understanding also means sensing how much a particular physical implementation can impact performance, for better or for worse. This is why I shall try to", + "source": "The Art of SQL.pdf", + "chunk_id": 98 + }, + { + "text": "design a reliable architecture without an understanding of all the various conditions, good and bad, with or against which the design will have to function. Understanding also means sensing how much a particular physical implementation can impact performance, for better or for worse. This is why I shall try to give you an idea, first of some of the practical problems DBMS implementers have had to face to help improve the speed of queries and changes to the database (of which more will be said in Chapter 9), and second of some of the answers they have found. From a practical point of view, though, be aware that some of the features presented in this chapter are not available with all database systems. Or, if they are available, they may require separate licensing. One last word before we begin. I have tried to establish some points of comparison between various commercial products. To that end, this chapter presents a number of actual test results. However, it is by no means the purpose of this book to organize a beauty contest between various database products, especially as the balance may change between versions. Similarly, absolute values have no meaning, since they depend very strongly on your hardware and the design of the database. This is why I have chosen to present only relative values, and why I have also chosen (with one exception) to compare variations for only one particular DBMS. www.it-ebooks.info 108 C H A P T E R F I V E The Conflicting Goals There are often two conflicting goals when trying to optimize the physical layout of data for a system that expects a large number of active users, some of them reading and others writing data. One goal is to try to store the data in as compact a way as possible and to help queries find it as quickly as possible. The other goal is to try to spread the data, so that several processes writing concurrently do not impede one another and cause contention and competition for resources that cannot be shared. Even when there is no concurrency involved, there is always some tension when designing the physical aspect of a database, between trying to make both queries and updates (in the general sense of “changes to the data”) as fast as possible. Indexing is an obvious case in point: people often index in anticipation of queries using the indexed columns as selection criteria. However, as seen in Chapter 3, the cost of maintaining indexes is extremely high and inserting into an index is often much more expensive than inserting into the underlying table alone. Contention issues affect any data that has to be stored, especially in change-heavy transactional applications (I am using the generic term change to mean any insert, delete, and update operation). Various storage units and some very low layers of the operating system can take care of some contention issues. The files that contain the database data may be sliced,", + "source": "The Art of SQL.pdf", + "chunk_id": 99 + }, + { + "text": "stored, especially in change-heavy transactional applications (I am using the generic term change to mean any insert, delete, and update operation). Various storage units and some very low layers of the operating system can take care of some contention issues. The files that contain the database data may be sliced, mirrored, and spread all over the place to ensure data integrity in case of hardware failure, as well as to limit contention. Unfortunately, relying on the operating system alone to deal with contention is not enough. The base units of data that a DBMS handles (known as pages or blocks depending on the product) are usually, even at the lowest layers, atomic from a database perspective, especially as they are ultimately all scanned in memory. Even when everything is perfect for the systems engineer, there may be pure DBMS performance issues. To get the best possible response time, we must try to keep the number of data pages that have to be accessed by the database engine as low as possible. We have two principal means of decreasing the number of pages that will have to be accessed in the course of a query: • Trying to ensure a high data density per page • Grouping together those pieces of data most likely to be required during one retrieval process However, trying to squeeze the data into as few pages as possible may not be the optimum approach where the same page is being written by several concurrent processes and perhaps also being read at the same time. Where that single data page is the subject of multiple read or write attempts, conflict resolution takes on an altogether more complex and serious dimension. www.it-ebooks.info T E R R A I N 109 Many believe that the structure of a database is the exclusive responsibility of the database administrator. In reality, it is predominantly but not exclusively the responsibility of that very important person. The way in which you physically structure your data is extremely dependent on the nature of the data and its intended use. For example, partitioning can be a valuable aid in optimizing a physical design, but it should never be applied in a haphazard way. Because there is such an intimate relationship between process requirements and physical design, we often encounter profound conflicts between alternative designs for the same data when that data is shared between two or more business processes. This is just like the dilemma faced by the general on the battlefield, where the benefits of using alternative parts of his forces (infantry, cavalry, or artillery) have to be balanced against the suitability of the terrain across which he has to deploy them. The physical design of tables and indexes is one of those areas where database administrators and developers must work together, trying to match the available DBMS features in the best possible way against business requirements. The sections to follow introduce some different strategies and show their impact on queries and updates from", + "source": "The Art of SQL.pdf", + "chunk_id": 100 + }, + { + "text": "design of tables and indexes is one of those areas where database administrators and developers must work together, trying to match the available DBMS features in the best possible way against business requirements. The sections to follow introduce some different strategies and show their impact on queries and updates from a single-process perspective, which, in practice, is usually the batch program perspective. Reads and writes don’t live in harmony: readers want data clustered; and concurrent writers want data scattered. Considering Indexes as Data Repositories Indexes allow us to find quickly the addresses (references to some particular storage in persistent memory, typically file identifiers and offsets within the files) of the rows that contain a key we are looking for. Once we have an address, then it can be translated into a low-level, operating system reference which, if we are lucky, will direct us to the true memory address where the data is located. Alternatively, the index search will result in some input/output operation taking place before we have the data at our disposal in memory. As discussed previously in Chapter 3, when the value of a key we are looking for refers to a very large number of rows, it is often more efficient simply to scan the table from the beginning to the end and ignore the indexes. This is why, at least in a transactional database, it is useless to index columns with a low number of distinct values (i.e., a low cardinality) unless one value is highly selective and appears frequently in where clauses. Other indexes that we can dispose of are single-column indexes on columns that already participate in composite indexes as the leading column: there is no need whatsoever to www.it-ebooks.info 110 C H A P T E R F I V E index the same column twice in these circumstances. The very common tree-structured, or hierarchical, index can be efficiently searched even if we do not have the full key value, just as long as we have a sufficient number of leading bytes to ensure discrimination. The use of leading bytes rather than the full index key for querying an index introduces an interesting type of optimization. If there is an index on (c1, c2, c3), this index is usable even if we only specify the value of c1. Furthermore, if the key values are not compressed, then the index contains all the data held in the (c1, c2, c3) triplets that are present in the table. If we specify c1 to get the corresponding values of c2, or of c2 to find the corresponding c3, we find within the index itself all the data we need, without requiring further access to the actual table. For example, to take a very simple analogy, it’s exactly as though you were looking for William Shakespeare’s year of birth. Submitting the string William Shakespeare to any web search engine will return information such as you see in Figure 5-1. There is no need to visit any of", + "source": "The Art of SQL.pdf", + "chunk_id": 101 + }, + { + "text": "table. For example, to take a very simple analogy, it’s exactly as though you were looking for William Shakespeare’s year of birth. Submitting the string William Shakespeare to any web search engine will return information such as you see in Figure 5-1. There is no need to visit any of these sites (which may be a pity): we have found our answer in the data returned from the search engine index itself. The fourth entry tells us that Shakespeare was born in 1564. When an index is sufficiently loaded with information, going to the place it points to becomes unnecessary. This very same reasoning is at the root of an often used optimization tactic. We can improve the speed of a frequently run query by stuffing into an index additional columns (one or more) which of themselves have no part to play in the actual search criteria, but which crucially hold the data we need to answer our query. FIGURE 5-1. Searching the Web for \"William Shakespeare\" www.it-ebooks.info T E R R A I N 111 Thus the data that we require can be retrieved entirely from the index, cutting out completely the need to access the original source data. Some products such as DB2 are clever enough to let us specify that a unique index includes some other columns and check uniqueness of only a part of the composite key. The same result can be achieved with Oracle, in a somewhat more indirect fashion, by using a non-unique index to enforce a uniqueness or primary key constraint. Conversely, there have been cases of batch programs suddenly taking much more time to run to completion than previously, following what appears to be the most insignificant modification to the query. This minor change was the addition of another column to the list of columns returned by a select statement. Unfortunately, prior to the modification, the entire query could be satisfied by reference to the data returned from an index. The addition of the new column forced the database to go back to the table, resulting in a hugely significant increase in processor activity. Let’s look in more detail at the contrast between “index only” and “index plus table” retrieval performance. Figure 5-2 illustrates the performance impact of fetching one additional column absent from the index that is used to answer the query for three of the major database systems. The table used for the test was the same in all cases, having 12 columns populated with 250,000 rows. The primary key was defined as a three-column composite key, consisting of first an integer column with random values uniformly spread between 1 and 5,000, then a string of 8 to 10 characters, and then finally a datetime column. There is no other index on the table other than the unique index that implements the primary key. The reference query is fetching the second and third columns in the primary key on the basis of a random value of between 1 and 5,000", + "source": "The Art of SQL.pdf", + "chunk_id": 102 + }, + { + "text": "and then finally a datetime column. There is no other index on the table other than the unique index that implements the primary key. The reference query is fetching the second and third columns in the primary key on the basis of a random value of between 1 and 5,000 in the first column. The test measures the performance impact of fetching one more column—numeric and therefore relatively small—that doesn’t belong to the index. The results in Figure 5-2 are normalized such that the case of fetching two columns that are found in the index is always pegged at 100%. The case of having to go to the table and fetch an additional column absent from the index is then expressed as some percentage less than 100%. FIGURE 5-2. Performance impact of fetching a third column that has to be retrieved from the table www.it-ebooks.info 112 C H A P T E R F I V E Figure 5-2 shows that the performance impact of having to go to the table as well as to the index isn’t enormous (around 5 or 10%) but nevertheless it is noticeable, and it is much more so with some database products than with others. As always, the exact numbers may vary with circumstances, and the impact can be much more severe if the table access requires additional physical I/O operations, which isn’t the case here. Pushing to the extreme the principle of storing as much data as possible in the indexes, some database management systems, such as Oracle, allow you to store all of a table’s data into an index built on the primary key, thus getting rid of the table structure altogether. This approach saves storage and may save time. The table is the index, and is known as an index-organized table (I0T) as opposed to the regular heap structure. After the discussion in Chapter 3 about the cost penalty of index insertion, you might expect insertions into an index-organized table to be less costly than applying insertions to a table with no other index than the primary key enforcement index. In fact, in some circumstances the opposite is true, as you can see from Figure 5-3. It compares insertion rates for a regular table against those for an IOT. The tests used a total of four tables. Two table patterns with the same column definitions were created, once as a regular heap table, and once as an IOT. The first table pattern was a small table consisting of the primary key columns plus one additional column, and the second pattern a table consisting of the primary key columns plus nine other columns, all numeric. The (compound) primary key in every table was defined as a number column, a 10-character string, and a timestamp. For each case, two tests were performed. In the first test, the primary key was subjected to the insertion of randomly ordered primary key values. The second test involved the insertion into the leading primary key column of", + "source": "The Art of SQL.pdf", + "chunk_id": 103 + }, + { + "text": "was defined as a number column, a 10-character string, and a timestamp. For each case, two tests were performed. In the first test, the primary key was subjected to the insertion of randomly ordered primary key values. The second test involved the insertion into the leading primary key column of an increasing, ordered sequence of numbers. Where the table holds few columns other than the ones that define the primary key, it is indeed faster to insert into an IOT. However, if the table has even a moderate number of columns, all those columns that don’t pertain to the primary key also have to be stored in the index structure (sometimes to an overflow area). Since the table is the index, much more information is stored there than would otherwise be the case. Chapter 3 has also shown that inserting into an index is intrinsically more costly than inserting into a regular table. The byte-shuffling cost associated with inserting more data into a more complicated structure can lead to a severe performance penalty, unless the rows are inserted in the same or near-the-same order as the primary key index. The penalty is even worse with long character strings. In many cases the additional cost of insertion outweighs the benefit of not having to go to the table when fetching data through the primary key index.* * A reviewer remarked that implementation reasons that are beyond the scope of this book also make other indexes than the primary key index less efficient on an IOT than they would be on a regular table. www.it-ebooks.info T E R R A I N 113 There are, however, some other potential benefits linked to the strong internal ordering of indexes, as you shall see next. Some queries can be answered by retrieving only the index data. Forcing Row Ordering There is another aspect to an index-organized table than just finding all required data in the index itself without requiring an additional access to the table. Because IOTs, being indexes, are, first and foremost, strongly ordered structures, their rows are internally ordered. Although the notion of order is totally foreign to the relational theory, from a practical point of view whenever a query refers to a range of values, it helps to find them together instead of having to gather data scattered all over the table. The most common example of this sort of application is range searching on time series data, when you are looking for events that occurred between two particular dates. FIGURE 5-3. Relative cost of inserting into an Oracle index-organized table compared to a regular (heap- organized) table www.it-ebooks.info 114 C H A P T E R F I V E Most database systems manage to force such an ordering of rows by assigning to an index the role of defining the order of rows in the table. SQL Server and Sybase call such an index a clustered index. DB2 calls it a clustering index, and it has much the same", + "source": "The Art of SQL.pdf", + "chunk_id": 104 + }, + { + "text": "Most database systems manage to force such an ordering of rows by assigning to an index the role of defining the order of rows in the table. SQL Server and Sybase call such an index a clustered index. DB2 calls it a clustering index, and it has much the same effect in practice as an Oracle IOT. Some queries benefit greatly from this type of organization. But similar to index organized tables, updates to columns pertaining to the index that defines the order are obviously more costly because they entail a physical movement of the data to a new position corresponding to the “rank” of the new values. The ordering of rows inevitably favors one type of range-scan query at the expense of range scans on alternative criteria. As with IOTs that are defined by the primary key, it is safer to use the primary key index as the clustering index, since primary keys are never updated (and if your application needs to update your primary key, there is something very seriously wrong indeed with your design, and it won’t take long before there is something seriously wrong with the integrity of your data). In contrast to IOTs, an index other than the one that enforces the primary key constraint can be chosen as the clustering index. But remember that any ordering unduly favors some processes at the expense of others. The primary key, if it is a natural key, has a logical significance; the associated index is more equal than all the other indexes that may be defined on the table, even unique ones. If some columns must be given some particular prominence through the physical implementation, these are the ones. Figure 5-4 illustrates the kind of differences you may expect between clustered and non- clustered index performance in practice. If we take the same table as was used for the index-organized table in Figure 5-3’s example (a three-column primary key plus nine numeric columns), and if we insert rows in a totally random way, the cost of insertion into a table where the primary key index is clustered is quite high, since tests show that our insertion rate is about half the insertion rate obtained with a non-clustered primary key. But when we run a range scan test on about 50,000 rows, this clustered index provides really excellent performance. In this particular case, the clustered index allows us to outperform the non-clustered approach by a factor of 20. We should, of course, see no difference when fetching a single row. A structural optimization, such as a clustered index or an IOT, necessarily has some drawbacks. For one thing, such structures apply some strong, tree-based, and therefore hierarchical ordering to tables. This approach resurrects many of the flaws that saw hierarchical databases replaced by relational databases in the corporate world. Any hierarchical structure favors one vision of the data and one access path over all the others. One particular access path will be better than anything you could get", + "source": "The Art of SQL.pdf", + "chunk_id": 105 + }, + { + "text": "to tables. This approach resurrects many of the flaws that saw hierarchical databases replaced by relational databases in the corporate world. Any hierarchical structure favors one vision of the data and one access path over all the others. One particular access path will be better than anything you could get with a non- clustered table, but most other access paths are likely to be significantly worse. Updates may prove more costly. The initial tidy disposition of the data inside the database files may deteriorate faster at the physical level, due to chaining, overflow pages, and similar constructs, which take a heavy toll on performance. Clustered structures are excellent in www.it-ebooks.info T E R R A I N 115 some cases, boosting performance by an impressive factor. But they always need to be carefully tested, because there is a high probability that they will make many other processes run slower. One must judge their suitability while looking at the global picture—and not on the basis of one particular query. Range scanning on clustered data can give impressive performance, but other queries will suffer as a consequence. Automatically Grouping Data You have seen that finding all the rows together when doing a range scan can be highly beneficial to performance. There are, actually, other means to achieve a grouping of data than the somewhat constraining use of clustering indexes or index-organized tables. All database management systems let us partition tables and indexes—an application of the old principle of divide and rule. A large table may be split into more manageable chunks. Moreover, in terms of process architecture, partitioning allows an increased concurrency and parallelism, thus leading to more scalable architectures, as you shall see in Chapters 9 and 10. First of all, beware that this very word, partition, has a different meaning depending on the DBMS under discussion, sometimes even depending on the version of the DBMS. There was a time, long ago, when what is now known as an Oracle tablespace used to be referred to as a partition. FIGURE 5-4. How clustered indexes perform www.it-ebooks.info 116 C H A P T E R F I V E Round-Robin Partitioning In some cases, partitioning is a totally internal, non-data-driven mechanism. We arbitrarily define a number of partitions as distinct areas of disk storage, usually closely linked to the number of devices on which we want the data to be stored. One table may have one or more partitions assigned to it. When data is inserted, it is loaded to each partition according to some arbitrary method, in a round-robin fashion, so as to balance the load on disk I/O induced by the insertions. Incidentally, the scattering of data across several partitions may very well assist subsequent random searches. This mechanism is quite comparable to file striping over disk arrays. In fact, if your files are striped, the benefit of such a partitioning becomes slight and sometimes quite negligible. Round-robin scattering can be thought of as a mechanism designed only to arbitrarily", + "source": "The Art of SQL.pdf", + "chunk_id": 106 + }, + { + "text": "may very well assist subsequent random searches. This mechanism is quite comparable to file striping over disk arrays. In fact, if your files are striped, the benefit of such a partitioning becomes slight and sometimes quite negligible. Round-robin scattering can be thought of as a mechanism designed only to arbitrarily spread data irrespective of logical data associations, rather than to regroup data on the basis of those natural associations. However, with some products, Sybase being one of them, one transaction will always write to the same partition, thus achieving some business-process-related grouping of data. Data-Driven Partitioning There is, however, a much more interesting type of partitioning known as data-driven partitioning. With data-driven partitioning, it is the values, found in one or several columns, that defines the partition into which each row is inserted. As always, the more the DBMS knows about the data and how it is stored, the better. Most really large tables are large because they contain historical data. However, our interest in a particular news event quickly wanes as new and fresher events crowd in to demand our attention, so it is a safe assumption to make that the most-often-queried subset of historical data is the most recent one. It is therefore quite natural to try to partition data by date, separating the wheat from the chaff, the active data from the dormant data. For instance, a manual way to partition by date is to split a large figures table (containing data for the last twelve months) into twelve separate tables, one for each month, namely jan_figures, feb_figures...all the way to dec_figures. To ensure that a global vision of the year is still available for any queries that require it, we just have to define figures as the union of those twelve tables. Such a union is often given some kind of official endorsement at the database level as a partitioned view, or (in MySQL) a merge table. During the month of March, we’ll insert into the table mar_figures. Then we’ll switch to apr_figures for the following month. The use of a view as a blanket object over a set of similarly structured tables may appear an attractive idea, but it has drawbacks: www.it-ebooks.info T E R R A I N 117 • The capital sin is that such a view builds in a fundamental design flaw. We know that the underlying tables are logically related, but we have no way to inform the DBMS of their relationships except, in some cases, via the rather weak definition of the parti- tioned view. Such a multi-table design prevents us from correctly defining integrity constraints. We have no easy way to enforce uniqueness properly across all the under- lying tables, and as a matter of consequence, we would have to build multiple foreign keys referencing this “set” of tables, a situation that becomes utterly difficult and unnatural. All we can do in terms of integrity is to add a check constraint on the col- umn that determines partitioning. For", + "source": "The Art of SQL.pdf", + "chunk_id": 107 + }, + { + "text": "tables, and as a matter of consequence, we would have to build multiple foreign keys referencing this “set” of tables, a situation that becomes utterly difficult and unnatural. All we can do in terms of integrity is to add a check constraint on the col- umn that determines partitioning. For example, we could add a check constraint to sales_date, to ensure that sales_date in the jun_sales table cannot fall outside the June 1 to June 30 range. • Without specific support for partitioned views in your DBMS, it is rather inconvenient to code around such a set of tables, because every month we must insert into a differ- ent underlying table. This means that insert statements must be dynamically built to accommodate varying table names. The effect of dynamic statements is usually to sig- nificantly increase the complexity of programs. In our case, for instance, a program would have to get the date, either the current one or some input value, check it, determine the name of the table corresponding to that date, and build up a suitable SQL statement. However, the situation is much better with partitioned views, because insertions can then usually be performed directly through the view, and the DBMS takes care of where to insert the rows. In all cases, however, as a direct consequence of our flawed design, it is quite likely that after some unfortunate and incoherent insertions we shall be asked to code refer- ential integrity checks, thus further compounding a poor design with an increased development load—both for the developers and for the machine that runs the code. This will move the burden of integrity checking from the DBMS kernel to, in the best of cases, code in triggers and stored procedures and, in the worst of cases, to the appli- cation program. • There is a performance impact on queries when using blanket views. If we are inter- ested in the figures for a given month, we can query a single table. If we are inter- ested in the figures from the past 30 days, we will most often need to query two tables. For queries, then, the simplest and more maintainable way to code is to query the view rather than the underlying tables. If we have a partitioned view and if the column that rules the placement of rows belongs to our set of criteria, the DBMS opti- mizer will be able to limit the scope of our query to the proper subset of tables. If not, our query will necessarily be more complicated than it would be against a regular table, especially if it is a complex query involving subqueries or aggregates. The com- plexity of the query will continue to increase as more tables become involved in the union. The overhead of querying a large union view over directly querying a single table will quickly show in repeatedly executed statements. www.it-ebooks.info 118 C H A P T E R F I V E Historically, the first step", + "source": "The Art of SQL.pdf", + "chunk_id": 108 + }, + { + "text": "will continue to increase as more tables become involved in the union. The overhead of querying a large union view over directly querying a single table will quickly show in repeatedly executed statements. www.it-ebooks.info 118 C H A P T E R F I V E Historically, the first step taken by most database management systems towards partitioning has been the support of partitioned views. The next logical step has been support for true data-driven partitioning. With true partitioning, we have a single table at the logical level, with a true primary key able to be referenced by other tables. In addition, we have one or several columns that are defined as the partition key; their values are used to determine into which partition a row is inserted. We have all the advantages of partitioned views, transparency when operating on the table, and we can push back to the DBMS engine the task of protecting the integrity of the data, which is one of the primary functions of the DBMS. The kernel knows about partitioning, and the optimizer will know how to exploit such a physical structure, by either limiting operations to a small number of partitions (something known as partition pruning), or by operating on several partitions in parallel. The exact way partitioning is implemented and the number of available options is product-dependent. There are several different ways to partition data, which may be more or less appropriate to particular situations: Hash-partitioning Spreads data by determining the partition as the result of a computation on the partition key. It’s a totally arbitrary placement based entirely on an arithmetic computation, and it takes no account at all of the distribution of data values. Hash-partitioning does, however, ensure very fast access to rows for any specific value of the partition key. It is useless for range searching, because the hash function transforms consecutive key values into non-consecutive hash values, and it’s these hash values that translate to physical address. NOTE DB2 provides an additional mechanism called range-clustering, which, although not the same as partitioning, nevertheless uses the data from the key to determine physical location. It does this through a mechanism that, in contrast to hashing, preserves the order of data items. We then gain on both counts, with efficient specific accesses as well as efficient range scans. Range-partitioning Seeks to gather data into discrete groups according to continuous data ranges. It’s ideally suited for dealing with historical data. Range-partitioning is closest to the concept of partitioned views that we discussed earlier: a partition is defined as being dedicated to the storage of values falling within a certain range. An else partition is set up for catching everything that might slip through the net. Although the most common use of range partitioning is to partition by range of temporal values, whether it is hours or years or anything between, this type of partitioning is in no way restricted to a particular type of data. A multivolume www.it-ebooks.info T E R R", + "source": "The Art of SQL.pdf", + "chunk_id": 109 + }, + { + "text": "through the net. Although the most common use of range partitioning is to partition by range of temporal values, whether it is hours or years or anything between, this type of partitioning is in no way restricted to a particular type of data. A multivolume www.it-ebooks.info T E R R A I N 119 encyclopedia in which the articles in each volume would indeed be within the alphabetical boundaries of the volume but otherwise in no particular order pro- vides a good example of range partitioning. List-partitioning Is the most manual type of partitioning and may be suitable for tailor-made solutions. Its name says it all: you explicitly specify that rows containing a list of the possible values for the partition key (usually just one column) will be assigned to a particular partition. List-partitioning can be useful when the distri- bution of values is anything but uniform. The partitioning process can sometimes be repeated with the creation of subpartitions. A subpartition is merely a partition within a partition, giving you the ability to partition against a second dimension by creating, for instance, hash-partitions within a range-partition. Data partitioning is most valuable when it is based on the data values themselves. The Double-Edged Sword of Partitioning Despite the fact that partitioning spreads data from a table over multiple, somewhat independent partitions, data-driven partitioning is not a panacea for resolving concurrency problems. For example, we might partition a table by date, having one partition per week of activity. Doing so is an efficient way to spread data for one year over 52 logically distinct areas. The problem is that during any given week everybody will rush to the same partition to insert new rows. Worse, if our partitioning key is the current system date and time, all concurrent sessions will be directed towards the very same data block (unless some structural implementation tricks have been introduced, such as maintaining several lists of pages or blocks where we can insert). As a result, we may have some very awkward memory contention. Our large table will become a predominantly cold area, with a very hot spot corresponding to most current data. Such partitioning is obviously less than ideal when many processes are inserting concurrently. NOTE If all data is inserted through a single process, which is sometimes the case in data- warehousing environments, then we won’t have a hot spot to contend with, and our 52-week partitioning scheme won’t lead to concurrency problems. On the other hand, let’s assume that we choose to partition according to the geographical origin of a purchase order (we may have to carefully organize our partitioning if our products are more popular in some areas and suffer from heavier competition elsewhere). www.it-ebooks.info 120 C H A P T E R F I V E At any given moment, since sales are likely to come from nowhere in particular, our inserts will be more or less randomly spread over all our partitions. The performance impact from our partitioning will be", + "source": "The Art of SQL.pdf", + "chunk_id": 110 + }, + { + "text": "competition elsewhere). www.it-ebooks.info 120 C H A P T E R F I V E At any given moment, since sales are likely to come from nowhere in particular, our inserts will be more or less randomly spread over all our partitions. The performance impact from our partitioning will be quite noticeable when we are running geographical reports. Of course, because we have partitioned on spatial criteria, time-based reports will be less efficiently generated than if we had partitioned on time. Nevertheless, even time- based queries may, to some extent, benefit from partitioning since it is quite likely that on a multiprocessor box the various partitions will be searched in parallel and the subsequent results merged. There are therefore two sides to partitioning. On the one hand, it is an excellent way to cluster data according to the partitioning key so as to achieve faster data retrieval. On the other hand, it is a no-less-excellent way to spread data during concurrent inserts so as to avoid hot spots in the table. These two objectives can work in opposition to one another, so the very first thing to consider when partitioning is to identify the major problem, and partition against that. But it is important to check that the gain on one side is not offset by the loss on the other. The ideal case is when the clustering of data for selects goes hand in hand with suitably spread inserts, but this is unfortunately not the most common situation. Data partitioning can be used to scatter or cluster your data: it all depends on your requirements. Partitioning and Data Distribution You may be tempted to believe that if we have a very large table and want to avoid contention when many sessions are simultaneously writing to the database, then we are necessarily better off partitioning the data in one way or another. This is not always true. Suppose that we have a large table storing the details of orders passed by our customers. If, as sometimes happens, a single customer represents the bulk of our activity, partitioning on the customer identifier is not going to help us very much. We can very roughly divide our queries into two families: queries relating to our big customer and queries relating to the other, smaller customers. When we query the data relating to one small customer, an index on the customer identifier will be very selective and therefore efficient, without any compelling need for partitioning. A clever optimizer fed with suitable statistics about the distribution of keys will be able to detect the skewness and use the index. There will be little benefit to having those small customers stored into smallish partitions next to the big partition holding our main customer. www.it-ebooks.info T E R R A I N 121 Conversely, when querying the data attached to our major customer, the very same clever optimizer will understand that scanning the table is by far the most efficient way of proceeding. In that case,", + "source": "The Art of SQL.pdf", + "chunk_id": 111 + }, + { + "text": "to the big partition holding our main customer. www.it-ebooks.info T E R R A I N 121 Conversely, when querying the data attached to our major customer, the very same clever optimizer will understand that scanning the table is by far the most efficient way of proceeding. In that case, fully scanning a partition that comprises, for example, 80% of the total volume will not be much faster than doing a full table scan. The end users will hardly notice the performance advantage, whereas the purchasing department will most certainly notice the extra cost of the separately priced partitioning option. The biggest benefits to queries of table partitioning are obtained when data is uniformly spread in respect to the partitioning key. The Best Way to Partition Data Never forget that what dictates the choice of a nonstandard storage option such as partitioning is the global improvement of business operations. It may mean improving a business process that is perceived as being of paramount importance to the detriment of some other processes. For instance, it makes sense to optimize transactional processing that takes place during business hours at the expense of a nightly batch job that has ample time to complete. The opposite may also be true, and we may decide that we can afford to have very slightly less responsive transactions if it allows us to minimize a critical upload time during which data is unavailable to users. It’s a matter of balance. In general, you should avoid unduly favoring one process over another that needs to be run under similar conditions. In this regard, any type of storage that positions data at different locations based on the data value (for example both clustering indexes as well as partitioning) are very costly when that value is updated. What would have previously been an in situ update in a regular table, requiring hardly more than perhaps changing and shifting a few bytes in the table at an invariant physical address, becomes a delete on one part of the disk, followed by an insert somewhere else, with all the maintenance operations usually associated with indexes for this type of operation. Having to move data when we update partition keys seems, on the surface, to be a situation best avoided. Strangely, however, partitioning on a key that is updated may sometimes be preferable to partitioning on a key that is immutable once it has been inserted. For example, suppose that we have a table being used as a service queue. Some process inserts service requests into this table that are of different types (say type T1 to type Tn). New service requests are initially set to status W, meaning “waiting to be processed.” Server processes S1 to Sp regularly poll the table for requests with the W status, change the status of those requests to P (meaning “being processed”), and then, as each request is completed its status is set to D for “done.” www.it-ebooks.info 122 C H A P T E R", + "source": "The Art of SQL.pdf", + "chunk_id": 112 + }, + { + "text": "Server processes S1 to Sp regularly poll the table for requests with the W status, change the status of those requests to P (meaning “being processed”), and then, as each request is completed its status is set to D for “done.” www.it-ebooks.info 122 C H A P T E R F I V E Let’s further suppose that we have as many server processes as we have request types, and that each server process is dedicated to handling a particular type of request. Figure 5-5 shows the service queue as well as the processes. Of course, since we cannot let the table fill with “done” requests, there must be some garbage-collecting process, not shown, that removes processed requests after a suitable delay. Each server process regularly executes a select (actually, a select ... for update) query with two criteria, the type, which depends on the server, and a condition: and status = 'W' Let’s consider alternative ways of partitioning the service queue table. One way to partition the table, and possibly the most obvious, is to partition by request type. There is a big advantage here should any server process crash or fall behind in one way or another. The queue will lengthen for that process until it finally catches up, but the interruption to the processing of that queue will have no influence on the other processes. Another advantage of partitioning by request type is that we avoid having requests of any one type swamp the system. Without partitioning, the polling processes scan a queue that under normal circumstances contains very few rows of interest. If we have a common waiting line and all of a sudden we have a large number of requests of one type and status, all the processes will have more requests to inspect and therefore each will be slowed down. If we partition by type, we establish a watertight wall between the processing of different types. But there is another possible way to partition our service queue table, and that is by status. The downside is obvious: any status change will make a request migrate from one partition to the next. Can there be any advantage to such migration? Actually, there may FIGURE 5-5. A service queue www.it-ebooks.info T E R R A I N 123 indeed be benefit in this approach. Everything in partition W is ready and waiting to be processed. So there is no need to scan over requests being processed by another server or requests that have already been processed. Therefore, the cost of polling may be significantly reduced. Another advantage is that garbage collection will operate on a separate partition, and will not disturb the servers. We cannot say definitively that “partitioning must be by type” or “partitioning must be by status.” It depends on how many servers we have, their polling frequency, the relative rate at which data arrives, the processing time for each type of request, and how often we remove processed requests, and so on. We", + "source": "The Art of SQL.pdf", + "chunk_id": 113 + }, + { + "text": "definitively that “partitioning must be by type” or “partitioning must be by status.” It depends on how many servers we have, their polling frequency, the relative rate at which data arrives, the processing time for each type of request, and how often we remove processed requests, and so on. We must carefully test various hypotheses and consider the overall picture. But it is sometimes more efficient for the overall system to sacrifice outright performance for one particular operation, if by doing so other, more frequently running processes are able to obtain a net advantage, thus benefiting the global business operations. There may be several ways to partition tables, and the most obvious is not always the most efficient. Always consider the global picture. Pre-Joining Tables We have seen that physically grouping rows together is of most benefit when performing range scans, where we are obviously interested in a succession of logically adjacent rows. But our discussion so far has been with regard to retrieving data from only one table. Unless the database design is very, very, very bad, most queries will involve far more than one table. It may therefore seem somewhat questionable if we group all the data from one table into one physical location, only to have to complete the retrieval by visiting several randomly scattered locations for data from a second and subsequent tables. We need some method to group data from at least two tables into the same physical location on disk. The answer lies in pre-joined tables, a technique that is supported by some database systems. Pre-joining is not the same as summary tables or materialized views, which are themselves nothing other than redundant data, pre-digested results that are updated more or less automatically. Pre-joined tables are tables that are physically stored together, based on some criterion that will usually be the join condition. (Oracle calls such a set of pre-joined tables a cluster, which has nothing to do with either index clustering, as defined earlier in this chapter, nor with the MySQL clusters of databases, which are multiple servers accessing the same set of tables.) www.it-ebooks.info 124 C H A P T E R F I V E When tables are pre-joined, the basic unit of storage (a page or a block), normally devoted to the data from a single table, holds data from two or more tables, brought together on the basis of a common join key. This arrangement may be very efficient for one specific join. But it often proves to be a disaster for everything else. Here’s a review of some of the disadvantages of pre-joining tables: • Once the data from two or more tables starts to be shared within one page (or block), the amount of data from one table that can be held in one database page obviously falls, as the page is now sharing its fixed space between two or more tables. Conse- quently, there is a net increase in the number of pages needed to hold", + "source": "The Art of SQL.pdf", + "chunk_id": 114 + }, + { + "text": "one page (or block), the amount of data from one table that can be held in one database page obviously falls, as the page is now sharing its fixed space between two or more tables. Conse- quently, there is a net increase in the number of pages needed to hold all the data from that one table. More I/O is required than previously if a full table scan has to be performed. • Not only is data being shared across additional pages, but the effective size of those pages has been reduced from what was obviously judged to be the optimum at data- base creation time, and so overflow and chaining start to become significant problems. When this happens, the number of successive accesses required to reach the actual data also increases. • Moreover, as anybody who has ever shared an apartment will know, one person often expands space occupancy at the expense of the other. Database tables are just the same! If you want to address this problem by allocating strictly identical storage to each table per page in the cluster, the result is frequently storage waste and the use of even more pages. This particular type of storage should be used extremely sparingly to solve very specific issues, and then only by database administrators. Developers should forget about this technique. Pre-joining tables is a very specialized tactic to facilitate queries, but is often done to the detriment of just about every other database activity. Holy Simplicity It is reasonable and safe to assume that any storage option that is not the default one, however attractive it may look, can introduce a degree of complexity out of all proportion to the possible gains that may (or may not) be achieved. In the worst case, a poorly chosen storage option can dramatically degrade performance. Military history is full of impregnable fortresses built in completely the wrong places that failed to fill any useful purpose, and of many a Great Wall that never prevented any invasion because the www.it-ebooks.info T E R R A I N 125 enemy, a bad sport, failed to behave as planned. All organizations undergo changes, such as divisions and mergers. Business plans and processes may change, too. Careful plans may have to be scrapped and rebuilt from scratch. The trouble with structuring data in a particular way is that it is often done with a particular type of process in mind. One of the beauties of the relational model is its flexibility. By strongly structuring your data at the physical level, you may sacrifice, in a somewhat hidden way, some of this flexibility. Of course, some structures are less constraining than others, and data partitioning is almost unavoidable with enormous databases. But always test very carefully and keep in mind that changing the physical structure of a big database because it was poorly done initially can take days, if not weeks, to complete. The physical storage organization that works for us today may work against us", + "source": "The Art of SQL.pdf", + "chunk_id": 115 + }, + { + "text": "almost unavoidable with enormous databases. But always test very carefully and keep in mind that changing the physical structure of a big database because it was poorly done initially can take days, if not weeks, to complete. The physical storage organization that works for us today may work against us tomorrow. www.it-ebooks.info www.it-ebooks.info Chapter 6. C H A P T E R S I X The Nine Situations Recognizing Classic SQL Patterns Je pense que pour conserver la clarté dans le récit d’une action de guerre, il faut se borner à...ne raconter que les faits principaux et décisifs du combat. To preserve clarity in relating a military action, I think one ought to be content with... reporting only the facts that affected the decision. —Général Baron de Marbot (1782–1854) Mémoires, Book I, xxvi www.it-ebooks.info 128 C H A P T E R S I X A ny SQL statement that we execute has to examine some amount of data before identifying a result set that must be either returned or changed. The way that we have to attack that data depends on the circumstances and conditions under which we have to fight the battle. As I discuss in Chapter 4, our attack will depend on the amount of data from which we retrieve our result set and on our forces (the filtering criteria), together with the volume of data to be retrieved. Any large, complicated query can be divided into a succession of simpler steps, some of which can be executed in parallel, rather like a complex battle is often the combination of multiple engagements between various distinct enemy units. The outcome of these different fights may be quite variable. But what matters is the final, overall result. When we come down to the simpler steps, even when we do not reach a level of detail as small as the individual steps in the execution plan of a query, the number of possibilities is not much greater than the individual moves of pieces in a chess game. But as in a chess game, combinations can indeed be very complicated. This chapter examines common situations encountered when accessing data in a properly normalized database. Although I refer to queries in this chapter, these example situations apply to updates or deletes as well, as soon as a where clause is specified; data must be retrieved before being changed. When filtering data, whether it is for a simple query or to update or delete some rows, the following are the most typical situations—I call them the nine situations—that you will encounter: • Small result set from a few tables with specific criteria applied to those tables • Small result set based on criteria applied to tables other than the data source tables • Small result set based on the intersection of several broad criteria • Small result set from one table, determined by broad selection criteria applied to two or more additional tables • Large result set • Result set obtained by self-joining", + "source": "The Art of SQL.pdf", + "chunk_id": 116 + }, + { + "text": "applied to tables other than the data source tables • Small result set based on the intersection of several broad criteria • Small result set from one table, determined by broad selection criteria applied to two or more additional tables • Large result set • Result set obtained by self-joining on one table • Result set obtained on the basis of aggregate function(s) • Result set obtained by simple searching or by range searching on dates • Result set predicated on the absence of other data This chapter deals with each of these situations in turn and illustrates them with either simple, specific examples or with more complex real-life examples collected from different programs. Real-life examples are not always basic, textbook, one- or two-table affairs. But the overall pattern is usually fairly recognizable. www.it-ebooks.info T H E N I N E S I T U A T I O N S 129 As a general rule, what we require when executing a query is the filtering out of any data that does not belong in our final result set as soon as possible; this means that we must apply the most efficient of our search criteria as soon as possible. Deciding which criterion to apply first is normally the job of the optimizer. But, as I discuss in Chapter 4, the optimizer must take into account a number of variable conditions, from the physical implementation of tables to the manner in which we have written a query. Optimizers do not always “get it right,” and there are things we can do to facilitate performance in each of our nine situations. Small Result Set, Direct Specific Criteria The typical online transaction-processing query is a query returning a small result set from a few tables and with very specific criteria applied to those tables. When we are looking for a few rows that match a selective combination of conditions, our first priority is to pay attention to indexes. The trivial case of a single table or even a join between two tables that returns few rows presents no more difficulty than ensuring that the query uses the proper index. However, when many tables are joined together, and we have input criteria referring to, for instance, two distinct tables TA and TB, then we can either work our way from TA to TB or from TB to TA. The choice depends on how fast we can get rid of the rows we do not want. If statistics reflect the contents of tables with enough accuracy, the optimizer should, hopefully, be able to make the proper decision as to the join order. When writing a query to return few rows, and with direct, specific criteria, we must identify the criteria that are most efficient at filtering the rows; if some criteria are highly critical, before anything else, we must make sure that the columns corresponding to those criteria are indexed and that the indexes can be used by the query. Index Usability You’ve already", + "source": "The Art of SQL.pdf", + "chunk_id": 117 + }, + { + "text": "criteria, we must identify the criteria that are most efficient at filtering the rows; if some criteria are highly critical, before anything else, we must make sure that the columns corresponding to those criteria are indexed and that the indexes can be used by the query. Index Usability You’ve already seen in Chapter 3 that whenever a function is applied to an indexed column, a regular index cannot be used. Instead, you would have to create a functional index, which means that you index the result of the function applied to the column instead of indexing the column. Remember too that you don’t have to explicitly invoke a function to see a function applied; if you compare a column of a given type to a column or literal value of a different type, the DBMS may perform an implicit type conversion (an implicit call to a conversion function), with the performance hit that one can expect. Once we are certain that there are indexes on our critical search criteria and that our query is written in such a way that it can take full advantage of them, we must distinguish between unique index fetches of a single row, and other fetches—non-unique index or a range scan of a unique index. www.it-ebooks.info 130 C H A P T E R S I X Query Efficiency and Index Usage Unique indexes are excellent when joining tables. However, when the input to a query is a primary key and the value of the primary key is not a primitive input to the program, then you may have a poorly designed program on your hands. What I call primitive input is data that has been fed into the program, either typed in by a user or read from a file. If the primary key value has been derived from some primitive input and is itself the result of a query, the odds are very high that there is a massive design flaw in the program. Because this situation often means that the output of one query is used as the input to another one, you should check whether the two queries can be combined. Excellent queries don’t necessarily come from excellent programs. Data Dispersion When indexes are not unique, or when a condition on a unique index is expressed as a range, for instance: where customer_id between ... and ... or: where supplier_name like 'SOMENAME%' the DBMS must perform a range scan. Rows associated with a given key may be spread all over the table being queried, and this is something that a cost-based optimizer often understands. There are therefore cases when an index range scan would require the DBMS kernel to fetch, one by one, a large number of table data pages, each with very few rows of relevance to the query, and when the optimizer decides that the DBMS kernel is better off scanning the table and ignoring the index. You saw in Chapter 5 that many database systems offer facilities such", + "source": "The Art of SQL.pdf", + "chunk_id": 118 + }, + { + "text": "by one, a large number of table data pages, each with very few rows of relevance to the query, and when the optimizer decides that the DBMS kernel is better off scanning the table and ignoring the index. You saw in Chapter 5 that many database systems offer facilities such as table partitions or clustered indexes to direct the storage of data that we would like to retrieve together. But the mere nature of data insertion processes may well lead to clumping of data. When we associate a timestamp with each row and do mostly inserts into a table, the chances are that most rows will be inserted next to one another (unless we have taken special measures to limit contention, as I discuss in Chapter 9). The physical proximity of the inserted rows is not an absolute necessity and, in fact, the notion of order as such is totally foreign to relational algebra. But, in practice, it is what may happen. Therefore, www.it-ebooks.info T H E N I N E S I T U A T I O N S 131 when we perform a range scan on the index on the timestamp column to look for index entries close together in time, the chances are that the rows in question will be close together too. Of course, this will be even truer if we have tweaked the storage so as to get such a result. Now, if the value of a key bears no relation to any peculiar circumstance of insertion nor to any hidden storage trick, the various rows associated with a key value or with a range of key values can be physically placed anywhere on disk. The keys in the index are always, by construction, held in sorted order. But the associated rows will be randomly located in the table. In practice, we shall have to visit many more blocks to answer a query involving such an index than would be the case were the table partitioned or the index clustered. We can have, therefore, two indexes on the same table, with strictly identical degrees of selectivity, one of which gives excellent results, and the other one, significantly worse results, a situation that was mentioned in Chapter 3 and that it is now time to prove. To illustrate this case I have created a 1,000,000–row table with three columns c1, c2, and c3, c1 being filled with a sequence number (1 to 1,000,000), c2 with all different random numbers in the range 1 to 2,000,000, and c3 with random values that can be, and usually are, duplicated. On face value, and from a logical point of view, c1 and c2 are both unique and therefore have identical selectivity. In the case of the index on column c1, the order of the rows in the table matches the order in the index. In a real case, some activity against the table might lead to “holes” left by deletions and subsequently filled with out- of-order records due", + "source": "The Art of SQL.pdf", + "chunk_id": 119 + }, + { + "text": "have identical selectivity. In the case of the index on column c1, the order of the rows in the table matches the order in the index. In a real case, some activity against the table might lead to “holes” left by deletions and subsequently filled with out- of-order records due to new insertions. By contrast, the order of the rows in the table bears no relation to the ordering of the keys in the index on c2. When we fetch c3, based on a range condition of the type: where column_name between some_value and some_value + 10 it makes a significant difference whether we use c1 and its associated index (the ordered index, where keys are ordered as the rows in the table) or c2 and its associated index (the random index), as you can see in Figure 6-1. Don’t forget that we have such a difference because additional accesses to the table are required in order to fetch the value of c3; there would be no difference if we had two composite indexes, on (c1, c3) and (c2, c3), because then we could return everything from an index in which the keys are ordered. The type of difference illustrated in Figure 6-1 also explains why sometimes performance can degrade over time, especially when a new system is put into production with a considerable amount of data coming from a legacy system. It may happen that the initial data loading imposes some physical ordering that favors particular queries. If a few months of regular activity subsequently destroys this order, we may suffer over this period a mysterious 30–40% degradation of performance. www.it-ebooks.info 132 C H A P T E R S I X It should be clear by now that the solution “can’t the DBAs reorganize the database from time to time?” is indeed a fudge, not a solution. Database reorganizations were once quite in vogue. Ever-increasing volumes, 99.9999% uptime requirements and the like have made them, for the most part, an administrative task of the past. If the physical implementation of rows really is crucial for a critical process, then consider one of the self-organizing structures discussed Chapter 5, such as clustered indexes or index- organized tables. But keep in mind that what favors one type of query sometimes disadvantages another type of query and that we cannot win on all fronts. Performance variation between comparable indexes may be due to physical data dispersion. Criterion Indexability Understand that the proper indexing of specific criteria is an essential component of the “small set, direct specific criteria” situation. We can have cases when the result set is small and some criteria may indeed be quite selective, but are of a nature that isn’t suitable for indexing: the following real-life example of a search for differences among different amounts in an accounting program is particularly illustrative of a very selective criterion, yet unfit for indexing. In the example to follow, a table named glreport contains a column named amount_diff that ought to", + "source": "The Art of SQL.pdf", + "chunk_id": 120 + }, + { + "text": "that isn’t suitable for indexing: the following real-life example of a search for differences among different amounts in an accounting program is particularly illustrative of a very selective criterion, yet unfit for indexing. In the example to follow, a table named glreport contains a column named amount_diff that ought to contain zeroes. The purpose of the query is to track accounting errors, and identify where amount_diff isn’t zero. Directly mapping ledgers to tables and applying a logic that dates back to a time when these ledgers where inked with a quill is rather questionable when using a modern DBMS, but unfortunately one encounters FIGURE 6-1. Difference of performance when the order in the index matches the order of the rows in the table www.it-ebooks.info T H E N I N E S I T U A T I O N S 133 questionable databases on a routine basis. Irrespective of the quality of the design, a column such as amount_diff is typical of a column that should not be indexed: ideally amount_diff should contain nothing but zeroes, and furthermore, it is obviously the result of a denormalization and the object of numerous computations. Maintaining an index on a column that is subjected to computations is even costlier than maintaining an index on a static column, since a modified key will “move” inside the index, causing the index to undergo far more updates than from the simple insertion or deletion of nodes. All specific criteria are not equally suitable for indexing. In particular, columns that are frequently updated increase maintenance costs. Returning to the example, a developer came to me one day saying that he had to optimize the following Oracle query, and he asked for some expert advice about the execution plan: select total.deptnum, total.accounting_period, total.ledger, total.cnt, error.err_cnt, cpt_error.bad_acct_count from -– First in-line view (select deptnum, accounting_period, ledger, count(account) cnt from glreport group by deptnum, ledger, accounting_period) total, -– Second in-line view (select deptnum, accounting_period, ledger, count(account) err_cnt from glreport where amount_diff <> 0 www.it-ebooks.info 134 C H A P T E R S I X group by deptnum, ledger, accounting_period) error, -– Third in-line view (select deptnum, accounting_period, ledger, count(distinct account) bad_acct_count from glreport where amount_diff <> 0 group by deptnum, ledger, accounting_period ) cpt_error where total.deptnum = error.deptnum(+) and total.accounting_period = error.accounting_period(+) and total.ledger = error.ledger(+) and total.deptnum = cpt_error.deptnum(+) and total.accounting_period = cpt_error.accounting_period(+) and total.ledger = cpt_error.ledger(+) order by total.deptnum, total.accounting_period, total.ledger For readers unfamiliar with Oracle-specific syntax, the several occurrences of (+) in the outer query’s where clause indicate outer joins. In other words: select whatever from ta, tb where ta.id = tb.id (+) is equivalent to: select whatever from ta outer join tb on tb.id = ta.id The following SQL*Plus output shows the execution plan for the query: 10:16:57 SQL> set autotrace traceonly 10:17:02 SQL> / 37 rows selected. Elapsed: 00:30:00.06 www.it-ebooks.info T H E N I N E S I T U A T I O N S 135 Execution Plan ---------------------------------------------------------- 0 SELECT", + "source": "The Art of SQL.pdf", + "chunk_id": 121 + }, + { + "text": "= ta.id The following SQL*Plus output shows the execution plan for the query: 10:16:57 SQL> set autotrace traceonly 10:17:02 SQL> / 37 rows selected. Elapsed: 00:30:00.06 www.it-ebooks.info T H E N I N E S I T U A T I O N S 135 Execution Plan ---------------------------------------------------------- 0 SELECT STATEMENT Optimizer=CHOOSE (Cost=1779554 Card=154 Bytes=16170) 1 0 MERGE JOIN (OUTER) (Cost=1779554 Card=154 Bytes=16170) 2 1 MERGE JOIN (OUTER) (Cost=1185645 Card=154 Bytes=10780) 3 2 VIEW (Cost=591736 Card=154 Bytes=5390) 4 3 SORT (GROUP BY) (Cost=591736 Card=154 Bytes=3388) 5 4 TABLE ACCESS (FULL) OF 'GLREPORT' (Cost=582346 Card=4370894 Bytes=96159668) 6 2 SORT (JOIN) (Cost=593910 Card=154 Bytes=5390) 7 6 VIEW (Cost=593908 Card=154 Bytes=5390) 8 7 SORT (GROUP BY) (Cost=593908 Card=154 Bytes=4004) 9 8 TABLE ACCESS (FULL) OF 'GLREPORT' (Cost=584519 Card=4370885 Bytes=113643010) 10 1 SORT (JOIN) (Cost=593910 Card=154 Bytes=5390) 11 10 VIEW (Cost=593908 Card=154 Bytes=5390) 12 11 SORT (GROUP BY) (Cost=593908 Card=154 Bytes=5698) 13 12 TABLE ACCESS (FULL) OF 'GLREPORT' (Cost=584519 Card=4370885 Bytes=161722745) Statistics ---------------------------------------------------------- 193 recursive calls 0 db block gets 3803355 consistent gets 3794172 physical reads 1620 redo size 2219 bytes sent via SQL*Net to client 677 bytes received via SQL*Net from client 4 SQL*Net roundtrips to/from client 17 sorts (memory) 0 sorts (disk) 37 rows processed I must confess that I didn’t waste too much time on the execution plan, since its most striking feature was fairly apparent from the text of the query itself: it shows that the table glreport, a tiny 4 to 5 million–row table, is accessed three times, once per subquery, and each time through a full scan. Nested queries are often useful when writing complex queries, especially when you mentally divide each step, and try to match a subquery to every step. But nested queries are not silver bullets, and the preceding example provides a striking illustration of how easily they may be abused. The very first inline view in the query computes the number of accounts for each department, accounting period, and ledger, and represents a full table scan that we cannot avoid. We need to face realities; we have to fully scan the table, because we are including all rows when we check how many accounts we have. We need to scan the table once, but do we absolutely need to access it a second or third time? www.it-ebooks.info 136 C H A P T E R S I X If a full table scan is required, indexes on the table become irrelevant. What matters is to be able to not only have a very analytic view of processing, but also to be able to stand back and consider what we are doing in its entirety. The second inline view counts exactly the same things as the first one—except that there is a condition on the value of amount_diff. Instead of counting with the count( ) function, we can, at the same time as we compute the total count, add 1 if amount_diff is not 0, and 0 otherwise. This is very easy to write with", + "source": "The Art of SQL.pdf", + "chunk_id": 122 + }, + { + "text": "the first one—except that there is a condition on the value of amount_diff. Instead of counting with the count( ) function, we can, at the same time as we compute the total count, add 1 if amount_diff is not 0, and 0 otherwise. This is very easy to write with the Oracle-specific decode(u, v, w, x) function or using the more standard case when u = v then w else x end construct. The third inline view filters the same rows as the second one; however, here we want to count distinct account numbers. This counting is a little trickier to merge into the first subquery; the idea is to replace the account numbers (which, by the way, are defined as varchar2* in the table) by a value which is totally unlikely to occur when amount_diff is 0; chr(1) (Oracle-speak to mean the character corresponding to the ASCII value 1) seems to be an excellent choice (I always feel a slight unease at using chr(0) with something written in C like Oracle, since C terminates all character strings with a chr(0)). We can then count how many distinct accounts we have and, of course, subtract one to avoid counting the dummy chr(1) account. So this is the suggestion that I returned to the developer: select deptnum, accounting_period, ledger, count(account) nb, sum(decode(amount_diff, 0, 0, 1)) err_cnt, count(distinct decode(amount_diff, 0, chr(1), account)) – 1 bad_acct_count from glreport group by deptnum, ledger, accounting_period My suggestion was reported to be four times as fast as the initial query, which came as no real surprise since the three full scans had been replaced by a single one. Note that there is no longer any where clause in the query: we could say that the condition on amount_diff has “migrated” to both the logic performed by the decode( ) function inside the select list and the aggregation performed by the group by clause. The * To non-Oracle users, the varchar2 type is, for all practical purposes, the same as the varchar type. www.it-ebooks.info T H E N I N E S I T U A T I O N S 137 replacement of a filtering condition that looked specific with an aggregate demonstrates that we are here in another situation, namely a result set obtained on the basis of an aggregate function. In-line queries can simplify a query, but can result in excessive and duplicated processing if used without care. Small Result Set, Indirect Criteria A situation that is superficially similar to the previous one is when you have a small result set that is based on criteria applied to tables other than the data source tables. We want data from one table, and yet our conditions apply to other, related tables from which we don’t want any data to be returned. A typical example is the question of “which customers have ordered a particular item” that we amply discussed earlier in Chapter 4. As you saw in Chapter 4, this type of query can be", + "source": "The Art of SQL.pdf", + "chunk_id": 123 + }, + { + "text": "conditions apply to other, related tables from which we don’t want any data to be returned. A typical example is the question of “which customers have ordered a particular item” that we amply discussed earlier in Chapter 4. As you saw in Chapter 4, this type of query can be expressed in either of two ways: • As a regular join with a distinct to remove duplicate rows that are the result, for instance, of customers having ordered the same item several times • By way of either a correlated or uncorrelated subquery If there is some particularly selective criterion to apply to the table (or tables) from which we obtain the result set, there is no need to say much more than what has been said in the previous situation “Small Result Set, Direct Specific Criteria”: the query will be driven by the selective criterion. and the same reasoning applies. But if there is no such criterion, then we have to be much more careful. To take a simplified version of the example in Chapter 4, identifying the customers who have ordered a Batmobile, our typical case will be something like the following: select distinct orders.custid from orders join orderdetail on (orderdetail.ordid = orders.ordid) join articles on (articles.artid = orderdetail.artid) where articles.artname = 'BATMOBILE' In my view it is much better, because it is more understandable, to make explicit the test on the presence of the article in a customer’s orders by using a subquery. But should that subquery be correlated or uncorrelated? Since we have no other criterion, the answer should be clear: uncorrelated. If not, one would have to scan the orders table and fire the subquery for each row—the type of big mistake that passes unnoticed when we start with a small orders table but becomes increasingly painful as the business gathers momentum. www.it-ebooks.info 138 C H A P T E R S I X The uncorrelated subquery can either be written in the classic style as: select distinct orders.custid from orders where ordid in (select orderdetails.ordid from orderdetail join articles on (articles.artid = orderdetail.artid) where articles.artname = 'BATMOBILE') or as a subquery in the from clause: select distinct orders.custid from orders, (select orderdetails.ordid from orderdetail join articles on (articles.artid = orderdetail.artid) where articles.artname = 'BATMOBILE') as sub_q where sub_q.ordid = orders.ordid I find the first query more legible, but it is really a matter of personal taste. Don’t forget that an in( ) condition on the result of the subquery implies a distinct and therefore a sort, which takes us to the fringe of the relational model. Where using subqueries, think carefully before choosing either a correlated or uncorrelated subquery. Small Intersection of Broad Criteria The situation we talk about in this section is that of a small result set based on the intersection of several broad criteria. Each criterion individually would produce a large result set, yet the intersection of those individual, large sets is a very small, final result set returned by the", + "source": "The Art of SQL.pdf", + "chunk_id": 124 + }, + { + "text": "The situation we talk about in this section is that of a small result set based on the intersection of several broad criteria. Each criterion individually would produce a large result set, yet the intersection of those individual, large sets is a very small, final result set returned by the query. Continuing on with our query example from the preceding section, if the existence test on the article that was ordered is not selective, we must necessarily apply some other criteria elsewhere (otherwise the result set would no longer be a small result set). In this case, the question of whether to use a regular join, a correlated subquery, or an uncorrelated subquery usually receives a different answer depending on both the relative “strength” of the different criteria and the existing indexes. Let’s suppose that instead of checking people who have ordered a Batmobile, admittedly not our best-selling article, we look for customers who have ordered something that I hope is much less unusual, in this case some soap, but purchased last Saturday. Our query then becomes something like this: www.it-ebooks.info T H E N I N E S I T U A T I O N S 139 select distinct orders.custid from orders join orderdetail on (orderdetail.ordid = orders.ordid) join articles on (articles.artid = orderdetail.artid) where articles.artname = 'SOAP' and Quite logically, the processing flow will be the reverse of what we had with a selective article: get the article, then the order lines that contained the article, and finally the orders. In the case we’re currently discussing, that of orders for soap, we should first get the small number of orders placed during the relatively short interval of time, and then check which ones refer to the article soap. From a practical point of view, we are going to use a totally different set of indexes. In the first case, ideally, we would like to see one index on the article name and one on the article identifier in the orderdetail table, and then we would have used the index on the primary key ordid in the orders table. In the case of orders for soap, what we want to find is an index on the date in orders and then one on orderid in orderdetail, from which we can use the index on the primary key of articles—assuming, of course, that in both cases using the indexes is the best course to take. The obvious natural choice to get customers who bought soap last Saturday would appear to be a correlated subquery: select distinct orders.custid from orders where and exists (select 1 from orderdetail join articles on (articles.artid = orderdetail.artid) where articles.artname = 'SOAP' and orderdetails.ordid = orders.ordid) In this approach, we take for granted that the correlated subquery executes very quickly. Our assumption will prove true only if orderdetail is indexed on ordid (we shall then get the", + "source": "The Art of SQL.pdf", + "chunk_id": 125 + }, + { + "text": "exists (select 1 from orderdetail join articles on (articles.artid = orderdetail.artid) where articles.artname = 'SOAP' and orderdetails.ordid = orders.ordid) In this approach, we take for granted that the correlated subquery executes very quickly. Our assumption will prove true only if orderdetail is indexed on ordid (we shall then get the article through its primary key artid; therefore, there is no other issue). You’ve seen in Chapter 3 that indexes are something of a luxury in transactional databases, due to their high cost of maintenance in an environment of frequent inserts, updates, and deletes. This cost may lead us to opt for a “second-best” solution. The absence of the vital index on orderdetail and good reason for not creating further indexes might prompt us to consider the following: select distinct orders.custid from orders, (select orderdetails.ordid from orderdetail, articles www.it-ebooks.info 140 C H A P T E R S I X where articles.artid = orderdetail.artid and articles.artname = 'SOAP') as sub_q where sub_q.ordid = orders.ordid and In this second approach, the index requirements are different: if we don’t sell millions of articles, it is likely that the condition on the article name will perform quite satisfactorily even in the absence of any index on artname. We shall probably not need any index on the column artid of orderdetail either: if the article is popular and appears in many orders, the join between orderdetail and articles is probably performed in a more efficient manner by hash or merge join, rather than by a nested loop that would need such an index on artid. Compared to the first approach, we have here a solution that we could call a low index solution. Because we cannot afford to create indexes on each and every column in a table, and because we usually have in every application a set of “secondary” queries that are not absolutely critical but only require a decent response time, the low index approach may perform in a perfectly acceptable manner. Adding one extra search criterion to an existing query can completely change a previous construct: a modified query is a new query. Small Intersection, Indirect Broad Criteria An indirect criterion is one that applies to a column in a table that you are joining only for the purpose of evaluating the criterion. The retrieval of a small result set through the intersection of two or more broad criteria, as in the previous situation “Small Intersection of Broad Criteria,” is often a formidable assignment. Obtaining the intersection of the large intermediary result sets by joining from a central table, or even through a chain of joins, makes a difficult situation even more daunting. This situation is particularly typical of the “star schema” that I discuss in some detail in Chapter 10, but you’ll also encounter it fairly frequently in operational databases. When you are looking for that rare combination of multiple nonselective conditions on the columns of the row, you must expect to", + "source": "The Art of SQL.pdf", + "chunk_id": 126 + }, + { + "text": "This situation is particularly typical of the “star schema” that I discuss in some detail in Chapter 10, but you’ll also encounter it fairly frequently in operational databases. When you are looking for that rare combination of multiple nonselective conditions on the columns of the row, you must expect to perform full scans at some point. The case becomes particularly interesting when several tables are involved. The DBMS engine needs to start from somewhere. Even if it can process data in parallel, at some point it has to start with one table, index, or partition. Even if the resulting set defined by the intersection of several huge sets of data is very small, a boot-strapping full table scan, and possibly two scans, will be required—with a nested loop, hash join, or merge join performed on the result. The difficulty will then be to identify which www.it-ebooks.info T H E N I N E S I T U A T I O N S 141 combination of tables (not necessarily the smallest ones) will result in the least number of rows from which the final result set will be extracted. In other words, we must find the weakest point in the line of defense, and once we have eliminated it, we must concentrate on obtaining the final result set. Let me illustrate such a case with a real-life Oracle example. The original query is a pretty complicated query, with two tables each appearing twice in the from clause. Although none of the tables is really enormous (the biggest one contains about 700,000 rows), the problem is that none of the nine parameters that are passed to the query is really selective: select (data from ttex_a, ttex_b, ttraoma, topeoma, ttypobj, ttrcap_a, ttrcap_b, trgppdt, tstg_a) from ttrcapp ttrcap_a, ttrcapp ttrcap_b, tstg tstg_a, topeoma, ttraoma, ttex ttex_a, ttex ttex_b, tbooks, tpdt, trgppdt, ttypobj where ( ttraoma.txnum = topeoma.txnum ) and ( ttraoma.bkcod = tbooks.trscod ) and ( ttex_b.trscod = tbooks.permor ) and ( ttraoma.trscod = ttrcap_a.valnumcod ) and ( ttex_a.nttcod = ttrcap_b.valnumcod ) and ( ttypobj.objtyp = ttraoma.objtyp ) and ( ttraoma.trscod = ttex_a.trscod ) and ( ttrcap_a.colcod = :0 ) -- not selective and ( ttrcap_b.colcod = :1 ) -- not selective and ( ttraoma.pdtcod = tpdt.pdtcod ) and ( tpdt.risktyp = trgppdt.risktyp ) and ( tpdt.riskflg = trgppdt.riskflg ) and ( tpdt.pdtcod = trgppdt.pdtcod ) and ( trgppdt.risktyp = :2 ) -- not selective and ( trgppdt.riskflg = :3 ) -- not selective and ( ttraoma.txnum = tstg_a.txnum ) and ( ttrcap_a.refcod = :5 ) -- not selective and ( ttrcap_b.refcod = :6 ) -- not selective and ( tstg_a.risktyp = :4 ) -- not selective and ( tstg_a.chncod = :7) -- not selective and ( tstg_a.stgnum = :8 ) -- not selective www.it-ebooks.info 142 C H A P T E R S I X When run with suitable parameters (here indicated as :0 to :8), the query takes more than 25 seconds to return fewer than 20 rows, doing about 3,000", + "source": "The Art of SQL.pdf", + "chunk_id": 127 + }, + { + "text": "not selective and ( tstg_a.stgnum = :8 ) -- not selective www.it-ebooks.info 142 C H A P T E R S I X When run with suitable parameters (here indicated as :0 to :8), the query takes more than 25 seconds to return fewer than 20 rows, doing about 3,000 physical I/Os and hitting data blocks 3,000,000 times. Statistics correctly represent the actual contents of tables (one of the very first things to check), and a query against the data dictionary gives the number of rows of the tables involved: TABLE_NAME NUM_ROWS --------------------------- ---------- ttypobj 186 trgppdt 366 tpdt 5370 topeoma 12118 ttraoma 12118 tbooks 12268 ttex 102554 ttrcapp 187759 tstg 702403 A careful study of the tables and of their relationships allows us to draw the enemy position of Figure 6-2, showing our weak criteria represented as small arrows, and tables as boxes the size of which approximately indicates the number of rows. One thing is especially remarkable: the central position of the ttraoma table that is linked to almost every other table. Unfortunately, all of our criteria apply elsewhere. By the way, an interesting fact to notice is that we are providing two values to match columns risktyp and riskflg of trgppdt—which is joined to tpdt on those very two columns, plus pdtcod. In such a case, it can be worth contemplating reversing the flow—for example, comparing the columns of tpdt to the constants provided, and only then pulling the data from trgppdt. Most DBMS allow you to check the execution plan chosen by the optimizer, either through the explain command or sometimes by directly checking in memory how something has been executed. When this query took 25 seconds, the plan, although not especially atrocious, was mostly a full scan of ttraoma followed by a series of nested loops, FIGURE 6-2. The enemy position www.it-ebooks.info T H E N I N E S I T U A T I O N S 143 using the various indexes available rather efficiently (it would be tedious to detail the numerous indexes, but suffice to say that all columns we are joining on are correctly indexed). Is this full scan the reason for slowness? Definitely not. A simple test, fetching all the rows of ttraoma (without displaying them to avoid the time associated with displaying characters on a screen) proves that it takes just a tiny fraction, hardly measurable, of the elapsed time for the overall query. When we consider the weak criteria we have, our forces are too feeble for a frontal attack against tstg, the bulk of the enemy troops, and even ttrcap won’t lead us very far, because we have poor criteria against each instance of this table, which intervenes twice in the query. However, it should be obvious that the key position of ttraoma, which is relatively small, makes an attack against it, as a first step, quite sensible—precisely the decision that the optimizer makes without any prompting. If the full scan is not to blame, then", + "source": "The Art of SQL.pdf", + "chunk_id": 128 + }, + { + "text": "which intervenes twice in the query. However, it should be obvious that the key position of ttraoma, which is relatively small, makes an attack against it, as a first step, quite sensible—precisely the decision that the optimizer makes without any prompting. If the full scan is not to blame, then where did the optimizer go wrong? Have a look at Figure 6-3, which represents the query as it was executed. When we check the order of operations, it all becomes obvious: our criteria are so bad, on face value, that the optimizer chose to ignore them altogether. Starting with a pretty reasonable full scan of ttraoma, it then chose to visit all the smallish tables gravitating around ttraoma before ending with the tables to which our filtering criteria apply. This approach is the mistake. It is likely that the indexes of the tables we first visit look much more efficient to the optimizer, perhaps because of a lower average number of table rows per key or because the indexes more closely match the order of the rows in the tables. But postponing the application of our criteria is not how we cut down on the number of rows we have to process and check. FIGURE 6-3. What the optimizer chose to do www.it-ebooks.info 144 C H A P T E R S I X Once we have taken ttraoma and hold the key position, why not go on with the tables against which we have criteria instead? The join between those tables and ttraoma will help us eliminate unwanted rows from ttraoma before proceeding to apply joins with the other tables. This is a tactic that is likely to pay dividends since—and this is information we have but that is unknown to the optimizer—we know we should have, in all cases, very few resulting rows, which means that our combined criteria should, through the joins, inflict heavy casualties among the rows of ttraoma. Even when the number of rows to be returned is larger, the execution path I suggest should still remain relatively efficient. How then can we force the DBMS to execute the query as we want it to? It depends on the SQL dialect. As you’ll see in Chapter 11, most SQL dialects allow directives, or hints, to the optimizer, although each dialect uses different syntax for such hints—telling the optimizer, for instance, to take on the tables in the same order as they are listed in the from clause. The trouble with hints is that they are more imperative than their name suggests, and every hint is a gamble on the future—a bet that circumstances, volumes, database algorithms, hardware, and the rest will evolve in such a way that our forced execution path will forever remain, if not absolutely the best, at least acceptable. In the particular case of our example, since nested loops using indexes are the most efficient choice, and because nested loops don’t really benefit from parallelism, we are taking a rather small risk", + "source": "The Art of SQL.pdf", + "chunk_id": 129 + }, + { + "text": "way that our forced execution path will forever remain, if not absolutely the best, at least acceptable. In the particular case of our example, since nested loops using indexes are the most efficient choice, and because nested loops don’t really benefit from parallelism, we are taking a rather small risk concerning the future evolution of our tables by ordering tables as we want them processed and instructing the optimizer to obey. Explicitly forcing the order followed to visit tables was the approach actually taken in this real-life case, which resulted in a query running in a little less than one second, with hardly fewer physical I/Os than before (2,340 versus 3,000—not too surprising since we start with a full scan of the very same table) but since we “suggested” a more efficient path, logical I/Os fell dramatically—to 16,500, down from over 3,000,000— with a noticeable result on the response time. Remember that you should heavily document anything that forces the hand of the DBMS. Explicitly forcing the order in which to visit tables by using optimizer directives is a heavy-handed approach. A more gentle way to obtain the same result from the optimizer, provided that it doesn’t savagely edit our SQL clauses, may be to nest queries in the from clause, thus suggesting associations like parentheses would in a numerical expression: www.it-ebooks.info T H E N I N E S I T U A T I O N S 145 select (select list) from (select ttraoma.txnum, ttraoma.bkcod, ttraoma.trscod, ttraoma.pdtcod, ttraoma.objtyp, ... from ttraoma, tstg tstg_a, ttrcapp ttrcap_a where tstg_a.chncod = :7 and tstg_a.stgnum = :8 and tstg_a.risktyp = :4 and ttraoma.txnum = tstg_a.txnum and ttrcap_a.colcod = :0 and ttrcap_a.refcod = :5 and ttraoma.trscod = ttrcap_a.valnumcod) a, ttex ttex_a, ttrcapp ttrcap_b, tbooks, topeoma, ttex ttex_b, ttypobj, tpdt, trgppdt where ( a.txnum = topeoma.txnum ) and ( a.bkcod = tbooks.trscod ) and ( ttex_b.trscod = tbooks.permor ) and ( ttex_a.nttcod = ttrcap_b.valnumcod ) and ( ttypobj.objtyp = a.objtyp ) and ( a.trscod = ttex_a.trscod ) and ( ttrcap_b.colcod = :1 ) and ( a.pdtcod = tpdt.pdtcod ) and ( tpdt.risktyp = trgppdt.risktyp ) and ( tpdt.riskflg = trgppdt.riskflg ) and ( tpdt.pdtcod = trgppdt.pdtcod ) and ( tpdt.risktyp = :2 ) and ( tpdt.riskflg = :3 ) and ( ttrcap_b.refcod = :6 ) It is often unnecessary to be very specific about the way we want a query to be executed and to multiply esoteric hints; the right initial guidance is usually enough to put an optimizer on the right track. Nested queries making explicit some table associations have the further advantage of being quite understandable to a qualified human reader. A confused query can make the optimizer confused. Clarity and suggested joins are often enough to help the optimizer provide good performance. www.it-ebooks.info 146 C H A P T E R S I X Large Result Set The situation of a large result set includes any result, irrespective of how it is obtained (with the exception of the explicit cases discussed", + "source": "The Art of SQL.pdf", + "chunk_id": 130 + }, + { + "text": "joins are often enough to help the optimizer provide good performance. www.it-ebooks.info 146 C H A P T E R S I X Large Result Set The situation of a large result set includes any result, irrespective of how it is obtained (with the exception of the explicit cases discussed here) that might be described as “large” or, in other words, a result set which it would be sensible to generate in a batch environment. When you are looking for a very large number of rows, even if this number looks like a fraction of the total number of rows stored in the tables involved in the query, conditions are probably not very selective and the DBMS engine must perform full scans, except perhaps in some very special cases of data warehousing, which are discussed in Chapter 10. When a query returns tens of thousand of rows, whether as the final result or an intermediate step in a complex query, it is usually fairly pointless to look for a subtle use of indexes and fast jumps from an index to the table rows of interest. Rather, it’s time to hammer the data remorselessly through full scans, usually associated with hash or merge joins. There must, however, be intelligence behind the brute force. We always must try to scan the objects, whether they are tables, indexes, or partitions of either tables or indexes, for which the ratio of data returned to data scanned is highest. We must scan objects for which filtering is the most coarse, because the best justification for the “effort” of scanning is to make it pay by a rich data harvest. A situation when a scan is unavoidable is the major exception to the rule of trying to get rid of unnecessary data as soon as possible; but we must fall back to the usual rule as soon as we are done with the unavoidable scans. As ever, if we consider scanning rows of no interest to us as useless work, we must minimize the number of blocks we access. An approach often taken is to minimize accesses by hitting indexes rather than tables—even if the total volume of indexes is often bigger than the volume of data, each individual index is usually much smaller than its underlying table. Assuming that an index contains all the required information, scanning the index rather than the table makes a lot of sense. Implementation techniques such as adding columns to an index to avoid visiting the table can also show their worth. Processing very large numbers of rows, whether you need to return them or simply have to check them, requires being very careful about what you do when you process each row. Calling a suboptimal, user-defined function, for instance, is not extremely important when you do it in the select list of a query that returns a small result set or when it comes as an additional criterion in a very selective where clause. But when you call such", + "source": "The Art of SQL.pdf", + "chunk_id": 131 + }, + { + "text": "process each row. Calling a suboptimal, user-defined function, for instance, is not extremely important when you do it in the select list of a query that returns a small result set or when it comes as an additional criterion in a very selective where clause. But when you call such a function hundreds of thousands of times, the DBMS is no longer forgiving, and a slight awkwardness in the code can bring your server to its knees. This is a time for lean and mean code. www.it-ebooks.info T H E N I N E S I T U A T I O N S 147 One key point to watch is the use of subqueries. Correlated subqueries are the death toll of performance when we are processing massive amounts of rows. When we can identify several subqueries within a query, we must let each of them operate on a distinct and “self-sufficient” subset, removing any dependence of one subquery on the result set of another. Dependencies between the various datasets separately obtained must be solved at the latest stage of query execution through hash joins or set operators. Relying on parallelism may also be a good idea, but only when there are very few concurrently active sessions—typically in a batch job. Parallelism as it is implemented by a DBMS consists in splitting, when possible, one query into multiple subtasks, which are run in parallel and coordinated by a dedicated task. With a very high number of users, parallelism comes naturally with many similar tasks being executed concurrently, and adding DBMS parallelism to de facto parallelism often makes throughput worse rather than better. Generally speaking, processing very large volumes of information with a very high number of concurrent sessions qualifies as a situation in which the best you can aim for is an honorable fight and in which the solution is often to throw more hardware into the ring. Response times are, lest we forget about the various waits for the availability of a resource in the course of processing, mostly dependent on the amount of data we have to browse through. But don’t forget that, as you saw in Chapter 4, the subjective vision of an end user may be utterly different from a cold analysis of the size of the haystack: the only interest to the end user is the needle. Self-Joins on One Table In a correctly designed relational database (third normal form or above), all non-key columns are about the key, the whole key, and nothing but the key, to use an excellent and frequently quoted formula.* Each row is both logically consistent and distinct from all other rows in the same table. It is this design characteristic that enables join relationships to be established within the same table. You can therefore select in the same query different (not necessarily disjoint) sets of rows from the same table and join them as if those rows came from several different tables. In this section, I’ll discuss the simple", + "source": "The Art of SQL.pdf", + "chunk_id": 132 + }, + { + "text": "characteristic that enables join relationships to be established within the same table. You can therefore select in the same query different (not necessarily disjoint) sets of rows from the same table and join them as if those rows came from several different tables. In this section, I’ll discuss the simple self- join and exclude the more complex examples of nested hierarchies that I discuss later in Chapter 7. Self-joins—tables joined to themselves—are much more common than hierarchies. In some cases, it is simply because the data is seen in an identical way, but from two * I have seen this elegant formula credited only once—to a 1983 paper by William Kent, available at http://www.bkent.net. www.it-ebooks.info 148 C H A P T E R S I X different angles; for instance, we can imagine that a query listing air flights would refer to the airports table twice, once to find the name of the departure airport, and once to find the name of the arrival airport. For example: select f.flight_number, a.airport_name departure_airport, b.airport_name arrival_airport from flights f, airports a, airports b where f.dep_iata_code = a.iata_code and f.arr_iata_code = b.iata_code In such a case, the usual rules apply: what matters is to ensure that highly efficient index access takes place. But what if the criteria are such that efficient access is not possible? The last thing we want is to do a first pass on the table, then a second one to pick up rows that were discarded during the first pass. In that case, what we should do is a single pass, collect all the rows of interest, and then use a construct such as the case statement to display separately rows from the two sets; I show examples of this “single-pass” approach in Chapter 11. There are subtle cases that only superficially look like the airport case. Imagine that we store in some table cumulative values taken at regular intervals* and we want to display by how much the counter increased between two successive snapshots. In such a case, we have a relationship between two different rows in the same table, but instead of having a strong relationship coming from another table, such as the flights table that links the two instances of airports together, we have a weak, internal relationship: we define that two rows are related not because their keys are associated in another table, but because the timestamp of one row happens to be the timestamp which immediately follows the timestamp of another row. For instance, if we assume that snapshots are taken every five minutes, with a timestamp expressed in seconds elapsed since a reference date, we might issue the following query: select a.timestamp, a.statistic_id, (b.counter – a.counter)/5 hits_per_minute from hit_counter a, hit_counter b where b.timestamp = a.timestamp + 300 and b.statistic_id = a.statistic_id order by a.timestamp, a.statistic_id * This is exactly what happens when you collect values from the V$ views in Oracle, which contain monitoring information. www.it-ebooks.info T H E N I N E", + "source": "The Art of SQL.pdf", + "chunk_id": 133 + }, + { + "text": "– a.counter)/5 hits_per_minute from hit_counter a, hit_counter b where b.timestamp = a.timestamp + 300 and b.statistic_id = a.statistic_id order by a.timestamp, a.statistic_id * This is exactly what happens when you collect values from the V$ views in Oracle, which contain monitoring information. www.it-ebooks.info T H E N I N E S I T U A T I O N S 149 There is a significant flaw in this script: if the second snapshot has not been taken exactly five minutes after the first one, down to the second, we may be unable to join the two rows. We may therefore choose to express the join condition as a range condition. For example: select a.timestamp, a.statistic_id, (b.counter – a.counter) * 60 / (b.timestamp – a.timestamp) hits_per_minute from hit_counter a, hit_counter b where b.timestamp between a.timestamp + 200 and a.timestamp + 400 and b.statistic_id = a.statistic_id order by a.timestamp, a.statistic_id One side effect of this approach is the risk of having bigger data gaps than needed when, for one reason or another (such as a change in the sampling frequency), two successive records are no longer collected between 200 and 400 seconds of each other. We may play it even safer and use an OLAP function that operates on windows of rows. It is indeed difficult to imagine something less relational in nature, but such a function can come in handy as the final shine on a query, and it can even make a noticeable difference in performance. Basically, OLAP functions allow the consideration of different subsets of the final result set, through the use of the partition clause. Sorts, sums, and other similar functions can be applied separately to these individual result subsets. We can use the row_number( ) OLAP function to create one subset by statistic_id, and then assign to each different statistic successive integer numbers that increase as timestamps do. When these numbers are generated by the OLAP function, we can join on both statistic_id and two sequential numbers, as in the following example: select a.timestamp, a.statistic_id, (b.counter - a.counter) * 60 / (b.timestamp - a.timestamp) from (select timestamp, statistic_id, counter, row_number( ) over (partition by statistic_id order by timestamp) rn from hit_counter) a, (select timestamp, statistic_id, counter, row_number( ) over (partition by statistic_id order by timestamp) rn from hit_counter) b where b.rn = a.rn + 1 and a.statistic_id = b.statistic_id order by a.timestamp, a.statistic_id www.it-ebooks.info 150 C H A P T E R S I X We may even do better—about 25% faster than the previous query—if our DBMS implements, as Oracle does, a lag(column_name, n) OLAP function that returns the nth previous value for column_name, on the basis of the specified partitioning and ordering: select timestamp, statistic_id, (counter - prev_counter) * 60 / (timestamp - prev_timestamp) from (select timestamp, statistic_id, counter, lag(counter, 1) over (partition by statistic_id order by timestamp) prev_counter, lag(timestamp, 1) over (partition by statistic_id order by timestamp) prev_timestamp from hit_counter) a order by a.timestamp, a.statistic_id In many cases we don’t have such symmetry in", + "source": "The Art of SQL.pdf", + "chunk_id": 134 + }, + { + "text": "(counter - prev_counter) * 60 / (timestamp - prev_timestamp) from (select timestamp, statistic_id, counter, lag(counter, 1) over (partition by statistic_id order by timestamp) prev_counter, lag(timestamp, 1) over (partition by statistic_id order by timestamp) prev_timestamp from hit_counter) a order by a.timestamp, a.statistic_id In many cases we don’t have such symmetry in our data, as is shown by the flight example. Typically, a query looking for all the data associated with the smallest, or the largest, or the oldest, or the most recent value of a specific column, first needs to find the actual smallest, largest, oldest, or most recent value in the column used for filtering (this is the first pass, which compares rows), and then search the table again in a second pass, using as a search criterion the value identified in the first pass. The two passes can be made (at least superficially) into one through the use of OLAP functions that operate on sliding windows. Queries applied to data values associated to timestamps or dates are a special case of sufficient importance to deserve further discussion later in this chapter as the situation “Simple or Range Searching on Dates.” When multiple selection criteria are applied to different rows in the same table, functions that operate on sliding windows may be of assistance. Result Set Obtained by Aggregation An extremely common situation is the case in which the result set is a dynamically computed summary of the detailed data from one or more main tables. In other words, we are facing an aggregation of data. When data is aggregated, the size of the result set isn’t dependent on the precision of the criteria that are provided, but merely on the cardinality of the columns that we group by. As in the first situation of the small result set obtained through precise criteria (and as you’ll see again in Chapter 11), aggregate functions (or aggregates) are also often quite useful for obtaining in a single pass on the table results that are not truly aggregated but that would otherwise require self-joins and www.it-ebooks.info T H E N I N E S I T U A T I O N S 151 multiple passes. In fact, the most interesting SQL uses of aggregates are not the cases in which sums or averages are an obvious part of the requirements, but situations in which a clever use of aggregates provides a pure SQL alternative to a procedural processing. I stress in Chapter 2 that one of the keys to efficient SQL coding is a swashbuckling approach to code execution, testing for success after the deed rather than executing preliminary queries to check if, by chance, the really useful query we want to execute may fail: you cannot win a swimming race by tiptoeing carefully into the water. The other key point is to try to pack as much “action” as possible into an SQL query, and it is in respect to this second key point that aggregate functions can be particularly useful.", + "source": "The Art of SQL.pdf", + "chunk_id": 135 + }, + { + "text": "execute may fail: you cannot win a swimming race by tiptoeing carefully into the water. The other key point is to try to pack as much “action” as possible into an SQL query, and it is in respect to this second key point that aggregate functions can be particularly useful. Much of the difficulty of good SQL programming lies in seeing how a problem can translate, not into a succession of queries to a database, but into very few queries. When, in a program, you need a lot of intermediate variables to hold values you get from the database before reinjecting them into the database as input to other queries, and if you perform against those variables nothing but very simple tests, you can bet that you have the algorithm wrong. And it is a striking feature of poorly written SQL programs to see the high number of lines of code outside of SQL queries that are simply devoted to summing up, multiplying, dividing, and subtracting inside loops what is painfully returned from the database. This is a totally useless and utterly inefficient job: we have SQL aggregate functions for that sort of work. NOTE Aggregate functions are very useful tools for solving SQL problems (and we will revisit them in Chapter 11, when I talk about stratagems); however, it often appears to me that developers use only the least interesting aggregate function of all, namely count( ), the real usefulness of which is often, at best, dubious in most programs. Chapter 2 shows that using count(*) to decide whether to update an existing row or insert a new one is wasteful. You can misuse count(*) in reports as well. A test for existence is sometimes implemented as a mock-Boolean value such as: case count(*) when 0 then 'N' else 'Y' end Such an implementation gets, when rows are found, all the rows that match the condition in order to obtain a precise count, whereas finding only one is enough to decide whether Y or N must be displayed. You can usually write a much more effective statement by using a construct that either limits the number of rows returned or tests for existence, effectively stopping processing as soon as a row that matches the condition is found. www.it-ebooks.info 152 C H A P T E R S I X But when the question at hand is about the most, the least, the greatest, or even the first or the last, it is likely that aggregate functions (possibly used as OLAP functions) will provide the best answer. If you believe that aggregate functions should be used only when counts, sums, maxima, minima, or averages are explicitly required, then you risk seriously underusing them. Interestingly, aggregate functions are extremely narrow in scope. If you exclude the computation of maximum and minimum values, the only thing they can really do is simple arithmetic; a count( ) is nothing more than adding 1s for each row encountered. Similarly, the computation of avg( )", + "source": "The Art of SQL.pdf", + "chunk_id": 136 + }, + { + "text": "seriously underusing them. Interestingly, aggregate functions are extremely narrow in scope. If you exclude the computation of maximum and minimum values, the only thing they can really do is simple arithmetic; a count( ) is nothing more than adding 1s for each row encountered. Similarly, the computation of avg( ) is just, on one hand, adding up the values in the column it is applied to and, on the other hand, adding 1s, and then dividing. But it is sometimes wonderful what you can do with simple sums. If you’re mathematically inclined, you’ll remember how easily you can switch between sums and products by the magic of logarithms and power functions. And if you’re logically inclined, you know well how much OR owes to sums and AND to products. I’ll show the power of aggregation with a simple example. Assume that we have a number of shipments to make and that each shipment is made of a number of different orders, each of which has to be separately prepared; it is only when each order in a shipment is complete that the shipment itself is ready. The problem is how to detect when all the orders comprising a shipment are complete. As is so often the case, there are several ways to determine the shipments that are complete. The worst approach would probably be to loop on all shipments, inside a second loop on each shipment count how many orders have N as value for the order_ complete column, and return shipment IDs for which the count is 0. A much better solution would be to recognize the test on the nonexistence of an N value for what it is, and use a subquery, correlated or uncorrelated; for instance: select shipment_id from shipments where not exists (select null from orders where order_complete = 'N' and orders.shipment_id = shipments.shipment_id) This approach is pretty bad if we have no other condition on the shipments table. Following is a query that may be much more efficient if we have a large shipments table and few uncompleted orders: select shipment_id from shipments where shipment_id not in (select shipment_id from orders where order_complete = 'N') www.it-ebooks.info T H E N I N E S I T U A T I O N S 153 This query can also be expressed as follows, as a variant that an optimizer may like better but that wants an index on the column shipment_id of the table orders: select shipments.shipment_id from shipments left outer join orders on orders.shipment_id = shipments.shipment_id and orders.order_complete = 'N' where orders.shipment_id is null Another alternative is a massive set operation that will operate on the primary key index of shipments on one hand, and that will perform a full table scan of orders on the other hand: select shipment_id from shipments except select shipment_id from orders where order_complete = 'N' Be aware that not all DBMS implement the except operator, sometimes known as minus. But there is still another way to express our query.", + "source": "The Art of SQL.pdf", + "chunk_id": 137 + }, + { + "text": "that will perform a full table scan of orders on the other hand: select shipment_id from shipments except select shipment_id from orders where order_complete = 'N' Be aware that not all DBMS implement the except operator, sometimes known as minus. But there is still another way to express our query. What we are doing, basically, is to return the identifiers of all shipments for which a logical AND operation on all orders which have been completed returns TRUE. This kind of operation happens to be quite common in the real world. As hinted previously, there is a very strong link between AND and multiplication, and between OR and addition. The key is to convert flags such as Y and N to 0s and 1s. This conversion is a trivial operation with the case construct. To get just order_complete as a 0 or 1 value, we can write: select shipment_id, case when order_complete = 'Y' then 1 else 0 end flag from orders So far, so good. If we always had a fixed number of orders per shipment, it would be easy to sum the calculated column and check if the result is the number of orders we expect. However, what we want here is to multiply the flag values per shipment and check whether the result is 0 or 1. That approach works, because even one incomplete order, represented by a 0, will cause the final result of all the multiplication to also be 0. The multiplication can be done with the help of logarithms (although 0s are not the easiest values to handle with logarithms). But in this particular case, our task is even easier. www.it-ebooks.info 154 C H A P T E R S I X What we want are the shipments for which the first order is completed and the second order is completed and...the nth order is completed. Logic and the laws of de Morgan* tell us that this is exactly the same as stating that we do not have (first order not completed or second order not completed...or nth order not completed). Since their kinship to sums makes ORs much easier to process with aggregates than ANDs, checking that a list of conditions linked by OR is false is much easier than checking that a list of conditions linked by AND is true. What we must consider as our true predicate is “the order is not completed” rather than the reverse, and convert the order_complete flag to 1 if it is N, and 0 if it is Y. In that way, we can easily check that we have 0s (or yeses) everywhere by summing up values—if the sum is 0, then all orders are completed; otherwise, we are at various stages of incompletion. Therefore we can also express our query as: select shipment_id from (select shipment_id, case when order_complete = 'N' then 1 else 0 end flag from orders) s group by shipment_id having sum(flag) =0 And it can be expressed in an even more", + "source": "The Art of SQL.pdf", + "chunk_id": 138 + }, + { + "text": "otherwise, we are at various stages of incompletion. Therefore we can also express our query as: select shipment_id from (select shipment_id, case when order_complete = 'N' then 1 else 0 end flag from orders) s group by shipment_id having sum(flag) =0 And it can be expressed in an even more concise way as: select shipment_id from orders group by shipment_id having sum(case when order_complete = 'N' then 1 else 0 end) =0 There is another way to write this query that is even simpler, using another aggregate function, and without any need to convert flag values. Noticing that Y is, from an alphabetical point of view, greater than N, it is not too difficult to infer that if all values are Y then the minimum will necessarily be Y too. Hence: select shipment_id from orders group by shipment_id having min(order_complete) = 'Y' * The India-born Augustus de Morgan (1806–1871) was a British mathematician who contributed to many areas of mathematics, but most significantly to the field of logic. The de Morgan laws state that the complement of the intersection of any number of sets equals the union of their comple- ments and that the complement of the union of any number of sets equals the intersection of their complements. If you remember that SQL is about sets, and that negating a condition returns the complement of the result set returned by the initial condition (if you have no null values), you’ll understand why these laws are particularly useful to the SQL practitioner. www.it-ebooks.info T H E N I N E S I T U A T I O N S 155 This approach of depending on Y to be greater than N may not be as well grounded mathematically as the flag-to-number conversion, but it is just as efficient. Of course we must see how the query that uses a group by and a condition on the minimum value for order_complete compares to the other versions that use subqueries or except instead of an aggregate function. What we can say is that it has to fully sort the orders table to aggregate the values and check whether the sum is or is not 0. As I’ve specified the problem, this solution involving a non-trivial use of an aggregate function is likely to be faster than the other queries, which hit two tables (shipments and orders), and usually less efficiently. I have made an extensive use of the having clause in the previous examples. As already mentioned in Chapter 4, a common example of careless SQL statements involves the use of the having clause in aggregate statements. Such an example is illustrated in the following (Oracle) query, which attempts to obtain the sales per product per week during the past month: select product_id, trunc(sale_date, 'WEEK'), sum(sold_qty) from sales_history group by product_id, trunc(sale_date, 'WEEK') having trunc(sale_date, 'WEEK') >= add_month(sysdate, -1) The mistake here is that the condition expressed in the having clause doesn’t depend on the aggregate. As a result, the DBMS", + "source": "The Art of SQL.pdf", + "chunk_id": 139 + }, + { + "text": "the sales per product per week during the past month: select product_id, trunc(sale_date, 'WEEK'), sum(sold_qty) from sales_history group by product_id, trunc(sale_date, 'WEEK') having trunc(sale_date, 'WEEK') >= add_month(sysdate, -1) The mistake here is that the condition expressed in the having clause doesn’t depend on the aggregate. As a result, the DBMS has to process all of the data in sales_history, sorting it and aggregating against each row, before filtering out ancient figures as the last step before returning the required rows. This is the kind of mistake that can go unnoticed until sales_history grows really big. The proper approach is, of course, to put the condition in a where clause, ensuring that the filtering occurs at an early stage and that we are working afterwards on a much reduced set of data. I should note that when we apply criteria to views, which are aggregated results, we may encounter exactly the same problem if the optimizer is not smart enough to reinject our filter before aggregation. You can have slightly more subtle variants of a filter applied later than it should be. For instance: select customer_id from orders where order_date < add_months(sysdate, -1) group by customer_id having sum(amount) > 0 In this query, the following condition looks at first glance like a reasonable use of having: having sum(amount) > 0 www.it-ebooks.info 156 C H A P T E R S I X However, this use of having does not really make sense if amount is always a positive quantity or zero. In that event, we might be better using the following condition: where amount > 0 We have two possibilities here. Either we keep the group by: select customer_id from orders where order_date < add_months(sysdate, -1) and amount > 0 group by customer_id or we notice that group by is no longer required to compute any aggregate and replace it with a distinct that in this case performs the same task of sorting and eliminating duplicates: select distinct customer_id from orders where order_date < add_months(sysdate, -1) and amount > 0 Placing the condition in the where clause allows unwanted rows to be filtered at an earlier stage, and therefore more effectively. Aggregate as little data as you can. Simple or Range Searching on Dates Among search criteria, dates (and times) hold a particular place that is all their own. Dates are extremely common, and more likely than other types of data to be subjected to range conditions, whether they are bounded (“between this date and that date”) or only partially bounded (“before this date”). Very often, and what this situation describes, the result set is derived from searches against date values referenced to the current date (e.g., “six months earlier than the current date,” etc.). The example in the previous section, “Result Set Obtained by Aggregation,” refers to a sales_history table; our condition was on an amount, but it is much more common with this type of table to have conditions on date, especially to get a snapshot of the data", + "source": "The Art of SQL.pdf", + "chunk_id": 140 + }, + { + "text": "the current date,” etc.). The example in the previous section, “Result Set Obtained by Aggregation,” refers to a sales_history table; our condition was on an amount, but it is much more common with this type of table to have conditions on date, especially to get a snapshot of the data either at a given date or between two dates. When you are looking for a value on a given date in a table containing historical data, you must pay particular attention to the way you identify current data. The way you handle current data may happen to be a special case of data predicated on an aggregate condition. www.it-ebooks.info T H E N I N E S I T U A T I O N S 157 I have already pointed out in Chapter 1 that the design of a table destined to store historical data is a tricky affair and that there is no easy, ready-made solution. Much depends on what you plan to do with your data, whether you are primarily interested in current values or in values as of a particular date. The best solution also depends on how fast data becomes outdated. If you are a retailer and wish to keep track of the wares you sell, it is likely that, unless your country suffers severe hyper-inflation, the rate of change of your prices will be pretty slow. The rate of change will be higher, possibly much higher, if you are recording the price of financial instruments or monitoring network traffic. To a large extent, what matters most with history tables is how much historical data you keep on average per item: you may store a lot of historical information for very few items, or have few historical records for a very large number of items, or anything in between. The point here is that the selectivity of any item depends on the number of items being tracked, the frequency of sampling (e.g., either once per day or every change during the day), and the total time period over which the tracking takes place (infinite, purely annual, etc.). We shall therefore first consider the case when we have many items with few historical values, then the opposite case of few items with a rich history, and then, finally, the problem of how to represent the current value. Many Items, Few Historical Values If we don’t keep an enormous amount of historical data per item, the identification of an item is quite selective by itself. Specifying the item under study restricts our “working set” to just a few historical rows, and it then becomes fairly easy to identify the value at a given reference date (the current or a previous date) as the value recorded at the closest date prior to the reference date. In this case, we are dealing once again with aggregate values. Unless some artificial, surrogate key has been created (and this is a case where there is no real need for a surrogate", + "source": "The Art of SQL.pdf", + "chunk_id": 141 + }, + { + "text": "or a previous date) as the value recorded at the closest date prior to the reference date. In this case, we are dealing once again with aggregate values. Unless some artificial, surrogate key has been created (and this is a case where there is no real need for a surrogate key), the primary key will generally be a composite key on the identifier of items (item_id) and the date associated with the historical value (record_date). We mostly have two ways of identifying the rows that store values that were current as of a given reference date: subqueries and OLAP functions. Using subqueries If we are looking for the value of one particular item as of a given date, then the situation is relatively simple. In fact, the situation is deceptively simple, and you’ll often encounter a reference to the value that was current for a given item at a given date coded as: select whatever from hist_data as outer where outer.item_id = somevalue and outer.record_date = (select max(inner.record_date) from hist_data as inner where inner.item_id = outer.item_id and inner.record_date <= reference_date) www.it-ebooks.info 158 C H A P T E R S I X It is interesting to see what the consequences of this type of construct suggest in terms of the execution path. First of all, the inner query is correlated to the outer one, since the inner query references the item_id of the current row returned by the outer query. Our starting point is therefore the outer query. Logically, from a theoretical point of view, the order of the columns in a composite primary key shouldn’t matter much. In practice, it is critical. If we have made the mistake of defining the primary key as (record_date, item_id) instead of (item_id, record_date), we desperately need an additional index on item_id for the inner query; otherwise, we will be unable to efficiently descend the tree-structured index. And we know how costly each additional index can be. Starting with our outer query and finding the various rows that store the history of item_ id, we will then use the current value of item_id to execute the subquery each time. Wait! This inner query depends only on item_id, which is, by definition, the same for all the rows we check! The logical conclusion: we are going to execute exactly the same query, returning exactly the same result for each historical row for item_id. Will the optimizer notice that the query always returns the same value? The answer may vary. It is better not to take the chance. There is no point in using a correlated subquery if it always returns the same value for all the rows for which it is evaluated. We can easily uncorrelate it: select whatever from hist_data as outer where outer.item_id = somevalue and outer.record_date = (select max(inner.record_date) from hist_data as inner where inner.item_id = somevalue and inner.record_date <= reference_date) Now the subquery can be executed without accessing the table: it finds everything it requires inside the primary key index.", + "source": "The Art of SQL.pdf", + "chunk_id": 142 + }, + { + "text": "easily uncorrelate it: select whatever from hist_data as outer where outer.item_id = somevalue and outer.record_date = (select max(inner.record_date) from hist_data as inner where inner.item_id = somevalue and inner.record_date <= reference_date) Now the subquery can be executed without accessing the table: it finds everything it requires inside the primary key index. It may be a matter of personal taste, but a construct that emphasizes the primary key is arguably preferable to the preceding approach, if the DBMS allows comparing several columns to the output of a subquery (a feature that isn’t supported by all products): select whatever from hist_data as outer where (outer.item_id, outer.record_date) in (select inner.item_id, max(inner.record_date) from hist_data as inner where inner.item_id = somevalue and inner.record_date <= reference_date group by inner.item_id) The choice of a subquery that precisely returns the columns matching a composite primary key is not totally gratuitous. If we now need to return values for a list of items, www.it-ebooks.info T H E N I N E S I T U A T I O N S 159 possibly the result of another subquery, this version of the query naturally suggests a good execution path. Replace somevalue in the inner query by an in( ) list or a subquery, and the overall query will go on performing efficiently under the very same assumptions that each item has a relatively short history. We have also replaced the equality condition by an in clause: in most cases the behavior will be exactly the same. As usual, it is at the fringes that you encounter differences. What happens if, for instance, the user mistyped the identification of the item? The in( ) will return that no data was found, while the equality may return a different error. Using OLAP functions With databases, OLAP functions such as row_number( ) that we have already used in the self-joins situation can provide a satisfactory and sometimes even a more efficient way to answer the same question “what was the current value for one particular item at a given date?” (remember that OLAP functionality does, however, introduce a distinctly non- relational aspect to the proceedings*). NOTE OLAP functions belong to the non-relational layer of SQL. They represent the final, or almost final, step in query execution, since they have to operate on the post- retrieval result set after the filtering has completed. With a function such as row_number( ) we can assign a degree of freshness (one meaning most recent) to the data by ranking on date: select row_number( ) over (partition by item_id order by record_date desc) as freshness, whatever from hist_data where item_id = somevalue and record_date <= reference_date Selecting the freshest data is then simply a matter of only retaining the rows with a value of one for freshness: select x. from (select row_number( ) over (partition by item_id order by record_date desc) as freshness, whatever from hist_data where item_id = somevalue and record_date <= reference_date) as x where x.freshness = 1 * ...even if the term OLAP was coined", + "source": "The Art of SQL.pdf", + "chunk_id": 143 + }, + { + "text": "the rows with a value of one for freshness: select x. from (select row_number( ) over (partition by item_id order by record_date desc) as freshness, whatever from hist_data where item_id = somevalue and record_date <= reference_date) as x where x.freshness = 1 * ...even if the term OLAP was coined by Dr. E.F. Codd himself in a 1993 paper. www.it-ebooks.info 160 C H A P T E R S I X In theory, there should be hardly any difference between the OLAP function approach and the use of subqueries. In practice, an OLAP function hits the table only once, even if the usual sorting happens behind the scene. There is no need for additional access to the table, even a fast one that uses the primary key. The OLAP function approach may therefore be faster (albeit only slightly so). Many Historical Values Per Item The picture may be different when we have a very large number of historical values—for instance, a monitoring system in which metrics are collected at a rather high frequency. The difficulty here lies in the fact that all the intermediate sorting required for identifying the value at or nearest a given date may have to operate on a really large amount of data. Sorting is a costly operation. If we apply the principles of Chapter 4, the only way we have to reduce the thickness of the non-relational layer is by doing a bit more work at the relational level—by increasing the amount of filtering. In such a case, it is very important to narrow our scope by bracketing the date (or time) more precisely for which we want the data. If we only provide an upper boundary, then we shall have to scan and sort the full history since the beginning of ages. If data is collected at a high frequency, it is then reasonable to give a lower limit. If we succeed in restraining the “working set” of rows to a manageable size, we are back to the case in which we have relatively few historical values per item. If specifying both an upper boundary (such as the current date) and a lower boundary isn’t an option, our only hope is in partitioning per item; operating on a single partition will take us closer to the “large result set” situation. Current Values When we are predominantly interested in the most recent or current values, it is very tempting to design a way to avoid either the nested subquery or the OLAP function (which both entail a sort), and hit the proper values directly. We mentioned in Chapter 1 that one solution to this problem is to associate each value with some “end date”—the kind of “best before” you find on your cereal boxes—and to say that for current values that end date is far, far away into the future (let’s say December 31, 2999). We also mentioned that there were some practical issues associated with such a design and the time has now come", + "source": "The Art of SQL.pdf", + "chunk_id": 144 + }, + { + "text": "of “best before” you find on your cereal boxes—and to say that for current values that end date is far, far away into the future (let’s say December 31, 2999). We also mentioned that there were some practical issues associated with such a design and the time has now come to explore these issues. With a fixed date, it certainly becomes extremely easy to find the current value. Our query simply becomes: select whatever from hist_data where item_id = somevalue and record_date = fixed_date_in_the future www.it-ebooks.info T H E N I N E S I T U A T I O N S 161 We then hit the right row, spot on, through the primary key. And of course, nothing prevents us from using either subqueries or OLAP functions whenever we need to refer to a date other than the current one. There are, however, two main drawbacks to this approach—an obvious one and a more subtle one: • The obvious drawback is that each insertion of a new historical value will first require updating what used to be the current value with, for example, today’s date, to mean that it used to be the current value until today. Then the new value can be inserted with the later date, to mean that it is now the current value until further notice. This process leads to double the amount of work, which is bad enough. Moreover, since in the relational theory the primary key is what identifies a row, the combination (item_ id, record_date) can be unique but cannot be the primary key since we have to par- tially update it. We therefore need a surrogate key to be referenced by foreign keys (identity column or sequence), which further complicates programs. The trouble with big historical tables is that usually, to grow that big, they also undergo a high rate of insertion. Does the benefit of faster querying offset the disadvantage of inserting more slowly? It’s difficult to say, but definitely a question worth asking. • The subtle drawback has to do with the optimizer. The optimizer relies on statistics that may be of variable detail, with the result that it is not unusual for it to check the lowest and highest value in a column to try to assess the spread of values. Let us say that our historical table contains values since January 1, 2000. Our data will therefore consist of perhaps 99.9% historical data, spread over several, but relatively few, years, and 0.1% of current data, officially as of December 31, 2999. The view of the opti- mizer will be of data spread over one millennium. This skewness on the part of the optimizer view of the data range is because it is being misled by the upper boundary date in the query (\"and record_date = fixed_date_in_the future\"). The problem is then that when you search for something other than current values (for instance if you want to collect variations over time for statistical purposes), the optimizer", + "source": "The Art of SQL.pdf", + "chunk_id": 145 + }, + { + "text": "data range is because it is being misled by the upper boundary date in the query (\"and record_date = fixed_date_in_the future\"). The problem is then that when you search for something other than current values (for instance if you want to collect variations over time for statistical purposes), the optimizer may well incorrectly decide that since you are accessing such a tiny fraction of the millennium, then using indexes is the thing to do, but what you really need is to scan the data. Skewness can lead to totally wrong execution plans, which are not easy to correct. You must understand your data and your data distributions if you are to understand how the optimizer views your system. Result Set Predicated on Absence of Data It is a common occurrence to look for rows in one table for which there is no matching data in another table—usually for identifying exceptions. There are two solutions people most often think of when having to deal with this type of problem: using either not in ( ) www.it-ebooks.info 162 C H A P T E R S I X with an uncorrelated subquery or not exists ( ) with a correlated subquery. Popular wisdom says that you should use not exists. Since a correlated subquery is efficient when used to mop up after the bulk of irrelevant data has been cleared out by efficient filtering, popular wisdom has it right when the subquery comes after the strong forces of efficient search criteria, and totally wrong when the subquery happens to be the only criterion. One sometimes encounters more exotic solutions to the problem of finding rows in one table for which there is no matching data in another. The following example is a real-life case that monitoring revealed to be one of the costliest queries performed against a database (note that question marks are placeholders, or bind variables, for constant values that are passed to the query on successive executions): insert into ttmpout(custcode, suistrcod, cempdtcod, bkgareacod, mgtareacod, risktyp, riskflg, usr, seq, country, rating, sigsecsui) select distinct custcode, ?, ?, ?, mgtareacod, ?, ?, usr, seq, country, rating, sigsecsui from ttmpout a where a.seq = ? and 0 = (select count(*) from ttmpout b where b.suistrcod = ? and b.cempdtcod = ? and b.bkgareacod = ? and b.risktyp = ? and b.riskflg = ? and b.seq = ?) This example must not be understood as an implicit unconditional endorsement of temporary tables! As a passing remark, I suspect that the insert statement was part of a loop. Proper performance improvement would probably be achieved by removing the loop. www.it-ebooks.info T H E N I N E S I T U A T I O N S 163 An insertion into a table based on a select on the very same table as in the current example is a particular and yet not uncommon case of self-reference, an insertion derived from existing rows and conditional on the absence of the row to be created. Using", + "source": "The Art of SQL.pdf", + "chunk_id": 146 + }, + { + "text": "N S 163 An insertion into a table based on a select on the very same table as in the current example is a particular and yet not uncommon case of self-reference, an insertion derived from existing rows and conditional on the absence of the row to be created. Using count(*) to test whether something exists or doesn’t exist is a bad idea: to count, the DBMS must search and find all rows that match. We should use exists in such a case, which stops as soon as the first match is encountered. Arguably, it does not make much difference if the filtering criterion happens to be the primary key. But it may make a very significant difference in other cases—and anyway from a semantic point of view there is no reason to say this: and 0 = (select count(*) ...) when we mean this: and not exists (select 1 ...) If we use count(*) as a test for existence, we may be lucky enough to benefit from the “invisible hand” of a smart optimizer, which will turn our query into something more suitable. But this will not necessarily be the case, and it will never be the case if the rows are counted into some variable as an independent step, because then even the smartest of optimizers cannot guess for which purpose we are counting: the result of the count( ) could be a critical value that absolutely has to be displayed to the end user! In such a case when we want to create new, unique rows derived from rows already present in the table, however, the right construct to use is probably a set operator such as except (sometimes known as minus). insert into ttmpout(custcode, suistrcod, cempdtcod, bkgareacod, mgtareacod, risktyp, riskflg, usr, seq, country, rating, sigsecsui) (select custcode, ?, ?, ?, mgtareacod, ?, ?, usr, seq, www.it-ebooks.info 164 C H A P T E R S I X country, rating, sigsecsui from ttmpout where seq = ? except select custcode, ?, ?, ?, mgtareacod, ?, ?, usr, seq, country, rating, sigsecsui from ttmpout where suistrcod = ? and cempdtcod = ? and bkgareacod = ? and risktyp = ? and riskflg = ? and seq = ?) The big advantage of set operators is that they totally break the time frame imposed by subqueries, whether they are correlated or uncorrelated. What does breaking the time frame mean? When you have correlated subqueries, you must run the outer query, and then you must execute the inner query for each row that passes through all other filtering criteria. Both queries are extremely dependent on each other, since the outer query feeds the inner one. The picture is slightly brighter with uncorrelated subqueries, but not yet totally rosy: the inner query must be executed, and in fact completed, before the outer query can step in and gather steam (something similar occurs even if the optimizer chooses to execute the global query as a hash join, which is the smart thing", + "source": "The Art of SQL.pdf", + "chunk_id": 147 + }, + { + "text": "with uncorrelated subqueries, but not yet totally rosy: the inner query must be executed, and in fact completed, before the outer query can step in and gather steam (something similar occurs even if the optimizer chooses to execute the global query as a hash join, which is the smart thing for it to do, because to execute a hash join, the SQL engine first has to scan one of the tables involved to build a hash array). With set operators, on the contrary, whether they are union, intersect or except, none of the components in the query depends on any other. As a result, the different parts of the query can run in parallel. Of course, parallelism is of hardly any benefit if one of the steps is very slow while all the others are very fast; and it will be of no benefit at all if much of the work in one part is strictly identical to the work in another part, because then you are duplicating, rather than sharing, the work between processes. But in a favorable case, it is much more efficient to have all parts run in parallel before the final step, which combines the partial result sets—divide and rule. www.it-ebooks.info T H E N I N E S I T U A T I O N S 165 There is an additional snag with using set operators: they require each part of the query to return compatible columns—an identical number of columns of identical types. A case such as the following (another real-life case, coming from a billing program) is typically unsuited to set operators: select whatever, sum(d.tax) from invoice_detail d, invoice_extractor e where (e.pga_status = 0 or e.rd_status = 0) and suitable_join_condition and (d.type_code in (3, 7, 2) or (d.type_code = 4 and d.subtype_code not in (select trans_code from trans_description where trans_category in (6, 7)))) group by what_is_required having sum(d.tax) != 0 I am always fascinated by the final condition: sum(d.tax) != 0 and the way it evokes yellow brick roads and fantasy worlds where taxes are negative. A condition such as: and d.tax > 0 might have been more appropriate in the where clause, as already demonstrated. In such a case a set operator would be rather awkward, since we would have to hit the invoice_detail table—as we can guess, not a lightweight table—several times. However, depending on the selectivity of the various criteria provided, typically if type_code=4 is a rare and therefore selective attribute condition, an exists might be more appropriate than a not in ( ). If, however, trans_description happens to be, at least relatively, a small table, then there is no doubt that trying to improve the query by playing on the existence test alone is a dead end. Another interesting way to express nonexistence—and often quite an efficient one—is to use outer joins. The purpose of outer joins is basically to return, in a join, all information from one table, including rows for which no match is found in", + "source": "The Art of SQL.pdf", + "chunk_id": 148 + }, + { + "text": "on the existence test alone is a dead end. Another interesting way to express nonexistence—and often quite an efficient one—is to use outer joins. The purpose of outer joins is basically to return, in a join, all information from one table, including rows for which no match is found in the joined table. As it happens, when we are looking for data that has no match in another table, it is precisely these rows that are of interest to us. How can we identify them? By checking the joined table columns: when there is no match, they are replaced with null values. Something such as: select whatever from invoice_detail where type_code = 4 www.it-ebooks.info 166 C H A P T E R S I X and subtype_code not in (select trans_code from trans_description where trans_category in (6, 7)) can therefore be rewritten: select whatever from invoice_detail outer join trans_description on trans_description.trans_category in (6, 7) and trans_description.trans_code = invoice_detail.subtype_code where trans_description.trans_code is null I have purposely included the condition on trans_category in the join clause. Whether it should rightly appear in this clause or in the where clause is debatable but, in fact, filtering before the join or after the join is result-neutral (of course, from a performance point of view, it can make a difference, depending on the relative selectivity of this condition and of the join condition itself). However, we have no such latitude with the condition on the null value, since this is something that can only be checked after the join. Apart from the fact that the outer join may in some cases require a distinct, in practice there should be very little difference between checking the absence of data through an outer join or a not in ( ) uncorrelated subquery, since the column which is used for the join happens to be the very same column that is compared to the result set of the subquery. But SQL is famous for being a language in which the manner of the query expression often has a very real effect on the pattern of execution, even if the theory says otherwise. It all depends on the degree of sophistication of the optimizer, and whether it processes both types of queries in a similar way or not. In other words, SQL is not a truly declarative language, even if the enhancement of optimizers with each new version slowly improves its reliability. Before closing this topic, watch out for the perennial SQL party-poopers—null values. Although in an in ( ) subquery a null value joining the flow of non-null values does not bother the outer query, with a not in ( ) subquery, any null value returned by the inner query causes the not in ( ) condition to be evaluated as false. It does not cost much to ensure that a subquery returns no null value—and doing so will save you a lot of grief. Data sets can be compared using various techniques, but outer joins and set", + "source": "The Art of SQL.pdf", + "chunk_id": 149 + }, + { + "text": "inner query causes the not in ( ) condition to be evaluated as false. It does not cost much to ensure that a subquery returns no null value—and doing so will save you a lot of grief. Data sets can be compared using various techniques, but outer joins and set operators are likely to be efficient. www.it-ebooks.info Chapter 7. C H A P T E R S E V E N Variations in Tactics Dealing with Hierarchical Data The golden rule is that there are no golden rules. —George Bernard Shaw (1856–1950) Man and Superman/Maxims for Revolutionists www.it-ebooks.info 168 C H A P T E R S E V E N Y ou have seen in the previous chapter that queries sometimes refer to the same table several times and that results can be obtained by joining a row from one table to another row in the same table. But there is a very important case in which a row is not only related to another row, but is dependent upon it. That latter row is itself dependent on another one—and so forth. I am talking here of the representation of hierarchies. Tree Structures Relational theory struck the final blow to hierarchical databases as the main repositories for structured data. Hierarchical databases were historically the first attempt at structuring data that had so far been stored as records in files. Instead of having linear sequences of identical records, various records were logically nested. Hierarchical databases were excellent for some queries, but their strong structure made one feel as if in a straitjacket, and navigating them was painful. They first bore the brunt of the assault by network, or CODASYL, databases, in which navigation was still difficult but that were more flexible, until the relational theory proved that database design was a science and not a craft. However, hierarchies, or at least hierarchical representations, are extremely common— which probably accounts for the resilience of the hierarchical model, still alive today under various names such as Lightweight Directory Access Protocol (LDAP) and XML. The handling of hierarchical data, also widely known as the Bill of Materials (BOM) problem, is not the simplest of problems to understand. Hierarchies are complicated not so much because of the representation of relationships between different components, but mostly because of the way you walk a tree. Walking a tree simply means visiting all or some of the nodes and usually returning them in a given order. Walking a tree is often implemented, when implemented at all, by DBMS engines in a procedural way—and that procedurality is a cardinal relational sin. Tree Structures Versus Master/Detail Relationships Many designers tend, not unnaturally, to consider that a parent/child link is in itself not very different from a master/detail relationship—the classical orders/order_detail relationship, in which the order_detail table stores (as part of its own key) the reference of the order it relates to. There are, however, at least four major differences between the parent/child link and the master/detail relationship: Single table", + "source": "The Art of SQL.pdf", + "chunk_id": 150 + }, + { + "text": "in itself not very different from a master/detail relationship—the classical orders/order_detail relationship, in which the order_detail table stores (as part of its own key) the reference of the order it relates to. There are, however, at least four major differences between the parent/child link and the master/detail relationship: Single table The first difference is that when we have a tree representing a hierarchy, all the nodes are of the very same nature. The leaf nodes, in other words the nodes that have no child node, are sometimes different, as happens in file management sys- tems with folders—regular nodes and files—leaf nodes, but I’ll set that case apart www.it-ebooks.info V A R I A T I O N S I N T A C T I C S 169 for the time being. Since all nodes are of the same nature, we describe them in the same way, and they will be represented by rows in the same table. Putting it another way, we have a kind of master/detail relationship, not between two dif- ferent tables holding rows of different nature, but between a table and itself. Depth The second difference is that in the case of a hierarchy, how far you are from the top is often significant information. In a master/detail relationship, you are always either the master or the detail. Ownership The third difference is that in a master/detail relationship you can have a clean foreign key integrity constraint; for instance, every order identifier in the order_ detail table must correspond to an existing identifier in the orders table, plain and simple. Such is not the case with hierarchical data. You can decide to say that, for instance, the manager number must refer to an existing employee number. Except that you then have a problem with the top manager, who in truth reports to the representatives of shareholders—the board, not an employee. This leaves us with that endless source of difficulties, a null value. And you may have several such “special case” rows, since we may need to describe in the same table several independent trees, each with its own root— something that is called a forest. Multiple parents Associating a “child” with the identifier of a “parent” assumes that a child can have only one parent. In fact, there are many real-life situations when this is not the case, whether it is investments, ingredients in formulae, or screws in mechanical parts. A case when a child has multiple parents is arguably not a tree in the mathematical sense; unfortunately, many real-life trees, including genea- logical trees, are more complex than simple parent-child relationships, and may even require the handling of special cases (outside the scope of this book) such as cycles in a line of links. In his excellent book, Practical Issues in Database Management (Addison Wesley), Fabian Pascal explains that the proper relational view of a tree is to understand that we have two distinct entity types, the nodes (for which we may have a special", + "source": "The Art of SQL.pdf", + "chunk_id": 151 + }, + { + "text": "book) such as cycles in a line of links. In his excellent book, Practical Issues in Database Management (Addison Wesley), Fabian Pascal explains that the proper relational view of a tree is to understand that we have two distinct entity types, the nodes (for which we may have a special subtype of leaf nodes, bearing more information) and the links between the nodes. I should point out that this design approach solves the question of integrity constraints, since one only describes links that actually exist. Pascal’s approach also solves the case of the “child” that appears in the descent of numerous “parents.” This case is quite common in the industry and yet so rare in textbooks, which usually stick to the employee/manager example. Pascal, following ideas of Chris Date, suggests that there should be an explode( ) operator to flatten, on the fly, a hierarchy, by providing a view which would make explicit the implicit links between nodes. The only snag is that this operator has never been www.it-ebooks.info 170 C H A P T E R S E V E N implemented. DBMS vendors have quite often implemented specialized processes such as the handling of spatial data or full-text indexing, but the proper implementation of hierarchical data has oscillated between the nonexistent and the feeble, thus leaving most of the burden of implementation just where it doesn’t belong: with the developer. As I have already hinted, the main difficulty when dealing with hierarchical data lies in walking the tree. Of course, if your aim is just to display a tree structure in a graphical user interface, each time the user clicks on a node to expand it, you have no particular problem: issuing a query that returns all the children of the node for which you pass the identifier as argument is a straightforward task. Practical Examples of Hierarchies In real life, you meet hierarchies very often, but the tasks applied to them are rarely simple. Here are just three examples of real-life problems involving hierarchies, from different industries: Risk exposure When you attempt to compute your exposure to risk in a financial structure such as a hedge fund, the matter becomes hierarchically complex. These finan- cial structures invest in funds that themselves may hold shares in other funds. Archive location If you are a big retail bank, you are likely to face a nontrivial task if you want to retrieve from your archives the file of a loan signed by John Doe seven years ago, because files are stored in folders, which are in boxes, which are on shelves, which are in cabinets in an alley in some room of some floor of some building. The nested “containers” (folders, boxes, shelves, etc.) form a hierarchy. Use of ingredients If you work for the pharmaceutical industry, identifying all of the drugs you manufacture that contain an ingredient for which a much cheaper equivalent has just been approved and can now be used presents the very same type of SQL", + "source": "The Art of SQL.pdf", + "chunk_id": 152 + }, + { + "text": "(folders, boxes, shelves, etc.) form a hierarchy. Use of ingredients If you work for the pharmaceutical industry, identifying all of the drugs you manufacture that contain an ingredient for which a much cheaper equivalent has just been approved and can now be used presents the very same type of SQL challenge in a totally unrelated area. It is important to understand that these hierarchical problems are indeed quite distinct in their fundamental characteristics. A task such as finding the location of a file in an archive means walking a tree from the bottom to the top (that is, from a position of high granularity to one of increasing aggregation), because you start from some single file reference, that will point you to the folder in which it is stored, where you will find the identification of a box, and so forth on up to the room in a building, and so on, thus determining the exact location of the file. Finding all the products that contain a given ingredient also happens to be a bottom-up walk, although in that case our number of starting points may be very high— and we have to repeat the walk each time. By contrast, risk exposure analysis means, first, a top-down walk to find all investments, followed by computations on the way back up to the top. It is a kind of aggregation, only more complicated. www.it-ebooks.info V A R I A T I O N S I N T A C T I C S 171 In general, the number of levels in trees tends to be rather small. This is, in fact, the main beauty of trees and the reason why they can be efficiently searched. If the number of levels is fixed, the only thing we have to do is to join the table containing a tree with itself as many times as we have levels. Let’s take the case of archives and say that the inventory table shows us in which folder our loan file is located. This folder identifier will take us to a location table, that points us the identifier of the box which contains the folder, the shelf upon which the box is laid, the cabinet to which the shelf belongs, the alley where we can find this cabinet, the room which contains the alley, the floor on which the room is located, and, finally, the building. If the location table treats folders, boxes, shelves, and the like as generic “locations,” a query returning all the components in the physical location of a file might look like this: select building.name building, floor.name floor, room.name room, alley.name alley, cabinet.name cabinet, shelf.name shelf, box.name box, folder.name folder from inventory, location folder, location box, location shelf, location cabinet, location alley, location room, location floor, location building where inventory.id = 'AZE087564609' and inventory.folder = folder.id and folder.located_in = box.id and box.located_in = shelf.id and shelf.located_in = cabinet.id and cabinet.located_in = alley.id and alley.located_in = room.id and room.located_in = floor.id and floor.located_in", + "source": "The Art of SQL.pdf", + "chunk_id": 153 + }, + { + "text": "location folder, location box, location shelf, location cabinet, location alley, location room, location floor, location building where inventory.id = 'AZE087564609' and inventory.folder = folder.id and folder.located_in = box.id and box.located_in = shelf.id and shelf.located_in = cabinet.id and cabinet.located_in = alley.id and alley.located_in = room.id and room.located_in = floor.id and floor.located_in = building.id This type of query, in spite of an impressive number of joins, should run fast since each successive join will use the unique index on location (that is, the index on id), presumably the primary key. But yes, there is a catch: the number of levels in a hierarchy is rarely constant. Even in the rather sedate world of archives, the contents of boxes are often moved after the passage of time to new containers (which may be more compact and, therefore, provide cheaper storage). Such activity may well replace two levels in a hierarchy with just one, as containers will replace both boxes and shelves. What should we do when we don’t know the number of levels? How best do we query such a hierarchy? Do we use a union? An outer-join? www.it-ebooks.info 172 C H A P T E R S E V E N Links between objects of the same nature should be modeled as trees as soon as the number of levels between two objects is no longer a constant. Representing Trees in an SQL Database Trees are generally represented in the SQL world by one of three models: Adjacency model The adjacency model is thus called because the identifier of the closest ancestor up in the hierarchy (the parent row) is given as an attribute of the child row. Two adjacent nodes in the tree are therefore clearly associated. The adjacency model is often illustrated by the employee number of the manager being speci- fied as an attribute of each employee managed. (The direct association of the manager to the employee is in truth a poor design, because the manager identi- fication should be an attribute of the structure that is managed. There is no rea- son that, when the head of a department is changed, one should update the records of all the people who work in the department to indicate the new man- ager). Some products implement special operators for dealing with this type of model, such as Oracle’s connect by (introduced as early as Oracle version 4 around the mid 1980s) or the more recent recursive with statement of DB2 and SQL Server. Without any such operator, the adjacency model is very hard to manage. Materialized path model The idea here is to associate with each node in the tree a representation of its position within the tree. This representation takes the form of a concatenated list of the identifiers of all the node’s ancestors, from the root of the tree down to its immediate parent, or as a list of numbers indicating the rank within siblings of a given ancestor at one generation (a method frequently used by genealogists).", + "source": "The Art of SQL.pdf", + "chunk_id": 154 + }, + { + "text": "takes the form of a concatenated list of the identifiers of all the node’s ancestors, from the root of the tree down to its immediate parent, or as a list of numbers indicating the rank within siblings of a given ancestor at one generation (a method frequently used by genealogists). These lists are usually stored as delimited strings. For instance, '1.2.3.2' means (right to left) that the node is the second child of its parent (the path of which is '1.2.3'), which itself is the third child of the grandparent ('1.2'), and so forth. Nested set model In this model, devised by Joe Celko,* a pair of numbers (defined as a left number and a right number) is associated to each node in such a fashion that they define an interval which always contains the interval associated with any of the descen- dents. The upcoming subsection “Nested Sets Model (After Celko)” under “Practi- cal Implementation of Trees” gives a practical example of this intricate scheme. * First introduced in articles in DBMS Magazine (circa 1996), and much later developed in Trees and Hierarchies in SQL for Smarties (Morgan-Kauffman). www.it-ebooks.info V A R I A T I O N S I N T A C T I C S 173 There is a fourth, less well-known model, presented by its author, Vadim Tropashko, who calls it the nested interval model, in a very interesting series of papers.* The idea behind this model is, to put it very simply, to encode the path of a given node with two numbers, which are interpreted as the numerator and the denominator of a rational number (a fraction to those uncomfortable with the vocabulary of mathematics) instead of an interval. Unfortunately, this model is heavy on computations and stored procedures and, while it looks promising for a future implementation of good tree-handling functions (perhaps the explode( ) operator) in a DBMS, it is in practice somewhat difficult to implement and not the fastest you can do, which is why I shall focus on the three aforementioned models. To keep in tone with our general theme, and to generate a reasonable amount of data, I have created a test database of the organizations of the various armies that were opposed in 1815 at the famous battle of Waterloo in Belgium, near Brussels† (known as orders of battle), which describe the structure of the Anglo-Dutch, Prussian, and French armies involved—corps, divisions, and brigades down to the level of the regiments. I use this data, and mostly the descriptions of the various units and the names of their commanders, as the basis for many of the examples that you’ll see in this chapter. I must hasten to say that the point of what follows in this chapter is to demonstrate various ways to walk hierarchies and that the design of my tables is, to say the least, pretty slack. Typically, a proper primary key for a fighting unit should be an understandable and standardized code, not a description", + "source": "The Art of SQL.pdf", + "chunk_id": 155 + }, + { + "text": "that the point of what follows in this chapter is to demonstrate various ways to walk hierarchies and that the design of my tables is, to say the least, pretty slack. Typically, a proper primary key for a fighting unit should be an understandable and standardized code, not a description that may suffer from data entry errors. Please understand that any reference to a surrogate id is indeed shorthand for an implicit, sound primary key. The main difficulty with hierarchies is that there is no “best representation.” When our interest is mostly confined to the ancestors of a few elements (a bottom-up walk), either connect by or the recursive with is, at least functionally and in terms of performance, sufficiently satisfactory. However, if we scratch the surface, connect by in particular is of course a somewhat ugly, non-relational, procedural implementation, in the sense that we can only move gradually from one row to the next one. It is much less satisfactory when we want to return either a bottom-up hierarchy for a very large number of items, or when we need to return a very large number of descendants in a top-down walk. As is so often the case with SQL, the ugliness that you can hide with a 14–row table becomes painfully obvious when you are dealing with millions, not to say billions, of rows, and that nice little SQL trick now shows its limits in terms of performance. * Initially published on http://www.dbazine.com. † Using, with his permission, the data compiled by Peter Kessler, at http://www.kessler-web.co.uk. www.it-ebooks.info 174 C H A P T E R S E V E N My example table, which contains a little more than 800 rows, is a bit larger than the usual examples, although it is in no way comparable to what you can regularly find in the industry. However, it is big enough to point out the strengths and weaknesses of the various models. The SQL implementation of trees is DBMS dependent; use what your DBMS has to offer. Practical Implementation of Trees The following subsections provide examples of each of the three hierarchy models. In each case, rows have been inserted into the example tables in the same order (ordered by commander) in an attempt to divorce the physical order of the rows from the expected result. Remember that the design is questionable, and that the purpose is to show in as simple a way as possible how to handle trees according to the model under discussion. Adjacency Model The following table describes the hierarchical organization of an army using the adjacency model. The table name I’ve chosen to use is, appropriately enough, ADJACENCY_ MODEL. Each row in the table describes a military unit. The parent_id points upward in the tree to the enclosing unit: Name Null? Type ------------------------------- -------- -------------- ID NOT NULL NUMBER PARENT_ID NUMBER DESCRIPTION NOT NULL VARCHAR2(120) COMMANDER VARCHAR2(120) Table ADJACENCY_MODEL has three indexes: a unique index on id (the primary key), an index on parent_id,", + "source": "The Art of SQL.pdf", + "chunk_id": 156 + }, + { + "text": "describes a military unit. The parent_id points upward in the tree to the enclosing unit: Name Null? Type ------------------------------- -------- -------------- ID NOT NULL NUMBER PARENT_ID NUMBER DESCRIPTION NOT NULL VARCHAR2(120) COMMANDER VARCHAR2(120) Table ADJACENCY_MODEL has three indexes: a unique index on id (the primary key), an index on parent_id, and an index on commander. Here are a few sample lines from ADJACENCY_MODEL: ID PARENT_ID DESCRIPTION COMMANDER --- --------- ---------------------------- ----------------------------- 435 0 French Armée du Nord of 1815 Emperor Napoleon Bonaparte 619 435 III Corps Général de Division Dominique Vandamme 620 619 8th Infantry Division Général de Division Baron Etienne-Nicolas Lefol 621 620 1st Brigade Général de Brigade Billard (d.15th) 622 621 15th Rgmt Léger Colonel Brice 623 621 23rd Rgmt de Ligne Colonel Baron Vernier www.it-ebooks.info V A R I A T I O N S I N T A C T I C S 175 624 620 2nd Brigade Général de Brigade Baron Corsin 625 624 37th Rgmt de Ligne Colonel Cornebise 626 620 Division Artillery 627 626 7/6th Foot Artillery Captain Chauveau Materialized Path Model Table MATERIALIZED_PATH_MODEL stores the same hierarchy as ADJACENCY_MODEL but with a different representation. The (id, parent_id) pair of columns associating adjacent nodes is replaced with a single materialized_path column that records the full “ancestry” of the current row: Name Null? Type ----------------------------------- -------- ---------------- MATERIALIZED_PATH NOT NULL VARCHAR2(25) DESCRIPTION NOT NULL VARCHAR2(120) COMMANDER VARCHAR2(120) Table MATERIALIZED_PATH_MODEL has two indexes, a unique index on materialized_path (the primary key), and an index on commander. In a real case, the choice of the path as the primary key is, of course, a very poor one, since people or objects rarely have as a defining characteristic their position in a hierarchy. In a proper design, there should be at least some kind of id, as in table ADJACENCY_MODEL. I have suppressed it simply because I had no use for it in my limited tests. However, my questionable choice of materialized_path as the key was also made with the idea of checking in that particular case the benefit of the special implementations discussed in Chapter 5, in particular, what happens when the table that describes a tree happens to map the tree structure of an index? In fact, in this particular example such mapping makes no difference. Here are the same sample lines as in the adjacency model, but with the materialized path: MATERIALIZED_PATH DESCRIPTION COMMANDER ----------------- ---------------------------- -------------------------- F French Armée du Nord of 1815 Emperor Napoleon Bonaparte F.3 III Corps Général de Division Dominique Vandamme F.3.1 8th Infantry Division Général de Division Baron Etienne-Nicolas Lefol F.3.1.1 1st Brigade Général de Brigade Billard (d.15th) F.3.1.1.1 15th Rgmt Léger Colonel Brice F.3.1.1.2 23rd Rgmt de Ligne Colonel Baron Vernier F.3.1.2 2nd Brigade Général de Brigade Baron Corsin F.3.1.2.1 37th Rgmt de Ligne Colonel Cornebise F.3.1.3 Division Artillery F.3.1.3.1 7/6th Foot Artillery Captain Chauveau www.it-ebooks.info 176 C H A P T E R S E V E N Nested Sets Model (After Celko) With the nested set", + "source": "The Art of SQL.pdf", + "chunk_id": 157 + }, + { + "text": "Colonel Baron Vernier F.3.1.2 2nd Brigade Général de Brigade Baron Corsin F.3.1.2.1 37th Rgmt de Ligne Colonel Cornebise F.3.1.3 Division Artillery F.3.1.3.1 7/6th Foot Artillery Captain Chauveau www.it-ebooks.info 176 C H A P T E R S E V E N Nested Sets Model (After Celko) With the nested set model, we have two columns, left_num and right_num, which describe how each row relates to other rows in the hierarchy. I’ll show shortly how those two numbers are used to specify a hierarchical position: Name Null? Type ----------------------------------- -------- ------------- DESCRIPTION VARCHAR2(120) COMMANDER VARCHAR2(120) LEFT_NUM NOT NULL NUMBER RIGHT_NUM NOT NULL NUMBER Table NESTED_SETS_MODEL has a composite primary key, (left_num, right_num) plus an index on commander. As with the materialized path model, this is a poor choice but it is adequate for our present tests. It is probably time now to explain how the mysterious numbers, left_num and right_num, are obtained. Basically, one starts from the root of the tree, assigning 1 to left_num for the root node. Then all child nodes are recursively visited, as shown in Figure 7-1, and a counter increases at each call. You can see the counter on the line in the figure. It begins with 1 for the root node and increases by one as each node is visited. Say that we visit a node for the very first time. For instance, in the example of Figure 7-1, after having assigned the integer 1 to the left_num value of the 1st Corps node, we encounter (for the first time) the node 1st British Guards Division. We increase our counter and assign 2 to left_num. Then we visit the node’s children, encountering for the first time 1st Guards Brigade and assigning the value of our counter, 3 at this stage, to left_num. But this node, on this example, has no child. Because there is no child, we increment our counter and assign its value to right_num, which in this case takes the value 4. Then we move on to the node’s sibling, 2nd Guards Brigade. It is the same story with this sibling. Finally, we return—our second visit—to the parent node 1st British Guards Division and can assign the new value of our counter, which has now reached 7, to its right_num. We then proceed to the next sibling, 3rd Anglo-German Division, and so on. As mentioned earlier, you can see that the [left_num, right_num] pair of any node is enclosed within the [left_num, right_num] pair of any of its ascendants—hence the name FIGURE 7-1. How nested sets numbers are assigned www.it-ebooks.info V A R I A T I O N S I N T A C T I C S 177 of nested sets. Since, however, we have three independent trees (the Anglo-Dutch, Prussian, and French armies), which is called in technical terms a forest, I have had to create an artificial top level that I have called Armies of 1815. Such an artificial top level is not required by the other models. Here", + "source": "The Art of SQL.pdf", + "chunk_id": 158 + }, + { + "text": "Since, however, we have three independent trees (the Anglo-Dutch, Prussian, and French armies), which is called in technical terms a forest, I have had to create an artificial top level that I have called Armies of 1815. Such an artificial top level is not required by the other models. Here is what we get from our example after having computed all numbers: DESCRIPTION COMMANDER LEFT_NUM RIGHT_NUM ---------------------------- -------------------------- -------- ---------- Armies of 1815 1 1622 French Armée du Nord of 1815 Emperor Napoleon Bonaparte 870 1621 III Corps Général de Division 1237 1316 Dominique Vandamme 8th Infantry Division Général de Division Baron 1238 1253 Etienne-Nicolas Lefol 1st Brigade Général de Brigade Billard 1239 1244 (d.15th) 15th Rgmt Léger Colonel Brice 1240 1241 23rd Rgmt de Ligne Colonel Baron Vernier 1242 1243 2nd Brigade Général de Brigade Baron 1245 1248 Corsin 37th Rgmt de Ligne Colonel Cornebise 1246 1247 Division Artillery 1249 1252 7/6th Foot Artillery Captain Chauveau 1250 1251 The rows in our sample that are at the bottom level in the hierarchy can be spotted by noticing that right_num is equal to left_num + 1. The author of this clever method claims that it is much better than the adjacency model because it operates on sets and that is what SQL is all about. This is perfectly true, except that SQL is all about unbounded sets, whereas his method relies on finite sets, in that you must count all nodes before being able to assign the right_num value of the root. And of course, whenever you insert a node somewhere, you must renumber both the left_num and right_num values of all the nodes that should be visited after the new node, as well as the right_num value of all its ascendants. The necessity to modify many other items when you insert a new item is exactly what happens when you store an ordered list into an array: as soon as you insert a new value, you have to shift, on average, half the array. The nested set model is imaginative, no doubt, but a relational nightmare, and it is difficult to imagine worse in terms of denormalization. In fact, the nested sets model is a pointer-based solution, the very quagmire from which the relational approach was designed to escape. Walking a Tree with SQL In order to check efficiency and performance, I have compared how each model performed with respect to the following two problems: 1. To find all the units under the command of the French general Dominique Vandamme (a top-down query), if possible as an indented report (which requires keeping track of www.it-ebooks.info 178 C H A P T E R S E V E N the depth within the tree) or as a simple list. Note that in all cases we have an index on the commander’s name. I refer to this problem as the Vandamme query. 2. To find, for all regiments of Scottish Highlanders, the various units they belong to, once again with and", + "source": "The Art of SQL.pdf", + "chunk_id": 159 + }, + { + "text": "depth within the tree) or as a simple list. Note that in all cases we have an index on the commander’s name. I refer to this problem as the Vandamme query. 2. To find, for all regiments of Scottish Highlanders, the various units they belong to, once again with and without proper indentation (a bottom-up query). We have no index on the names of units (column description in the tables), and our only way to spot Scottish Highlanders is to look for the Highland string in the name of the unit, which of course means a full scan in the absence of any full-text indexing. I refer to this problem as the Highlanders query. To ensure that the only variation from test to test was in the model used, my comparisons are all done using the same DBMS, namely Oracle. Top-Down Walk: The Vandamme Query In the Vandamme query, we start with the commander of the French Third Corps, General Vandamme, and want to display in an orderly fashion all units under his command. We don’t want a simple list: the structure of the army corps must be clear, as the corps is made of divisions that are themselves made of brigades that are themselves usually composed of two regiments. Adjacency model Writing the Vandamme query with the adjacency model is fairly easy when using Oracle’s connect by operator. All you have to specify is the node you wish to start from (start with) and how each two successive rows returned relate to each other (connect by = prior , or connect by = prior , depending on whether you are walking down or up the tree). For indentation, Oracle maintains a pseudo-column named level that tells you how many levels away from the starting point you are. I am using this pseudo-column and left-padding the description with as many spaces as the current value of level. My query is: select lpad(description, length(description) + level) description, commander from adjacency_model connect by parent_id = prior id start with commander = 'Général de Division Dominique Vandamme' And the results are: DESCRIPTION COMMANDER ------------------------------- ----------------------------------------------- III Corps Général de Division Dominique Vandamme 8th Infantry Division Général de Division Baron Etienne-Nicolas Lefol 2nd Brigade Général de Brigade Baron Corsin 37th Rgmt de Ligne Colonel Cornebise 1st Brigade Général de Brigade Billard (d.15th) 23rd Rgmt de Ligne Colonel Baron Vernier 15th Rgmt Léger Colonel Brice ... www.it-ebooks.info V A R I A T I O N S I N T A C T I C S 179 10th Infantry Division Général de Division Baron Pierre-Joseph Habert 2nd Brigade Général de Brigade Baron Dupeyroux 70th Rgmt de Ligne Colonel Baron Maury 22nd Rgmt de Ligne Colonel Fantin des Odoards 2nd (Swiss) Infantry Rgmt Colonel Stoffel 1st Brigade Général de Brigade Baron Gengoult 88th Rgmt de Ligne Colonel Baillon 34th Rgmt de Ligne Colonel Mouton", + "source": "The Art of SQL.pdf", + "chunk_id": 160 + }, + { + "text": "Baron Pierre-Joseph Habert 2nd Brigade Général de Brigade Baron Dupeyroux 70th Rgmt de Ligne Colonel Baron Maury 22nd Rgmt de Ligne Colonel Fantin des Odoards 2nd (Swiss) Infantry Rgmt Colonel Stoffel 1st Brigade Général de Brigade Baron Gengoult 88th Rgmt de Ligne Colonel Baillon 34th Rgmt de Ligne Colonel Mouton Division Artillery 18/2nd Foot Artillery Captain Guérin 40 rows selected. Now, what about the other member in the adjacency family, the recursive with statement?* With this model, a recursive-factorized statement is defined, which is made of the union (the union all, to be precise) of two select statements: • The select that defines our starting point, which in this particular case is: select 1 level, id, description, commander from adjacency_model where commander = 'Général de Division Dominique Vandamme' What is this solitary 1 for? It represents, as the alias indicates, the depth in the tree. In contrast to the Oracle connect by implementation, this DB2 implementation has no sys- tem pseudo-variable to tell us where we are in the tree. We can compute our level, however, and I’ll explain more about that in just a moment. • The select which defines how each child row relates to its parent row, as it is returned by this very same query that we can call, with a touch of originality, recursive_query: select parent.level + 1, child.id, child.description, child.comander from recursive_query parent, adjacency_model child where parent.id = child.parent_id Notice in this query that we add 1 to parent.level. Each execution of this query repre- sents a step down the tree. For each step down the tree, we increment our level, thus keeping track of our depth. All that’s left is to fool around with functions to nicely indent the description, and here is our final query: with recursive_query(level, id, description, commander) as (select 1 level, id, description, * Using this time the first product that implemented it, namely DB2. www.it-ebooks.info 180 C H A P T E R S E V E N commander from adjacency_model where commander = 'Général de Division Dominique Vandamme' union all select parent.level + 1, child.id, child.description, child.commander from recursive_query parent, adjacency_model child where parent.id = child.parent_id) select char(concat(repeat(' ', level), description), 60) description, commander from recursive_query Of course, you have to be a real fan of the recursive with to be able to state without blushing that the syntax here is natural and obvious. However, it is not too difficult to understand once written; and it’s even rather satisfactory, except that the query first returns General Vandamme as expected, but then all the officers directly reporting to him, and then all the officers reporting to the first one at the previous level, followed by all officers reporting to the second one at the previous level, and so on. The result is not quite the nice top-to-bottom walk of the connect by, showing exactly who reports to whom. I’ll hasten to say that since ordering doesn’t belong to the relational theory, there is nothing wrong with the ordering", + "source": "The Art of SQL.pdf", + "chunk_id": 161 + }, + { + "text": "to the second one at the previous level, and so on. The result is not quite the nice top-to-bottom walk of the connect by, showing exactly who reports to whom. I’ll hasten to say that since ordering doesn’t belong to the relational theory, there is nothing wrong with the ordering that you get from with, but that ordering does raise an important question: in practice, how can we order the rows from a hierarchical query? Ordering the rows from a hierarchical query using recursive with is indeed possible if, for instance, we make the not unreasonable assumption that one parent node never has more than 99 children and that the tree is not monstrously deep. Given these caveats, what we can do is associate with each node a number that indicates where it is located in the hierarchy—say 1.030801—to mean the first child (the two rightmost digits) of the eighth child (next two digits, from right to left) of the third child of the root node. This assumes, of course, that we are able to order siblings, and we may not always be able to assign any natural ordering to them. Sometimes it is necessary to arbitrarily assign an order to each sibling using, possibly, an OLAP function such as row_number( ). We can therefore slightly modify our previous query to arbitrarily assign an order to siblings and to use the just-described technique for ordering the result rows: with recursive_query(level, id, rank, description, commander) as (select 1, id, cast(1 as double), description, commander from adjacency_model where commander = 'Général de Division Dominique Vandamme' union all www.it-ebooks.info V A R I A T I O N S I N T A C T I C S 181 select parent.level + 1, child.id, parent.rank + ranking.sn / power(100.0, parent.level), child.description, child.commander from recursive_query parent, (select id, row_number( ) over (partition by parent_id order by description) sn from adjacency_model) ranking, adjacency_model child where parent.id =child.parent_id and child.id = ranking.id) select char(concat(repeat(' ', level), description), 60) description, commander from recursive_query order by rank We might fear that the ranking query that appears as a recursive component of the query would be executed for each node in the tree that we visit, returning the same result set each time. This isn’t the case. Fortunately, the optimizer is smart enough not to execute the ranking query more than is necessary, and we get: DESCRIPTION COMMANDER ----------------------------- ---------------------------------------------- III Corps Général de Division Dominique Vandamme 10th Infantry Division Général de Division Baron Pierre-Joseph Habert 1st Brigade Général de Brigade Baron Gengoult 34th Rgmt de Ligne Colonel Mouton 88th Rgmt de Ligne Colonel Baillon 2nd Brigade Général de Brigade Baron Dupeyroux 22nd Rgmt de Ligne Colonel Fantin des Odoards 2nd (Swiss) Infantry Rgmt Colonel Stoffel 70th Rgmt de Ligne Colonel Baron Maury Division Artillery 18/2nd Foot Artillery Captain Guérin 11th Infantry Division Général de Division Baron Pierre Berthézène ... 23rd Rgmt de Ligne Colonel Baron Vernier 2nd Brigade Général de Brigade Baron Corsin 37th Rgmt de Ligne Colonel", + "source": "The Art of SQL.pdf", + "chunk_id": 162 + }, + { + "text": "Odoards 2nd (Swiss) Infantry Rgmt Colonel Stoffel 70th Rgmt de Ligne Colonel Baron Maury Division Artillery 18/2nd Foot Artillery Captain Guérin 11th Infantry Division Général de Division Baron Pierre Berthézène ... 23rd Rgmt de Ligne Colonel Baron Vernier 2nd Brigade Général de Brigade Baron Corsin 37th Rgmt de Ligne Colonel Cornebise Division Artillery 7/6th Foot Artillery Captain Chauveau Reserve Artillery Général de Division Baron Jérôme Doguereau 1/2nd Foot Artillery Captain Vollée 2/2nd Rgmt du Génie The result is not strictly identical to the connect by case, simply because we have ordered siblings by alphabetical order on the description column, while we didn’t order siblings at all with connect by (we could have ordered them by adding a special clause). But otherwise, the very same hierarchy is displayed. www.it-ebooks.info 182 C H A P T E R S E V E N While the result of the with query is logically equivalent to that of the connect by query, the with query is a splendid example of nightmarish, obfuscated SQL, which in comparison makes the five-line connect by query look like a model of elegant simplicity. And even if on this particular example performance is more than acceptable, one can but wonder with some anguish at what it might be on very large tables. Must we disregard the recursive with as a poor, substandard implementation of the superior connect by? Let’s postpone conclusions until the end of this chapter. The ranking number we built in the recursive query is nothing more than a numerical representation of the materialized path. It is therefore time to check how we can display the troops under the command of General Vandamme using a simple materialized path implementation. Materialized path model Our query is hardly more difficult to write under the materialized path model—but for the level, which is derived from the path itself. Let’s assume just for an instant that we have at hand a function named mp_depth( ) that returns the number of hierarchical levels between the current node and the top of the tree. We can write a query as: select lpad(a.description, length(a.description) + mp_depth(...)) description, a.commander from materialized_path_model a, materialized_path_model b where a.materialized_path like b.materialized_path || '%' and b.commander = 'Général de Division Dominique Vandamme') order by a.materialized_path Before dealing with the mp_depth( ) function, I’ll note a few traps. In my example, I have chosen to start the materialized path with A for the Anglo-Dutch army, P for the Prussian one, and F for the French one. That first letter is then followed by dot-separated digits. Thus, the 12th Dutch line battalion, under the command of Colonel Bagelaar, is A.1.4.2.3, while the 11th Régiment of Cuirassiers of Colonel Courtier is F.9.1.2.2. Ordering by materialized path can lead to the usual problems of alphabetical sorts of strings of digits, namely that 10.2 will be returned before 2.3; however, I should stress that, since the separator has a lower code (in ASCII at least) than 0, then the order of levels will be", + "source": "The Art of SQL.pdf", + "chunk_id": 163 + }, + { + "text": "Ordering by materialized path can lead to the usual problems of alphabetical sorts of strings of digits, namely that 10.2 will be returned before 2.3; however, I should stress that, since the separator has a lower code (in ASCII at least) than 0, then the order of levels will be respected. The sort may not, however respect the order of siblings implied by the path. Does that matter? I don’t believe that it does because sibling order is usually information that can be derived from something other than the materialized path itself (for instance, brothers and sisters can be ordered by their birth dates, rather than by the path). Be careful with the approach to sorting that I’ve used here. The character encoding used by your database might throw off the results. www.it-ebooks.info V A R I A T I O N S I N T A C T I C S 183 What about our mysterious mp_depth( ) function now? The hierarchical difference between any commander under General Vandamme and General Vandamme himself can be defined as the difference between the absolute levels (i.e., counting down from the root of the tree) of the unit commanded by General Vandamme and any of the underlying units. How then can we determine the absolute level? Well, by counting the dots. To count the dots, the easiest thing to do is probably to start with suppressing them, with the help of the replace( ) function that you find in the SQL dialect of all major products. All you have to do next is subtract the length of the string without the dots from the length of the string with the dots, and you get exactly what you want, the dot-count: length((materialized_path) – length(replace(materialized_path, '.', '')) If we check the result of our dot-counting algorithm for the author of the epigraph that adorns Chapter 6 (a cavalry colonel at the time), here is what we get: SQL> select materialized_path, 2 length(materialized_path) len_w_dots, 3 length(replace(materialized_path, '.', '')) len_wo_dots, 4 length(materialized_path) - 5 length(replace(materialized_path, '.', '')) depth, 6 commander 7 from materialized_path_model 8 where commander = 'Colonel de Marbot' 9 / MATERIALIZED_PATH LEN_W_DOTS LEN_WO_DOTS DEPTH COMMANDER ----------------- ---------- ----------- ---------- ------------------ F.1.5.1.1 9 5 4 Colonel de Marbot Et voilà. Nested sets model Finding all the units under the command of General Vandamme is very easy under the nested sets model, since the model requires us to have numbered our nodes in such a way that the left_num and right_num of a node bracket are the left_num and right_num of all descendants. All we have to write is: select a.description, a.commander from nested_sets_model a, nested_sets_model b where a.left_num between b.left_num and b.right_num and b.commander = 'Général de Division Dominique Vandamme' All? Not quite. We have no indentation here. How do we get the level? Unfortunately, the only way we have to get the depth of a node (from which indentation is derived) is by counting how many nodes we have between that node and the", + "source": "The Art of SQL.pdf", + "chunk_id": 164 + }, + { + "text": "'Général de Division Dominique Vandamme' All? Not quite. We have no indentation here. How do we get the level? Unfortunately, the only way we have to get the depth of a node (from which indentation is derived) is by counting how many nodes we have between that node and the root. There is no way to derive depth from left_num and right_num (in contrast to the materialized path model). www.it-ebooks.info 184 C H A P T E R S E V E N If we want to display an indented list under the nested sets model, then we must join a third time with our nested_sets_model table, for the sole purpose of computing the depth: select lpad(description, length(description) + depth) description, commander from (select count(c.left_num) depth, a.description, a.commander, a.left_num from nested_sets_model a, nested_sets_model b, nested_sets_model c where a.left_num between c.left_num and c.right_num and c.left_num between b.left_num and b.right_num and b.commander = 'Général de Division Dominique Vandamme' group by a.description, a.commander, a.left_num) order by left_num The simple addition of the indentation requirement makes the query, as with (sic) the recursive with( ), somewhat illegible. Comparing the Vandamme query under the various models After having checked that all queries were returning the same 40 rows properly indented, I then ran each of the queries 5,000 times in a loop (thus returning a total of 200,000 rows). I have compared the number of rows returned per second, taking the adjacency model as our 100-mark reference. You see the results in Figure 7-2. As Figure 7-2 shows, for the Vandamme query, the adjacency model, in which the tree is walked using connect by, outperforms the competition despite the procedural nature of connect by. The materialized path makes a decent show, but probably suffers from the function calls to compute the depth and therefore the indentation. The cost of a nicely indented output is even more apparent with the nested sets model, where the obvious FIGURE 7-2. Performance comparison for the Vandamme query www.it-ebooks.info V A R I A T I O N S I N T A C T I C S 185 performance killer is the computation of the depth through an additional join and a group by. One might cynically suggest that, since this model is totally hard-wired, static, and non-relational, we might as well go whole hog in ignoring relational design tenets and store the depth of each node relative to the root. Doing so would certainly improve our query’s performance, but at a horrendous cost in terms of maintenance. Bottom-Up Walk: The Highlanders Query As I said earlier, looking for the Highland string within the description attributes will necessarily lead to a full scan of the table. But let’s write our query with each of the models in turn, and then we’ll consider the resulting performance implications. Adjacency model The Highlanders query is very straightforward to write using connect by, and once again we use the dynamically computed level pseudo-column to indent our result properly. Note that level was previously", + "source": "The Art of SQL.pdf", + "chunk_id": 165 + }, + { + "text": "our query with each of the models in turn, and then we’ll consider the resulting performance implications. Adjacency model The Highlanders query is very straightforward to write using connect by, and once again we use the dynamically computed level pseudo-column to indent our result properly. Note that level was previously giving the depth, and now it returns the height since it is always computed from our starting point, and that we now return the parent after the child: select lpad(description, length(description) + level) description, commander from adjacency_model connect by id = prior parent_id start with description like '%Highland%' And here is the result that we get: DESCRIPTION COMMANDER ---------------------------------- ---------------------------------------- 2/73rd (Highland) Rgmt of Foot Lt-Colonel William George Harris 5th British Brigade Major-General Sir Colin Halkett 3rd Anglo-German Division Lt-General Count Charles von Alten I Corps Prince William of Orange The Anglo-Allied Army of 1815 Field Marshal Arthur Wellesley, Duke of Wellington 1/71st (Highland) Rgmt of Foot Lt-Colonel Thomas Reynell British Light Brigade Major-General Frederick Adam 2nd Anglo-German Division Lt-General Sir Henry Clinton II Corps Lieutenant-General Lord Rowland Hill The Anglo-Allied Army of 1815 Field Marshal Arthur Wellesley, Duke of Wellington 1/79th (Highland) Rgmt of Foot Lt-Colonel Neil Douglas 8th British Brigade Lt-General Sir James Kempt 5th Anglo-German Division Lt-General Sir Thomas Picton (d.18th) General Reserve Duke of Wellington The Anglo-Allied Army of 1815 Field Marshal Arthur Wellesley, Duke of Wellington 1/42nd (Highland) Rgmt of Foot Colonel Sir Robert Macara (d.16th) 9th British Brigade Major-General Sir Denis Pack 5th Anglo-German Division Lt-General Sir Thomas Picton (d.18th) General Reserve Duke of Wellington www.it-ebooks.info 186 C H A P T E R S E V E N The Anglo-Allied Army of 1815 Field Marshal Arthur Wellesley, Duke of Wellington 1/92nd (Highland) Rgmt of Foot Lt-Colonel John Cameron 9th British Brigade Major-General Sir Denis Pack 5th Anglo-German Division Lt-General Sir Thomas Picton (d.18th) General Reserve Duke of Wellington The Anglo-Allied Army of 1815 Field Marshal Arthur Wellesley, Duke of Wellington 25 rows selected. The non-relational nature of connect by appears plainly enough: our result is not a relation, since we have duplicates. The name of the Duke of Wellington appears eight times, but in two different capacities, five times (as many times as we have Highland regiments) as commander-in-chief, and three as commander of the General Reserve. Twice—once as commander of the General Reserve and once as commander-in-chief— would have been amply sufficient. Can we easily remove the duplicates? No we cannot, at least not easily. If we apply a distinct, the DBMS will sort our result to get rid of the duplicate rows and will break the hierarchical order. We get a result that somehow answers the question. But you can take it or leave it according to the details of your requirements. Materialized path model The Highlanders query is slightly more difficult to write under the materialized path model. Identifying the proper rows and indenting them correctly is easy: select lpad(a.description, length(a.description) + mp_depth(b.materialized_path) - mp_depth(a.materialized_path)) description, a.commander from materialized_path_model a,", + "source": "The Art of SQL.pdf", + "chunk_id": 166 + }, + { + "text": "it or leave it according to the details of your requirements. Materialized path model The Highlanders query is slightly more difficult to write under the materialized path model. Identifying the proper rows and indenting them correctly is easy: select lpad(a.description, length(a.description) + mp_depth(b.materialized_path) - mp_depth(a.materialized_path)) description, a.commander from materialized_path_model a, materialized_path_model b where b.materialized_path like a.materialized_path || '%' and b.description like '%Highland%') However, we have two issues to solve: • We have duplicates, as with the adjacency model. • The order of rows is not the one we want. Paradoxically, the second issue is the reason why we can solve the first one easily; since we shall have to find a means of correctly ordering anyway, adding a distinct will break nothing in this case. How can we order correctly? As usual, by using the materialized path as our sort key. By adding these two elements and pushing the query into the from clause so as to be able to sort by materialized_path without displaying the column, we get: www.it-ebooks.info V A R I A T I O N S I N T A C T I C S 187 select description, commander from (select distinct lpad(a.description, length(a.description) + mp_depth(b.materialized_path) - mp_depth(a.materialized_path)) description, a.commander, a.materialized_path from materialized_path_model a, materialized_path_model b where b.materialized_path like a.materialized_path || '%' and b.description like '%Highland%') order by materialized_path desc which displays: DESCRIPTION COMMANDER ---------------------------------- ---------------------------------------- 1/92nd (Highland) Rgmt of Foot Lt-Colonel John Cameron 1/42nd (Highland) Rgmt of Foot Colonel Sir Robert Macara (d.16th) 9th British Brigade Major-General Sir Denis Pack 1/79th (Highland) Rgmt of Foot Lt-Colonel Neil Douglas 8th British Brigade Lt-General Sir James Kempt 5th Anglo-German Division Lt-General Sir Thomas Picton (d.18th) General Reserve Duke of Wellington 1/71st (Highland) Rgmt of Foot Lt-Colonel Thomas Reynell British Light Brigade Major-General Frederick Adam 2nd Anglo-German Division Lt-General Sir Henry Clinton II Corps Lieutenant-General Lord Rowland Hill 2/73rd (Highland) Rgmt of Foot Lt-Colonel William George Harris 5th British Brigade Major-General Sir Colin Halkett 3rd Anglo-German Division Lt-General Count Charles von Alten I Corps Prince William of Orange The Anglo-Allied Army of 1815 Field Marshal Arthur Wellesley, Duke of Wellington 16 rows selected. This is a much nicer and more compact result than is achieved with the adjacency model. However, I should point out that a condition such as: where b.materialized_path like a.materialized_path || '%' where we are looking for a row in the table aliased by a, knowing the rows in the table aliased by b, is something that, generally speaking, may be slow because we can’t make efficient use of the index on the column. What we would like to do, to make efficient use of the index, is the opposite, looking for b.materialized_path knowing a.materialized_path. There are ways to decompose a materialized path into the list of the materialized paths of the ancestors of the node (see Chapter 11), but that operation is not without cost. On our sample data, the query we have here was giving far better results than decomposing the material path", + "source": "The Art of SQL.pdf", + "chunk_id": 167 + }, + { + "text": "There are ways to decompose a materialized path into the list of the materialized paths of the ancestors of the node (see Chapter 11), but that operation is not without cost. On our sample data, the query we have here was giving far better results than decomposing the material path so as to perform a more efficient join with the materialized path of each ancestor. However, this might not be true against several million rows. www.it-ebooks.info 188 C H A P T E R S E V E N Nested sets model Once again, what hurts this model is that the depth must be dynamically computed, and that computation is a rather heavy operation. Since the Highlanders query is a bottom-up query, we must take care not to display the artificial root node (easily identified by left_ num = 1) that we have had to introduce. Moreover, I have had to hard-code the maximum depth (6) to be able to indent properly. In our display, top levels are more indented than bottom levels, which means that padding is inversely proportional to depth. Since the depth is difficult to get, defining the indentation as 6 – depth was the simplest way to achieve the required result. As with the materialized path model, we have to reorder anyway, so we have no scruple about applying a distinct to get rid of duplicate rows. Here’s the query: select lpad(description, length(description) + 6 - depth) description, commander from (select distinct b.description, b.commander, b.left_num, (select count(c.left_num) from nested_sets_model c where b.left_num between c.left_num and c.right_num) depth from nested_sets_model a, nested_sets_model b where a.description like '%Highland%' and a.left_num between b.left_num and b.right_num and b.left_num > 1) order by left_num desc This query displays exactly the same result as does the materialized path query in the preceding section. Comparing the various models for the Highlanders query I have applied the same test to the Highlanders query as to the Vandamme query earlier, running each of the queries 5,000 times, with a minor twist: the adjacency model, as we have seen, returns duplicate rows that we cannot get rid of. My test returns 5,000 times 25 rows for the adjacency model, and 5,000 times 16 rows with the other models, because they are the only rows of interest. If we measure performance as a simple number of rows returned by unit of time, with the adjacency model we are also counting many rows that we are not interested in. I have therefore added an adjusted adjacency model, for which performance is measured as the number of rows of interest—the rows returned by the other two models—per unit of time. The result is given in Figure 7-3. It is quite obvious from Figure 7-3 that the adjacency model outperforms the two other models by a very wide margin before adjustment, and still by a very comfortable margin after adjustment. Also notice that the materialized path model is still faster than the nested sets model, but only marginally so. www.it-ebooks.info", + "source": "The Art of SQL.pdf", + "chunk_id": 168 + }, + { + "text": "quite obvious from Figure 7-3 that the adjacency model outperforms the two other models by a very wide margin before adjustment, and still by a very comfortable margin after adjustment. Also notice that the materialized path model is still faster than the nested sets model, but only marginally so. www.it-ebooks.info V A R I A T I O N S I N T A C T I C S 189 We therefore see that, in spite of its procedural nature, the implementation of the connect by works rather well, both for top-down and bottom-up queries, provided of course that columns are suitably indexed. However, the return of duplicate rows in bottom-up queries when there are several starting points can prove to be a practical nuisance. When connect by or a recursive with is not available, the materialized path model makes a good substitute. It is interesting to see that it performs better than the totally hard-wired nested sets model. When designing tables to store hierarchical data, there are a number of mistakes to avoid, some of which are made in our example: The materialized path should in no way be the key, even if it is unique. It is true that strong hierarchies are not usually associated with dynamic envi- ronments, but you are not defined by your place in a hierarchy. The materialized path should not imply any ordering of siblings. Ordering does not belong to a relational model; it is simply concerned with the presentation of data. You must not have to change anything in other rows when you insert a new node or delete an existing one (which is probably the biggest practical reason, forgetting about all theoretical reasons, for not using the nested sets model). It is always easy to insert a node as the parents’ last child. You can order everything first by sorting on the materialized path of the parent, and then on whichever attribute looks suitable for ordering the siblings. The choice of the encoding is not totally neutral. The choice is not neutral because whether you must sort by the materialized path or by the parent’s materialized path, you must use that path as a sort key. The safest approach is probably to use numbers left padded with zeroes, for instance 001.003.004.005 (note that if we always use three positions for each number, the separator can go). You might be afraid of the materialized path’s length; but if we assume that each parent never has more than 100 children numbered from 0 to 99, 20 characters allow us to store a materialized path for up to 10 levels, or trees containing up to 10010 nodes—probably more than needed. FIGURE 7-3. Performance comparison for the Highlanders query www.it-ebooks.info 190 C H A P T E R S E V E N Walking trees, whether down from the root or up from a leaf node, is by nature a sequential and therefore slow operation. Aggregating Values from Trees Now that you know how to", + "source": "The Art of SQL.pdf", + "chunk_id": 169 + }, + { + "text": "for the Highlanders query www.it-ebooks.info 190 C H A P T E R S E V E N Walking trees, whether down from the root or up from a leaf node, is by nature a sequential and therefore slow operation. Aggregating Values from Trees Now that you know how to deal with trees, let’s look at how you can aggregate values held in tree structures. Most cases for the aggregation of values held in hierarchical structures fall into two categories: aggregation of values stored in leaf nodes and propagation of percentages across various levels in the tree. Aggregation of Values Stored in Leaf Nodes In a more realistic example than the one used to illustrate the Vandamme and Highlanders queries, nodes carry information—especially the leaf nodes. For instance, regiments should hold the number of their soldiers, from which we can derive the strength of every fighting unit. Modeling head counts If we take the same example we used previously, restricting it to a subset of the French Third Corps of General Vandamme and only descending to the level of brigades, a reasonably correct representation (as far as we can be correct) would be the tables described in the following subsections. UNITS. Each row in the units table describes the various levels of aggregation (army corps, division, brigade) as in tables adjacency_model, materialized_path_models, or nested_ sets_model, but without any attribute to specify how each unit relates to a larger unit: ID NAME COMMANDER -- -------------------------- ----------------------------------------------- 1 III Corps Général de Division Dominique Vandamme 2 8th Infantry Division Général de Division Baron Etienne-Nicolas Lefol 3 1st Brigade Général de Brigade Billard 4 2nd Brigade Général de Brigade Baron Corsin 5 10th Infantry Division Général de Division Baron Pierre-Joseph Habert 6 1st Brigade Général de Brigade Baron Gengoult 7 2nd Brigade Général de Brigade Baron Dupeyroux 8 11th Infantry Division Général de Division Baron Pierre Berthézène 9 1st Brigade Général de Brigade Baron Dufour 10 2nd Brigade Général de Brigade Baron Logarde 11 3rd Light Cavalry Division Général de Division Baron Jean-Simon Domont 12 1st Brigade Général de Brigade Baron Dommanget 13 2nd Brigade Général de Brigade Baron Vinot 14 Reserve Artillery Général de Division Baron Jérôme Doguereau Since the link between units is no longer stored in this table, we need an additional table to describe how the different nodes in the hierarchy relate to each other. www.it-ebooks.info V A R I A T I O N S I N T A C T I C S 191 UNIT_LINKS_ADJACENCY. We may use the adjacency model once more, but this time links between the various units are stored separately from other attributes, in an adjacency list, in other words a list that associates to the (technical) identifier of each row, id, the identifier of the parent row. Such a list isolates the structural information. Our unit_ links_adjacency table looks like this: ID PARENT_ID ---------- ---------- 2 1 3 2 4 2 5 1 6 5 7 5 8 1 9 8 10", + "source": "The Art of SQL.pdf", + "chunk_id": 170 + }, + { + "text": "associates to the (technical) identifier of each row, id, the identifier of the parent row. Such a list isolates the structural information. Our unit_ links_adjacency table looks like this: ID PARENT_ID ---------- ---------- 2 1 3 2 4 2 5 1 6 5 7 5 8 1 9 8 10 8 11 1 12 11 13 11 14 1 UNIT_LINKS_PATH. But you have seen that an adjacency list wasn’t the only way to describe the links between the various nodes in a tree. Alternatively, we may as well store the materialized path, and we can put that into the unit_links_path table: ID PATH ---------- ----------------- 1 1 2 1.1 3 1.1.1 4 1.1.2 5 1.2 6 1.2.1 7 1.2.2 8 1.3 9 1.3.1 10 1.3.2 11 1.4 12 1.4.1 13 1.4.2 14 1.5 UNIT_STRENGTH. Finally, our historical source has provided us with the number of men in each of the brigades—the lowest unit level in our sample. We’ll put that information into our unit_strength table: ID MEN ---------- ---------- 3 2952 4 2107 6 2761 www.it-ebooks.info 192 C H A P T E R S E V E N 7 2823 9 2488 10 2050 12 699 13 318 14 152 Computing head counts at every level With the adjacency model, it is typically quite easy to retrieve the number of men we have recorded for the third corps; all we have to write is a simple query such as: select sum(men) from unit_strength where id in (select id from unit_links_adjacency connect by prior id = parent_id start with parent_id = 1) Can we, however, easily get the head count at each level, for example, for each division (the battle unit composed of two brigades) as well? Certainly, in the very same way, just by changing the starting point—using the identifier of each division each time instead of the identifier of the French Third Corps. We are now facing a choice: either we have to code procedurally in our application, looping on all fighting units and summing up what needs to be summed up, or we have to go for the full SQL solution, calling the query that computes the head count for each and every row returned. We need to slightly modify the query so as to return the actual head count each time the value is directly known, for example, for our lowest level, the brigade. For instance: select u.name, u.commander, (select sum(men) from unit_strength where id in (select id from unit_links_adjacency connect by parent_id = prior id start with parent_id = u.id) or id = u.id) men from units u It is not very difficult to realize that we shall be hitting again and again the very same rows, descending the very same tree from different places. Understandably, on large volumes, this approach will kill performance. This is where the procedural nature of connect by, which leaves us without a key to operate on (something I pointed out when I could not get rid of duplicates without", + "source": "The Art of SQL.pdf", + "chunk_id": 171 + }, + { + "text": "same rows, descending the very same tree from different places. Understandably, on large volumes, this approach will kill performance. This is where the procedural nature of connect by, which leaves us without a key to operate on (something I pointed out when I could not get rid of duplicates without destroying the order I wanted), leaves us no other choice than to adopt procedural processing when performance becomes a critical issue; “for all they that take the procedure shall perish with the procedure.” www.it-ebooks.info V A R I A T I O N S I N T A C T I C S 193 We are in a slightly better position with the materialized path here, if we are ready to allow a touch of black magic that I shall explain in Chapter 11. I have already referred to the explosion of links; it is actually possible, even if it is not a pretty sight, to write a query that explodes unit_links_path. I have called this view exploded_links_path and here is what it displays when it is queried: SQL> select * from exploded_links_path; ID ANCESTOR DEPTH ---------- ---------- ---------- 14 1 1 13 1 2 12 1 2 11 1 1 10 1 2 9 1 2 8 1 1 7 1 2 6 1 2 5 1 1 4 1 2 3 1 2 2 1 1 4 2 1 3 2 1 7 5 1 6 5 1 10 8 1 9 8 1 13 11 1 12 11 1 depth gives the generation gap between id and ancestor. When you have this view, it becomes a trivial matter to sum up over all levels (bar the bottom one in this case) in the hierarchy: select u.name, u.commander, sum(s.men) men from units u, exploded_links_path el, unit_strength s where u.id = el.ancestor and el.id = s.id group by u.name, u.commander which returns: NAME COMMANDER MEN -------------------------- -------------------------------------- ----- III Corps Général de Division Dominique Vandamme 16350 8th Infantry Division Général de Division Baron Etienne- 5059 Nicolas Lefol 10th Infantry Division Général de Division Baron Pierre 5584 Joseph Habert www.it-ebooks.info 194 C H A P T E R S E V E N 11th Infantry Division Général de Division Baron Pierre 4538 Berthézène 3rd Light Cavalry Division Général de Division Baron Jean-Simon 1017 Domont (We can add, through a union, a join between units and unit_strength to see units displayed for which nothing needs to be computed.) I ran the query 5,000 times to determine the numerical strength for all units, and then I compared the number of rows returned per unit time. As might be expected, the result shows that the adjacency model, which had so far performed rather well, bites the dust, as is illustrated in Figure 7-4. Simpler tree implementation sometimes performs quite well when computing aggregates. Propagation of Percentages Across Different Levels Must we conclude that with a materialized path and a pinch of adjacency where available we can solve anything more or less elegantly and", + "source": "The Art of SQL.pdf", + "chunk_id": 172 + }, + { + "text": "well, bites the dust, as is illustrated in Figure 7-4. Simpler tree implementation sometimes performs quite well when computing aggregates. Propagation of Percentages Across Different Levels Must we conclude that with a materialized path and a pinch of adjacency where available we can solve anything more or less elegantly and efficiently? Unfortunately not, and our last example will really demonstrate the limits of some SQL implementations when it comes to handling trees. For this case, let’s take a totally different example, and we will assume that we are in the business of potions, philters, and charms. Each of them is composed of a number of ingredients—and our recipes just list the ingredients and their percentage composition. Where is the hierarchy? Some of our recipes share a kind of “base philter” that appears as a kind of compound ingredient, as in Figure 7-5. FIGURE 7-4. Performance comparison when computing the head count of each unit www.it-ebooks.info V A R I A T I O N S I N T A C T I C S 195 Our aim is, in order to satisfy current regulations, to display on the package of Philter #5 the names and proportions of all the basic ingredients. First, let’s consider how we can model such a hierarchy. In such a case, a materialized path would be rather inappropriate. Contrarily to fighting units that have a single, well-defined place in the army hierarchy, any ingredient, including compound ones such as Potion #9, can contribute to many preparations. A path cannot be an attribute of an ingredient. If we decide to “flatten” compositions and create a new table to associate a materialized path to each basic ingredient in a composition, any change brought to Potion #9 would have to ripple through potentially hundreds of formulae, with the unacceptable risk in this line of business of one change going wrong. The most natural way to represent such a structure is therefore to say that our philter contains so much of powdered unicorn horn, so much of asphodel, and so much of Potion #9 and so forth, and to include the composition of Potion #9. Figure 7-6 illustrates one way we can model our database. We have a generic components table with two subtypes, recipes and basic_ingredients, and a composition table storing the quantity of a component (a recipe or a basic ingredient) that appears in each recipe. FIGURE 7-5. Don’t try this at home FIGURE 7-6. The model for recipes www.it-ebooks.info 196 C H A P T E R S E V E N However, Figure 7-6’s design is precisely where an approach such as connect by becomes especially clunky. Because of the procedural nature of the connect by operator, we can include only two levels, which could be enough for the case of Figure 7-5, but not in a general case. What do I mean by including two levels? With connect by we have the visibility of two levels at once, the current level and the parent level,", + "source": "The Art of SQL.pdf", + "chunk_id": 173 + }, + { + "text": "operator, we can include only two levels, which could be enough for the case of Figure 7-5, but not in a general case. What do I mean by including two levels? With connect by we have the visibility of two levels at once, the current level and the parent level, with the possible exception of the root level. For instance: SQL> select connect_by_root recipe_id root_recipe, 2 recipe_id, 3 prior pct, 4 pct 5 component_id 6 from composition 7 connect by recipe_id = prior component_id 8 / ROOT_RECIPE RECIPE_ID PRIORPCT PCT COMPONENT_ID ----------- ---------- ---------- ------------ ------------ 14 14 5 3 14 14 20 7 14 14 15 8 14 14 30 9 14 14 20 10 14 14 10 2 15 15 30 14 15 14 30 5 3 15 14 30 20 7 15 14 30 15 8 15 14 30 30 9 ... In this example, root_recipe refers to the root of the tree. We can handle simultaneously the percentage of the current row and the percentage of the prior row, in tree-walking order, but we have no easy way to sum up, or in this precise case, to multiply values across a hierarchy, from top to bottom. The requirement for propagating percentages across levels is, however, a case where a recursive with statement is particularly useful. Why? Remember that when we tried to display the underlings of General Vandamme we had to compute the level to know how deep we were in the tree, carrying the result from level to level across our walk. That approach might have seemed cumbersome then. But that same approach is what will now allow us to pull off an important trick. The great weakness of connect by is that at one given point in time you can only know two generations: the current row (the child) and its parent. If we have only two levels, if Potion #9 contains 15% of Mandragore and Philter #5 contains 30% of Potion #9, by accessing simultaneously the child (Potion #9) and the parent (Philter #5) we can easily say that we actually have 15% of 30%—in other words, 4.5% of Mandragore in Philter #5. But what if we have more than two www.it-ebooks.info V A R I A T I O N S I N T A C T I C S 197 levels? We may find a way to compute how much of each individual ingredient is contained in the final products with procedures, either in the program that accesses the database, or by invoking user-defined functions to store temporary results. But we have no way to make such a computation through plain SQL. “What percentage of each ingredient does a formula contain?” is a complicated question. The recursive with makes answering it a breeze. Instead of computing the current level as being the parent level plus 1, all we have to do is compute the actual percentage as being the current percentage (how much Mandragore we have in Potion #9) multiplied by", + "source": "The Art of SQL.pdf", + "chunk_id": 174 + }, + { + "text": "is a complicated question. The recursive with makes answering it a breeze. Instead of computing the current level as being the parent level plus 1, all we have to do is compute the actual percentage as being the current percentage (how much Mandragore we have in Potion #9) multiplied by the parent percentage (how much Potion #9 we have in Philter #5). If we assume that the names of the components are held in the components table, we can write our recursive query as follows: with recursive_composition(actual_pct, component_id) as (select a.pct, a.component_id from composition a, components b where b.component_id = a.recipe_id and b.component_name = 'Philter #5' union all select parent.pct * child.pct, child.component_id from recursive_composition parent, composition child where child.recipe_id = parent.component_id) Let’s say that the components table has a component_type column that contains I for a basic ingredient and R for a recipe. All we have to do in our final query is filter (with an f) recipes out, and, since the same basic ingredient can appear at various different levels in the hierarchy, aggregate per ingredient: select x.component_name, sum(y.actual_pct) from recursive_composition y, components x where x.component_id = y.component_id and x.component_type = 'I' group by x.component_name As it happens, even if the adjacency model looks like a fairly natural way to represent hierarchies, its two implementations are in no way equivalent, but rather complementary. While connect by may superficially look easier (once you have understood where prior goes) and is convenient for displaying nicely indented hierarchies, the somewhat tougher recursive with allows you to process much more complex questions relatively easily—and those complex questions are the type more likely to be encountered in real life. You only have to check the small print on a cereal box or a toothpaste tube to notice some similarities with the previous example of composition analysis. www.it-ebooks.info 198 C H A P T E R S E V E N In all other cases, including that of a DBMS that implements a connect by, our only hope of generating the result from a “single SQL statement” is by writing a user-defined function, which has to be recursive if the DBMS cannot walk the tree. A more complex tree walking syntax may make a more complex question easier to answer in pure SQL. While the methods described in this chapter can give reasonably satisfactory results against very small amounts of data, queries using the same techniques against very large volumes of data may execute “as slow as molasses.” In such a case, you might consider a denormalization of the model and a trigger-based “flattening” of the data. Many, including myself, frown upon denormalization. However, I am not recommending that you consider denormalizing for the oft-cited inherent slowness of the relational model, so convenient for covering up incompetent programming, but because SQL still lacks a truly adequate, scaleable processing of tree structures. www.it-ebooks.info Chapter 8. C H A P T E R E I G H T Weaknesses and Strengths Recognizing and Handling Difficult Cases", + "source": "The Art of SQL.pdf", + "chunk_id": 175 + }, + { + "text": "oft-cited inherent slowness of the relational model, so convenient for covering up incompetent programming, but because SQL still lacks a truly adequate, scaleable processing of tree structures. www.it-ebooks.info Chapter 8. C H A P T E R E I G H T Weaknesses and Strengths Recognizing and Handling Difficult Cases No one can guarantee success in war, but only deserve it. —Sir Winston Churchill (1874–1965) www.it-ebooks.info 200 C H A P T E R E I G H T T here are a number of cases when one has either to fight on unfavorable ground, or to attack a formidable amount of data with feeble weapons. In this chapter, I am going to try to describe a number of these difficult cases; first to try to sketch some tactics to disentangle oneself with honor from a perilous situation, and, perhaps more importantly, to be able to recognize as soon as possible those options that may just lead us into a trap. In mechanics, the larger the number of moving parts, the greater the odds that something will break. This is an observation that applies to complex architectures as well. Unfortunately, snappy, exciting new techniques—or indeed revamped, dull old ones— often make us forget this important principle: keep things simple. Simpler often means faster and always means more robust. But simpler for the database doesn’t always mean simpler for the developer, and simplicity often requires more skills than complexity. In this chapter, we shall first consider a case when a criterion that looks efficient proves rather weak but can be reinvigorated, and then we shall consider the dangers of abstract “persistency” layers and distributed systems. We shall finally look in some detail at a PHP/MySQL example showing the subtleties of combining flexibility with efficiency when a degree of freedom is left to the program user for the choice of search criteria. Deceiving Criteria I already mentioned in Chapter 6 that in some queries we have a very selective combination of criteria that individually are not very selective. I noted that this was a rather difficult situation from which to achieve good performance. Another interesting case, but one in which we are not totally helpless, is a criterion that at first sight looks efficient, that has the potential for becoming an efficient criterion, but that requires some attention to fulfill its potential. Credit card validation procedures provide a good example of such a criterion. As you may know, a credit card number encodes several pieces of information, including credit card type, issuer, and so on. By way of example, let’s look at the problem of achieving a first level of control for payments made at a toll road in one of the most visited Western European countries. This means checking a very large number of credit cards, supplied by a large number of international issuers, each with its own unique method of encoding. Credit card numbers can have a maximum of 19 digits, with some exceptions, such as the cards issued by", + "source": "The Art of SQL.pdf", + "chunk_id": 176 + }, + { + "text": "most visited Western European countries. This means checking a very large number of credit cards, supplied by a large number of international issuers, each with its own unique method of encoding. Credit card numbers can have a maximum of 19 digits, with some exceptions, such as the cards issued by MasterCard (16 digits), Visa (16 or 13), and American Express (15 digits), to mention just three well-known issuers. The first six digits in all cases indicate who the issuer is, and the last digit is a kind of checksum to spot any mistyping. A first, coarse level of control could be to check that the issuer is known, and that the checksum is correct. However the checksum algorithm is public knowledge (it can be found on the www.it-ebooks.info W E A K N E S S E S A N D S T R E N G T H S 201 Internet) and can easily be faked. A more refined level of control also checks that the prefix of the card number belongs, for one given issuer, to a valid range of values for this issuer, together with an additional control on the number of digits in the card. In our case, we are provided with a list of about 200,000 valid prefixes of varying lengths. How do we write the query to test a given card number against the valid ranges of values for the card’s issuer? The following is easy enough: select count(*) from credit_card_check where ? like prefix + '%' The where ? indicates the card number to check and here + denotes string concatenation, often done via || or concat( ). We just have to index the prefix column, and we will get a full table scan each time. Why is a full table scan happening? Haven’t we seen that an index was usable when we were addressing only the leftmost part of the key? True enough, but saying that the value we want to check is the leftmost part of the full key is not the same as saying, as here, that the full key is the leftmost part of the value we want to check. The difference may seem subtle, but the two cases are mirror images of each other. Suppose that the credit card number to verify is 4000 0012 3456 7899*. Now imagine that our credit_card_checks table holds values such as 312345, 3456 and 40001. We can see those three values as prefixes and, more or less implicitly, we see them as being in sorted order. First of all, they are in ascending order if they are stored as strings of characters, but not if they are stored as numbers. But there is yet more to worry about. When we descend a tree (our index), we have a value to compare to the keys stored in the tree. If the value is equal to the key stored into the current node, we are done. Otherwise, we have to search a subtree that", + "source": "The Art of SQL.pdf", + "chunk_id": 177 + }, + { + "text": "is yet more to worry about. When we descend a tree (our index), we have a value to compare to the keys stored in the tree. If the value is equal to the key stored into the current node, we are done. Otherwise, we have to search a subtree that depends on whether our value is smaller or greater than that key. If we had a prefix of fixed length, we would have no difficulty: we should only take the suitable number of digits from our card number (the current prefix), and compare it to the prefixes stored in the index. But when the length of the prefix varies, which is our case, we must compare a different number of characters each time. This is not a task that a regular SQL index search knows how to perform. Is there any way out? Fortunately, there is one. An operator such as like actually selects a range of values. If we want to check, say, that a 16-digit Visa card number is like 4000%, it actually means that we expect to find it between 4000000000000000 and 400099999999999. If we had a composite index on these lower and upper boundary * An invalid card number, in case you were wondering.... www.it-ebooks.info 202 C H A P T E R E I G H T numbers, then we could very easily check the card number by checking the index. That is, if all card numbers had 16 digits. But a varying number of digits is a problem that is easy to solve. All cards have a maximum number of 19 digits. If we right-pad our Visa card number with three more 0s, thus bringing its total number of digits to 19, we can as validly check whether 4000001234567899000 is between 4000000000000000000 and 400099999999999999. Instead of storing prefixes, we need to have two columns: lower_bound and upper_bound. The first one, the lower_bound, is obtained by right-padding our prefix to the maximum length of 19 with 0s, and upper_bound is obtained by right-padding with 9s. Granted, this is denormalization of a sort. However, this is a real read-only reference table, which makes our sin slightly more forgivable. We just have to index (lower_bound, upper_bound) and write our condition as the following to see our query fly: where substring(? + '0000000000000000000', 1, 19) between lower_bound and upper_bound Many products directly implement an rpad( ) function for right-padding. When we have a variable-length prefix to check, the solution is to get back to a common access case—the index range scan. Try to express unusual conditions such as comparisons on a prefix or a part of a key in known terms of range condition; whenever possible, try to ensure that there is a lower and an upper bound. Abstract Layers It is a common practice to create a succession of abstract layers over a suite of software primitives, ostensibly for maintenance reasons and software reuse. This is a worthy practice and provides superb material for exciting management presentations.", + "source": "The Art of SQL.pdf", + "chunk_id": 178 + }, + { + "text": "ensure that there is a lower and an upper bound. Abstract Layers It is a common practice to create a succession of abstract layers over a suite of software primitives, ostensibly for maintenance reasons and software reuse. This is a worthy practice and provides superb material for exciting management presentations. Unfortunately, this approach can very easily be abused, especially when the software primitives consist of database accesses. Of course, such an industrial aspect of software engineering is usually associated with modern, object-oriented languages. I am going to illustrate how not to encapsulate database accesses with some lines from a real-life program. Interestingly for a book entitled The Art of SQL, the following fragment of C# code (of questionable sharpness...) contains only bits of an SQL statement. It is nevertheless extremely relevant to our topic, for deplorable reasons. 1 public string Info_ReturnValueUS(DataTable dt, 2 string codeForm, 3 string infoTxt) 4 { 5 string returnValue = String.Empty ; www.it-ebooks.info W E A K N E S S E S A N D S T R E N G T H S 203 6 try 7 { 8 infoTxt = infoTxt.Replace(\"'\",\"''\"); 9 string expression = ComparisonDataSet.FRM_CD 10 + \" = '\" + codeForm 11 + \"' and \" + ComparisonDataSet.TXT_US 12 + \" = '\" + infoTxt + \"'\" ; 13 DataRow[] drsAttr = dt.Select(expression); 14 15 foreach (DataRow dr in drsAttr) 16 { 17 if (dr[ComparisonDataSet.VALUE_US].ToString().ToUpper().Trim( ) 18 != String.Empty) 19 { 20 returnValue = dr[ComparisonDataSet.VALUE_US].ToString( ) ; 21 break; 22 } 23 } 24 } 25 catch (MyException myex) 26 { 27 throw myex ; 28 } 29 catch (Exception ex) 30 { 31 throw new MyException(\"Info_ReturnValueUS \" + ex.Message) ; 32 } 33 return returnValue ; 34 } There is no need to be a C# expert to grasp the purpose of the above method, at least in general terms. The objective is to return the text associated with a message code. That text is to be returned in a given language (in this case American English, as US suggests). This code is from a multilingual system, and there is a second, identical function, in which two other letters replace the letters U and S. No doubt when other languages will be required, the same lines of code will be copied as many times as we have different languages, and the suitable ISO code substituted for US each time. Will it ease maintenance, when each change to the program has to be replicated to umpteen identical functions (...but for the ISO code)? I may be forgiven for doubting it, in spite of my legendary faith in what exciting management presentations promise modern languages to deliver. But let’s study the program a little more closely. The string expression in lines 9–12 is an example of shameless hardcoding, before being passed in line 13 to a Select( ) method that can reasonably be expected to perform a query. In fact, it would seem that two different types of elements are hardcoded: column", + "source": "The Art of SQL.pdf", + "chunk_id": 179 + }, + { + "text": "a little more closely. The string expression in lines 9–12 is an example of shameless hardcoding, before being passed in line 13 to a Select( ) method that can reasonably be expected to perform a query. In fact, it would seem that two different types of elements are hardcoded: column names (stored in attributes ComparisonDataSet.FRM_CD and ComparisonDataSet.TXT_US—and here, apparently, there is one column per supported language, which is a somewhat dubious design) and actual values passed to the query (codeForm and infoTxt). Column names can only be hardcoded, but www.it-ebooks.info 204 C H A P T E R E I G H T there should not be a very great number of different combinations of column names, so that the number of different queries that can be generated will necessarily be small and we will have no reason to worry about this. The same cannot be said of actual values: we may query as many different values as we have rows in the table; in fact we may even query more, generating queries that may return nothing. The mistake of hard-coding values from codeForm and infoTxt into the SQL statement is serious because this type of “give me the associated label” query is likely to be called a very high number of times. As it is written, each call will trigger the full mechanism of parsing, determining the best execution plan, and so on—for no advantage. The values should be passed to the query as bind variables—just like arguments are passed to a function. The loop of lines 15–23 is no less interesting. The program is looking for the first value that is not empty in the dataset just returned—dare we say the first value that is not null? Why code into an external application something that the SQL language can do perfectly well? Why return from the server possibly many more rows than are required, just to discard them afterwards? This is too much work. The database server will do more work, because even if we exit the loop at the first iteration, it is quite common to pre-fetch rows in order to optimize network traffic. The server may well have already returned tens or hundreds of rows before our application program begins its first loop. The application server does more work too, because it has to filter out most of what the database server painstakingly returned. Needless to say, the developer has written more code than is required. It is perfectly easy to add a suitable condition to expression, so that unneeded rows are not returned. As the C# code generates the query, the server has no idea that we are interested only in the first non-null value and will simply do as instructed. If we were to try and check on the database side for a clue indicating wrongly written code, the only thing that may possibly hint at a problem in the code will be the multitude of nearly identical hardcoded statements. This anomaly is, however,", + "source": "The Art of SQL.pdf", + "chunk_id": 180 + }, + { + "text": "and will simply do as instructed. If we were to try and check on the database side for a clue indicating wrongly written code, the only thing that may possibly hint at a problem in the code will be the multitude of nearly identical hardcoded statements. This anomaly is, however, only a part of the larger problem. One can write very poor code in any language, from plain old COBOL down to the coolest object-oriented language. But the greater the degree of independence between each layer of software, the better written those layers must each be. The problem here is that a succession of software layers may be called. No matter how skilled the developer who assembles these layers into the overall module, the final performance will be constrained by the weakest layer. The problem of the weakest layer is all the more perverse when you inherit bad libraries—as with inheriting bad genes, there is not much you can do about it. Rewriting inefficient low-level layers is rarely allowed by schedules or budgets. I once learned about a case in which a basic operator in a programming language had been “overloaded” (redefined) and was performing a database access each time it was used by unsuspecting developers! It is all the more complicated to correct such a situation, because it is quite www.it-ebooks.info W E A K N E S S E S A N D S T R E N G T H S 205 likely that individual queries, as seen from the database server, will look like conspicuously plain queries, not like the bad sort of SQL query that scans millions of rows and attracts immediate attention. Cool database access libraries are not necessarily efficient libraries. Distributed Systems Whether you refer to federated systems, a linked server, or a database link, the principle is the same: in distributed queries, you are querying data that is not physically managed inside the server (or database to the Oracle crowd) you are connected to. Distributed queries are executed through complex mechanisms, especially for remote updates, in which transaction integrity has to be preserved. Such complexity comes at a very heavy cost, of which many people are not fully aware. By way of example, I have run a series of tests against an Oracle database, performing massive inserts and selects against a very simple local table, and then creating database links and timing the very same operations with each database link. I have created three different database links: Inter-process A link made by connecting through inter-process communications—typically what one might do to query data located in another database* on the same host. No network was involved. Loop-back A link connecting through TCP, but specifying the loop-back address (127.0.0.1) to limit our foray into the network layers. IP address A link specifying the actual IP address of the machine—but once again without really using a network, so there is no network latency involved. The result of my tests, as it appears in Figure 8-1,", + "source": "The Art of SQL.pdf", + "chunk_id": 181 + }, + { + "text": "specifying the loop-back address (127.0.0.1) to limit our foray into the network layers. IP address A link specifying the actual IP address of the machine—but once again without really using a network, so there is no network latency involved. The result of my tests, as it appears in Figure 8-1, is revealing. In my case, there is indeed a small difference linked to my using inter-process communications or TCP in loop-back or regular mode. But the big performance penalty comes from using a database link in the very first place. With inserts, the database link divides the number of rows inserted per second by five, and with selects it divides the number of rows returned per second by a factor of 2.5 (operating in each case on a row-by-row basis). * Remember that what Oracle calls a database is what is known in most other database systems as a server. www.it-ebooks.info 206 C H A P T E R E I G H T When we have to execute transactions across heterogeneous systems, we have no other choice than to use database links or their equivalent. If we want data integrity, then we need to use mechanisms that preserve data integrity, whatever the cost. There are, however, many cases when having a dedicated server is an architectural choice, typically for some reference data. The performance penalty is quite acceptable for the odd remote reference. It is quite likely that if at connection time some particular credentials are checked against a remote server, nobody will really notice, as long as the remote server is up. If, however, we are massively loading data into a local database and performing some validation check against a remote server for each row loaded locally, then you can be sure to experience extremely slow performance. Validating rows one by one is in itself a bad idea (in a properly designed database, all validation should be performed through integrity constraints): remote checks will be perhaps two or three times slower than the same checks being carried out on the same local server. Distributed queries, involving data from several distinct servers, are also usually painful. First of all, when you send a query to a DBMS kernel, whatever that query is, the master of the game is the optimizer on that kernel. The optimizer will decide how to split the query, to distribute the various parts, to coordinate remote and local activity, and finally to put all the different pieces together. Finding the appropriate path is already a complicated-enough business when everything happens on the local server. We should take note that the notion of “distribution” is more logical than physical: part of the performance penalty comes from the unavailability of remote dictionary information in the local cache. The cost penalty will be considerably higher with two unrelated databases hosted by the same machine than with two databases hosted by two different servers but participating in a common federated database and sharing data dictionary information. FIGURE 8-1. The", + "source": "The Art of SQL.pdf", + "chunk_id": 182 + }, + { + "text": "the unavailability of remote dictionary information in the local cache. The cost penalty will be considerably higher with two unrelated databases hosted by the same machine than with two databases hosted by two different servers but participating in a common federated database and sharing data dictionary information. FIGURE 8-1. The cost of faking being far away www.it-ebooks.info W E A K N E S S E S A N D S T R E N G T H S 207 There is much in common between distributed and parallelized queries (when a query is split into a number of independent chunks that can be run in parallel) with, as you have seen, the additional difficulties of the network layers slowing down significantly some of the operations, and of the unavailability at one place of all dictionary information making the splitting slightly more hazardous. There is also an additional twist here: when sources are heterogeneous—for example when a query involves data coming from an Oracle database as well as data queried from an SQL Server database, all the information the optimizer usually relies on may not be available. Certainly, most products gather the same type of information in order to optimize queries. But for several reasons, they don’t work in a mutually cooperative fashion. First, the precise way each vendor’s optimizer works is a jealously guarded secret. Second, each optimizer evolves from version to version. Finally, the Oracle optimizer will never be able to take full advantage of SQL Server specifics and vice versa. Ultimately, only the greatest common denominator can be meaningfully shared between different product optimizers. Even with homogeneous data sources, the course of action is narrowly limited. As we have seen, fetching one row across a network costs considerably more than when all processes are done locally. The logical inference for the optimizer is that it should not take a path which involves some kind of to and fro switching between two servers, but rather move as much filtering as close to the data as it can. The SQL engine should then either pull or push the resulting data set for the next step of processing. You have already seen in Chapters 4 and 6 that a correlated subquery was a dreadfully bad way to test for existence when there is no other search criterion, as in for instance, the following example: select customer_name from customers where exists (select null from orders, orderdetails where orders.customer_id = customers.customer_id and orderdetails.order_id = orders.order_id and orderdetails.article_id = 'ANVIL023') Every row we scan from customers fires a subquery against orders and orderdetails. It is of course even worse when customers happens to be hosted by one machine and orders and orderdetails by another. In such a case, given the high cost of fetching a single row, the reasonable solution looks like a transformation (in the ideal case, by the optimizer) of the above correlated subquery into an uncorrelated one, to produce the following instead: select customer_name from customers where customer_id in (select", + "source": "The Art of SQL.pdf", + "chunk_id": 183 + }, + { + "text": "another. In such a case, given the high cost of fetching a single row, the reasonable solution looks like a transformation (in the ideal case, by the optimizer) of the above correlated subquery into an uncorrelated one, to produce the following instead: select customer_name from customers where customer_id in (select orders.customer_id www.it-ebooks.info 208 C H A P T E R E I G H T from orders, orderdetails where orderdetails.article_id = 'ANVIL023' and orderdetails.order_id = orders.order_id) Furthermore, the subquery should be run at the remote site. Note that this is also what should be performed even if you write the query as like this: select distinct customer_name from customers, orders, orderdetails where orders.customer_id = customers.customer_id and orderdetails.article_id = 'ANVIL023' and orders.order_id = orderdetails.order_id Now will the optimizer choose to do it properly? This is another question, and it is better not to take the chance. But obviously the introduction of remote data sources narrows the options we have in trying to find the most efficient query. Also, remember that the subquery must be fully executed and all the data returned before the outer query can kick in. Execution times will, so to speak, add up, since no operation can be executed concurrently with another one. The safest way to ensure that joins of two remote tables actually take place at the remote site is probably to create, at this remote site, a view defined as this join and to query the view. For instance, in the previous case, it would be a good idea to define a view vorders as: select orders.customer_id, orderdetails.article_id from orders, orderdetails where orderdetails.order_id = orders.order_id By querying vorders we limit the risks of seeing the DBMS separately fetching data from all the remote tables involved in the query, and then joining everything locally. Needless to say, if in the previous case, customers and orderdetails were located on the same server and orders were located elsewhere, we would indeed be in a very perilous position. The optimizer works well with what it knows well: local data. Extensive interaction with remote data sinks performance. Dynamically Defined Search Criteria One of the most common causes for awful visible performance (as opposed to the common dismal performance of batch programs, which can often be hidden for a while) is the use of dynamically defined search criteria. In practice, such criteria are a www.it-ebooks.info W E A K N E S S E S A N D S T R E N G T H S 209 consequence of the dreaded requirement to “let the user enter the search criteria as well as the sort order via a screen interface.” The usual symptoms displayed by this type of application is that many queries perform reasonably well, but that unfortunately from time to time a query that seems to be almost the same as a well-performing query happens to be very, very slow. And of course the problem is difficult to fix, since everything is so dynamic. Dynamic-search applications are often", + "source": "The Art of SQL.pdf", + "chunk_id": 184 + }, + { + "text": "that many queries perform reasonably well, but that unfortunately from time to time a query that seems to be almost the same as a well-performing query happens to be very, very slow. And of course the problem is difficult to fix, since everything is so dynamic. Dynamic-search applications are often designed as a two-step drill-down query, as in Figure 8-2. Basically, a first screen is displayed to the user with a large choice of criteria and an array of possible conditions such as exclude or date between ... and .... These criteria are used to dynamically build a query that returns a list with some identifier and description, from which you can view all the associated details by selecting one particular item in the list. When the same columns from the same tables are queried with varying search criteria, the key to success usually lays in a clever generation of SQL queries by the program that accesses the database. I am going to illustrate my point in detail with a very simple example, a movie database, and we shall only be concerned with returning a list of movie titles that satisfy a number of criteria. The environment used in this example is a widely popular combination, namely PHP and MySQL. Needless to say, the techniques shown in this chapter are in no way specific to PHP or to MySQL—or to movie databases. Designing a Simple Movie Database and the Main Query Our central table will be something such as the following: Table MOVIES movie_id int(10) (auto-increment) movie_title varchar(50) movie_country char(2) movie_year year(4) movie_category int(10) movie_summary varchar(250) FIGURE 8-2. A typical multi-criteria search www.it-ebooks.info 210 C H A P T E R E I G H T We certainly need a categories table (referenced by a foreign key on movie_category) to hold the different genres, such as Action, Drama, Comedy, Musical, and so forth. It can be argued that some movies sometimes span several categories, and a better design would involve an additional table representing a many-to-many relationship (meaning that one genre can be associated with several movies and that each movie can be associated with several genres as well), but for the sake of simplicity we shall admit that a single, main genre is enough for our needs in this example. Do we need one table for actors and another for directors? Creating two tables would be a design mistake, because it is quite common to see actors-turned-directors, and there is no need to duplicate personal information. From time to time one even finds a movie directed by one of the lead actors. We therefore need three more tables: people to store information such as name, first name, sex, year of birth, and so on; roles to define how people may contribute to a movie (actor, director, but also composer, director of photography, and the like); and movie_ credits to state who was doing what in which movie. Figure 8-3 shows our complete movie schema. Let’s suppose now that we", + "source": "The Art of SQL.pdf", + "chunk_id": 185 + }, + { + "text": "of birth, and so on; roles to define how people may contribute to a movie (actor, director, but also composer, director of photography, and the like); and movie_ credits to state who was doing what in which movie. Figure 8-3 shows our complete movie schema. Let’s suppose now that we want to let people search movies in our database by specifying either: words from the title, the name of the director, or up to three names of any of the actors. Following is the source of our prototype page, which I have built in HTML to act as our screen display: Movie Database FIGURE 8-3. The movie database schema www.it-ebooks.info W E A K N E S S E S A N D S T R E N G T H S 211

Please fill the form to query our database and click on Search when you are done...

Movie Title :
Director :
Actor :
Actor :
Actor :
This prototype page shows on screen as in Figure 8-4. First, let me make a few remarks: • Although we want to store the first and last names separately in our database (defi- nitely more convenient if we want to generate a listing ordered by last name), we don’t want our entry form to look like a passport renewal form: we just want a single entry field for each individual. • We want our query input values to be case-insensitive. www.it-ebooks.info 212 C H A P T E R E I G H T Certainly the thing not to do is to generate a query containing a criterion such as: and upper() = concat(upper(people_firstname), ' ', upper(people_name)) As shown in Chapter 3, the right part of the equality in such a criterion would prevent us from using any regular index we might have logically created on the name. Several products allow the creation of functional indexes and index the result of expressions, but the simplest and therefore best solution is probably as follows: 1. Systematically store in uppercase any character column that is likely to be queried (we can always write a function to beautify it before output). 2. Split the entry field into first name and (last) name before passing it to the query. The first point simply means inserting upper(string) instead of string, which is easy enough. Keep the second point in mind for the time being: I’ll come back to it in just a bit. If users were to fill all entry fields, all the time, then our resulting main query could be something such as: select movie_title, movie_year from movies inner join movie_credits mc1 on mc1.movie_id =", + "source": "The Art of SQL.pdf", + "chunk_id": 186 + }, + { + "text": "second point in mind for the time being: I’ll come back to it in just a bit. If users were to fill all entry fields, all the time, then our resulting main query could be something such as: select movie_title, movie_year from movies inner join movie_credits mc1 on mc1.movie_id = movies.movie_id inner join people actor1 on mc1.people_id = actor1.people_id inner join roles actor_role on mc1.role_id = actor_role.role_id and mc2.role_id = actor_role.role_id and mc3.role_id = actor_role.role_id FIGURE 8-4. The movie database search screen www.it-ebooks.info W E A K N E S S E S A N D S T R E N G T H S 213 inner join movie_credits mc2 on mc2.movie_id = movies.movie_id inner join people actor2 on mc2.people_id = actor2.people_id inner join movie_credits mc3 on mc3.movie_id = movies.movie_id inner join people actor3 on mc3.people_id = actor3.people_id inner join movie_credits mc4 on mc4.movie_id = movies.movie_id inner join people director on mc4.people_id = director.people_id inner join roles director_role on mc4.role_id = director_role.role_id where actor_role.role_name = 'ACTOR' and director_role.role_name = 'DIRECTOR' and movies.movie_title like 'CHARULATA%' and actor1.people_firstname = 'SOUMITRA' and actor1.people_name = 'CHATTERJEE' and actor2.people_firstname = 'MADHABI' and actor2.people_name = 'MUKHERJEE' and actor3.people_firstname = 'SAILEN' and actor3.people_name = 'MUKHERJEE' and director.people_name = 'RAY' and director.people_firstname = 'SATYAJIT' Unfortunately, will somebody who can name the title, director and the three main actors of a film (most typically a movie buff) really need to use our database? This is very unlikely. The most likely search will probably be when a single field or possibly two, at most, will be populated. We must therefore anticipate blank fields, asking the question: what will we do when no value is passed? A common way of coding one’s way out of a problematic situation like this is to keep the select list unchanged; then to join together all the tables that may intervene in one way or another, using suitable join conditions; and then to replace the straightforward conditions from the preceding example with a long series of: and column_name = coalesce(?, column_name) where ? will be associated with the value from an entry field, and coalesce( ) is the function that returns the first one of its arguments that is non null. If a value is provided, then a filter is applied; otherwise, all values in the column pass the test. All values? Not really; if a column contains a NULL, the condition for that column will evaluate to false. We cannot say that something we don’t know is equal to something we don’t know, even if it is the same something (nothing?). If one condition in our long series of conditions linked by and evaluates to false, the query will return nothing, which is certainly not what we want. There is a solution though, which is to write: and coalesce(column_name, constant) = coalesce(?, column_name, constant) www.it-ebooks.info 214 C H A P T E R E I G H T This solution would be absolutely perfect if only it did not mean forfeiting the use of", + "source": "The Art of SQL.pdf", + "chunk_id": 187 + }, + { + "text": "not what we want. There is a solution though, which is to write: and coalesce(column_name, constant) = coalesce(?, column_name, constant) www.it-ebooks.info 214 C H A P T E R E I G H T This solution would be absolutely perfect if only it did not mean forfeiting the use of any index on column_name when a parameter is specified. Must we sacrifice the correctness of results to performance, or performance to the correctness of results? The latter solution is probably preferable, but unfortunately both of them might also mean sacrificing our job, a rather unpleasant prospect. A query that works in all cases, whatever happens, is quite difficult to write. The commonly adopted solution is to build such a query dynamically. What we can do in this example scenario is to store in a string everything up to the where and the fixed conditions on role names, and then to concatenate to this string the conditions which have been input by our program user—and only those conditions. A variable number of search criteria calls for dynamically built queries. Assuming that a user searched our database for movies starring Amitabh Bachchan, the resulting, dynamically written query might be something like the following: select distinct movie_title, movie_year from movies inner join movie_credits mc1 on mc1.movie_id = movies.movie_id inner join people actor1 on mc1.people_id = actor1.people_id inner join roles actor_role on mc1.role_id = actor_role.role_id and mc2.role_id = actor_role.role_id and mc3.role_id = actor_role.role_id inner join movie_credits mc2 on mc2.movie_id = movies.movie_id inner join people actor2 on mc2.people_id = actor2.people_id inner join movie_credits mc3 on mc3.movie_id = movies.movie_id inner join people actor3 on mc3.people_id = actor3.people_id inner join movie_credits mc4 on mc4.movie_id = movies.movie_id inner join people director on mc4.people_id = director.people_id inner join roles director_role on mc4.role_id = director_role.role_id where actor_role.role_name = 'ACTOR' and director_role.role_name = 'DIRECTOR' and actor1.people_firstname = 'AMITABH' and actor1.people_name = 'BACHCHAN' order by movie_title, movie_year www.it-ebooks.info W E A K N E S S E S A N D S T R E N G T H S 215 First, let me make two remarks: • We have to make our select a select distinct. We do this because we keep the joins without any additional condition. Otherwise, as many rows would be returned for each movie as we have actors and directors recorded for the movie. • It is very tempting when building the query to concatenate the values that we receive to the SQL text under construction proper, thus obtaining a query exactly as above. This is not, in fact, what we should do. I have already mentioned the subject of bind variables; it is now time to explain how they work. The proper course is indeed to build the query with placeholders such as ? (it depends on the language), and then to call a special func- tion to bind the actual values to the placeholders. It may seem more work for the devel- oper, but in fact it will mean less work for the DBMS engine.", + "source": "The Art of SQL.pdf", + "chunk_id": 188 + }, + { + "text": "build the query with placeholders such as ? (it depends on the language), and then to call a special func- tion to bind the actual values to the placeholders. It may seem more work for the devel- oper, but in fact it will mean less work for the DBMS engine. Even if we rebuild the query each time, the DBMS usually caches the statements it executes as a part of its standard optimization routines. If the SQL engine is given a query to process that it finds in its cache, the DBMS has already parsed the SQL text and the optimizer has already determined the best execution path. If we use placeholders, all queries that are built on the same pattern (such as searches for movies starring one particular actor) will use the same SQL text, irrespective of the actor’s name. All the setup is done, the query can be run immediately, and the end user gets the response faster. Besides performance, there is also a very serious concern associated with dynamically built hardcoded queries, a security concern: such queries present a wide-open door to the technique known as SQL injection. What is SQL injection? Let’s say that we run a commercial operation, and that only subscribers are allowed to query the full database while access to movies older than 1960 is free to everybody. Suppose that a malicious non-subscriber enters into the movie_title field something such as: X' or 1=1 or 'X' like 'X When we simply concatenate entry fields to our query text we shall end up with a condition such as: where movie_title like 'X' or 1=1 or 'X' like 'X%' and movie_year < 1960 which is always true and will obviously filter nothing at all! Concatenating the entry field to the SQL statement means that in practice anybody will be able to download our full database without any subscription. And of course some information is more sensitive than movie databases. Binding variables protects from SQL injection. SQL injection is a very real security matter for anyone running an on-line database, and great care should be taken to protect against its malicious use. When using dynamically built queries, use parameter markers and pass values as bind variables, for both performance and security (SQL injection) reasons. www.it-ebooks.info 216 C H A P T E R E I G H T A query with prepared joins and dynamically concatenated filtering conditions executes very quickly when the tables are properly indexed. But there is nevertheless something that is worrisome. The preceding example query is a very complicated query, particularly when we consider the simplicity of both the output result and of what we provided as input. Right-Sizing Queries In fact, the complexity of the query is just one part of the issue. What happens, in the case of the final query in the preceding section, if we have not recorded the name of the director in our database, or if we know only the names of the two lead actors?", + "source": "The Art of SQL.pdf", + "chunk_id": 189 + }, + { + "text": "the complexity of the query is just one part of the issue. What happens, in the case of the final query in the preceding section, if we have not recorded the name of the director in our database, or if we know only the names of the two lead actors? The query will return no rows. All right, can we not use outer joins then, which return matching values when there is one and NULL when there is none? Using outer joins might be a solution, except that we don’t know what exactly will be queried. What if we only have the name of the director in our database? In fact, we would need outer joins everywhere—and putting them everywhere is often, logically, impossible. We therefore have an interesting case, in which we are annoyed by missing information even if all of our attributes are defined as mandatory and we have absolutely no NULL values in the database, simply because our query so far assumes joins that may be impossible to satisfy. In fact, in the particular case when only one actor name is provided, we need a query no more complicated than the following: select movie_title, movie_year from movies inner join movie_credits mc1 on mc1.movie_id = movies.movie_id inner join people actor1 on mc1.people_id = actor1.people_id inner join roles actor_role on mc1.role_id = actor_role.role_id where actor_role.role_name = 'ACTOR' and actor1.people_firstname = 'AMITABH' and actor1.people_name = 'BACHCHAN' order by movie_title, movie_year This “tight-fit” query assumes nothing about our also knowing the name of the director, nor of a sufficient number of other actors, and hence there is no need for outer joins. Since we have already begun building our query dynamically, why not try to inject a little more intelligence in our building exercise, so as to obtain a query really built to order, exactly tailored to our needs? Our code will no doubt be more complicated. Is the complication worth it? The simple fact that we are now certain to return all the information available when given an actor’s name, even when we don’t know who directed a film, should be reason enough for an unqualified “yes.” But performance reasons also justify taking this step. www.it-ebooks.info W E A K N E S S E S A N D S T R E N G T H S 217 Nothing is as convincing as running a query in a loop a sufficient number of times to show the difference between two approaches: our “tight-fit” query is five times faster than the “one-size-fits-all” query. All other things aside, does it matter if our query executes in 0.001 second instead of 0.005 second? Not much, if our database is only queried now and then. But there may be a day when queries arrive at a rate higher than we can service and keep up with, and then we’ll have a problem. Queries will have to be queued, and the queue length will increase very quickly—as fast as the number of complaints about", + "source": "The Art of SQL.pdf", + "chunk_id": 190 + }, + { + "text": "and then. But there may be a day when queries arrive at a rate higher than we can service and keep up with, and then we’ll have a problem. Queries will have to be queued, and the queue length will increase very quickly—as fast as the number of complaints about poor database performance. Simply put, going five times faster enables five times as many queries to be processed on the same hardware. (We will consider these issues in more detail in Chapter 9.) Matching criteria with dynamically built queries improves performance by minimizing joins, and eliminates the issue of missing values. Wrapping SQL in PHP Let’s first start our PHP page with a smattering of regular HTML before the real PHP code: Query result
prepare($query)) { /* * Bind parameters for markers * * This is the messiest part. * We can have anything between 1 and 9 parameters in all (all strings) */ switch ($paramcnt) { case 1 : $stmt->bind_param(\"s\", $params[0]); break; case ... ... break; www.it-ebooks.info 222 C H A P T E R E I G H T case 9 : $stmt->bind_param(\"sssssssss\", $params[0], $params[1], $params[2], $params[3], $params[4], $params[5], $params[6], $params[7], $params[8]); break; default : break; } Et voilà! We are done and just have to execute the query and display the result: /* execute query */ $stmt->execute( ); /* fetch values */ $stmt->bind_result($mt, $my); while ($row = $stmt->fetch( )) { printf (\"\\n\", $mt, $my); } /* close statement */ $stmt->close( ); } else { printf(\"Error: %s\\n\", $mysqli->sqlstate); } ?>
TitleYear
%s%d
Obviously, the code here is significantly more complicated than if we had tried to have one single query. It may seem surprising, after I have advocated pushing as much work as possible onto the DBMS side, to now find me defending the use of complicated code to build as simple a SQL statement as possible. Doing as much work on the SQL side as possible makes sense when it is work that has to be performed. But joining three times as many tables as are needed in the average query, with some of these useless joins not necessarily being very efficient (especially when they happen to be against complex views) makes no sense at all. By intelligently building the query, we tightly control what is executed in terms of security, correctness of the result, and performance. Any simpler solution bears risks of sacrificing at least one of these aspects. www.it-ebooks.info To summarize, there are at least three mistakes that are very commonly made in queries that take a variable number of search criteria: • First of all, it is quite common to see the values against which the columns of the tables are compared being concatenated with the statement-in-making, thus resulting in a magnificent, totally hardcoded statement. Even where queries are supposed to be absolutely unpredictable, you usually find a few queries that are issued again and again by the users, with only the constants varying. Some constants are susceptible to a high degree of variability (such as entity identifiers, as opposed to date formats or even status codes). It isn’t much work to replace these constants by a parameter marker, the syntax of which", + "source": "The Art of SQL.pdf", + "chunk_id": 194 + }, + { + "text": "again and again by the users, with only the constants varying. Some constants are susceptible to a high degree of variability (such as entity identifiers, as opposed to date formats or even status codes). It isn’t much work to replace these constants by a parameter marker, the syntax of which depends on the language (for instance '?') and then to bind the actual value to this parameter marker. This will result in much less work for the server, which will not need to re-analyze the statement each time it is issued, and in particular will not need to determine each time a best execution plan, that will always be the same. And no user will be able to bypass any additional restriction you may want to add to the query, which means that by binding variables you will plug a seri- ous security issue at the same time. • A second mistake is usually to try to include in the query everything that may matter. It is not because a search criterion may refer to data stored in one table that this table must appear in the from clause. I have already alluded to this issue in the previous chapters, but the from clause should only contain the tables from which we return data, as well as the tables enabling us to join them together. As we have seen in Chapter 6, existence tests should be solved by subqueries—which are no more diffi- cult to generate dynamically than a regular condition in a where clause. • The most important mistake is the one-size-fits-all philosophy. Behind every generic query are usually hidden three or four families of queries. Typically, input data is made up of identifiers, status values, or some ranges of dates. The input values may be strong, efficient criteria, or weak ones, or indeed anything in between (sometimes an additional criterion may reinforce a weak one by narrowing the scope). From here, trying to build several alternate queries in an intelligent fashion, as in the various cases of Chapter 6, is the only sound way out, even if it looks more complicated. More intelligence in the dynamic construction of an SQL statement makes for a more efficient SQL statement. www.it-ebooks.info www.it-ebooks.info Chapter 9fm. C H A P T E R N I N E Multiple Fronts Tackling Concurrency Yet to their General’s Voice they soon obey’d Innumerable. —John Milton (1608–1674) Paradise Lost, Book I www.it-ebooks.info 226 C H A P T E R N I N E W hen we have a lot of sessions running concurrently, all accessing one database, we may encounter difficulties that can remain hidden when running single-user tests. Contention occurs, and locks may be held for unpredictable periods of time. This chapter discusses how to face the situation when users advance in overwhelming numbers. There are several different issues associated with a large number of concurrent users. One of the most obvious is contention when updating (sometimes reading) data and the consequent requirement for", + "source": "The Art of SQL.pdf", + "chunk_id": 195 + }, + { + "text": "held for unpredictable periods of time. This chapter discusses how to face the situation when users advance in overwhelming numbers. There are several different issues associated with a large number of concurrent users. One of the most obvious is contention when updating (sometimes reading) data and the consequent requirement for locks at one level or another. But users are not only fighting for the right to modify bytes at one place in the system without any interference from others; they are also competing for processing power, access to disks, workspace in memory, and network bandwidth. Very often difficulties that are latent with a few users become blatant with many. Increases in the number of users are not always as smooth as one might expect them to be. Sudden increases can come through the meteoritic success of your company, but fast-paced increases more often happen through the gradual deployment of applications—or sometimes as a result of mergers or buyouts. The Database Engine as a Service Provider You might be tempted to consider the DBMS as an intelligent and dedicated servant that rushes to forestall your slightest desire and bring data at the exact time when you need it. Reality is slightly less exalted than the intelligent servant model, and at times a DBMS looks closer to a waiter in a very busy restaurant. If you take your time to choose from the menu, chances are that the waiter will tell you “I’ll let you choose, and I’ll come back later to take your order” before disappearing for a long time. A DBMS is a service provider or, perhaps more precisely, a collection of service providers. The service is simply to execute some operation against the data, fetching it or updating it—and the service may be requested by many concurrent sessions at the same time. It is only when each session queries efficiently that the DBMS can perform efficiently. The Virtues of Indexes Let’s execute some fairly basic tests against a very simple table with three columns. The first two are integer columns (each populated with distinct values from 1 to 50,000), one being declared as the primary key and the second without an index. The third column (named label) is a text column consisting of random strings thirty to fifty characters long. If we generate random numbers between 1 and 50,000 and use these random numbers as query identifiers to return the label column, you might be surprised to discover that on any reasonably powerful machine, the following query: select label from test_table where indexed_column = random value www.it-ebooks.info M U L T I P L E F R O N T S 227 as well as this one: select label from test_table where unindexed_column = random value provide virtually instant results. How is this possible? A query using an unindexed column should be much slower, surely? Actually, a 50,000–row table is rather small, and if it has as few columns as is the case in our example, the number of", + "source": "The Art of SQL.pdf", + "chunk_id": 196 + }, + { + "text": "from test_table where unindexed_column = random value provide virtually instant results. How is this possible? A query using an unindexed column should be much slower, surely? Actually, a 50,000–row table is rather small, and if it has as few columns as is the case in our example, the number of bytes to scan is not that enormous, and a modern machine can perform the full scan very rapidly. We indeed have, on one hand, a primary key index search, and on the other hand, a full-table scan. What’s happening is that the difference between indexed and unindexed access is too small for a human to perceive. To really test the benefit of an index, I have run our queries continuously for one minute, and then I have checked on how many queries I was able to process by unit of time. The result is reassuringly familiar: on the machine on which I ran the test, the query using the indexed column can be performed 5,000 times per second, while the query using the unindexed column can only be performed 25 times per second. A developer running single user tests may not really notice a difference, but there is one, and it is truly massive. Even sub-second response times sometimes hide major performance issues. Don’t trust unitary tests. A Just-So Story Continuing with the example from the preceding section, let’s have a look at what may very well happen in practice. Suppose that instead of being a number, the key of our table happens to be a string of characters. During development, somebody notices that a query has unexpectedly returned the wrong result. A quick investigation shows that the key column contains both uppercase and lowercase characters. Under pressure to make a quick fix, a developer modifies the where clause in the query and applies an upper( ) function to the key column—thus forfeiting the index. The developer runs the query, the correct result set is returned, and anyone other than a native of the planet Krypton cannot possibly notice any significant difference in response time. All appears to be for the best, and we can ship the code to production. Now we have hordes of users, all running our query again, again and again. Chapter 2 makes the point that in our programs we should not execute queries inside loops, whether they are cursor loops or the more traditional for or while constructs. Sadly, we very often find queries nested inside loops on the result set of other queries, and as a result, our query can be run at a pretty high rate, even without having tens of thousands www.it-ebooks.info 228 C H A P T E R N I N E of concurrent users. Let’s see now what happens to our test table when we run the query at a high rate, with a set number of executions per unit of time, occurring at random intervals. When we execute our query at the relatively low rate of 500 per", + "source": "The Art of SQL.pdf", + "chunk_id": 197 + }, + { + "text": "N E of concurrent users. Let’s see now what happens to our test table when we run the query at a high rate, with a set number of executions per unit of time, occurring at random intervals. When we execute our query at the relatively low rate of 500 per minute, everything appears normal whether we use the index or not, as you can see in Figure 9-1. All our queries complete in under 0.2 seconds, and nobody will complain. We actually have to increase our execution rate 10 times, to a relatively high rate of 5,000 executions per minute, to notice in Figure 9-2 that we may occasionally have a slow response when we use the unindexed column as key. This, however, affects only a very low percentage of our queries. In fact, 97% of them perform in 0.3 seconds or less. But at 5,000 queries per minute, we are unaware that we are tottering on the brink of catastrophe. If we push the rate up to a very high 10,000 executions per minute, you can see in Figure 9-3 that a very significant proportion of the queries will execute noticeably more slowly, some taking as long as 4 seconds to complete. If in another test we run the queries that use the index at the same high rate, all queries execute imperturbably in 0.1 seconds or less. Of course, when some queries that used to run fast start to take much longer, users are going to complain; and other users who, unprompted, would otherwise have noticed nothing will probably grumble as well, out of sympathy. The database is slow—can’t it be tuned? Database administrators and system engineers will tweak parameters, gaining a few weeks of relief, until the evidence will finally impose itself, in all its glorious simplicity: we need a more powerful server. FIGURE 9-1. Response time of a simple query against a 50,000–row table, low query rate www.it-ebooks.info M U L T I P L E F R O N T S 229 FIGURE 9-2. Response time of a simple query against a 50,000–row table, high query rate FIGURE 9-3. Response time of a simple query against a 50,000–row table, very high query rate www.it-ebooks.info 230 C H A P T E R N I N E An increasing load may not cause performance problems, but may actually reveal them, suggesting program improvements as an alternative to upgrading the hardware. Get in Line One can take a fairly realistic view of a DBMS engine by imagining it to be like a post office staffed by a number of clerks serving customers with a wide array of requests—our queries. Obviously, a very big post office will have many counters open at the same time and will be able to serve several customers all at the same time. We may also imagine that young hypercaffeinated clerks will work faster than older, sedate, herbal-tea types. But we all know that what will make the biggest difference, especially at peak", + "source": "The Art of SQL.pdf", + "chunk_id": 198 + }, + { + "text": "many counters open at the same time and will be able to serve several customers all at the same time. We may also imagine that young hypercaffeinated clerks will work faster than older, sedate, herbal-tea types. But we all know that what will make the biggest difference, especially at peak hours, is the requests actually presented by each customer. These will vary between the individual who has prepared the exact change to buy a stamp book and the one who inquires at length about the various rates at which to send a parcel to a remote country, involving the completion of customs forms, and so on. What is most irritating is of course when someone with a mildly complicated request spends several minutes looking for a purse when the moment for payment arrives. But fortunately, in post offices, you never encounter the case that is so frequent in real database applications: the man with 20 letters who joins the queue on 20 separate successive occasions, buying only one stamp in each visit to the counter. It is important to understand that there are two components that determine how quickly one is served at the counter: • The performance of, in our example, the clerk. In the case of a database application, this equates to a combination of database engine, hardware, and I/O subsystems. • The degree of complexity of the request itself, and to a large extent how the request is presented, its lucidity and clarity, such that the clerk can easily understand the request, and accordingly make a quick and complete answer. In the database world, the first component is the domain of system engineers and database administrators. The second component belongs squarely within the business requirements and development arena. The more complicated the overall system, the more important becomes the collaboration between the different parties involved when you want to get the best out of your hardware and software investment. With the post-office image in mind, we can understand what happened in our query test. What matters is the ratio of the number of customers arriving (e.g., the rate of execution of queries), to the average time required to answer the query. As long as the rate of arrival is low enough to enable everyone to find a free counter, nobody will complain. However, as soon as customers arrive faster than they can be serviced, queues will start to lengthen, just as much for the fast queries as for the slow ones. www.it-ebooks.info M U L T I P L E F R O N T S 231 There is a threshold effect, very similar to what one of Charles Dickens’s characters says in David Copperfield: Annual income twenty pounds, annual expenditure nineteen six, result happiness. Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery. This can easily be demonstrated by running our two queries simultaneously, the one using the indexed column and the other using the unindexed column, at a rate of 5,000", + "source": "The Art of SQL.pdf", + "chunk_id": 199 + }, + { + "text": "twenty pounds, annual expenditure nineteen six, result happiness. Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery. This can easily be demonstrated by running our two queries simultaneously, the one using the indexed column and the other using the unindexed column, at a rate of 5,000 times per second. The compound result of Figure 9-4 is noticeably different from Figure 9-2, in which results were shown for the two queries running separately, not concurrently. As appears clearly from Figure 9-4, the performance of the fast query has deteriorated because of the simultaneous presence of slow queries. System performance crashes when statements arrive faster than they can be serviced; all queries are affected, not only slow ones. Concurrent Data Changes When you change data, the task of maintaining a good level of performance becomes even more difficult as the level of activity increases. For one thing, any change is by essence a more costly operation than a mere query, since it involves both getting the data and then writing it back to the database. In the case of inserts, only the latter operation FIGURE 9-4. Fast and slower queries running together, both at a high query rate www.it-ebooks.info 232 C H A P T E R N I N E applies. Therefore, data modification, whether updates, deletes, or inserts, intrinsically requires a longer service time than the equivalent query-only task. This longer service time is made worse by one mechanism and one situation that are often confused. The mechanism is locking, and the situation is contention. Locking When several users want to modify the same data at once—for instance to book the very last seat on a flight—the only solution available to the DBMS is to block all but one user, who is usually the first person to present the request. The necessity of sequentializing access to critical resources is a problem that is as old as multiuser systems themselves. It existed with files and records long before database systems began to be adopted. One user acquires a lock over a resource, and the other users who also want to lock the same resource either have to queue up, waiting patiently for the lock to be released, or handle the error code that they will receive. In many ways, the situation is entirely analogous to our fictitious post office when several customers require the use of a single photocopier— people must wait patiently for their turn (or turn away and come back later). Locking granularity One of the most important practical questions to address when attempting to change the contents of the database will be to determine exactly where the locks will be applied. Locks can impact any or all of the following: • The entire database • The physical subset of the database where the table is stored • The table identified for modification • The particular block or page (unit of storage) containing the target data • The table row containing the affected data •", + "source": "The Art of SQL.pdf", + "chunk_id": 200 + }, + { + "text": "any or all of the following: • The entire database • The physical subset of the database where the table is stored • The table identified for modification • The particular block or page (unit of storage) containing the target data • The table row containing the affected data • The column(s) in the row As you can see, how much users interfere with each other is a question that relates to the granularity of the locking procedures. The type of locking that can be applied varies with the DBMS. Locking granularity is an area where “big products,” designed for large information systems, are significantly different from “small products” that have more limited ambitions. When locks apply to a restricted amount of data, several concurrent processes can happily change data in the very same table at the same time without much affecting each other. Instead of having to wait until another process has finished with its transaction to get ahold of a lock, there can be some overlap between the various processes—which means that from a hardware point of view you can have more processors working, thus making www.it-ebooks.info M U L T I P L E F R O N T S 233 better use of your hardware resources. The benefit of a finer granularity can be seen quite clearly in Figure 9-5, which shows the contrast in the total throughput of a number of concurrent sessions updating a table, first in table-locking mode and then second in row- locking mode. In each case the DBMS server is the same one. In the table-locking case, throughput increases slightly with two sessions, because the server, in the sense of service provider, is not saturated. But two sessions generate the maximum number of updates we can sequentially execute per unit of time, and from then on the curve is flat—actually, very slowly decreasing because system resources are required to handle more sessions, and this is detrimental to the system resources required to perform the updates. The situation of table locking can be contrasted with the situation of row locking, where changes applied to the same table can occur simultaneously as long as they do not affect identical rows. As in the case of table locking we will eventually reach a point where we saturate the server, but this point is reached both later and for a much higher number of concurrent updates. If your DBMS is rather heavy-handed when locking resources, your only hope to cope with a sudden increase of activity, optimistically assuming that everything else has been tried, is to buy better hardware. “Better hardware” must of course be qualified. If locking is the bottleneck, more processors will not help, because the critical resource is access to the data. However, faster processors may speed up execution, reduce the time locks are actually held, and therefore allow the processing of more changes per unit of time. Processing still remains strictly sequential, of course, and the same number of locks is", + "source": "The Art of SQL.pdf", + "chunk_id": 201 + }, + { + "text": "help, because the critical resource is access to the data. However, faster processors may speed up execution, reduce the time locks are actually held, and therefore allow the processing of more changes per unit of time. Processing still remains strictly sequential, of course, and the same number of locks is still applied. FIGURE 9-5. Update performance for table versus row locking www.it-ebooks.info 234 C H A P T E R N I N E Lock handling Locking mechanisms are an integral part of the implementation of a DBMS and there is not much that we can do about them. We are limited to just two directions in dealing with locks: Try not to lock tables in a haphazard way. It goes without saying that we should not run a program that massively updates rows in a table by the million at the same time as many users are trying to exe- cute very short update transactions against the same table. Try to hold locks for as short a time as possible. When we are in a situation where several users are concurrently attempting to access a resource that cannot be shared, speed matters not to one, but to all transactions. There is little benefit in running a fast update that has to wait for a slow one to release a lock before it can do its work. Everything must be fast, or else everything will be slow: “A chain is only as strong as its weakest link.” The overwhelming majority of update and delete statements contain a where clause, and so any rewrite of the where clause that speeds up a select statement will have the same effect on a data manipulation statement with the very same where clause. If a delete statement has no where clause (in other words the entire table is being deleted!), then it is likely that we would be better off using a truncate statement, which empties a table (or a partition) much more efficiently. We mustn’t forget, though, that indexes also have to be maintained, and that updating an indexed column is costly; we may have to arbitrate between the speed of fetches and the speed of changes. The index that might be helpful in the where clause may prove to be a nuisance when rows are changed. Concerning insert statements, a number of them may actually be insert...select constructs in which the link between select performance and insert performance is naturally obvious. You’ve seen in Chapter 2 that impeccable statement performance doesn’t necessarily rhyme with good program performance. When changing data, we have a particular scope to consider: the transaction or, in other words, the duration of a logical unit of work. We shall have to retain locks on a particular part of the database for most if not all of the transaction. Everything that need not be done within the transaction, especially if it is a slow activity, should be excluded from that transaction. The start of a transaction may sometimes be", + "source": "The Art of SQL.pdf", + "chunk_id": 202 + }, + { + "text": "shall have to retain locks on a particular part of the database for most if not all of the transaction. Everything that need not be done within the transaction, especially if it is a slow activity, should be excluded from that transaction. The start of a transaction may sometimes be implicit with the first data manipulation language (DML) statement issued. The end of a transaction is always obvious, as it is marked by a commit or rollback statement. With this background, some practices are just common sense. Inside a transaction: www.it-ebooks.info M U L T I P L E F R O N T S 235 • Avoid looping on SQL statements as much as possible. • Keep round-trips between the program and the database, whether running as an application server or as a mere client, to a minimum, since these add network latency to the overall elapsed time. • Exploit to the full whatever mechanisms the DBMS offers to minimize the number of round-trips (e.g., take advantage of stored procedures or array fetching). • Keep any nonessential SQL statements that are not strictly necessary within the logi- cal unit of work outside of it. For instance, it is quite common to fetch error messages from a table, especially in localized applications. If we encounter an error, we should end our transaction with a rollback first, and then query the error message table, not the reverse: doing so will release locks earlier, and therefore help to maximize throughput. As simple a transaction as one that inserts a new row in both a master table and a slave table provides ample ground for mistakes. An example for this type of transaction is typically the creation of a new customer order (in the master table) and of the first item in our shopping basket (held in an order_detail table). The difficulty usually comes as a result of using system-generated identifiers for the orders. The primary mistake to avoid is to store into a table the “next value to use.” Such a table is mercilessly locked by all concurrent processes updating it, thus becoming the major bottleneck in the whole application. Depending on the DBMS you are using, a system- generated identifier is either the value of an auto-incremented column, which will take for each new row inserted the value of the previous row plus one, or the next value of a database object such as a sequence, which is in essence very similar to an auto- incremented column but without the explicit reference to a column in an existing table. We have nothing to do to generate a new identifier for each new order other than to grab the value generated by the system. The snag is that we must know this value to be able to link the items in the basket to a particular order. In other words, we have to insert this value into table order_detail as well as the master table. Some DBMS products that use auto-incremented columns", + "source": "The Art of SQL.pdf", + "chunk_id": 203 + }, + { + "text": "the system. The snag is that we must know this value to be able to link the items in the basket to a particular order. In other words, we have to insert this value into table order_detail as well as the master table. Some DBMS products that use auto-incremented columns provide either a system variable (as @@IDENTITY with Transact-SQL), or a function (such as MySQL’s last_insert_ id( )) to retrieve the value that was last generated by the session. Fail to use facilities provided by your DBMS, and you are condemned to run useless queries to perform the same task in the middle of a transaction, thus wasting resources and slowing down your transaction. Using functions or variables referring implicitly to the last generated value requires a little discipline in executing statements in the proper order, particularly if one is juggling several auto-incremented columns simultaneously. www.it-ebooks.info 236 C H A P T E R N I N E For some unknown reason, there is a marked tendency among developers who are using sequences to first issue a .nextval call to the database to get a new value, and then to store it in a program variable for future reference. There is actually a .currval call (or previous value for with DB2), and as its name implies its purpose is to return the last value that was generated for the given sequence. In most cases, there is no need to use a program variable to store the current value, and even less to precede true action with a special get a new sequence value call. In the worst case, some DBMS extensions can prove useful. For instance, Oracle (and PL/SQL) users can use the returning ... into ... clause of insert and update statements to return system-generated values without requiring a new round-trip to the server. Running one special statement to get the next sequence value and adding one more round-trip to the database generates overhead that can globally amount to a very significant percentage for simple and often executed transactions. Where transactional activity is high, it is vital that locks are never held for operations that don’t require them. Locking and committing If we try and hold locks for the minimum possible time, we are bound to have to make frequent commits. Committing is a very costly process, since it means writing to persistent memory (journal files), and therefore initiating physical I/O operations. If we commit changes after absolutely every logical unit of work, we add a lot of overhead as can be seen in Figure 9-6. The figure shows the performance impact of committing every 1, 2, 3...12 rows in the case of a very fast update executed by a single user process running on an empty test machine. Depending on the statement and the number of rows affected, figures may of course vary but the trend is always the same. If a batch update program commits every transaction, it can easily take two to", + "source": "The Art of SQL.pdf", + "chunk_id": 204 + }, + { + "text": "fast update executed by a single user process running on an empty test machine. Depending on the statement and the number of rows affected, figures may of course vary but the trend is always the same. If a batch update program commits every transaction, it can easily take two to three times as long to complete as when it commits less frequently. In the case of batch programs in which concurrency control is not an issue, it is advisable to avoid committing changes too often. The snag with not committing zillions of changes, besides the impact of holding the inevitable locks, is that the system has to record the pre-change data image for a hypothetical undo operation, which will put some serious strain on resources. If the process fails for any reason, rolling back the changes may take a considerable amount of time. There are two schools of thought on this topic. One favors committing changes at regular intervals so as to moderate demands on the system in terms of resources, as well as reduce the amount of work which might have to be done in case of a database change failure. The other school is frankly more gung ho and argues, www.it-ebooks.info M U L T I P L E F R O N T S 237 not without some reason, that system resources are here to sustain business processes, not the other way around. For the disciples of this school, if higher throughput can be achieved by less frequent commits—and if they can afford the occasional failure and still have processes completed properly and faster—then there is benefit in less frequent commits. Their case is further strengthened if the DBMS features some “pass or break” mode that shuns the generation of undo data. The commit-once-when-we’re-done approach implicitly assumes that redoing everything from scratch when something has totally failed is often simpler than trying to fix something that only partially worked. Both schemes have advantages and disadvantages, and the final choice may often be linked to operational constraints—or perhaps even to politics. In any case, a batch program committing once in a while may block interactive users. Likewise, it is possible for interactive users to block batch programs. Even when the locking granularity is at the row level, a mechanism such as lock escalation that is applied by some database systems (in which many fine-grain locks are automatically replaced by a coarser-grain lock) may lead to a hung system. Even without lock escalation, a single uncommitted change may block a massive update. One thing is clear: concurrency and batch programs are not a happy match, and we must think about our transactions in a different way according to whether they are interactive or batch. The greater the number of concurrent users, the shorter should be the commit intervals. FIGURE 9-6. Impact of committing on performance www.it-ebooks.info 238 C H A P T E R N I N E Locking and scalability When comparing table and row locking, you have", + "source": "The Art of SQL.pdf", + "chunk_id": 205 + }, + { + "text": "they are interactive or batch. The greater the number of concurrent users, the shorter should be the commit intervals. FIGURE 9-6. Impact of committing on performance www.it-ebooks.info 238 C H A P T E R N I N E Locking and scalability When comparing table and row locking, you have seen that the latter facilitates a much better throughput. However, just as with table locking, the performance curve quickly reaches its ceiling (the point at which performance refuses to improve), and from then on the curve is rather flat. Do all products behave in the same way? As a matter of fact, they don’t, as Figure 9-7 shows. To really compare how the various systems were behaving under increased concurrency, irrespective of speed on a particular example, I performed two series of updates against a large table: first, fast updates with a condition on the primary key, and second, slow updates with a condition on an unindexed column. These updates were repeated with a varying number of sessions, and the total number of updates performed was recorded each time. None of the products displays a strong dependency of throughput on the number of sessions with fast updates, probably because of a saturation of hardware resources. What is interesting, though, is checking whether there is a benefit attached to running a larger number of sessions in parallel. Can increased concurrency somewhat compensate for speed? This is exactly the same type of question as asking “is it better to have a server machine with few fast processors or a higher number of slower processors?” FIGURE 9-7. Row locking and concurrency with three database systems www.it-ebooks.info M U L T I P L E F R O N T S 239 Figure 9-7 shows how the ratio of the number of slow updates to the number of fast updates evolves as we increase the number of sessions. The DBMS1 product stands out for two reasons: • Slow updates are not that slow relatively to fast ones (hence a higher ratio than the other products). • As the steep decrease between 1 and 3 concurrent sessions shows, slow updates also suffer relatively more of increased concurrency. The product to watch, though, is not DBMS1. Even if row locking were in use in all systems, we see that one of the products, DBMS3 on the figure, will scale much better than the others, because the ratio slowly but significantly improves as more and more concurrent sessions enter the fray. This observation may have a significant impact on hardware choices and architectures; products such as DBMS1 and DBMS2 would probably get the most benefit from faster processors, not more numerous ones. From a software point of view, they would also benefit from query pooling on a small number of sessions. On the other hand, a product such as DBMS3 would better profit from additional processors at the same speed and, to some extent, from a higher number of concurrent sessions. How can I explain such differences", + "source": "The Art of SQL.pdf", + "chunk_id": 206 + }, + { + "text": "view, they would also benefit from query pooling on a small number of sessions. On the other hand, a product such as DBMS3 would better profit from additional processors at the same speed and, to some extent, from a higher number of concurrent sessions. How can I explain such differences between DBMS3 and the other products? Mostly by two factors: Saturation of hardware resources This probably partly explains what occurs in the case of DBMS1, which achieves excellent overall results in terms of global throughput, but that simply cannot do better on this particular hardware. Contention Remember that we have the same locking granularity in all three cases (row- level locks). Exactly the same statements where executed and committed. There is in fact more than data locking that limits the amount of work that several ses- sions can perform in parallel. To take a mechanical analogy, we could say that there is more friction in the case of DBMS1 and DBMS2 than in the case of DBMS3. This friction can also be called contention. Concurrency depends on integrity protection mechanisms that include locking as well as other controls that vary from product to product. www.it-ebooks.info 240 C H A P T E R N I N E Contention Rows in tables are not the only resources that cannot be shared. For instance, when one updates a value, the prior or original value (undo data) must be saved somewhere in case the user decides to roll back the change. On a loaded system, there may actually be some kind of competition between two or more processes trying to write undo data into the same physical location, even if these processes are operating on totally unrelated rows in different tables. Such a situation requires some kind of serialization to control events. Likewise, when changes are committed and written to transaction logfiles or in-memory buffers before being flushed to a file, there must be some means of preventing processes from overwriting each other’s bytes. The examples I’ve just given are examples of contention. More than contention, locking is a mechanism that tends to be a defining characteristic of particular DBMS architectures, leaving us little choice other than to try and keep to an absolute minimum the time that the lock is held. Contention, however, is linked to low-level implementation, and there are several actions that can be undertaken to tune contention. Some of these actions can be performed by systems engineers, for example by carefully locating transaction log files on disks. Database administrators can also help to improve the situation by playing with database parameters and storage options. Finally we can, as developers, address these problems in the way we build our applications. To show how we can try to code so as to limit contention, I shall walk you through a case in which contention is usually at its most visible: during multiple, concurrent inserts. Insertion and contention Let’s take as an example a 14–column table with two unique indexes. The primary", + "source": "The Art of SQL.pdf", + "chunk_id": 207 + }, + { + "text": "To show how we can try to code so as to limit contention, I shall walk you through a case in which contention is usually at its most visible: during multiple, concurrent inserts. Insertion and contention Let’s take as an example a 14–column table with two unique indexes. The primary key constraint is defined on a system-generated number (a surrogate key), and a unique constraint (enforced by a unique index of course) is applied to a “natural” compound key, the combination of some short string of characters and a datetime value. We can now proceed to run a series of insert operations for an increasing number of simultaneous sessions. As Figure 9-8 shows, although we are operating in row-locking mode, adding more processes inserting in parallel doesn’t do much to improve the number of rows inserted by unit time. The figure displays the median and the minimum and maximum values for 10 one-minute runs for each of the different numbers of concurrent processes. As you can see, there is much variability in the results—but the best result is obtained for four concurrent processes (which, by some happy coincidence, is not totally unrelated to the number of processors on the machine). Must we conclude that we are saturating the hardware resources? The answer of course is yes, but the real question is “can’t we make better use of these resources?” There is, in a www.it-ebooks.info M U L T I P L E F R O N T S 241 case like this, not much we can do about locking, because we never have two processes trying to access the same row. However, we do have contention when trying to access the data containers. In this situation contention can occur at two places within the database: in the table and in the index. There may be other contention issues at the system level, but these often derive from choices made at the database level. Contention consumes CPU, because there is the execution of code that is required for handling that very same contention issue, with possibly some active waits involved or idle loops while waiting for a resource held by a process running on another processor to be released. Can we try to lower contention and divert some of the CPU cycles to our inserts proper? I have run my example to generate Figure 9-8 on Oracle, one of the database systems that provides the widest range of possible options to try to limit contention. Basically, database-centric solutions to a contention issue will fall into one or more of the following three categories: • DBA solutions • Architectural solutions • Development solutions The following sections review each of these categories. DBA solutions A database administrator often has scant knowledge of business processes. What we call DBA solutions are changes that are applied to the containers themselves. They are application-neutral (as it is near impossible to be absolutely application-neutral, it would probably be more exact to say that the impact is minimal", + "source": "The Art of SQL.pdf", + "chunk_id": 208 + }, + { + "text": "solutions A database administrator often has scant knowledge of business processes. What we call DBA solutions are changes that are applied to the containers themselves. They are application-neutral (as it is near impossible to be absolutely application-neutral, it would probably be more exact to say that the impact is minimal on processes other than the insertion process we are trying to improve). FIGURE 9-8. Concurrent sessions inserting into a regular table www.it-ebooks.info 242 C H A P T E R N I N E There are two main zones in which Oracle DBAs can try and improve a contention issue with a minimum of fuss: Transaction space The first weapon is playing with the number of transaction entry slots reserved in the blocks that constitute the actual physical storage of tables and indexes. A transaction entry slot can be understood as the embodiment of a low-level lock. Without going into arcane detail, let me say that competition for these slots usu- ally figures prominently among the reasons for contention when several ses- sions are competing for write access to the same block. A DBA can try to improve the situation by allocating more space for transaction management. The only impact on the rest of the application is that less space will be available for data in table or index blocks; the direct consequence of such a situation is that more blocks will be required to store the same amount of data, and operations such as full scans and, to a much lesser degree, index searches will have to access more blocks. Free lists The second weapon is trying to force insertions to be directed to different blocks, something that can be done if some degree of control is retained on storage management. For each table, Oracle maintains one or several lists of blocks where new rows can be inserted. By default, there is only one list, but if there are several such lists, then insertions are assigned in a round-robin fashion to blocks coming from the various lists. This solution is not as neutral as allocating more space to transaction management; remember that the clustering of data has a significant impact on the performance of queries, and therefore while we may improve insertion performance, we may degrade some other queries. Architectural solutions Architectural solutions are those based on a modification of the physical disposition of data using the facilities of the DBMS. They may have a much more profound impact, to the disadvantage of our other processes. The three most obvious architectural solutions are: Partitioning Range partitioning will of course defeat our purpose if our goal is to spread update activity over the table—unless, for instance, each process is inserting data for one particular month, and we could assign one process to one partition, but this is not the situation in our current example. Hash partitioning, however, might help. If we compute a hash value from our system-generated (sequence) value, successive values will be arbitrarily assigned to different partitions. Unfor-", + "source": "The Art of SQL.pdf", + "chunk_id": 209 + }, + { + "text": "data for one particular month, and we could assign one process to one partition, but this is not the situation in our current example. Hash partitioning, however, might help. If we compute a hash value from our system-generated (sequence) value, successive values will be arbitrarily assigned to different partitions. Unfor- tunately, there are limitations to what we can do to an index used to enforce a constraint, and therefore it’s only contention at the table level that we can hope to improve. Moreover, this is a solution that unclusters data, which may impact on the performance of other queries. www.it-ebooks.info M U L T I P L E F R O N T S 243 Reverse index Chapter 3 shows that reversing the bytes in index keys can disperse the entries of keys that would otherwise have been in close proximity to one another, into unrelated leaves of the index, and that is a good way to minimize index conten- tion (although it will do nothing for table contention). The disadvantage is that using a reverse index will prevent us from performing range scans on the index, which can be a very serious hindrance. Index organized table Organizing our table as an index will allow us to get rid of one of the sources of contention. It will do nothing for the second one by itself, but instead of stum- bling from one point of contention—the table block—to a second point of con- tention—the index block—we will have everybody fighting in one place. Development solutions Development solutions are in the sole hands of the developer and require no change to the physical structure of the database. Here are two examples where the developer can influence matters: Adjusting parallelization The attempt at varying the number of concurrent processes shows clearly that there is a peak at 4 concurrent sessions and that adding more sessions doesn’t help. There is no benefit in assigning 10 people to a task that 4 people can han- dle perfectly well; it makes coordination more complicated, and some simple subtasks are sooner performed than assigned. Figure 9-8 showed that the effect of adding extra sessions beyond a hardware-dependent number is, in the best of cases, worthless. Removing them would put less strain on the system. Not using system-generated values Do we really need sequential values for a surrogate key? This is not always the case. Sequential values are of interest if we want to process ranges of values, because they allow us to use operators such as > or between. But if all we need is a unique identifier that can be used as a foreign key value in some other tables, why should it belong to a particular range? Let’s consider a possible alterna- tive—namely to simply use a random number—and regenerate a new one if we hit a value we have already used. Results Figure 9-9 shows the insertion rates we obtained with 10 concurrent sessions, using each of the methods just described. Once again", + "source": "The Art of SQL.pdf", + "chunk_id": 210 + }, + { + "text": "a particular range? Let’s consider a possible alterna- tive—namely to simply use a random number—and regenerate a new one if we hit a value we have already used. Results Figure 9-9 shows the insertion rates we obtained with 10 concurrent sessions, using each of the methods just described. Once again there is a significant variability of results (each test was run 10 times, as before). We cannot conclude that a technique that works well in this case will behave as well in any other one, nor, conversely, that a technique which gives disappointing results here will not one day surpass all expectations. But the result is nevertheless interesting. www.it-ebooks.info 244 C H A P T E R N I N E First, the DBA techniques gave results that were positive, but not particularly remarkable. Architectural choices are, in this example, rather inefficient. It is worth mentioning that our two indexes are enforcing constraints, a situation that limits the number of options applicable to them. Therefore, some of the techniques may improve contention at the table level when most of the contention occurs within indexes. This is typically the case with the index organized table, in which table contention is eliminated by the simple expedient of removing the table; unfortunately, because we now have more data to store inside the index, index contention increases and offsets the benefit of no longer having the table. This is also a situation in which we find that the system resource most in demand happens to be the CPU. This situation puts at a disadvantage all the techniques that use extra CPU—such as computing hash values or reversing index keys. Finally, random values provided both the worst and the best results. In the worst case, the (integer) random value was generated between 1 and a number equal to about twice the number of rows we were expecting to insert during the test. As a result, a significant number of values were generated more than once, causing primary key constraint violation and the necessity to generate a new random number. This was of course a waste of time, resulting in excessive consumption of resources—plus, since violation is detected when inserting the primary key index, and since this index stores the physical address, violation is detected after the row has been inserted into the table, so an operation must then be undone, again at additional cost. In the best case, the random number was generated out of an interval 100 times greater than in the worst case. The improvement is striking. But since having 10 concurrent sessions is no more efficient than having 4 concurrent sessions, what would have been the result with only 4 sessions? Figure 9-10 provides the answer. FIGURE 9-9. Tactics for limiting insert contention www.it-ebooks.info M U L T I P L E F R O N T S 245 Very interestingly, all techniques give significantly better results, even if they rank identically in terms of improved throughput (e.g., their relative performances remain", + "source": "The Art of SQL.pdf", + "chunk_id": 211 + }, + { + "text": "9-10 provides the answer. FIGURE 9-9. Tactics for limiting insert contention www.it-ebooks.info M U L T I P L E F R O N T S 245 Very interestingly, all techniques give significantly better results, even if they rank identically in terms of improved throughput (e.g., their relative performances remain largely similar). The comparison of the results between Figures 9-9 and 9-10 teaches some interesting lessons: • In our case study, the bottleneck is the primary key index. Techniques that should strongly limit contention at the table level (hash partitioning, IOT) bring no benefit; actually, the IOT provides worse performance on this example than does the combina- tion of a regular table and a primary key index. On the contrary, techniques that reduce contention on both table and index (such as allowing more room for transac- tion management) or only improve the situation at the index level (reverse index, random surrogate key) all bring benefits. • The comparison of 10 sessions with 4 sessions shows that some of the techniques require additional (and scarce) CPU resources from a machine already running flat out and consequentially show no improvement. • The best way to avoid contention is not to use a sequentially generated surrogate key! Instead of considering how much performance we can gain by adopting various mea- sures, let’s consider how much performance loss is (inadvertently?) introduced by the use of a sequential key with the resulting contention on the primary key index. Solely because of contention on the primary key index, our insertion rate drops from a rate of 180 to 100 insertions per unit of time; in other words, it is divided by a factor of almost 2! The lesson is clear: we are better off without auto-incremented columns where they are not required, such as for tables that are not referenced by other tables or that do not have a very long natural primary key. FIGURE 9-10. The impact of contention limiting techniques with fewer sessions www.it-ebooks.info 246 C H A P T E R N I N E Can we recommend randomly generated surrogate keys? The difference in performance between a key generated out of a very large interval of values and a key generated out of too narrow a range of values shows that it can be dangerous and not really efficient if we expect a final number of rows greater than perhaps one hundredth of the total possible number of values. Generating random integer values between 1 and 2 billion (a common range for integer values) can prove hazardous for a large table; unfortunately, tables that are subject to heavy insertion traffic have a tendency to grow big rather quickly. However, if your system supports “long long integers,” they can be a good solution—if you really need a surrogate key. In contrast to locking, database contention can be improved upon. Architects, developers, and administrators can all design so as to limit contention. www.it-ebooks.info Chapter 10. C H A P T E R T E", + "source": "The Art of SQL.pdf", + "chunk_id": 212 + }, + { + "text": "supports “long long integers,” they can be a good solution—if you really need a surrogate key. In contrast to locking, database contention can be improved upon. Architects, developers, and administrators can all design so as to limit contention. www.it-ebooks.info Chapter 10. C H A P T E R T E N Assembly of Forces Coping with Large Volumes of Data Thenne entryd in to the bataylle Iubance a geaunt and fought and slewe doune ryght and distressyd many of our knyghtes. —Sir Thomas Malory (d.1471) Le Morte D’Arthur, V, 11 www.it-ebooks.info 248 C H A P T E R T E N T his chapter deals with the particular challenges that are facing us when data volumes swell. Those challenges include searching gigantic tables effectively, but also avoiding the sometimes distressing performance impact of even a moderate volume increase. We’ll first look at the impact of data growth and a very large number of rows on SQL queries in the general case. Then we’ll examine what happens in the particular environments of data warehousing and decision-support systems. Increasing Volumes Some applications see the volume of data they handle increase in considerable proportion over time. In particular, any application that requires keeping online, for regulatory or business analysis purposes, several months or even years of mostly inactive data, often passes through phases of crisis when (mostly) batch programs tend to overshoot the time allocated to them and interfere with regular, human activity. When you start a new project, the volume of data usually changes, as shown in Figure 10-1. Initially, hardly anything other than a relatively small amount of reference data is loaded into the database. As a new system replaces an older one, data inherited from the legacy system is painfully loaded into the new one. First, because of the radical rethink of the application, conversion from the old system to the new system is fraught with difficulties. When deadlines have to be met and some noncritical tasks have to be postponed, the recovery of legacy data is a prime candidate for slipping behind schedule. As a result, this recovery goes on for some time after the system has become operational and teething problems have been solved. Second, the old system is usually much poorer from a functional perspective than the new one (otherwise, the cost of the new project would have made for difficult acceptance up the food chain). All this means that the volume of prehistoric data will be rather small compared to the data handled by the new system, and several months’ worth of old data will probably be equivalent to a few weeks of new data at most. Meanwhile, operational data accumulates. Usually, one encounters the first serious performance issues about midway before the volume that the database is expected to hold at cruising speed. Bad queries and bad algorithms are almost invisible, from an end-user perspective, when volumes are low or moderate. The raw power of hardware often hides gigantic mistakes and may give comfortable", + "source": "The Art of SQL.pdf", + "chunk_id": 213 + }, + { + "text": "first serious performance issues about midway before the volume that the database is expected to hold at cruising speed. Bad queries and bad algorithms are almost invisible, from an end-user perspective, when volumes are low or moderate. The raw power of hardware often hides gigantic mistakes and may give comfortable sub-second response times for full scans of tables that contain several hundreds of thousands of rows. You may be seriously misusing the hardware, balancing gross programming mistakes with power—but nobody will see that until your volume becomes respectable. At the first crisis point of the project, “expert tuning” is usually required to add a couple of indexes that should have been there from the start. The system then wobbles until it www.it-ebooks.info A S S E M B L Y O F F O R C E S 249 reaches the target volume. There are usually two target volumes: a nominal one (which has been grossly overestimated and which is the volume the system has officially been designed to manage) and the real target volume (which the system just manages to handle and which is often exceeded at some point because archiving of older data has been relegated to the very last lot in the project). The second and more serious crisis often comes in the wake of reaching that point. When archival has finally been put into production, architectural weaknesses reviewed, and some critical processes vigorously rewritten, the system finally reaches cruising speed, with an increase of data related to the natural growth of business—a growth that can lie anywhere between flatness and exponential exuberance. This presentation of the early months in the life of a new database application is partly caricature; but it probably bears more resemblance to reality than it often should, because the simple mistakes that lead to this caricature are not often avoided. However rigorously one tries to work, errors are made, because of pressure, lack of time for adequate testing, and ambiguous specifications. The only errors that can bring irredeemable failure are those linked to the design of the database and to the choice of the global architecture— two topics that are closely related and that are the foundation of a system. If the foundation is not sturdy enough, you need to pull the whole building down before reconstructing. Other mistakes may require a more or less deep overhaul of what is in place. Most crises, however, need not happen. You must anticipate volume increases when coding. And you must quickly identify and rewrite a query that deteriorates in performance too quickly in the face of increasing data volumes. FIGURE 10-1. The evolution of data in a new application www.it-ebooks.info 250 C H A P T E R T E N Sensitivity of Operations to Volume Increases All SQL operations are not equally susceptible to variations in performance when the number of rows to be processed increases. Some SQL operations are insensitive to volume increases, some see performance decrease linearly with volume, and some", + "source": "The Art of SQL.pdf", + "chunk_id": 214 + }, + { + "text": "P T E R T E N Sensitivity of Operations to Volume Increases All SQL operations are not equally susceptible to variations in performance when the number of rows to be processed increases. Some SQL operations are insensitive to volume increases, some see performance decrease linearly with volume, and some perform very badly with large volumes of data. Insensitivity to volume increase Typically, there will be no noticeable difference in a search on the primary key, whether you are looking for one particular key among 1,000 or one among 1,000,000. The common B-tree indexes are rather flat and efficient structures, and the size of the underlying table doesn’t matter for a single-row, primary-key search. But insensitivity to volume increase doesn’t mean that single primary-key searches are the ultimate SQL search method. When you are looking for a large number of rows, the “transactional” single-row operation can be significantly inefficient. Just consider the following, somewhat artificial, Oracle examples, each showing a range scan on a sequence-generated primary key: SQL> declare 2 n_id number; 3 cursor c is select customer_id 4 from orders 5 where order_id between 10000 and 20000; 6 begin 7 open c; 8 loop 9 fetch c into n_id; 10 exit when c%notfound; 11 end loop; 12 close c; 13 end; 14 / PL/SQL procedure successfully completed. Elapsed: 00:00:00.27 SQL> declare 2 n_id number; 3 begin 4 for i in 10000 .. 20000 5 loop 6 select customer_id 7 into n_id 8 from orders 9 where order_id = i; 10 end loop; 11 end; 12 / www.it-ebooks.info A S S E M B L Y O F F O R C E S 251 PL/SQL procedure successfully completed. Elapsed: 00:00:00.63 The cursor in the first example, which does an explicit range scan, runs twice as fast as the iteration on a single row. Why? There are multiple technical reasons (“soft parsing,” a fast acknowledgement at each iteration that the DBMS engine has already met this statement and knows how to execute it, is one of them), but the single most important one is that in the first example the B-tree is descended once, and then the ordered list of keys is scanned and row addresses found and used to access the table; while in the second example, the B-tree is descended for each searched value in the order_id column. The most efficient way to process a large number of rows is not to iterate and apply the single- row process. Linear sensitivity to volume increases End users usually understand well that if twice as many rows are returned, a query will take more time to run; but many SQL operations double in time when operating on double the number of rows without the underlying work being as obvious to the end user, as in the case of a full table scan returning rows one after the other. Consider the case of aggregate functions; if you compute a max( ), that aggregation will always return a single row, but the", + "source": "The Art of SQL.pdf", + "chunk_id": 215 + }, + { + "text": "of rows without the underlying work being as obvious to the end user, as in the case of a full table scan returning rows one after the other. Consider the case of aggregate functions; if you compute a max( ), that aggregation will always return a single row, but the number of rows the DBMS will have to operate on may vary wildly over the life of the application. Perfectly understandable, but end users will always see a single-row returned, so they may complain of performance degradation over time. The only way to ensure that the situation will not go from bad to worse is to put an upper bound on the number of rows processed by using another criterion such as a date range. Placing an upper bound keeps data volumes under control. In the case of max( ), the idea might be to look for the maximum since a given date, and not necessarily since the beginning of time. Adding a criterion to a query is not a simple technical issue and definitely depends on business requirements, but limiting the scope of queries is an option that certainly deserves to be pointed out to, and debated with, the people who draft specifications. Non-linear sensitivity to volume increases Operations that perform sorts suffer more from volume increases than operations that just perform a scan, because sorts are complex and require on average a little more than a single pass. Sorting 100 randomly ordered rows is not 10 times costlier than sorting 10 rows, but about 20 times costlier—and sorting 1,000 rows is, on average, something like 300 times costlier than sorting 10 rows. www.it-ebooks.info 252 C H A P T E R T E N In real life, however, rows are rarely randomly stored, even when techniques such as clustering indexes (Chapter 5) are not used. A DBMS can sometimes use sorted indexes for retrieving rows in the expected order instead of sorting rows after having fetched them, and performance degradation resulting from retrieving a larger sorted set, although real, is rarely shocking. Be careful though. Performance degradation from sorts often proceeds by fits and starts, because smaller sorts will be fully executed in memory, while larger sorts will result from the merge of several sorted subsets that have each been processed in memory before being written to temporary storage. There are, therefore, some “dangerous surroundings” where one switches from a relatively fast full-memory mode to a much slower memory-plus-temporary-storage mode. Adjusting the amount of memory allocated to sorts is a frequent and efficient tuning technique to improve sort- heavy operations when flirting with the dangerous limit. By way of example, Figure 10-2 shows how the fetch rate (number of rows fetched per unit of time) of a number of queries evolves as a table grows. The table used in the test is a very simple orders table defined as follows: order_id bigint(20) (primary key) customer_id bigint(20) order_date datetime order_shipping char(1) order_comment varchar(50) The queries are first a simple", + "source": "The Art of SQL.pdf", + "chunk_id": 216 + }, + { + "text": "of rows fetched per unit of time) of a number of queries evolves as a table grows. The table used in the test is a very simple orders table defined as follows: order_id bigint(20) (primary key) customer_id bigint(20) order_date datetime order_shipping char(1) order_comment varchar(50) The queries are first a simple primary key–based search: select order_date from orders where order_id = ? then a simple sort: select customer_id from orders order by order_date then a grouping: select customer_id, count(*) from orders group by customer_id having count(*) > 3 then the selection of the maximum value in a nonindexed column: select max(order_date) from orders and finally, the selection of the “top 5” customers by number of orders: select customer_id from (select customer_id, count(*) from orders www.it-ebooks.info A S S E M B L Y O F F O R C E S 253 group by customer_id order by 2 desc) as sorted_customers limit 5 (SQL Server would replace the closing limit 5 with an opening select top 5, and Oracle would replace limit 5 with where rownum <= 5.) The number of rows in the table has varied between 8,000 and around 1,000,000, while the number of distinct customer_id values remained constant at about 3,000. As you can see in Figure 10-2, the primary key search performs almost as well with one million rows as with 8,000. There seems to be some very slight degradation at the higher number, but the query is so fast that the degradation is hardly noticeable. By contrast, the sort suffers. The performance (measured by rows returned by unit of time, and therefore independent of the actual number of rows fetched) of the sorting query decreases by 40% when the number of rows goes from 8,000 to over one million. The degradation of performance, though, is even more noticeable for all the queries that, while returning the very same number of aggregated rows, have a great deal more rows to visit to get the relatively few rows to be returned. These queries are typically the type of queries that are going to draw the most complaints from end users. Note that the DBMS doesn’t perform that badly: the performance decrease is very close to proportional to the number of rows, even for the two queries that require a sort (the queries labeled “Group by” and “Top” in Figure 10-2). But end users simply see the same amount of data—just returned much more slowly. FIGURE 10-2. How some simple queries behave when the queried table grows www.it-ebooks.info 254 C H A P T E R T E N All database operations are not equally sensitive to volume increases. Anticipate how queries will perform on target volumes. Putting it all together The main difficulty in estimating how a query will behave when data volumes increase is that high sensitivity to volume may be hidden deep inside the query. Typically, a query that finds the “current value” of an item by running a subquery that looks for the last time", + "source": "The Art of SQL.pdf", + "chunk_id": 217 + }, + { + "text": "all together The main difficulty in estimating how a query will behave when data volumes increase is that high sensitivity to volume may be hidden deep inside the query. Typically, a query that finds the “current value” of an item by running a subquery that looks for the last time the price was changed, and then performs a max( ) over the price history, is highly sensitive. If we accumulate a large number of price changes, we shall probably suffer a slow performance degradation of the subquery, and by extension of the outer query as well. The degradation will be much less sensitive with an uncorrelated subquery, executed only once, than with a correlated subquery that will compound the effect by its being fired each time it is evaluated. Such degradation may be barely perceptible in a single-item operation, but will be much more so in batch programs. NOTE The situation will be totally different if we are tracking, for instance, the current status of orders in a merchant system, because max( ) will apply to a narrow number of possible states. Even if the number of orders doubles, max( ) will in that case always operate on about the same number of rows for one order. Another issue is sorts. We have seen that an increase in the number of rows sorted leads to a quite perceptible degradation of performance. Actually, what matters is not so much the number of rows proper as the number of bytes—in other words, the total amount of data to be sorted. This is why joins with what is mostly informational data, such as user-friendly labels associated with an obscure code (as opposed to the data involved in the filtering conditions driving the query), should be postponed to the very last stage of a query. Let’s take a simple example showing why some joins should be delayed until the end of a query. Getting the names and addresses of our 10 biggest customers for the past year will require joining the orders and order_detail tables to get the amount ordered by each customer, and joining to a customers table to get each customer’s details. If we only want to get our 10 biggest customers, we must get everybody who has bought something from us in the past year, sort them by decreasing amount, and then limit the output to the first ten resulting rows. If we join all the information from the start, we will have to sort the names and addresses of all our customers from the past year. We don’t need to operate on such a large amount of data. What we must do is keep the amount of data to be sorted to the strict minimum—the customer identifier and the amount. Once everything is www.it-ebooks.info A S S E M B L Y O F F O R C E S 255 sorted, we can join the 10 customer_ids we are left with to the customers table to return all the", + "source": "The Art of SQL.pdf", + "chunk_id": 218 + }, + { + "text": "sorted to the strict minimum—the customer identifier and the amount. Once everything is www.it-ebooks.info A S S E M B L Y O F F O R C E S 255 sorted, we can join the 10 customer_ids we are left with to the customers table to return all the information that is required. In other words, we mustn’t write something like: select * from (select c.customer_name, c.customer_address, c.customer_postal_code, c.customer_state, c.customer_country sum(d.amount) from customers c, orders_o, order_detail d where c.customer_id = o.customer_id and o.order_date >= some date expression and o.order_id = d.order_id group by c.customer_name, c.customer_address, c.customer_postal_code, c.customer_state, c.customer_country order by 6 desc) as A limit 10 but rather something like: select c.customer_name, c.customer_address, c.customer_postal_code, c.customer_state, c.customer_country b.amount from (select a.customer_id, a.amount from (select o.customer_id, sum(d.amount) as amount from orders_o, order_detail d where o.order_date >= some date expression and o.order_id = d.order_id group by o.customer_id order by 2 desc) as a limit 10) as b, customers c where c.customer_id = b.customer_id order by b.amount desc The second sort is a safeguard in case the join modifies the order of the rows resulting from the inner subquery (remember that relational theory knows nothing about sorts and that the DBMS engine is perfectly entitled to process the join as the optimizer finds most efficient). We have two sorts instead of one, but the inner sort operates on “narrower” rows, while the outer one operates on only 10 rows. www.it-ebooks.info 256 C H A P T E R T E N Remember what was said in Chapter 4: we must limit the “thickness” of the non- relational layer of SQL queries. The thickness depends on the number and complexity of operations, but also on the amount of data involved. Since sorts and limits of all kinds are non-relational operations, the optimizer will probably not rewrite a query to execute a join after having cut the number of customer identifiers to the bare minimum. Although an attentive reading of two queries may make it obvious that they will return the same result, mathematically proving that they always return the same result borders on mission impossible. An optimizer always plays it safe; a DBMS cannot afford to return wrong results by attempting daring rewrites, especially since it knows hardly anything about semantics. Our example is therefore a case in which the optimizer will limit its action to perform the join in inner queries as efficiently as possible. But ordering and aggregates put a stop to mingling inner and outer queries, and therefore the query will for the most part run as it is written. The query that performs the sort of amounts before the joins is, no doubt, very ugly. But this ugly SQL code is the way to write it, because it is the way the SQL engine should execute it if we want resilience to a strong increase in the number of customers and orders. To reduce the sensitivity of your queries to increases in the volume of data, operate only on the", + "source": "The Art of SQL.pdf", + "chunk_id": 219 + }, + { + "text": "is the way to write it, because it is the way the SQL engine should execute it if we want resilience to a strong increase in the number of customers and orders. To reduce the sensitivity of your queries to increases in the volume of data, operate only on the data that is strictly necessary at the deeper levels of a query. Keep ancillary joins for the outer level. Disentangling subqueries As I have said more than once, correlated subqueries must be fired for each row that requires their evaluation. They are often a major issue when volume increases transform a few shots into sustained rounds of fire. In this section, a real-life example will illustrate both how ill-used correlated subqueries can bog a process down and how one can attempt to save such a situation. The issue at hand, in an Oracle context, is a query that belongs to an hourly batch to update a security management table. Note that this mechanism is already in itself a fudge to speed up security clearance checks on the system in question. Over time, the process takes more and more time, until reaching, on the production server, 15 minutes—which for an hourly process that suspends application availability is a bit too much. The situation sends all bells ringing and all whistles blowing. Red alert! The slowness of the process has been narrowed down to the following statement: insert /*+ append */ into fast_scrty ( emplid, rowsecclass, access_cd, empl_rcd, name, www.it-ebooks.info A S S E M B L Y O F F O R C E S 257 last_name_srch, setid_dept, deptid, name_ac, per_status, scrty_ovrd_type) select distinct emplid, rowsecclass, access_cd, empl_rcd, name, last_name_srch, setid_dept, deptid, name_ac, per_status, 'N' from pers_search_fast Statistics are up to date, so we must focus our attack on the query. As it happens, the ill- named pers_search_fast is a view defined by the following query: 1 select a.emplid, 2 sec.rowsecclass, 3 sec.access_cd, 4 job.empl_rcd, 5 b.name, 6 b.last_name_srch, 7 job.setid_dept, 8 job.deptid, 9 b.name_ac, 10 a.per_status 11 from person a, 12 person_name b, 13 job, 14 scrty_tbl_dept sec 15 where a.emplid = b.emplid 16 and b.emplid = job.emplid 17 and (job.effdt= 18 ( select max(job2.effdt) 19 from job job2 20 where job.emplid = job2.emplid 21 and job.empl-rcd = job2.empl_rcd 22 and job2.effdt <= to_date(to_char(sysdate, 23 'YYYY-MM-DD'),'YYYY-MM-DD')) 24 and job.effseq = 25 ( select max(job3.effseq) 26 from job job3 27 where job.emplid = job3.emplid 28 and job.empl_rcd = job3.empl_rcd 29 and job.effdt = job3.effdt ) ) 30 and sec.access_cd = 'Y' 31 and exists 32 ( select 'X' 33 from treenode tn www.it-ebooks.info 258 C H A P T E R T E N 34 where tn.setid = sec.setid 35 and tn.setid = job.setid_dept 36 and tn.tree_name = 'DEPT_SECURITY' 37 and tn.effdt = sec.tree_effdt 38 and tn.tree_node = job.deptid 39 and tn.tree_node_num between sec.tree_node_num 40 and sec.tree_node_num_end 41 and not exists 42 ( select 'X' 43 from scrty_tbl_dept sec2 44 where sec.rowsecclass = sec2.rowsecclass 45 and sec.setid = sec2.setid 46 and", + "source": "The Art of SQL.pdf", + "chunk_id": 220 + }, + { + "text": "tn.setid = job.setid_dept 36 and tn.tree_name = 'DEPT_SECURITY' 37 and tn.effdt = sec.tree_effdt 38 and tn.tree_node = job.deptid 39 and tn.tree_node_num between sec.tree_node_num 40 and sec.tree_node_num_end 41 and not exists 42 ( select 'X' 43 from scrty_tbl_dept sec2 44 where sec.rowsecclass = sec2.rowsecclass 45 and sec.setid = sec2.setid 46 and sec.tree_node_num <> sec2.tree_node_num 47 and tn.tree_node_num between sec2.tree_node_num 48 and sec2.tree_node_num_end 49 and sec2.tree_node_num between sec.tree_node_num 50 and sec.tree_node_num_end )) This type of “query of death” is, of course, too complicated for us to understand at a glance! As an exercise, though, it would be interesting for you to pause at this point, consider carefully the query, try to broadly define its characteristics, and try to identify possible performance stumbling blocks. If you are done pondering the query, let’s compare notes. There are a number of interesting patterns that you may have noticed: • A high number of subqueries. One subquery is even nested, and all are correlated. • No criterion likely to be very selective. The only constant expressions are an unbounded comparison with the current date at line 22, which is likely to filter hardly anything at all; a comparison to a Y/N field at line 30; and a condition on tree_name at line 36 that looks like a broad categorization. And since the insert statement that has been brought to our attention contains no where clause, we can expect a good many rows to be processed by the query. • Expressions such as between sec.tree_node_num and sec.tree_node_num_end ring a familiar bell. This looks like our old acquaintance from Chapter 7, Celko’s nested sets! Finding them in an Oracle context is rather unusual, but commercial off-the-shelf (COTS) packages often make admirable, if not always totally successful, attempts at being por- table and therefore often shun the useful features of a particular DBMS. • More subtly perhaps, when we consider the four tables (actually, one of them, person_ name, is a view) in the outer from clause, only three of them, person, person_name, and job, are cleanly joined. There is a condition on scrty_tbl_dept, but the join proper is indirect and hidden inside one of the subqueries, lines 34 to 38. This is not a recipe for efficiency. One of the very first things to do is to try to get an idea about the volumes involved; person_name is a view, but querying it indicates no performance issue. The data dictionary tells us how many rows we have: www.it-ebooks.info A S S E M B L Y O F F O R C E S 259 TABLE_NAME NUM_ROWS ------------------------------ ---------- TREENODE 107831 JOB 67660 PERSON 13884 SCRTY_TBL_DEPT 568 None of these tables is really large; it is interesting to notice that one need not deal with hundreds of millions of rows to perceive a significant degradation of performance as tables grow. The killing factor is how we are visiting tables. Finding out on the development server (obviously not as fast as the server used in production) how many rows", + "source": "The Art of SQL.pdf", + "chunk_id": 221 + }, + { + "text": "notice that one need not deal with hundreds of millions of rows to perceive a significant degradation of performance as tables grow. The killing factor is how we are visiting tables. Finding out on the development server (obviously not as fast as the server used in production) how many rows are returned by the view is not very difficult but requires steel nerves: SQL> select count(*) from PERS_SEARCH_FAST; COUNT(*) ---------- 264185 Elapsed: 01:35:36.88 A quick look at indexes shows that both treenode and job are over-indexed, a common flaw of COTS packages. We do not have here a case of the “obviously missing index.” Where must we look to find the reason that the query is so slow? We should look mostly at the lethal combination of a reasonably large number of rows and of correlated subqueries. The cascading exists/not exists in particular, is probably what does us in. NOTE In real life, all this analysis took me far more time than it is taking you now to read about it. Please understand that the paragraphs that follow summarize several hours of work and that inspiration didn’t come as a flashing illumination! Take a closer look at the exists/not exists expression. The first level subquery introduces table treenode. The second level subquery again hits table scrty_tbl_dept, already present in the outer query, and compares it both to the current row of the first level subquery (lines 47 and 48) and to the current row of the outer subquery (lines 44, 45, 46, 49, and 50)! If we want to get tolerable performance, we absolutely must disentangle these queries. Can we understand what the query is about? As it happens, treenode, in spite of its misleading name, doesn’t seem to be the table that stores the “nested sets.” The references to a range of numbers are all related to scrty_tbl_dept; treenode looks more like a denormalized flat list (sad words to use in a supposedly relational context) of the “nodes” described in scrty_tbl_dept. Remember that in the nested set implementation of tree structures, two values are associated with each node and computed in such a way that the values associated with a child node are always between the values associated www.it-ebooks.info 260 C H A P T E R T E N with the parent node. If the two values immediately follow each other, then we necessarily have a leaf node (the reverse is not true, because a subtree may have been pruned and value recomputation skipped, for obvious performance reasons). If we try to translate the meaning of lines 31 to 50 in English (sort of), we can say something like: There is in treenode a row with a particular tree_name that matches job on both setid_dept and deptid, as well as matching scrty_tbl_dept on setid and tree_effdt, and that points to either the current “node” in scrty_tbl_dept or to one of its descendents. There is no other node (or descendent) in scrty_tbl_dept that the cur- rent treenode row points", + "source": "The Art of SQL.pdf", + "chunk_id": 222 + }, + { + "text": "tree_name that matches job on both setid_dept and deptid, as well as matching scrty_tbl_dept on setid and tree_effdt, and that points to either the current “node” in scrty_tbl_dept or to one of its descendents. There is no other node (or descendent) in scrty_tbl_dept that the cur- rent treenode row points to, that matches the current one on setid and rowsecclass, and that is a descendent of that node. Dreadful jargon, especially when one has not the slightest idea of what the data is about. Can we try to express the same thing in a more intelligible way, in the hope that it will lead us to more intelligible and efficient SQL? The key point is probably in the there is no other node part. If there is no descendent node, then we are at the bottom of the tree for the node identified by the value of tree_node_num in treenode. The subqueries in the initial view text are hopelessly mingled with the outer queries. But we can write a single contained query that “forgets” for the time being about the link between treenode and job and computes, for every node of interest in scrty_tbl_dept (a small table, under 600 rows), the number of children that match it on setid and rowsecclass: select s1.rowsecclass, s1.setid, s1.tree_node_num, tn.tree_node, count(*) – 1 children from scrty_tbl_dept s1, scrty_tbl_dept s2, treenode tn where s1.rowsecclass = s2.rowsecclass and s1.setid = s2.setid and s1.access_cd = 'Y' and tn.tree_name = 'DEPT_SECURITY' and tn.setid = s1.setid and tn.effdt = s1.tree_effdt and s2.tree_node_num between s1.tree_node_num and s1.tree_node_num_end and tn.tree_node_num between s2.tree_node_num and s2.tree_node_num_end group by s1.rowsecclass, s1.setid, s1.tree_node_num, tn.tree_node (The count(*) – 1 is for not counting the current row.) The resulting set will be, of course, small, at most a few hundred rows. We shall filter out nodes that are not leaf nodes (in our context) by using the preceding query as an inline view, and applying a filter: and children = 0 www.it-ebooks.info A S S E M B L Y O F F O R C E S 261 From here, and only from here, we can join to job and properly determine the final set. Giving the final text of the view would not be extremely interesting. Let’s just point out that the first succession of exists: and (job.effdt= ( select max(job2.effdt) from job job2 where job.emplid = job2.emplid and job.empl-rcd = job2.empl_rcd and job2.effdt <= to_date(to_char(sysdate,'YYYY-MM-DD'), 'YYYY-MM-DD')) and job.effseq = ( select max(job3.effseq) from job job3 where job.emplid = job3.emplid and job.empl_rcd = job3.empl_rcd and job.effdt = job3.effdt ) ) is meant to find, for the most recent effdt for the current (emplid, empl_rcd) pair, the row with the highest effseq value. This condition is not, particularly in comparison to the other nested subquery, so terrible. Nevertheless, OLAP (or should we say analytical, since we are in an Oracle context?) functions can handle, when they are available, this type of “top of the top” case slightly more efficiently. A query such as: select emplid, empl_rcd,", + "source": "The Art of SQL.pdf", + "chunk_id": 223 + }, + { + "text": "not, particularly in comparison to the other nested subquery, so terrible. Nevertheless, OLAP (or should we say analytical, since we are in an Oracle context?) functions can handle, when they are available, this type of “top of the top” case slightly more efficiently. A query such as: select emplid, empl_rcd, effdt, effseq from (select emplid, empl_rcd, effdt, effseq row_number() over (partition by emplid, empl_rcd order by effdt desc, effseq desc) rn from job where effdt <= to_date(to_char(sysdate,'YYYY-MM-DD'),'YYYY-MM-DD')) where rn = 1 will easily select the (emplid, empl_rcd) values that we are really interested in and will be easily reinjected into the main query as an inline view that will be joined to the rest. In real life, after rewriting this query, the hourly process that had been constantly lengthening fell from 15 to under 2 minutes. Minimize the dependencies of correlated subqueries on elements from outer queries. www.it-ebooks.info 262 C H A P T E R T E N Partitioning to the Rescue When the number of rows to process is on the increase, index searches that work wonders on relatively small volumes become progressively inefficient. A typical primary key search requires the DBMS engine to visit 3 or 4 pages, descending the index, and then the DBMS must visit the table page. A range scan will be rather efficient, especially when applied to a clustering index that constrains the table rows to be stored in the same order as the index keys. Nevertheless, there is a point at which the constant to-and-fro between index page and table page becomes costlier than a plain linear search of the table. Such a linear search can take advantage of parallelism and read-ahead facilities made available by the underlying operating system and hardware. Index-searches that rely on key comparisons are more sequential by nature. Large numbers of rows to inspect exemplify the case when accesses should be thought of in terms of sweeping scans, not isolated incursions, and joins performed through hashes or merges, not loops (all this was discussed in Chapter 6). Table scans are all the more efficient when the ratio of rows that belong to the result set to rows inspected is high. If we can split our table, using the data-driven partitioning introduced in Chapter 5, in such a way that our search criteria can operate on a well defined physical subset of the table, we maximize scan efficiency. In such a context, operations on a large range of values are much more efficient when applied brutishly to a well-isolated part of a table than when the boundaries have to be checked with the help of an index. Of course, data-driven partitioning doesn’t miraculously solve all volume issues: • For one thing, the repartition of the partitioning keys must be more or less uniform; if we can find one single value of the partitioning key in 90% of rows, then scanning the table rather than the partition will hardly make any difference for that key; and for the others,", + "source": "The Art of SQL.pdf", + "chunk_id": 224 + }, + { + "text": "For one thing, the repartition of the partitioning keys must be more or less uniform; if we can find one single value of the partitioning key in 90% of rows, then scanning the table rather than the partition will hardly make any difference for that key; and for the others, they will probably be accessed more efficiently by index. The benefit of using an index that operates against a partitioned table will be slight for selective val- ues. Uniformity of distribution is the reason why dates are so well suited to partition- ing, and why range partitioning by date is by far the most popular method of partitioning. • A second point, possibly less obvious but no less important, is that the boundaries of ranges must be well defined, in both their lower value and upper values. This isn’t a peculiarity of partitioned tables, because the same can be said of index range scans. A half-bounded range, unless we are looking for values greater than a value close to the maximum in the table or lesser than a value close to the minimum, will provide no help in significantly reducing the rows we have to inspect. Similarly, a range defined as: where date_column_1 >= some value and date_column_2 <= some other value www.it-ebooks.info A S S E M B L Y O F F O R C E S 263 will not enable us to use either partitioning or indexing any more efficiently than if only one of the conditions was specified. It’s by specifying a between (or any semantic equivalent) encompassing a small number of partitions that we shall make best usage of partitioning. Half-bounded conditions make a poor use of both indexes and partitions. Data Purges Archival and data purges are too often considered ancillary matters, until they are seen as the very last hope for retrieving those by-and-large satisfactory response times of six months ago. Actually, they are extremely sensitive operations that, poorly handled, can put much strain on a system and contribute to pushing a precarious situation closer to implosion. The ideal case is when tables are partitioned (true partitioning or partitioned view) and when archival and purges operate on partitions. If partitions can be simply detached, in one way or another, then an archival (or purge) operation is trivial: a partition is archived and a new empty one possibly created. If not, we are still in a relatively strong position: the query that selects rows for archival will be a simple one, and afterwards it will be possible to truncate a partition—truncate being a way of emptying a table or partition that bypasses most of the usual mechanisms and is therefore much faster than regular deletes. NOTE Because truncate bypasses so much of the work that delete performs, you should use caution. The use of truncate may impact your backups, and it may also have other side effects, such as the invalidation of some indexes. Any use of truncate should always be discussed with your", + "source": "The Art of SQL.pdf", + "chunk_id": 225 + }, + { + "text": "deletes. NOTE Because truncate bypasses so much of the work that delete performs, you should use caution. The use of truncate may impact your backups, and it may also have other side effects, such as the invalidation of some indexes. Any use of truncate should always be discussed with your DBAs. The less-than-ideal, but oh-so-common case is when archival is triggered by age and other conditions. Accountants, for instance, are often reluctant to archive unpaid invoices, even when rather old. This makes the rather simple and elegant partition shuffling or truncation look too crude. Must we fall back on the dull-but-trusted delete? It is at this point interesting to try to rank data manipulation operations (inserts, updates, and deletes) in terms of overall cost. We have seen that inserts are pretty costly, in large part because when you insert a new row, all indexes on the table have to be maintained. Updates require only maintenance of the indexes on the updated columns. Their weakness, compared to inserts, is two-fold: first, they are associated with a search (a where clause) that www.it-ebooks.info 264 C H A P T E R T E N can be as disastrous as with a select, with the aggravating circumstance that in the meanwhile locks are held. Second, the previous value, inexistent in the case of an insert, must be saved somewhere so as to be available in case of rollback. Deletes combine all the shortcomings: they affect all indexes, are usually associated with a where clause that can be slow, and need to save the values for a possible transaction rollback. Of all operations that change data, deletes offer the greatest potential for trouble. If we can therefore save on deletes, even at the price of other operations, we are likely to end up on the winning side. When a table is partitioned and archival and purge are dependent mostly on a date condition with strings attached, we can consider a three stage purge: 1. Insert into a temporary table those old rows that we want to keep. 2. Truncate partitions. 3. Insert back from the temporary table those rows that should be retained. Without partitioning, the situation is much more difficult. In order to limit lock duration—and assuming of course that once a row has attained the “ready for archival” state, no operation whatsoever can put it back to the “no, wait, I have second thought” state—we can consider a two-step operation. This two-step operation will be all the more advantageous given that the query that identifies rows for archiving is a slow-running one. What we may do in that case is: 1. Build a list of the identifiers of the rows to archive. 2. Join on this list for both archival and purge, rather than running the same slow where clause twice, once in a select statement and once in a delete statement. A major justification for temporary tables is to enable massive, table-oriented operations that would outperform row-wise operations. Data Warehousing The", + "source": "The Art of SQL.pdf", + "chunk_id": 226 + }, + { + "text": "2. Join on this list for both archival and purge, rather than running the same slow where clause twice, once in a select statement and once in a delete statement. A major justification for temporary tables is to enable massive, table-oriented operations that would outperform row-wise operations. Data Warehousing The purpose of this book is not to devote half a chapter to covering the complex issues linked to data warehousing. Many books on the topic of data warehousing have been written, some of them generic (Ralph Kimball’s The Data Warehouse Toolkit and Bill Inmon’s Building the Data Warehouse, both published by John Wiley & Sons, are probably www.it-ebooks.info A S S E M B L Y O F F O R C E S 265 the two best-known titles), some of them specific to a DBMS engine. There has been something of a religious war between the followers of Inmon, who advocates a clean 3NF design of enormous data repositories used by decision-support systems, and the supporters of Kimball, who believes that data warehouses are a different world with different needs, and that therefore the 3NF model, in spite of its qualities in the operational world, is better replaced with dimensional modeling, in which reference data is happily denormalized. As most of this book advocates and assumes a clean 3NF design, I will deal hereafter more specifically with dimensional models, to study their strengths and the reason for their popularity, but also their weaknesses. I will, in particular, examine the interactions between operational data stores (“production databases” to the less enlightened) and decision-support systems, since data doesn’t fall from heaven, unless you are working for NASA or a satellite operating company, and what you load into dimensional models has to come from somewhere. Understand that it is not because one is using the SQL language against “tables” that one is operating in the relational world. Facts and Dimensions: the Star Schema The principle of dimensional modeling is to store measurement values, whether they are quantities, amounts, or whatever you can imagine into big fact tables. Reference data is stored into dimension tables that mostly contain self-explanatory labels and that are heavily denormalized. There are typically 5 to 15 dimensions, each with a system- generated primary key, and the fact table contains all the foreign keys. Typically, the date associated with a series of measures (a row) in the fact table will not be stored as a date column in the fact table, but as a system-generated number that will reference a row in the date_dimension table in which the date will be declined under all possible forms. If we take, for instance, the traditional starting date of the Unix world, January 1, 1970, it would typically be stored in date_dimension as: Every row that refers to something having occurred on January 1, 1970 in the fact table would simply store the 12345 key. The rationale behind such an obviously non-normalized way of storing data is that, although normalization is highly", + "source": "The Art of SQL.pdf", + "chunk_id": 227 + }, + { + "text": "January 1, 1970, it would typically be stored in date_dimension as: Every row that refers to something having occurred on January 1, 1970 in the fact table would simply store the 12345 key. The rationale behind such an obviously non-normalized way of storing data is that, although normalization is highly important in environments where data is changed, because it is the only way to ensure data integrity, the overhead of storing redundant information in a data warehouse is relatively negligible since dimension tables contain very few rows compared to the normalized fact table. For instance, a one- century date dimension would only hold 36,525 rows. Moreover, argues Dr. Kimball, having only a fact table surrounded by dimension tables as in Figure 10-3 (hence the “star schema” name) makes querying that data extremely simple. Queries against the data tend to require very few joins, and therefore are very fast to execute. date_key date_value date_description day month year quarter holiday 12345 01/01/1970 January 1, 1970 Thursday January 1970 Q1 1970 Holiday www.it-ebooks.info 266 C H A P T E R T E N Anybody with a little knowledge of SQL will probably be startled by the implication that the fewer the joins, the faster a query runs. Jumping to the defense of joins is not, of course, to recommend joining indiscriminately dozens of tables, but unless you have had a traumatic early childhood experience with nested loops on big, unindexed tables, it is hard to assert seriously that joins are the reason queries are slow. The slowness comes from the way queries are written; in this light, dimensional modeling can make a lot of sense, and you’ll see why as you progress through this chapter. The design constraints of dimensional modeling are deliberately read-oriented, and consequently they frequently ignore the precepts of relational design. Query Tools The problem with decision-support systems is that their primary users have not the slightest idea how to write an SQL query, not even a terrible one. They therefore have to use query tools for that purpose, query tools that present them with a friendly interface and build queries for them. You saw in Chapter 8 that dynamically generating an efficient query from a fixed set of criteria is a difficult task, requiring careful thought and even more careful coding. It is easy to understand that when the query can actually be anything, a tool can only generate a decent query when complexity is low. The following piece of code is one that I saw actually generated by a query tool (it shows some of the columns returned by a subquery in a from clause): ... FROM (SELECT ((((((((((((t2.\"FOREIGN_CURRENCY\" || CASE WHEN 'tfp' = 'div' THEN t2.\"CODDIV\" WHEN 'tfp' = 'ac' THEN t2.\"CODACT\" WHEN 'tfp' = 'gsd' THEN t2.\"GSD_MNE\" WHEN 'tfp' = 'tfp' THEN t2.\"TFP_MNE\" ELSE NULL FIGURE 10-3. A simple star schema, showing primary keys (PK) and foreign keys (FK) www.it-ebooks.info A S S E M B L Y O F F O R C E", + "source": "The Art of SQL.pdf", + "chunk_id": 228 + }, + { + "text": "WHEN 'tfp' = 'ac' THEN t2.\"CODACT\" WHEN 'tfp' = 'gsd' THEN t2.\"GSD_MNE\" WHEN 'tfp' = 'tfp' THEN t2.\"TFP_MNE\" ELSE NULL FIGURE 10-3. A simple star schema, showing primary keys (PK) and foreign keys (FK) www.it-ebooks.info A S S E M B L Y O F F O R C E S 267 END ) || CASE WHEN 'Y' = 'Y' THEN TO_CHAR ( TRUNC ( t2.\"ACC_PCI\" ) ) ELSE NULL END ) || CASE WHEN 'N' = 'Y' THEN t2.\"ACC_E2K\" ELSE NULL END ) || CASE WHEN 'N' = 'Y' THEN t2.\"ACC_EXT\" ELSE NULL END ) || CASE ... It seems obvious from this sample’s select list that at least some “business intelligence” tools invest so much intelligence on the business side that they have nothing left for generating SQL queries. And when the where clause ceases to be trivial—forget about it! Declaring that it is better to avoid joins for performance reasons is quite sensible in this context. Actually, the nearer you are to the “text search in a file” (a.k.a. grep) model, the better. And one understands why having a “date dimension” makes sense, because having a date column in the fact table and expecting that the query tool will transform references to “Q1” into “between January 1 and March 31” to perform an index range scan requires the kind of faith you usually lose when you stop believing in the Tooth Fairy. By explicitly laying out all format variations that end users are likely to use, and by indexing all of them, risks are limited. Denormalized dimensions, simple joins, and all- round indexing increase the odds that most queries will execute in a tolerable amount of time, which is usually the case. Weakly designed queries may perform acceptably against dimensional models because the design complexity is much lower than in a typical transactional model. Extraction, Transformation, and Loading In order for business users to be able to proactively leverage strategic cost-generating opportunities (if data warehousing literature is to be believed), it falls on some poor souls to ensure the mundane task of feeding the decision-support system. And even if tools are available, this feeding is rarely an easy task. www.it-ebooks.info 268 C H A P T E R T E N Data extraction Data extraction is not usually handled through SQL queries. In the general case, purpose- built tools are used: either utilities or special features provided by the DBMS, or dedicated third-party products. In the unlikely event that you would want to run your own SQL queries to extract information to load into a data warehouse, you typically fall into the case of having large volumes of information, where full table scans are the safest tactic. You must do your best in such a case to operate on arrays (if your DBMS supports an array interface—that is fetching into arrays or passing multiple values as a single array), so as to limit round-trips between the DBMS kernel and the downloading program. Transformation Depending on your SQL", + "source": "The Art of SQL.pdf", + "chunk_id": 229 + }, + { + "text": "You must do your best in such a case to operate on arrays (if your DBMS supports an array interface—that is fetching into arrays or passing multiple values as a single array), so as to limit round-trips between the DBMS kernel and the downloading program. Transformation Depending on your SQL skills, the source of the data, the impact on production systems, and the degree of transformation required, you can use the SQL language to perform a complex select that will return ready-to-load data, use SQL to modify the data in a staging area, or use SQL to perform the transformation at the same time as the data is uploaded into the data warehouse. Transformations often include aggregates, because the granularity required by decision support systems is usually coarser than the level of detail provided by production databases. Typically, values may be aggregated by day. If transformation is not more complicated than aggregation, there is no reason for performing it as a separate operation. Writing to the database is much costlier than reading from it, and updating the staging area before updating the data warehouse proper may be adding an unwanted costly step. Such an extra step may be unavoidable, though, when data has to be compounded from several distinct operational systems; I can list several possible reasons for having to get data from different unrelated sources: • Acute warlordism within the corporation • A recently absorbed division still using its pre-acquisition information system • A migration spread over time, meaning that at some point you have, for instance, domestic operations still running on an old information system while international ones are already using a new system that will later be used everywhere The assemblage of data from several sources should be done, as much as possible, in a single step, using a combination of set operators such as union and of in-line views— subqueries in the from clause. Multiple passes carry a number of risks and should not be directly applied to the target data warehouse. The several-step update of tables, with null columns being suddenly assigned values is an excellent recipe for wreaking havoc at the physical level. When data is stored in variable length, as is often the case with character www.it-ebooks.info A S S E M B L Y O F F O R C E S 269 information and sometimes with numeric information as well (Oracle is an example of such a storage strategy), it will invariably lead to some of the data being relegated to overflow pages, thus compromising the efficiency of both full scans and indexed accesses, since indexes usually point to the head part of a row. Any pointer to an overflow area will mean visiting more pages than would otherwise be necessary to answer a given question, and will be costly. If the prepared data is very simply inserted into the target data warehouse tables, data will be properly reorganized in the process. It is also quite common to see several updates", + "source": "The Art of SQL.pdf", + "chunk_id": 230 + }, + { + "text": "will mean visiting more pages than would otherwise be necessary to answer a given question, and will be costly. If the prepared data is very simply inserted into the target data warehouse tables, data will be properly reorganized in the process. It is also quite common to see several updates applied to different columns of the same table in turn. Whenever possible, perhaps with help from the case construct, always update as many columns in one statement as possible. Multiple massive updates applied to a table often wreak havoc at the physical level. Loading If you build your data warehouse (or data mart, as others prefer to say) according to the rules of dimensional modeling, all dimensions will use artificial, system-generated keys for the purpose of keeping a logical track over time of items that may be technically different but logically identical. For instance, if you manufacture consumer electronics, a new model with a new reference may have been designed to replace an older model, now discontinued. By using the same artificial key for both, you can consider them as a single logical entity for analysis. The snag is that the primary keys in your operational database will usually have different values from the dimension identifiers used in the decision support system, which becomes an issue not with dimension tables but with fact tables. You have no reason to use surrogate keys for dates in your operational system. In the same way, the operational system doesn’t necessarily need to record which electronic device model is the successor to another. Dimension tables are, for the most part, loaded once and rarely updated. Dimensional modeling rests partly on the assumption that the fast-changing values are the ones stored in fact tables. As a result, for every row you need to insert into the fact table, you must retrieve (from the operational database primary key) the value of the corresponding surrogate, system-generated key for each of the dimensions—which necessarily means as many joins as there are different dimensions. Queries against the decision support system may require fewer joins, but loading into the decision support system will require many more joins because of the mapping between operational and dimensional keys. www.it-ebooks.info 270 C H A P T E R T E N The advantage of simpler queries against dimensional models is paid for by the disadvantage of complex preparation and loading of the data. Integrity constraints and indexes When a DBMS implements referential integrity checking, it is sensible to disable that checking during data load operations. If the DBMS engine needs to check for each row that the foreign keys exist, the engine does double the amount of work, because any statement that uploads the fact table has to look for the parent surrogate key anyway. You might also significantly speed up loading by dropping most indexes and rebuilding them after the load, unless the rows loaded represent a small percentage of the size of the table that you are loading, as rebuilding indexes", + "source": "The Art of SQL.pdf", + "chunk_id": 231 + }, + { + "text": "the fact table has to look for the parent surrogate key anyway. You might also significantly speed up loading by dropping most indexes and rebuilding them after the load, unless the rows loaded represent a small percentage of the size of the table that you are loading, as rebuilding indexes on very large tables can be prohibitively expensive in terms of resources and time. It would however be a potentially lethal mistake to disable all constraints, and particularly primary keys. Even if the data being loaded has been cleaned and is above all reproach, it is very easy to make a mistake and load the same data twice—much easier than trying to remove duplicates afterwards. The massive upload of decision-support systems is one of the rare cases when temporarily altering a schema may be tolerated. Querying Dimensions and Facts: Ad Hoc Reports If query tools are seriously helped by removing anything that can get in their way, such as evil joins and sophisticated subqueries, there usually comes a day when business users require answers that a simplistic schema cannot provide. The dimensional model is then therefore duly “embellished” with mini-dimensions, outriggers, bridge tables, and all kinds of bells and whistles until it begins to resemble a clean 3NF schema, at which point query tools are beginning to suffer. One day, a high-ranking user tries something daring—and the next day the problem is on the desk of a developer, while the tool-generated query is still running. Time for ad hoc queries and shock SQL! It is when you have to write ad hoc queries that it is time to get back to dimensional modeling and see the SQL implications. Basically, dimensions represent the breaks in a report. If an end user often wants to see sales by product, by store, and by month, then we have three dimensions involved: the date dimension that has been previously introduced, the product dimension, and the store dimension. Product and store can be denormalized to include information such as product line, brand, and category in one case, and region, surface, or whatever criterion is deemed to be relevant in the other case. Sales amounts are, obviously, facts. www.it-ebooks.info A S S E M B L Y O F F O R C E S 271 A key characteristic of the star schema is that we are supposed to attack the fact table through the dimensions such as in Figure 10-4; in the previous example, we might for instance want to see sales by product, store, and month for dairy products in the stores located in the Southwest and for the third quarter. Contrarily to the generally recommended practice in normalized operational databases, dimension tables are not only denormalized, but are also strongly indexed. Indexing all columns means that, whatever the degree of detail required (the various columns in a location dimension, such as city, state, region, country, area, can be seen as various levels of detail, and the same is true of a date dimension),", + "source": "The Art of SQL.pdf", + "chunk_id": 232 + }, + { + "text": "not only denormalized, but are also strongly indexed. Indexing all columns means that, whatever the degree of detail required (the various columns in a location dimension, such as city, state, region, country, area, can be seen as various levels of detail, and the same is true of a date dimension), an end user who is executing a query will hit an index. Remember that dimensions are reference tables that are rarely if ever updated, and therefore there is no frightful maintenance cost associated with heavy indexing. If all of your criteria refer to data stored in dimension tables, and if they are indexed so as to make any type of search fast, you should logically hit dimension tables first and then locate the relevant values in the fact table. Hitting dimensions first has very strong SQL implications that we must well understand. Normally, one accesses particular rows in a table through search criteria, finds some foreign keys in those rows, and uses those foreign keys to pull information from the tables those keys reference. To take a simple example, if we want to find the phone number of the assistant in the department where Owens works, we shall query the table of employees basing our search on the “Owens” name, find the department number, and use the primary key on the table of departments to find the phone number. This is a classic, nested-loop join case. With dimensional modeling, the picture totally changes. Instead of going from the referencing table (the employees) to the referenced table (the departments), naturally FIGURE 10-4. The usual way of querying tables in the dimensional model www.it-ebooks.info 272 C H A P T E R T E N following foreign keys, we start from the reference tables—the dimensions. To go where? There is no foreign key linking a dimension to the fact table: the opposite is true. It is like looking for the names of all the employees in a department when all you know is the phone number of the assistant. When joining the fact table to the dimension, the DBMS engine will have to go through a mechanism other than the usual nested loop—perhaps something such as a hash join. Another peculiarity of queries on dimensional models is that they often are perfect examples of the association of criteria that’s not too specific, with a relatively narrow intersection to obtain a result set that is not, usually, enormous. The optimizer can use a couple of tactics for handling such queries. For instance: Determining which is the most selective of all these not-very-selective criteria, joining the associated dimension to the fact table, and then checking each of the other dimensions Such a tactic is fraught with difficulties. First of all, the way dimensions are built may give the optimizer wrong ideas about selectivity. Suppose we have a date dimension that is used as the reference for many different dates: the sales date, but also, for instance, the date on which each store was first", + "source": "The Art of SQL.pdf", + "chunk_id": 233 + }, + { + "text": "fraught with difficulties. First of all, the way dimensions are built may give the optimizer wrong ideas about selectivity. Suppose we have a date dimension that is used as the reference for many different dates: the sales date, but also, for instance, the date on which each store was first opened, a fact that may be useful to compare how each store is doing after a given number of months of activity. Since the date dimension will never be a giant table, we may have decided to fill it from the start with seventy years’ worth of dates. Seventy years give us, on one hand, enough “historical” dates to be able to refer to even the opening of the humble store of the present chairman’s grandfather and, on the other hand, enough future dates so as to be able to forget about maintaining this dimension for quite a while. Inevitably, a reference to the sales of last year’s third quarter will make the criterion look much more selective than it really is. The problem is that if we truly had a “sales date” inside the fact table, it would be straightforward to determine the useful range of dates. If we just have a “date reference” pointing to the date dimension, the starting point for evaluation is the dimension, not the fact table. Scanning the fact table and discarding any row that doesn’t satisfy any of the various criteria Since fact tables contain all the measurement or metrics, they are very large. If they are partitioned, it will necessarily be against a single dimension (two if you use subpartitioning). Any query involving three or more dimensions will require a full scan of a table that can contain millions of rows. Scanning the fact table isn’t the most attractive of options. In such a case, visiting the fact table at an early stage, which also means after a first dimension, may be a mistake. Some products such as Oracle implement an interesting algorithm, known in Oracle’s case as the “star transformation.” We are going to look next at this transformation in some detail, including characteristics that are peculiar to Oracle, before discussing how such an algorithm may be emulated in non-Oracle environments. www.it-ebooks.info A S S E M B L Y O F F O R C E S 273 Dimensional modeling is built on the premise that dimensions are the entry points. Facts must be accessed last. The star transformation The principle behind the star transformation is, as a very first step, to separately join the fact table to each of the dimensions for which we have a filtering condition. The transformation makes it appear that we are joining several times to the fact table, but appearances are deceiving. What we really want is to get the addresses of rows from the fact table that match the condition on each dimension. Such an address, also known as a rowid (accessible as a pseudo-column with Oracle; Postgres has a functionally equivalent oid)", + "source": "The Art of SQL.pdf", + "chunk_id": 234 + }, + { + "text": "to the fact table, but appearances are deceiving. What we really want is to get the addresses of rows from the fact table that match the condition on each dimension. Such an address, also known as a rowid (accessible as a pseudo-column with Oracle; Postgres has a functionally equivalent oid) is stored in indexes. All we need to join, therefore, are three objects: • The index on the column from the dimension table that we use as a filtering condi- tion—for instance, the quarters column in date_dimension • The date_dimension itself, in which we find the system-generated artificial primary key date_key • The index on the column in the fact table that is defined as a foreign key referencing date_key (star transformations work best when the foreign keys in the fact table are indexed) Even though the fact table appears several times in a star query, we will not hit the same data or index pages repeatedly. All the separate joins will involve different indexes, and all storing rowids referring to the same table—but otherwise those indexes are perfectly distinct objects. As soon as we have the result of two joins, we can combine the two resulting sets of rowids, discarding everything that doesn’t belong to the intersection of the two sets for an and condition or retaining everything for an or condition. This step is further simplified if we are using bitmap indexes, for which simple bit-wise operations are all that is required to select our final set of rowids that refer to rows satisfying our conditions. Once we have our final, relatively small set of resulting rowids, then we can fetch the corresponding rows from the fact table that we are actually visiting for the very first time. Bitmap indexes, as their name says, index values by keeping bitmaps telling which rows contain a particular value and which do not. Bitmap indexes are particularly appropriate to index low-cardinality columns; in other words, columns in which there are few distinct values, even if the distribution of those values is not particularly skewed. Bitmap indexes were not mentioned in previous chapters for an excellent reason: they are totally inappropriate for general database operations. There is a major reason for avoiding them in a database that incurs normal update activity: when you update a bitmap, you have to www.it-ebooks.info 274 C H A P T E R T E N lock it. Since this type of index is designed for columns with few distinct values, you end up preventing changes to many, many rows, and you get a behavior that lies somewhere between page locking and table locking, but much closer to table locking. For read-only databases, however, bitmap indexes may prove useful. Bitmap indexes are quickly built during bulk loads and take much less storage than regular indexes. Emulating the star transformation Although automated star transformation is a feature that enables even poorly generated queries to perform efficiently, it is quite possible to write a query in a way that", + "source": "The Art of SQL.pdf", + "chunk_id": 235 + }, + { + "text": "prove useful. Bitmap indexes are quickly built during bulk loads and take much less storage than regular indexes. Emulating the star transformation Although automated star transformation is a feature that enables even poorly generated queries to perform efficiently, it is quite possible to write a query in a way that will induce the DBMS kernel to execute it in a similar, if not exactly identical, fashion. I must plead guilty to writing SQL statements that are geared at one particular result. From a relational point of view, I would deserve to be hanged high. On the other hand, dimensional modeling has nothing to do with the relational theory. I am therefore using SQL in a shamelessly unrelational way. Let’s suppose that we have a number of dimension tables named dim1, dim2, ... dimn. These dimension tables surround our fact table that we shall imaginatively call facts. Each row in facts is composed of key1, key2, ... keyn, foreign keys respectively pointing to one dimension table, plus a number of values (the facts) val1, val2, ... valp. The primary key of facts is defined as a composite key, and is simply made of key1 to keyn. Let’s further imagine that we need to execute a query that satisfies conditions on some columns from dim1, dim2, and dim3 (they may, for instance, represent a class of products, a store location, and a time period). For simplicity, say that we have a series of and conditions, involving col1 in dim1, col2 in dim2 and col3 in dim3. We shall ignore any transformation, aggregate or whatever, and limit our creative exercise to returning the appropriate set of rows in as effective a way as possible. The star transformation mostly aims to obtain in an efficient way the identifiers of the rows from the fact table that will belong to our result set, which may be the final result set or an intermediate result set vowed to further ordeals. If we start with joining dim2 to facts, for instance: select ... from dim2, facts where dim2.key2 = facts.key2 and dim2.col2 = some value then we have a major issue if we have no Oracle rowid, because the identifiers of the appropriate rows from facts are precisely what we want to see returned. Must we return the primary key from facts to properly identify the rows? If we do, we hit not only the index on facts(key2), but also table facts itself, which defeats our initial purpose. Remember that the frequently used technique to avoid an additional visit to the table is www.it-ebooks.info A S S E M B L Y O F F O R C E S 275 to store the information we need in the index by adding to the index the columns we want to return. So, must we turn our index on facts(key2) into an index on facts(key2, key3 ... keyn)? If we do that, then we must apply the same recipe to all foreign keys! We will end up with", + "source": "The Art of SQL.pdf", + "chunk_id": 236 + }, + { + "text": "in the index by adding to the index the columns we want to return. So, must we turn our index on facts(key2) into an index on facts(key2, key3 ... keyn)? If we do that, then we must apply the same recipe to all foreign keys! We will end up with n indexes that will each be of a size in the same order of magnitude as the facts table itself, something that is not acceptable and that forces us to read large amounts of data while scanning those indexes, thus jeopardizing performance. What we need for our facts table is a relatively small row identifier—a surrogate key that we may call fact_id. Although our facts table has a perfectly good primary key, and although it is not referenced by any other table, we still need a compact technical identifier—not to use in other tables, but to use in indexes. With our system-generated fact_id column, we can have indexes on (key1, fact_id), (key2, fact_id) ... (keyn, fact_id) instead of on the foreign keys alone. We can now fully write our previous query as: select facts.fact_id from dim2, facts where dim2.key2 = facts.key2 and dim2.col2 = some value This version of the query no longer needs the DBMS engine to visit anything but the index on col2, the dimension table dim2, and the facts index on (key2, fact_id). Note that by applying the same trick to dim2 (and of course the other dimension tables), systematically appending the key to indexes on every column, the query can be executed by only visiting indexes. Repeating the query for dim1 and dim3 provides us with identifiers of facts that satisfy the conditions associated with these dimensions. The final set of identifiers satisfying all conditions can easily be obtained by joining all the queries: select facts1.fact_id from (select facts.fact_id from dim1, facts where dim1.key1 = facts.key1 and dim1.col1 = some value) facts1, (select facts.fact_id from dim2, facts where dim2.key2 = facts.key2 and dim2.col2 = some other value) facts2, (select facts.fact_id from dim3, facts where dim3.key3 = facts.key3 and dim3.col3 = still another value) facts3 where facts1.fact_id = facts2.fact_id and facts2.fact_id = facts3.fact_id www.it-ebooks.info 276 C H A P T E R T E N Afterwards, we only have to collect from facts the rows, the identifiers of which are returned by the previous query. The technique just described is, of course, not specific to decision-support systems. But I must point out that we have assumed some very heavy indexing, a standard fixture of data marts and, generally speaking, read-only databases. In such a context, putting more information into indexes and adding a surrogate key column can be considered as “no impact” changes. You should be most reluctant in normal (including in the relational sense!) circumstances to modify a schema so significantly to accommodate queries. But if most of the required elements are already in place, as in a data warehousing environment, you can certainly take advantage of them. Querying a star schema the way it is", + "source": "The Art of SQL.pdf", + "chunk_id": 237 + }, + { + "text": "reluctant in normal (including in the relational sense!) circumstances to modify a schema so significantly to accommodate queries. But if most of the required elements are already in place, as in a data warehousing environment, you can certainly take advantage of them. Querying a star schema the way it is not intended to be queried As you have seen, the dimensional model is designed to be queried through dimensions. But what happens when, as in Figure 10-5, our input criteria refer to some facts (for instance, that the sales amount is greater than a given value) as well as to dimensions? We can compare such a case to the use of a group by. If the condition on the fact table applies to an aggregate (typically a sum or average), we are in the same situation as with a having clause: we cannot provide a result before processing all the data, and the condition on the fact table is nothing more than an additional step over what we might call regular dimensional model processing. The situation looks different, but it isn’t. If, on the contrary, the condition applies to individual rows from the fact table, we should consider whether it would be more efficient to discard unwanted facts rows earlier in the process, in the same way that it is advisable to filter out unwanted rows in the where FIGURE 10-5. A maverick usage of the dimension model www.it-ebooks.info A S S E M B L Y O F F O R C E S 277 clause of a query, before the group by, rather than in the having clause that is evaluated after the group by. In such a case, we should carefully study how to proceed. Unless the fact column that is subjected to a condition is indexed—a condition that is both unlikely and unadvisable—our entry point will still be through one of the dimensions. The choice of the proper dimension to use depends on several factors; selectivity is one of them, but not necessarily the most important one. Remember the clustering factor of indexes, and how much an index that corresponds to the actual, physical order of rows in the table outperforms other indexes (whether the correspondence is just a happy accident of the data input process, or whether the index has been defined as constraining the storage of rows in the table). The same phenomenon happens between the fact table and the dimensions. The order of fact rows may happen to match a particular date, simply because new fact rows are appended on a daily basis, and therefore those rows have a strong affinity to the date dimension. Or the order of rows may be strongly correlated to the “location dimension” because data is provided by numerous sites and processed and loaded on a site-by-site basis. The star schema may look symmetrical, just as the relational model knows nothing of order. But implementation hazards and operational processes often result in a break-up of the star schema’s", + "source": "The Art of SQL.pdf", + "chunk_id": 238 + }, + { + "text": "correlated to the “location dimension” because data is provided by numerous sites and processed and loaded on a site-by-site basis. The star schema may look symmetrical, just as the relational model knows nothing of order. But implementation hazards and operational processes often result in a break-up of the star schema’s theoretical symmetry. It’s important to be able to take advantage of this hidden dissymmetry whenever possible. If there is a particular affinity between one of the dimensions to which a search filter must be applied and to the fact table, the best way to proceed is probably to join that dimension to the fact table, especially if the criterion that is applied to the fact table is reasonably selective. Note that in this particular case we must join to the actual fact table, obviously through the foreign key index, but not limit our access to the index. This will allow us to directly get a superset of our target collection of rows from the fact table at a minimum cost in terms of visited pages, and check the condition that directly applies to fact rows early. The other criteria will come later. The way data is loaded to a star schema can favor one dimension over all others. A (Strong) Word of Caution Dimensional modeling is a technique, not a theory, and it is popular because it is well- suited to the less-than-perfect (from an SQL* perspective) tools that are commonly used in decision support systems, and because the carpet-indexing (as in “carpet-bombing”) it requires is tolerable in a read-only system—read-only after the loading phase, that is. The * Some will say—with some reason—that SQL itself is not above reproach. www.it-ebooks.info 278 C H A P T E R T E N problem is that when you have 10 to 15 dimensions, then you have 10 to 15 foreign keys in your fact table, and you must index all those keys if you want queries to perform tolerably well. You have seen that dimensions are mostly static and not enormous, so indexing all columns in all dimensions is no real issue. But indexing all columns may be much more worrisome with a fact table, which can grow very big: just imagine a large chain of grocery stores recording one fact each time they sell one article. New rows have to be inserted into the fact table very regularly. You saw in Chapter 3 that indexes are extremely costly when inserting; 15 indexes, then, will very significantly slow down loading. A common technique to load faster is to drop indexes and then recreate them (if possible in parallel) once the data is loaded. That technique may work for a while, but re- indexing will inexorably take more time as the base table grows. Indexing requires sorting, and (as you might remember from the beginning of this chapter) sorts belong to the category of operations that significantly suffer when the number of rows increases. Sooner or later, you will discover that the", + "source": "The Art of SQL.pdf", + "chunk_id": 239 + }, + { + "text": "re- indexing will inexorably take more time as the base table grows. Indexing requires sorting, and (as you might remember from the beginning of this chapter) sorts belong to the category of operations that significantly suffer when the number of rows increases. Sooner or later, you will discover that the re-creation of indexes takes way too much time, and you may well also be told that you have users who live far, far away who would like to access the data warehouse in the middle of the night. Users that want the database to be accessible during a part of the night mean a smaller maintenance window for loading the decision support system. Meanwhile, because recreating indexes takes longer as data accumulates into the decision support database, loading times have a tendency to increase. Instead of loading once every night, wouldn’t it be possible to have a continuous flow of data from operational systems to the data warehouse? But then, denormalization can become an issue, because the closer we get to a continuous flow, the closer we are to a transactional model, with all the integrity issues that only proper normalization can protect against. Compromising on normalization is acceptable in a carefully controlled environment, an ivory tower. When the headquarters are too close to the battlefield, they are submitted to the same rules. www.it-ebooks.info Chapter 11. C H A P T E R E L E V E N Stratagems Trying to Salvage Response Times But my doctrines and I begin to part company. —Thomas Hardy (1840–1928) Jude The Obscure, IV, ii www.it-ebooks.info 280 C H A P T E R E L E V E N I hope to have convinced you in Chapters 1 and 2 about the extent to which performance depends, first and foremost, on a sound database design, and second, on a clear strategy and well-designed programs. The sad truth is that when you are beginning to be acknowledged as a skilled SQL tuner, people will not seek your advice until they discover that they have performance problems. This happens—at best—during the final stages of acceptance testing, after man-months of haphazard development. You are then expected to work wonders on queries when table designs, program architectures, or sometimes even the requirements themselves may all be grossly inappropriate. Some of the most sensitive areas are related to interfacing legacy systems—in other words loading the database or downloading data to files. If there is one chapter in this book that should leave a small imprint on your memory, it should probably be this one. If you really want to remember something, I hope it will not be the recipes (those tricky and sometimes entertaining SQL queries of death) in this chapter, but the reasoning behind each recipe, which I have tried my best to make as explicit as possible. Nothing is better than getting things right from the very start; but there is some virtue in trying to get the best out of a rotten situation. You", + "source": "The Art of SQL.pdf", + "chunk_id": 240 + }, + { + "text": "death) in this chapter, but the reasoning behind each recipe, which I have tried my best to make as explicit as possible. Nothing is better than getting things right from the very start; but there is some virtue in trying to get the best out of a rotten situation. You will also find some possible answers to common problems that, sometimes surprisingly, seem to induce developers to resort to contorted procedures. These procedures are not only far less efficient, but also commonly far more obscure and harder to maintain than SQL statements, even complex ones. I shall end this chapter with a number of remarks about a commonly used stratagem indirectly linked to SQL proper, that of optimizer directives. Welcome to the heart of darkness. Turning Data Around The most common difficulty that you may encounter when trying to solve SQL problems is when you have to program against what might charitably be called an “unconventional” design. Writing a query that performs well is often the most visible challenge. However, I must underline that the complex SQL queries that are forced upon developers by a poor design only mirror the complication of programs (including triggers and stored procedures) that the same poor design requires in order to perform basic operations such as integrity checking. By contrast, a sound design allows you to declare constraints and let the DBMS check them for you, removing much of the risk associated with complexity. After all, ensuring data integrity is exactly what a DBMS, a rather fine piece of software, has been engineered to achieve. Unfortunately, haphazard designs will force you to spend days coding application controls. As a bonus, you get very high odds of letting software bugs creep in. Unlike popular software systems that are in daily use by www.it-ebooks.info S T R A T A G E M S 281 millions of users, where bugs are rapidly exposed and fixed, your home-grown software can hide bugs for weeks or months before they are discovered. Rows That Should Have Been Columns Rows that should have been originally specified as columns are most often encountered with that appalling “design” having the magical four attributes—entity_id, attribute_name, attribute_type, attribute_value—that are supposed to solve all schema evolution issues. Frighteningly, many supporters of this model seem to genuinely believe that it represents the ultimate sophistication in terms of normalization. You will find it under various, usually flattering, names—such as meta-design, or fact dimension with data warehouse designers. Proponents of the magical four attributes praise the “flexibility” of this model. There is an obvious confusion of flexibility with flabbiness. Being able to add “attributes” on the fly is not flexibility; those attributes need to be retrieved and processed meaningfully. The dubious benefit of inserting rows instead of painstakingly designing the database in the first place is absolutely negligible compared to the major coding effort that is required, first, to process those new rows, and second, to insure some minimal degree of integrity and data consistency. The proper way", + "source": "The Art of SQL.pdf", + "chunk_id": 241 + }, + { + "text": "meaningfully. The dubious benefit of inserting rows instead of painstakingly designing the database in the first place is absolutely negligible compared to the major coding effort that is required, first, to process those new rows, and second, to insure some minimal degree of integrity and data consistency. The proper way to deal with varying numbers of attributes is to define subtypes, as explained in Chapter 1. Subtypes let you define clean referential integrity constraints— checks that you will not need to code and maintain. A database should not be a mere repository where data is dumped without any thought to its semantic integrity. The predominant characteristic of queries against meta-design tables, as tables designed around our magical four attributes are sometimes called, is that you find the same table invoked a very high number of times in the from clause. Typically, queries will resemble something like: select emp_last_name.entity_id employee_id, emp_last_name.attribute_value last_name, emp_first_name.attribute_value first_name, emp_job.attribute_value job_description, emp_dept.attribute_value department, emp_sal.attribute_value salary from employee_attributes emp_last_name, employee_attributes emp_first_name, employee_attributes emp_job, employee_attributes emp_dept, employee_attributes emp_sal where emp_last_name.entity_id = emp_first_name.entity_id and emp_last_name.entity_id = emp_job.entity_id and emp_last_name.entity_id = emp_dept.entity_id and emp_last_name.entity_id = emp_sal.entity_id and emp_last_name.attribute_name = 'LASTNAME' and emp_first_name.attribute_name = 'FIRSTNAME' and emp_job.attribute_name = 'JOB' and emp_dept.attribute_name = 'DEPARTMENT' and emp_sal.attribute_sal = 'SALARY' order by emp_last_name.attribute_value www.it-ebooks.info 282 C H A P T E R E L E V E N Note how the same table is referenced five times in the from clause. The number of self- joins is usually much higher than in this simple example. Furthermore, such queries are frequently spiced up with outer joins as well. A query with a high number of self-joins performs extremely badly on large volumes; it is clear that the only reason for the numerous conditions in the where clause is to patch all the various “attributes” together. Had the table been defined as the more logical employees(employee_id, last_name, first_name, job_description, department, salary), our query would have been as simple as: select * from employees order by last_name And the best course for executing this query is obviously a plain table scan. The multiple joins and associated index accesses of the query against employee_attributes are performance killers. We can never succeed in making a query run as fast against a rotten design as it will run against a clean design. Any clever rewriting of a SQL query against badly designed tables will be nothing more than a wooden leg, returning only some degree of agility to a crippled query. However, we can often obtain spectacular results in comparison to the multiple joins approach by trying to achieve a single pass on the attribute table. We basically want one row with several attributes (reflecting what the table design should have been in the first place) instead of multiple rows, each with only one attribute of interest per row. Consolidating a multi-row result into a single row is a feat we know how to perform: aggregate functions do precisely this. The idea is therefore to proceed in two steps, as", + "source": "The Art of SQL.pdf", + "chunk_id": 242 + }, + { + "text": "have been in the first place) instead of multiple rows, each with only one attribute of interest per row. Consolidating a multi-row result into a single row is a feat we know how to perform: aggregate functions do precisely this. The idea is therefore to proceed in two steps, as shown in Figure 11-1: 1. Complete each row that contains only one value of interest, with as many dummy values as required to obtain the total number of attributes that we ultimately want. 2. Aggregate the different rows so as to keep only the single value of interest from each (the single value in each column). A function such as max( ), that has the advantage of being applicable to most data types, is perfect for this kind of operation. To be certain that max( ) will only retain meaningful values, we must use dummy values that will necessarily be smaller than any legitimate value we may have in a given column. It is probably better to use an explicit value rather than null as a dummy value, even though max( ) ignores null values according to the standard. If we apply the “recipe” illustrated in Figure 11-1 to our previous example, we can get rid of the numerous joins by writing: select employee_id, max(last_name) last_name, max(first_name) first_name, www.it-ebooks.info S T R A T A G E M S 283 max(job_description) job_desription, max(department) department, max(salary) salary from –- select all the rows of interest, returning -- as many columns as we have rows, one column -- of interest per row and values smaller -- than any value of interest everywhere else (select entity_id employee_id, case attribute_name when 'LASTNAME' then attribute_value else '' end last_name, case attribute_name when 'FIRSTNAME' then attribute_value else '' end first_name, case attribute_name when 'JOB' then attribute_value else '' end job_description, case attribute_name when 'DEPARTMENT' then attribute_value else -1 end department, case attribute_name when 'SALARY' then attribute_value else -1 end salary from employee_attributes where attribute_name in ('LASTNAME', 'FIRSTNAME', 'JOB', 'DEPARTMENT', 'SALARY')) as inner group by inner.employee_id order by 2 FIGURE 11-1. Transmogrification of several rows into one row www.it-ebooks.info 284 C H A P T E R E L E V E N The inner query is not strictly required—we could have used a series of max(case when ... end)—but the query as written makes the two steps appear more clearly. An aggregate is not, as you might expect, the best option in terms of performance. But in the kingdom of the blind, the one-eyed man is king, and this type of query just shown usually has no trouble outperforming one having a monstrous number of self-joins. A word of caution, though: in order to accommodate any unexpectedly lengthy attribute, the attribute_value column is usually a fairly large variable-length string. As a result, the aggregation process may require a significant amount of memory, and in some extreme cases you may run into difficulties if the number of attributes exceeds a few dozen. Multiple self-joins can often be", + "source": "The Art of SQL.pdf", + "chunk_id": 243 + }, + { + "text": "unexpectedly lengthy attribute, the attribute_value column is usually a fairly large variable-length string. As a result, the aggregation process may require a significant amount of memory, and in some extreme cases you may run into difficulties if the number of attributes exceeds a few dozen. Multiple self-joins can often be avoided by retrieving all rows in a single pass, spreading the values across separate columns, and using an aggregate function to collapse the many rows into one. Columns That Should Have Been Rows In contrast to the previous design in which rows have been defined for each attribute, another example of poor design occurs where columns are created instead of individual rows. The classic design mistake made by many beginners is to predefine a fixed number of columns for a number of variables, with some of the columns set to null when values are missing. A typical example is illustrated in Figure 11-2, with a very poorly designed movie database (compare this design to the correct design of Figure 8-3 in Chapter 8). Instead of using a movie_credits table as we did in Chapter 8 to link the movies table to the people table and record the nature of each individual’s involvement, the poor design shown in Figure 11-2 assumes that we will never need to record more than a fixed number of lead actors and one director. The first assumption is blatantly wrong and so is the second one since many sketch comedies have had multiple directors. As a representation of reality, this model is plainly flawed, which should already be sufficient reason to discard it. To make FIGURE 11-2. A badly designed movie database www.it-ebooks.info S T R A T A G E M S 285 matters worse, a poor design, in which data is stored as columns and yet reporting output obviously requires data to be presented as rows, often results in rather confusing queries. Unfortunately, writing queries against poor database designs seems to be as unavoidable as taxes and death in the world of SQL development. When you want different columns to be displayed as rows, you need a pivot table. Pivot tables are used to pivot, or turn sideways, tables where we want to see columns as rows. A pivot table is, in the context of SQL databases, a utility table that contains only one column, filled with incrementing values from 1 to whatever is needed. It can be a true table or a view—or even a query embedded in the from clause of a query. Using such a utility table is a favorite old trick of experienced SQL developers, and the next few subsections show how to create and use them. Creating a pivot table The constructs you have seen in Chapter 7 for walking trees are usually quite convenient for generating pivot table values; for instance, we can use a recursive with action with those database systems that support it. Here is a DB2 example to generate numbers from 1 to 50: with", + "source": "The Art of SQL.pdf", + "chunk_id": 244 + }, + { + "text": "table The constructs you have seen in Chapter 7 for walking trees are usually quite convenient for generating pivot table values; for instance, we can use a recursive with action with those database systems that support it. Here is a DB2 example to generate numbers from 1 to 50: with pivot(row_num) -- Generate 50 values -- 1 to 50, one value per row as (select 1 row_num from sysibm.sysdummy1 union all select row_num + 1 from pivot where row_num < 50) select row_num from pivot; Similar tricks are of course possible with Oracle’s connect by; for instance:* select level from dual connect by level <= 50 Using one of these constructs inside the from clause of a query can make that query particularly illegible, and it is therefore often advisable to use a regular table as pivot. But a recursive query can be useful to fill the pivot table (an alternate solution to fill a pivot table is to use Cartesian joins between existing tables). Typically, a pivot table would hold something like 1,000 rows. * Beware that such a construct may not work with some older versions of Oracle. www.it-ebooks.info 286 C H A P T E R E L E V E N Multiplying rows with a pivot table Now that we have a pivot table, what can we do with it? One way to look at a pivot table is to view it as a row-multiplying device. By combining a pivot to a table we want to see pivoted, we repeat each of the rows of the table to be transformed as many times as we wish. Specifying the number of times we want to see one row repeated is simply a matter of adding to the join a limiting condition on the pivot table, for instance: where pivot.row_num <= multiplying value We can thus multiply the three rows in a test employees table in a very simple way. First, here are the three rows: SQL> select name, job 2 from employees; NAME JOB ---------- ------------------------------ Tom Manager Dick Software engineer Harry Software engineer And now, here is the multiplication, by three, of those rows: SQL> select e.name, e.job, p.row_num 2 from employees e, 3 pivot p 4 where p.row_num <= 3; NAME JOB ROW_NUM ---------- ------------------------------ ---------- Tom Manager 1 Dick Software engineer 1 Harry Software engineer 1 Tom Manager 2 Dick Software engineer 2 Harry Software engineer 2 Tom Manager 3 Dick Software engineer 3 Harry Software engineer 3 9 rows selected. It’s best to index the only column in the pivot table so as not to fully scan this table when you need to use very few rows from it (as in the preceding example). Using pivot table values Besides the mere multiplying effect, the Cartesian join also allows us to associate a unique number in the range 1 to multiplying value for every copy of a row of the table we want to pivot. This value is simply the row_num column contributed by", + "source": "The Art of SQL.pdf", + "chunk_id": 245 + }, + { + "text": "example). Using pivot table values Besides the mere multiplying effect, the Cartesian join also allows us to associate a unique number in the range 1 to multiplying value for every copy of a row of the table we want to pivot. This value is simply the row_num column contributed by the pivot table, and it will enable us in turn to pick from each copy of a row only partial data. The full process of www.it-ebooks.info S T R A T A G E M S 287 multiplication of the source rows and selection is illustrated, with a single row, in Figure 11-3. If we want the initial row to finally appear as a single column (which by the way implicitly requires the data types of col1 ... coln to be consistent), we must pick just one column into each of the rows generated by the Cartesian product. By checking the number coming from the pivot table, we can specify with a case for each resulting row which column is to be displayed to the exclusion of all the others. For instance, we can decide to display col1 if the value coming from the pivot table is 1, col2 if it is 2, and so on. Needless to say, multiplying rows and discarding most of the columns we are dealing with is not the most efficient way of processing data; keep in mind that we are rowing upstream. An ideal database design would avoid the need for such multiplication and discarding. Interestingly, and still in the hypothetical situation of a poor (to put it mildly) database design, a pivot table can in some circumstances bring direct performance benefits. Let’s suppose that, in our badly designed movie database, we want to count how many different actors are recorded (note that none of the actor_... columns are indexed, and that we therefore have to fetch the values from the table). One way to write this query is to use a union: select count(*) from (select actor_1 from movies union select actor_2 from movies union select actor_3 from movies) as m FIGURE 11-3. Pivoting a row www.it-ebooks.info 288 C H A P T E R E L E V E N But we can also pivot the table to obtain something that looks more like a select on the movie_credits table of the properly designed database: select count(distinct actor_id) from –- Use a 3-row pivot to multiply -- the number of rows by 3 -- and return actor_1 the first row in each -- set of 3, actor_2 for the second one -- and actor_3 for the third one (select case pv.row_num when 1 then actor_1 when 2 then actor_2 else actor_3 end actor_id from movies as m, pivot as pv where pv.row_num <= 3) as m The second version runs about twice as fast as the first one—significantly faster. The pivot and unpivot operators As a possibly sad acknowledgment of the generally poor quality of database designs, SQL Server 2005 has introduced", + "source": "The Art of SQL.pdf", + "chunk_id": 246 + }, + { + "text": "actor_id from movies as m, pivot as pv where pv.row_num <= 3) as m The second version runs about twice as fast as the first one—significantly faster. The pivot and unpivot operators As a possibly sad acknowledgment of the generally poor quality of database designs, SQL Server 2005 has introduced two operators called pivot and unpivot to perform the toppling of rows into columns and vice-versa, respectively. The previous employee_ attributes example can be written as follows using the pivot operator: select entity_id as employee_id, [lastname], [firstname], [job], [department], [salary] from employee_attributes as employees pivot (max(attribute_value) for attribute_name in ([lastname], [firstname], [job], [department], [salary]) as pivoted_employees order by 2 The specific values in the attribute_name column that we want to appear as columns are listed in the for ... in clause, using a particular syntax that transforms the character data into column identifiers. There is an implicit group by applied to all the columns from employee_attributes that are not referenced in the pivot clause; we must be careful if we have other columns (for instance, an attribute_type column) than entity_id, as they may require an additional aggregation layer. The unpivot operator performs the reverse operation, and allows us to see the link between movie and actor as a more logical collection of (movie_id, actor_id) pairs by writing: www.it-ebooks.info S T R A T A G E M S 289 select movie_id, actor_type, actor_id from movies unpivot (actor_id for actor_type in ([actor_1], [actor_2], [actor_3])) as movie_actors Note that this query doesn’t exactly produce the result we want, since it introduces the name of the original column as a virtual actor_type column. There is no need to qualify actors as actor_1, actor_2, or actor_3, and once again the query may need to be wrapped into another query that only returns movie_id and actor_id. The use of a pivot table, or of the pivot and unpivot operators, is a very interesting technique that can help extricate us from more than one quagmire. The support for pivoting operators by major database systems is not, of course, to be interpreted as an endorsement of bad design, but as an example of realpolitik. Pivot tables and operators can be a useful technique in their own right, but they should never be used as a means of glossing over the inadequacies of a bad design. Single Columns That Should Have Been Something Else Some designers of our movie database may well have been sensitive to the limitation on the number of actors we may associate with one movie. Trying to solve design issues with a creative use of irrelevant techniques, someone may have come up with a “bright idea”: what about storing the actor identifiers as a comma-separated string in one wide actors column? For instance: first actor id, second actor id, ... And so much for the first normal form.... The big design mistake here is to store several pieces of data that we need to handle one by one into one column. There would be", + "source": "The Art of SQL.pdf", + "chunk_id": 247 + }, + { + "text": "comma-separated string in one wide actors column? For instance: first actor id, second actor id, ... And so much for the first normal form.... The big design mistake here is to store several pieces of data that we need to handle one by one into one column. There would be no issue if a complex string—for instance a lengthy XML message—were considered as an opaque object by the DBMS and handled as if it were an atomic item. But that’s not the case here. Here we have several values in one column, and we do want to treat and manipulate each value individually. We are in trouble. There are only two workable solutions with a creative design of this sort: • Scrapping it and rewriting everything. This is, of course, by far the best solution. • When delays, costs, and politics require a fast solution, the only way out may be to apply a creative SQL solution; once again, let me state that “solution” is probably not the best choice of words in this case, “fix” would be a better description. www.it-ebooks.info 290 C H A P T E R E L E V E N I’ll also point out that a more elaborate version of the same mistake could use an “XML type” column; I am going to use simple character-string manipulation functions in my example, but they could as well be XML-extracting functions. NOTE Be warned: “creative SQL” is often a euphemism for ugly SQL! First normal form on the fly Our problem is to extract various individual components from a string of characters and return them one by one on separate rows. This is easier with some database systems (for instance, Oracle has a very rich set of string functions that noticeably eases the work) than with others. Conventions such as systematically starting or ending the string with a comma may further help us. We are not wimps, but real SQL developers, and we are therefore going to take the north face route and assume the worst: 1. First, let’s assume that our lists of identifiers are in the following form: id1,id2,id3, ..., idn 2. Second, we shall also assume that the only sets of functions at our disposal are those common to the major database systems. We shall use Transact-SQL for our example and only use built-in functions. As you will see, a well-designed user function might ease both the writing and the performance of the resulting query. Let’s start with a (very small) movies table in which a list of actor identifiers is (wrongly) stored as an attribute of the movie: 1> select movie_id, actors 2> from movies 3> go movie_id actors --------------------- ---------------------------------------- 1 123,456,78,96 2 23,67,97 3 67,456 (3 rows affected) The first step is to use as many rows from our pivot table as we may have characters in the actors string—arbitrarily set to a maximum length of 50 characters. We are going to multiply the number of rows in the movies", + "source": "The Art of SQL.pdf", + "chunk_id": 248 + }, + { + "text": "123,456,78,96 2 23,67,97 3 67,456 (3 rows affected) The first step is to use as many rows from our pivot table as we may have characters in the actors string—arbitrarily set to a maximum length of 50 characters. We are going to multiply the number of rows in the movies table by this number, 50. We would naturally be rather reluctant to do something similar on millions of rows (as an aside, a function allowing us to return the position of the nth separator or the nth item in the string would make it necessary to multiply only by the maximum number of identifiers we can encounter, instead of by the maximum string length). www.it-ebooks.info S T R A T A G E M S 291 Our next move is to use the substring( ) function to successively get subsets (that can be null) of actors, starting at the first character, then moving to the second, and so forth, up to the last character (at most, the 50th character). We just have to use the row_num value from the pivot table to find the starting character of each substring. If we take for instance the string from the actors column that is associated to the movie identified by the value 1 for movie_id, we shall get something like: 123,456,78,96 associated to the row_num value 1 23,456,78,96 associated to the row_num value 2 3,456,78,96 associated to the row_num value 3 ,456,78,96 .... 456,78,96 56,78,96 6,78,96 ,78,96 78,96 .... We’ll compute these subsets in a column that we’ll call substring1. Having these successive substrings, we can now check the position of the first comma in them. Our next move is to return as a column called substring2 the content of substring1 shifted by one position. We also locate the position of the first comma in substring2. These operations are illustrated in Figure 11-4. Among the various resulting rows, the only ones to be of interest are those marking the beginning of a new identifier in the string: the first row in the series that is associated with the row_num value of 1, and all the rows for which we find a comma in first position of substring1. For all these rows, the position of the comma in substring2 tells us the length of the identifier that we are trying to isolate. FIGURE 11-4. Splitting-up a comma separated list www.it-ebooks.info 292 C H A P T E R E L E V E N Translated into SQL code, here is what we get: 1> select row_num, 2> movie_id, 3> actors, 4> first_sep, 5> next_sep 6> from (select row_num, 7> movie_id, 8> actors, 9> charindex(',', substring(actors, row_num, 10> char_length(actors))) first_sep, 11> charindex(',', substring(actors, row_num + 1, 12> char_length(actors))) + 1 next_sep 13> from movies, 14> pivot 15> where row_num <= 50) as q 16> where row_num = 1 17> or first_sep = 1 18> go row_num movie_id actors first_sep next_sep ----------- -------------- ------------------ ----------- ----------- 1 1 123,456,78,96 4 4 4 1 123,456,78,96 1", + "source": "The Art of SQL.pdf", + "chunk_id": 249 + }, + { + "text": "+ 1, 12> char_length(actors))) + 1 next_sep 13> from movies, 14> pivot 15> where row_num <= 50) as q 16> where row_num = 1 17> or first_sep = 1 18> go row_num movie_id actors first_sep next_sep ----------- -------------- ------------------ ----------- ----------- 1 1 123,456,78,96 4 4 4 1 123,456,78,96 1 5 8 1 123,456,78,96 1 4 11 1 123,456,78,96 1 1 1 2 23,67,97 3 3 3 2 23,67,97 1 4 6 2 23,67,97 1 1 1 3 67,456 3 3 3 3 67,456 1 1 (9 rows affected) If we accept that we must take some care to remove commas, and the particular cases of both the first and last identifiers in a list, getting the various identifiers is then reasonably straightforward, even if the resulting code is not for the faint-hearted: 1> select movie_id, 2> actors, 3> substring(actors, 4> case row_num 5> when 1 then 1 6> else row_num + 1 7> end, 8> case next_sep 9> when 1 then char_length(actors) 10> else 11> case row_num 12> when 1 then next_sep - 1 13> else next_sep - 2 14> end 15> end) as id 16> from (select row_num, www.it-ebooks.info S T R A T A G E M S 293 17> movie_id, 18> actors, 19> first_sep, 20> next_sep 21> from (select row_num, 22> movie_id, 23> actors, 24> charindex(',', substring(actors, row_num, 25> char_length(actors))) first_sep, 26> charindex(',', substring(actors, row_num + 1, 27> char_length(actors))) + 1 next_sep 28> from movies, 29> pivot 30> where row_num <= 50) as q 31> where row_num = 1 32> or first_sep = 1) as q2 33> go movie_id actors id ---------------- ------------------------------ ----------------- 1 123,456,78,96 123 1 123,456,78,96 456 1 123,456,78,96 78 1 123,456,78,96 96 2 23,67,97 23 2 23,67,97 67 2 23,67,97 97 3 67,456 67 3 67,456 456 (9 rows affected) We could have made the code slightly simpler by prepending and appending a comma to the actors column. I leave doing that as an exercise for the undaunted reader. Note that as the left alignment shows, the resulting id column is a string and should be explicitly converted to numeric before joining to the table that stores the actors’ names. The preceding case, besides being an interesting example of solving a SQL problem by successively wrapping queries, also comes as a healthy warning of what awaits us on the SQL side of things when tables are poorly designed. Lifting the veil on the Chapter 7 mystery path explosion You may remember that in Chapter 7 I described the materialized path model for tree representations. In that chapter I noted that it would be extremely convenient if we could “explode” a materialized path into the different materialized paths of all its ancestors. The advantage of this method is that when we want to walk a hierarchy from the bottom up, we can make efficient use of the index that should hopefully exist on the materialized path. If we don’t “explode” the materialized path, the only way we have to find the ancestors of", + "source": "The Art of SQL.pdf", + "chunk_id": 250 + }, + { + "text": "advantage of this method is that when we want to walk a hierarchy from the bottom up, we can make efficient use of the index that should hopefully exist on the materialized path. If we don’t “explode” the materialized path, the only way we have to find the ancestors of a given row is to specify a condition such as: and offspring.materialized_path like concat(ancestor.materialized_path, '%') www.it-ebooks.info 294 C H A P T E R E L E V E N Sadly, this is a construct that cannot use the index (for reasons that are quite similar to those in the credit card prefix problem of Chapter 8). How can we “explode” the materialized path? The time has come to explain how we can pull that rabbit out of the hat. Since our node will have, in the general case, several ancestors, the very first thing we have to do is to multiply the rows by the number of preceding generations. In this way we’ll be able to extract from the materialized path of our initial row (for example the row that represents the Hussar regiment under the command of Colonel de Marbot) the paths of the various ancestors. As always, the solution for multiplying rows is to use a pivot table. If we do it this time with MySQL, there is a function called substring_index( ) that very conveniently returns the substring of its first argument from the beginning up to the third argument occurrence of the second argument (hopefully, the example is easier to understand). To know how many rows we need from the pivot table, we just compute how many elements we have in the path in exactly the same way that we computed the depth in Chapter 7, namely by comparing the length of the path to the length of the same when separators have been stripped off. Here is the query, and the results: mysql> select mp.materialized_path, -> substring_index( mp.materialized_path, '.', p.row_num ) -> as ancestor_path -> from materialized_path_model as mp, -> pivot as p -> where mp.commander = 'Colonel de Marbot' -> and p.row_num <= 1 + length( mp.materialized_path ) -> - length(replace(mp.materialized_path, '.', '')); +-------------------+---------------+ | materialized_path | ancestor_path | +-------------------+---------------+ | F.1.5.1.1 | F | | F.1.5.1.1 | F.1 | | F.1.5.1.1 | F.1.5 | | F.1.5.1.1 | F.1.5.1 | | F.1.5.1.1 | F.1.5.1.1 | +-------------------+---------------+ 5 rows in set (0.00 sec) Querying with a Variable in List There is another, and rather important, use of pivot tables that I must now mention. In previous chapters I have underlined the importance of binding variables, in other words of passing parameters to SQL queries. Variable binding allows the DBMS kernel to skip the parsing phase (in other words, the compilation of the statement) after it has done it once. Keep in mind that parsing includes steps as potentially costly as the search for the best execution path. Even when SQL statements are dynamically constructed, it is quite possible, as you have seen in Chapter", + "source": "The Art of SQL.pdf", + "chunk_id": 251 + }, + { + "text": "phase (in other words, the compilation of the statement) after it has done it once. Keep in mind that parsing includes steps as potentially costly as the search for the best execution path. Even when SQL statements are dynamically constructed, it is quite possible, as you have seen in Chapter 8, to pass variables to them. There is, however, one www.it-ebooks.info S T R A T A G E M S 295 difficult case: when the end user can make multiple choices out of a combo box and pass a variable number of parameters for use in an in list. The selection of multiple values raises several issues: • Dynamically binding a variable number of parameters may not be possible with all languages (often you must bind all variables at once, not one by one) and will, in any case, be rather difficult to code. • If the number of parameters is different for almost every call, two statements that only differ by the number of bind variables will be considered to be different statements by the DBMS, and we shall lose the benefit of variable binding. The ability provided by pivot tables to split a string allows us to pass a list of values as a single string to the statement, irrespective of the actual number of values. This is what I am going to demonstrate with Oracle in this section. The following example shows how most developers would approach the problem of passing a list of values to an in list when that list of values is contained within a single string. In our case the string is v_list, and most developers would concatenate several strings together, including v_list, to produce a complete select statement: v_statement := 'select count(order_id)' || ' from order_detail' || ' where article_id in (' || v_list || ')'; execute immediate v_statement into n_count; This example looks dynamic, but for the DBMS it’s in fact all hardcoded. Two successive executions will each be different statements, both of which will have to be parsed before execution. Can we pass v_list as a parameter to the statement, instead of concatenating it into the statement? We can, by applying exactly the same techniques to the comma- separated value stored in variable v_list as we have applied to the comma-separated value stored in column actors in the example of on-the-fly normalization. A pivot table allows us to write the following somewhat wilder SQL statement: select count(od.order_id) into n_count from order_detail od, ( -- Return at many rows as we have items in the list -- and use character functions to return the nth item -- on the nth row select to_number(substr(v_list, case row_num when 1 then 1 else 1 + instr(v_list, ',', 1, row_num - 1) end, case instr(v_list, ',', 1, row_num) when 0 then length(v_list) else www.it-ebooks.info 296 C H A P T E R E L E V E N case row_num when 1 then instr(v_list, ',', 1, row_num) - 1 else instr(v_list, ',', 1, row_num)", + "source": "The Art of SQL.pdf", + "chunk_id": 252 + }, + { + "text": "1 + instr(v_list, ',', 1, row_num - 1) end, case instr(v_list, ',', 1, row_num) when 0 then length(v_list) else www.it-ebooks.info 296 C H A P T E R E L E V E N case row_num when 1 then instr(v_list, ',', 1, row_num) - 1 else instr(v_list, ',', 1, row_num) - 1 - instr(v_list, ',', 1, row_num - 1) end end)) article_id from pivot where instr(v_list||',', ',', 1, row_num) > 0 and row_num <= 250) x where od.article_id = x.article_id; You may need, if you are really motivated, to study this query a bit to figure out how it all works. The mechanism is all based on repeated use of the Oracle function instr( ). Let me just say that this function instr(haystack, needle, from_pos, count) returns the countth occurrence of needle in haystack starting at position from_pos (0 is returned when nothing is found), but the logic is exactly the same as with the previous examples. I have run the pivot and hardcoded versions of the query successively 1, 10, 100, 1,000, 10,000, and 100,000 times. Each time, I randomly generated a list of from 1 to 250 v_list values. The results are shown in Figure 11-5, and they are telling: the “pivoted” list is 30% faster as soon as the query is repeatedly executed. Remember that the execution of a hardcoded query requires parsing and then execution, while a query that takes parameters (bind variables) can be re-executed subsequently for only a marginal cost of the first execution. Even if this later query is noticeably more complicated, as long as the execution is faster than execution plus parsing for the hardcoded query, the later query wins hands-down in terms of performance. FIGURE 11-5. Performance of a hardcoded list versus a list transformed with a pivot table www.it-ebooks.info S T R A T A G E M S 297 There are actually two other benefits that don’t show up in Figure 11-5: • Parsing is a very CPU-intensive operation. If CPU happens to be the bottleneck, hard- coded queries can be extremely detrimental to other queries. • SQL statements are cached whether they contain parameters or whether they are totally hardcoded, because you can imagine having hardcoded statements that are repeatedly executed by different users, and it makes sense for the SQL engine to anticipate such a situation. To take, once again, the movie database example, even if the names of actors are hardcoded, a query referring to a very popular actor or actress could be executed a large number of times.* The SQL engine will therefore cache hardcoded statements like the others. Unfortunately, a repeatedly executed hardcoded statement is the exception rather than the rule. As a result, a succession of dynamically built hardcoded statements that may each be executed only once or a very few times will all accumulate in the cache before being overwritten as a result of the normal cache management activity. This cache management will require more work and is therefore an additional price", + "source": "The Art of SQL.pdf", + "chunk_id": 253 + }, + { + "text": "a succession of dynamically built hardcoded statements that may each be executed only once or a very few times will all accumulate in the cache before being overwritten as a result of the normal cache management activity. This cache management will require more work and is therefore an additional price to pay. Aggregating by Range (Bands) Some people have trouble writing SQL queries that return aggregates for bands. Such queries are actually quite easy to write using the case construct. By way of example, look at the problem of reporting on the distribution of tables by their total row counts. For instance, how many tables contain fewer than 100 rows, how many contain 100 to 10,000 rows, how many 10,000 to 1,000,000 rows, and how many tables store more than 1,000,000 rows? Information about tables is usually accessible through data dictionary views: for instance, INFORMATION_SCHEMA.TABLES, pg_statistic, and pg_tables, dba_tables, syscat.tables, sysobjects and systabstats, and so on. In my explanation here, I’ll assume the general case of a view named table_info, containing, among other things, the columns table_name and row_count. Using this table, a simple use of case and the suitable group by can give us the distribution by row_count that we are after: select case when row_count < 100 then 'Under 100 rows' when row_count >= 100 and row_count < 10000 then '100 to 10000' when row_count >= 10000 and row_count < 1000000 then '10000 to 1000000' else 'Over 1000000 rows' end as range, count(*) as table_count from table_info * Actually, the best optimization tactic in this particular case would be to cache the result of the query rather than the query. www.it-ebooks.info 298 C H A P T E R E L E V E N where row_count is not null group by case when row_count < 100 then 'Under 100 rows' when row_count >= 100 and row_count < 10000 then '100 to 10000' when row_count >= 10000 and row_count < 1000000 then '10000 to 1000000' else 'Over 1000000 rows' end There is only one snag here: group by performs a sort before aggregating data. Since we are associating a label with each of our aggregates, the result is, by default, alphabetically sorted on that label: RANGE TABLE_COUNT ----------------- ------------ 100 to 10000 18 10000 to 1000000 15 Over 1000000 rows 6 Under 100 rows 24 The ordering that would be logical to a human eye in such a case is to see Under 100 rows appear first, and then each band by increasing number of rows, with Over 1,000,000 rows coming last. Rather than trying to be creative with labels, the stratagem to solve this problem consists of two steps: 1. Performing the group by on two, instead of one, columns, associating with each label a dummy column, the only purpose of which is to serve as a sort key 2. Wrapping up the query as a query within the from clause, so as to mask the sort key thus created and ensure that only the data of", + "source": "The Art of SQL.pdf", + "chunk_id": 254 + }, + { + "text": "of one, columns, associating with each label a dummy column, the only purpose of which is to serve as a sort key 2. Wrapping up the query as a query within the from clause, so as to mask the sort key thus created and ensure that only the data of interest is returned Here is the query that results from applying the preceding two steps: select row_range, table_count from ( -- Build a sort key to have bands suitably ordered -- and hide it inside a subquery select case when row_count < 100 then 1 when row_count >= 100 and row_count < 10000 then 2 when row_count >= 10000 and row_count < 1000000 then 3 else 4 end as sortkey, case when row_count < 100 then 'Under 100 rows' when row_count >= 100 and row_count < 10000 then '100 to 10000' www.it-ebooks.info S T R A T A G E M S 299 when row_count >= 10000 and row_count < 1000000 then '10000 to 1000000' else 'Over 1000000 rows' end as row_range, count(*) as table_count from table_info where row_count is not null group by case when row_count < 100 then 'Under 100 rows' when row_count >= 100 and row_count < 10000 then '100 to 10000' when row_count >= 10000 and row_count < 1000000 then '10000 to 1000000' else 'Over 1000000 rows' end, case when row_count < 100 then 1 when row_count >= 100 and row_count < 10000 then 2 when row_count >= 10000 and row_count < 1000000 then 3 else 4 end) dummy order by sortkey; And following are the results from executing that query: ROW_RANGE TABLE_COUNT ----------------- ----------- Under 100 rows 24 100 to 10000 18 10000 to 1000000 15 Over 1000000 rows 6 Aggregating by range (bands) requires building an artificial sort key to display results in desired order. Superseding a General Case The technique of hiding a sort key within a query in the from clause, which I used in the previous section to display bands, can also be helpful in other situations. A particularly important case is when a table contains the definition of a general rule that happens to be superseded from time to time by a particular case defined in another table. I’ll illustrate by example. www.it-ebooks.info 300 C H A P T E R E L E V E N I mentioned in Chapter 1 that the handling of various addresses is a difficult issue. Let’s take the case of an online retailer, one that knows at most two addresses for each customer: a billing address and a shipping address. In most cases, the two addresses are the same. The retailer has decided to store the mandatory billing address in the customers table and to associate the customer_id identifier with the various components of the address (line_1, line_2, city, state, postal_code, country) in a different shipping_ addresses table for those few customers for whom the two addresses differ. The wrong way to get the shipping address when you know the customer identifier is", + "source": "The Art of SQL.pdf", + "chunk_id": 255 + }, + { + "text": "and to associate the customer_id identifier with the various components of the address (line_1, line_2, city, state, postal_code, country) in a different shipping_ addresses table for those few customers for whom the two addresses differ. The wrong way to get the shipping address when you know the customer identifier is to execute two queries: 1. Look for a row in shipping_addresses. 2. If nothing is found, then query customers. An alternate way to approach this problem is to apply an outer join on shipping_addresses and customers. You will then get two addresses, one of which will in most cases be a suite of null values. Either you check programmatically if you indeed have a valid shipping address, which is a bad solution, or you might imagine using the coalesce( ) function that returns its first non-null argument: select ... coalesce(shipping_address.line_1, customers.line_1), ... Such a use of coalesce( ) would be a very dangerous idea, because it implicitly assumes that all addresses have exactly the same number of non-null components. If you suppose that you do indeed have a different shipping address, but that its line_2 component is null while the line_2 component of the billing address is not, you may end up with a resulting invalid address that borrows components from both the shipping and billing addresses. A correct approach is to use case to check for a mandatory component from the address—which admittedly can result in a somewhat difficult to read query. An even better solution is probably to use the “hidden sort key” technique, combined with a limit on the number of rows returned (select top 1 ..., limit 1, where rownum = 1 or similar, depending on the DBMS) and write the query as follows: select * from (select 1 as sortkey, line_1, line_2, city, state, postal_code, country from shipping_addresses where customer_id = ? union select 2 as sortkey, line_1, www.it-ebooks.info S T R A T A G E M S 301 line_2, city, state, postal_code, country from customers where customer_id = ? order by 1) actual_shipping_address limit 1 The basic idea is to use the sort key as a preference indicator. The limit set on the number of rows returned will therefore ensure that we’ll always get the “best match” (note that similar ideas can be applied to several rows when a row_number( ) OLAP function is available). This approach greatly simplifies processing on the application program side, since what is retrieved from the DBMS is “certified correct” data. The technique I’ve just described can also be used in multilanguage applications where not everything has been translated into all languages. When you need to fetch a message, you can define a default language and be assured that you will always get at least some message, thus removing the need for additional coding on the application side. Selecting Rows That Match Several Items in a List An interesting problem is that of how to write queries based on some criteria referring to a varying list of values.", + "source": "The Art of SQL.pdf", + "chunk_id": 256 + }, + { + "text": "that you will always get at least some message, thus removing the need for additional coding on the application side. Selecting Rows That Match Several Items in a List An interesting problem is that of how to write queries based on some criteria referring to a varying list of values. This case is best illustrated by looking for employees who have certain skills, using the three tables shown in Figure 11-6. The skillset table links employees to skills, associating a 1 to 3 skill_level value to distinguish between honest competency, strong experience, and outright wizardry. Finding employees that have a level 2 or 3 SQL skill is easy enough: select e.employee_name from employees e where e.employee_id in (select ss.employee_id from skillset ss, skills s where s.skill_id = ss.skill_id and s.skill_name = 'SQL' and ss.skill_level >= 2) order by e.employee_name FIGURE 11-6. Tables used for querying employee skills www.it-ebooks.info 302 C H A P T E R E L E V E N (We can also write the preceding query with a simple join.) If we want to retrieve the employees who are competent with Oracle or DB2, all we need to do is write: select e.employee_name, s.skill_name, ss.skill_level from employees e, skillset ss, skills s where e.employee_id = ss.employee_id and s.skill_id = ss.skill_id and s.skill_name in ('ORACLE', 'DB2') order by e.employee_name No need to test for the skill level, since we will accept any level. However, we do need to display the skill name; otherwise, we won’t be able to tell why a particular employee was returned by the query. We also encounter a first difficulty, namely that people who are competent in both Oracle and DB2 will appear twice. What we can try to do is to aggregate skills by employee. Unfortunately, not all SQL dialects provide an aggregate function for concatenating strings (you can sometimes write it as a user-defined aggregate function, though). We can nevertheless perform a skill aggregate by using the simple stratagem of a double conversion. First we convert our value from string to number, then from number back to string once we have aggregated numbers. Skill levels are in the 1 through 3 range. We can therefore confidently represent any combination of Oracle and DB2 skills by a two-digit number, assigning for instance the first digit to DB2 and the second one to Oracle. This is easily done as follows: select e.employee_name, (case s.skill_name when 'DB2' then 10 else 1 end) * ss.skill_level as computed_skill_level from employees e, skillset ss, skills s where e.employee_id = ss.employee_id and s.skill_id = ss.skill_id and s.skill_name in ('ORACLE', 'DB2') computed_skill_level will result in 10, 20, or 30 for DB2 skill levels, while Oracle skill levels will remain 1, 2, and 3. We then can very easily aggregate our skill levels, and convert them back to a more friendly description: select employee_name, -- Decode the numerically encoded skill + skill level combination -- Tens are DB2 skill levels, and units Oracle skill levels case when aggr_skill_level >= 10 then", + "source": "The Art of SQL.pdf", + "chunk_id": 257 + }, + { + "text": "2, and 3. We then can very easily aggregate our skill levels, and convert them back to a more friendly description: select employee_name, -- Decode the numerically encoded skill + skill level combination -- Tens are DB2 skill levels, and units Oracle skill levels case when aggr_skill_level >= 10 then 'DB2:' + str(round(aggr_skill_level/10,0)) + ' ' end + case when aggr_skill_level % 10 > 0 then 'Oracle:' + str(aggr_skill_level % 10) end as skills www.it-ebooks.info S T R A T A G E M S 303 from (select e.employee_name, -- Numerically encode skill + skill level -- so that we can aggregate them sum((case s.skill_name when 'DB2' then 10 else 1 end) * ss.skill_level) as aggr_skill_level from employees e, skillset ss, skills s where e.employee_id = ss.employee_id and s.skill_id = ss.skill_id and s.skill_name in ('ORACLE', 'DB2') group by e.employee_name) as encoded_skills order by employee_name But now let’s try to answer a more difficult question. Suppose that the project we want to staff happens to be a migration from one DBMS to another one. Instead of finding people who know Oracle or DB2, we want people who know both Oracle and DB2. We have several ways to answer such a question. If the SQL dialect we are using supports it, the intersect operator is one solution: we find people who are skilled on Oracle on one hand, people who are skilled on DB2 on the other hand, and keep the happy few that belong to both sets. We certainly can also write the very same query with an in( ): select e.employee_name from employees e, skillset ss, skills s where s.skill_name = 'ORACLE' and s.skill_id = ss.skill_id and ss.employee_id = e.employee_id and e.employee_id in (select ss2.employee_id from skillset ss2, skills s2 where s2.skill_name = 'DB2' and s2.skill_id = ss2.skill_id) We can also use the double conversion solution and filter on the numerical aggregate by using the same expressions as we have been using for decoding the encoded_skills computed column. The double conversion stratagem has other advantages: • It hits tables only once. • It makes it easier to handle more complicated questions such as “people who know Oracle and Java, or MySQL and PHP.” • As we are only using a list of skills, we can use a pivot table and bind the list, thus improving performance of oft-repeated queries. The row_num pivot table column can help us encode since, if the list is reasonably short, we can multiply the skill_level value by 10 raised to the (row_num –1)th power. If we don’t care about the exact value of the skill level, and our DBMS implements bit-wise aggregate functions, we can even try to dynamically build a bit-map. www.it-ebooks.info 304 C H A P T E R E L E V E N Finding the Best Match Let’s conclude our adventures in the SQL wilderness by combining several of the techniques shown in this chapter and try to select employees on the basis of some rather complex and fuzzy conditions. We", + "source": "The Art of SQL.pdf", + "chunk_id": 258 + }, + { + "text": "H A P T E R E L E V E N Finding the Best Match Let’s conclude our adventures in the SQL wilderness by combining several of the techniques shown in this chapter and try to select employees on the basis of some rather complex and fuzzy conditions. We want to find, from among our employees, that one member of staff who happens to be the best candidate for a project that requires a range of skills across several different environments (for example, Java, .NET, PHP, and SQL Server). The ideal candidate is a guru in all environments; but if we issue a query asking for the highest skill level everywhere it shall probably return no row. In the absence of the ideal candidate, we are usually left with imperfect candidates, and we must identify someone who has the best competency in as many of our environments as possible and is therefore the best suited for the project. For instance, if our Java guru is a world expert, but knows nothing of PHP, that person is unlikely to be selected. “Best suited” implies a comparison between the various employees, or, in other words, a sort, from which the winner will emerge. Since we want only one winner, we shall have to limit the output of our list of candidates to the first row. You should already be beginning to see the query as a select ... from (select ... order by) limit 1 or whatever your SQL dialect permits. The big question is, of course, how we are going to order the employees. Who is going to get the preference between one who has a decent knowledge of three of the specified topics, and one who is an acknowledged guru of two subjects? It is likely, in a case such as we are discussing, that the width of knowledge is what matters more to us than the depth of knowledge. We can use a major sort key on the number of skills from the requirement list that are mastered, and a minor sort key on the sum of the various skill_level values by employee for the skills in the requirement list. Our inner query comes quite naturally: select e.employee_name, count(ss.skill_id) as major_key, sum(ss.skill_level) as minor_key from employees e, skillset ss, skills s where s.skill_name in ('JAVA', '.NET', 'PHP', 'SQL SERVER') and s.skill_id = ss.skill_id and ss.employee_id = e.employee_id group by e.employee_name order by 2, 3 This query, however, doesn’t tell us anything about the actual skill level of our best candidate. We should therefore combine this query with a double conversion to get an encoding of skills. I leave doing that as an exercise, assuming that you have not yet reached a semi-comatose state. www.it-ebooks.info S T R A T A G E M S 305 You should also note, from a performance standpoint, that we need not refer to the employees table in the inner query. The employee name is information that we need only when we display", + "source": "The Art of SQL.pdf", + "chunk_id": 259 + }, + { + "text": "yet reached a semi-comatose state. www.it-ebooks.info S T R A T A G E M S 305 You should also note, from a performance standpoint, that we need not refer to the employees table in the inner query. The employee name is information that we need only when we display the final result. We should therefore handle only employee_id values, and do the bulk of the processing using the tables skills and skillset. You should also think about the rare situation in which two candidates have exactly the same skills—do you really want to restrict output to one row? NOTE To paraphrase General Robert E. Lee, “It is well that SQL is so terrible, or we should grow too fond of it.” Optimizer Directives I shall conclude this chapter with a cautionary note about optimizer directives. An SQL optimizer can be compared to the program that computes shutter speed and exposure in an automated camera. There are conditions when the “auto” mode is no longer appropriate—for instance, when the subject of the picture is backlit or for the shooting of night scenes. Similarly, all database systems provide one way or another to override or at least direct decisions taken by the query optimizer in its quest for the Dream Execution Path. There are basically two techniques to constrain the optimizer: • Special settings in the session environment that are applied to all queries executed in the session until further notice. • Local directives explicitly written into individual statements. In the latter case the syntax between products varies, since you may have these directives written as an inherent part of the SQL statement (for instance force index(...) with MySQL or option loop join with Transact-SQL), or written as a special syntax comment (such as /*+ all_rows */ with Oracle). Optimizer directives have so far been mostly absent from this book, and for good reasons. Repeatedly executing queries against living data is, to some degree, similar to repeatedly photographing the same subject at various times of day: what is backlit in the morning may be in full light in the afternoon. Directives are destined to override particular quirks in the behavior of the optimizer and are better left alone. The most admissible directives are those directives specifying either the expected outcome, such as sql_small_result or sql_big_result with MySQL, or whether we are more interested in a fast answer, as is generally the case in transactional processing, with directives such as option fast 100 with SQL Server or /*+ first_rows(100) */ with Oracle. These directives, which we could compare to the “landscape” or “sports” mode of a camera, provide the optimizer with information that it would not otherwise be able to gather. They are directives that don’t www.it-ebooks.info 306 C H A P T E R E L E V E N depend on the volume or distribution of data; they are therefore stable in time, and they do add value. In any case, even directives that add value should not be employed", + "source": "The Art of SQL.pdf", + "chunk_id": 260 + }, + { + "text": "are directives that don’t www.it-ebooks.info 306 C H A P T E R E L E V E N depend on the volume or distribution of data; they are therefore stable in time, and they do add value. In any case, even directives that add value should not be employed unless they are required. The optimizer is able to determine a great deal about the best way to proceed when it is given a properly written query in the first place. The best and most simple example of implicit guidance of the optimizer is possibly the use of correlated versus uncorrelated subqueries. They are to be used under dissimilar circumstances to achieve functionally identical results. One of the nicest features of database optimizers is their ability to adapt to changing circumstances. Freezing their behavior by using constraining directives is indicative of a very short-term view that can be potentially damaging to performance in the future. Some directives are real time-bombs, such as those specifying indexes by name. If, for one reason or another a DBA renames an index used in a directive, the result can be disastrous. We can get a similarly catastrophic effect when a directive specifies a composite index, and this index is rebuilt one day with a different column order. NOTE Optimizer directives must be considered the private territory of database administrators. The DBA should use them to cope with the shortcomings of a particular DBMS release and then remove them if at all possible after the next upgrade. Let me add that it is common to see inexperienced developers trying to derive a query from an existing one. When the original query contains directives, beginners rarely bother to question whether these directives are appropriate to their new case. Beginners simply apply what they see as minor changes to the select list and the search criteria. As a result, you end up with queries that look like they have been fine-tuned, but that often follow a totally irrelevant execution path. The good plan that is forced upon a query today may be disastrous tomorrow. www.it-ebooks.info Chapter 12. C H A P T E R T W E L V E Employment of Spies Monitoring Performance And he that walketh in darkness knoweth not whither he goeth. Gospel according to St. John, 12:35 www.it-ebooks.info 308 C H A P T E R T W E L V E I ntelligence gathering has always been an essential part of war. All database systems include monitoring facilities, each with varying degrees of sophistication. Third-party offerings are also available in some cases. All these monitoring facilities are primarily aimed at database administrators. However, when they allow you to really see what is going on inside the SQL engine, they can become formidable spies in the service of the performance-conscious developer. I should note that when monitoring facilities lack the level of detail we require, it is usually possible to obtain additional information by turning on logging or tracing. Logging or tracing", + "source": "The Art of SQL.pdf", + "chunk_id": 261 + }, + { + "text": "is going on inside the SQL engine, they can become formidable spies in the service of the performance-conscious developer. I should note that when monitoring facilities lack the level of detail we require, it is usually possible to obtain additional information by turning on logging or tracing. Logging or tracing necessarily entails a significant overhead, which may not be a very desirable extra load on a busy production server that is already painfully clunking along. But during performance testing, logging can provide us with a wealth of information on what to expect in production. Detailing all or even some of the various monitoring facilities available would be both tedious and product-specific. Furthermore, such an inventory would be rapidly outdated. I shall concentrate instead on what we should monitor and why. This will provide you with an excellent opportunity for a final review of some of the key concepts introduced in previous chapters. The Database Is Slow Let’s first try to define the major categories of performance issues that we are likely to encounter in production—since our goal, as developers, is to anticipate and, if possible, avoid these situations. The very first manifestation of a performance issue on a production database is often a call to the database administrators’ desk to say that “the database is slow” (a useful piece of information for database administrators who may have hundreds of database servers in their care...). In a well-organized shop, the DBA will be able to check whether a monitoring tool does indeed report something unusual, and if that is the case, will be able to answer confidently “I know. We are working on the case.” In a poorly organized shop, the DBA may well give the same answer, lying diplomatically. In all cases, the end of the call will mean the beginning of a frantic scramble for clues. Such communications stating that “the database is slow” will usually have been motivated by one of the five following reasons: It’s not the database The network is stuttering or the host is totally overloaded by something else. Thanks for calling. Sudden global sluggishness All tasks slow down, suddenly, for all users. There are two cases to consider here: www.it-ebooks.info E M P L O Y M E N T O F S P I E S 309 • Either the performance degradation is really sudden, in which case it can often be traced to some system or DBMS change (software upgrade, parameter adjustment, or hardware configuration modification). • Or it results from a sudden inflow of queries. The first case is not a development issue, just one of those hazards that make the life of a systems engineer or DBA so exciting. The second case is a development or specifications issue. Remember the post office of Chapter 9: when customers arrive faster than they can be serviced, queues lengthen and performance tumbles down all of a sudden. Either the original specifications were tailored too tightly and the system is facing a load it", + "source": "The Art of SQL.pdf", + "chunk_id": 262 + }, + { + "text": "second case is a development or specifications issue. Remember the post office of Chapter 9: when customers arrive faster than they can be serviced, queues lengthen and performance tumbles down all of a sudden. Either the original specifications were tailored too tightly and the system is facing a load it wasn’t designed for, or the application has been insuffi- ciently stress-tested. In many cases, improving some key queries will massively decrease the average service time and may improve the situation for a negligible fraction of the cost of a hardware upgrade. Sudden global sluggishness is usually characterized by the first phone call being followed by many others. Sudden localized slowness If one particular task slows down all of a sudden, locking issues should be con- sidered. Database administrators can monitor locks and confirm that several tasks are competing for the same resources. This situation is a development and task-scheduling issue that can be improved by trying to release locks faster. A slow degradation of performance reaching a threshold The threshold may first be felt by one hypersensitive user. If the load has been steadily increasing over time, the crossing of the threshold may be a warning sign of an impending catastrophe and may relate to the lengthening service queues of a sudden global sluggishness. The crossing of a threshold may also be linked to the size increase of badly indexed tables or to a degradation of physical storage after heavy delete/update operations (hanging high-water mark of a table that has inflated then deflated, a Swiss cheese–like effect resulting in much too many pages or blocks to store the data, or chaining to overflow areas). If the problem is with indexes or physical storage (or outdated statistics taking the optimizer down a wrong path), a DBA may be able to help, but the necessity for a rescue operation on a regular basis is usually the sign of poorly designed processes. One particularly slow query If the application was properly tested, then the case to watch for is a dynami- cally built query provided with a highly unusual set of criteria. This is most likely to be a pure development issue. Many of these events can be foreseen and prevented. If you are able to identify what loads your server, and if you are able to relate database activity to business activity, you have all the required elements to identify the weakest spots in an application. You can then focus on those weak spots during performance testing and improve them. www.it-ebooks.info 310 C H A P T E R T W E L V E To anticipate live application performance, you must monitor activity very closely during stress tests and user acceptance trials. The Components of Server Load Load, in information technology, ultimately boils down to a combination of excessive CPU consumption, too many input/output operations and insufficient network speed or bandwidth. It’s quite similar to the “critical tasks” of project management, where one bottleneck can result in the whole system grinding", + "source": "The Art of SQL.pdf", + "chunk_id": 263 + }, + { + "text": "trials. The Components of Server Load Load, in information technology, ultimately boils down to a combination of excessive CPU consumption, too many input/output operations and insufficient network speed or bandwidth. It’s quite similar to the “critical tasks” of project management, where one bottleneck can result in the whole system grinding not to a halt, but to an unnaceptable level of slowness. If processes that are ready to run must wait for some other processes to release the CPU, the system is overloaded. If the CPU is idle, waiting for data to be sent across the network or to be fetched from persistent storage, the system is overloaded too. “Overloaded,” though, mustn’t be understood as an absolute notion. Systems may be compared to human beings in respect of the fact that load is not always directly proportional to the work accomplished. As C. Northcote Parkinson remarked in Parkinson’s Law, his famous satire of bureaucratic institutions: Thus, an elderly lady of leisure can spend the entire day in writing and dispatching a postcard [...]. The total effort that would occupy a busy man for three minutes all told may in this fashion leave another person prostrate after a day of doubt, anxi- ety, and toil. Poorly developed SQL applications can very easily bring a server to its knees and yet not achieve very much. Here are a few examples (there are many others) illustrating different ways to increase the load without providing any useful work: Hardcoding all queries This will force the DBMS to run parser and optimizer code for every execution, before actually performing any data access. This technique is remarkably effi- cient for swamping the CPU. Running useless queries This is a situation more common than one would believe. It includes queries that are absolutely useless, such as a dummy query to check that the DBMS is up and running before every statement (true story), or issuing a count(*) to check whether a row should be updated or inserted. Other useless queries also include repeatedly fetching information that is stable for the entire duration of a session, or issuing 400,000 times a day a query to fetch a currency exchange rate that is updated once every night. Multiplying round-trips Operating row-by-row, extensively using cursor loops, and banishing stored pro- cedures are all excellent ways to increase the level of “chatting” between the www.it-ebooks.info E M P L O Y M E N T O F S P I E S 311 application side and the SQL engine, wasting time on protocol issues, multiply- ing packets on the network and of course, as a side benefit, preventing the data- base optimizer from doing its work efficiently by keeping most of the mysteries of data navigation firmly hidden in the application. Let me underline that these examples of bad use of the DBMS don’t specifically include the “bad SQL query” that represents the typical SQL performance issue for many people. The queries described in the preceding list often run fast. But even", + "source": "The Art of SQL.pdf", + "chunk_id": 264 + }, + { + "text": "mysteries of data navigation firmly hidden in the application. Let me underline that these examples of bad use of the DBMS don’t specifically include the “bad SQL query” that represents the typical SQL performance issue for many people. The queries described in the preceding list often run fast. But even when they run at lightning speed, useless queries are always too slow: they waste resources that may be in short supply during peak activity. There are two components that affect the load on a database server. The visible component is made up of the slow “bad SQL queries” that people are desperate to have tuned. The invisible component is the background noise of a number of queries each of acceptable speed, perhaps even including some very fast ones, that are executed over and over again. The cumulative cost of the load generated by all this background noise routinely dwarfs the individual load of most of the big bad queries. As Sir Arthur Conan Doyle put in the mouth of Sherlock Holmes: It has long been an axiom of mine that the little things are infinitely the most important. As the background noise is spread over time, instead of happening all of a sudden, it passes unnoticed. It may nevertheless contribute significantly to reducing the “power reserve” that may be needed during occasional bursts of activity. Repetitive short-duration mediocre statements often load a server more than the big bad SQL queries that take a long time to run. Defining Good Performance Load is one thing, performance another. Good performance proves an elusive notion to define. Using CPU or performing a large number of I/O operations is not wrong in itself; your company, presumably, didn’t buy powerful hardware with the idea of keeping it idle. When the time comes to assess performance, there is a striking similarity between the world of databases and the world of corporate finance. You find in both worlds some longing for “key performance indicators” and magical ratio—and in both worlds, global indicators and ratios can be extremely misleading. A good average can hide distressing results during the peaks, and a significant part of the load may perhaps be traced back to a batch program that is far from optimal but that runs at a time of night when no one cares what the load is. To get a true appreciation of the real state of affairs, you must drill down to a lower level of detail. www.it-ebooks.info 312 C H A P T E R T W E L V E To a large extent, getting down to the details is an exercise similar to that which is known in managerial circles as “activity-based costing.” In a company, knowing in some detail how much you spend is relatively easy. However, relating costs to benefits is an exercise fraught with difficulties, notoriously for transverse operations such as information technology. Determining if you spend the right amount on hardware, software, and staff, as well as the rubber bands and", + "source": "The Art of SQL.pdf", + "chunk_id": 265 + }, + { + "text": "company, knowing in some detail how much you spend is relatively easy. However, relating costs to benefits is an exercise fraught with difficulties, notoriously for transverse operations such as information technology. Determining if you spend the right amount on hardware, software, and staff, as well as the rubber bands and duct tape required to hold everything together is extremely difficult, particularly when the people who actually earn money are “customers” of the IT department. Assessing whether you do indeed spend what you should has three prerequisites: • Knowing what you spend • Knowing what you get for the money • Knowing how your return on investment compares with acknowledged standards In the following subsections, I shall consider each of these points in turn in the context of database systems. Knowing What You Spend In the case of database performance, what we spend means, first and foremost, how many data pages we are hitting. The physical I/Os that some people tend to focus on are an ancillary matter. If you hit a very large number of different data pages, this will necessarily entail sustained I/O activity unless your database entirely fits in memory. But CPU load is also often a direct consequence of hitting the same data pages in memory again and again. Reducing the number of data pages accessed is not a panacea, as there are cases when the global throughput is higher when some queries hit a few more pages than is strictly necessary. But as far as single indicators go, the number of data pages hit is probably the most significant one. The other cost to watch is excessive SQL statement parsing, an activity that can consume an inordinate amount of CPU (massive hardcoded insertions can easily take 75% of the CPU available for parsing alone). The two most significant indicators of database load are the amount of CPU spent on statement parsing and the number of data pages visited when executing queries. Knowing What You Get There is a quote that is famous among advertisers, a quip attributed to John Wanamaker, a 19th-century American retailer: Half the money I spend on advertising is wasted; the trouble is I don’t know which half. www.it-ebooks.info E M P L O Y M E N T O F S P I E S 313 The situation is slightly better with database applications, but only superficially. You define what you get in terms of the number of rows (or bytes) returned by select statements; and similarly the number of rows affected by change operations. But such an apparently factual assessment is far from providing a true measure of the work performed on your behalf by the SQL engine, for a number of reasons: • First, from a practical point of view, all products don’t provide you with such statistics. • Second, the effort required to obtain a result set may not be in proportion to the size of the result set. As a general rule you can be suspicious of a", + "source": "The Art of SQL.pdf", + "chunk_id": 266 + }, + { + "text": "of reasons: • First, from a practical point of view, all products don’t provide you with such statistics. • Second, the effort required to obtain a result set may not be in proportion to the size of the result set. As a general rule you can be suspicious of a very large number of data page hits when only a few rows are returned. However such a proportion may be per- fectly legitimate when data is aggregated. It is impossible to give a hard-and-fast rule in this area. • Third, should data returned from the database for the sole purpose of using it as input to other queries be counted as useful work? What about systematically updating to N a column in a table without using a where clause when N already happens to be the value stored in most rows? In both cases, the DBMS engine performs work that can be mea- sured in terms of bytes returned or changed. Unfortunately, most of the work per- formed can be avoided. There are times when scanning large tables or executing a long-running query may be perfectly justified (or indeed inescapable). For instance, when you run summary reports on very large volumes of data, you cannot expect an immediate answer. If an immediate answer is required, then it is likely that the data model (the database representation of reality) is inappropriate to the questions you want to see answered. This is a typical case when a decision support database that doesn’t necessarily require the level of detail of the main operational database may be suitable. Remember what you saw in Chapter 1: correct modeling depends both on the data and what you want to do with the data. You may share data with your suppliers or customers and yet have a totally different database model than they do. Naturally, feeding a decision support system will require long and costly operations both on the source operational database and the target decision support database. Because what you do with the data matters so much, you cannot judge performance if you don’t relate the load to the execution of particular SQL statements. The global picture that may be available through monitoring utilities (that most often provide cumulative counters) is not of much interest if you cannot assign to each statement its fair share of the load. As a first stage in the process of load analysis, you must therefore capture and collect SQL statements, and try to determine how much each one contributes to the overall cost. It may not be important to capture absolutely every statement. Database activity is one of those areas where the 80/20 rule, the empirical assessment that 80% of the consequences result from 20% of the causes, often describes the situation rather well. Usually, much of www.it-ebooks.info 314 C H A P T E R T W E L V E the load comes from a small number of SQL statements. We must be careful not to overlook the", + "source": "The Art of SQL.pdf", + "chunk_id": 267 + }, + { + "text": "the consequences result from 20% of the causes, often describes the situation rather well. Usually, much of www.it-ebooks.info 314 C H A P T E R T W E L V E the load comes from a small number of SQL statements. We must be careful not to overlook the fact that hardcoded SQL statements may distort the picture. With hardcoded statements, the DBMS may record thousands of distinct statements where a properly coded query would be referenced only once, even though it might be called thousands of times, each time with differing parameters. Such a situation can usually be spotted quite easily by the great number of SQL statements, and sometimes by global statistics. For instance, a procedure such as sp_trace_setevent in Transact-SQL lets you obtain a precise count of executed cursors, reexecutions of prepared cursors, and so on. If nothing else is available and if you can access the SQL engine cache, a snapshot taken at a relatively low frequency of once every few minutes may in many cases prove quite useful. Big bad queries are usually hard to miss, as also are queries that are being executed dozens of times a minute. Global costs should in any case be checked in order to validate the hypothesis that what has been missed contributes only marginally to the global load. It’s when SQL statements are hardcoded that taking snapshots will probably give less satisfactory results; you should then try to get a more complete picture, either through logging (as already mentioned a high-overhead solution), or by use of less intrusive “sniffer” utilities. I should note that even if you catch all hardcoded statements, then they have to be “reverse soft-coded” by taking constant values out of the SQL text before being able to estimate the relative load, not of a single SQL statement, but of one particular SQL statement pattern. Identifying the statements that keep the DBMS busy, though, is only part of the story. You will miss much if you don’t then relate SQL activity to the essential business activity of the organization that is supported by the database. Having an idea of how many SQL statements are issued on average each time you are processing a customer order is more important to SQL performance than knowing the disk transfer rate or the CPU speed under standard conditions of temperature and pressure. For one thing, it helps you anticipate the effect of the next advertising campaign; and if the said number of SQL statements is in the hundreds, you can raise interesting questions about the program (could there be, by chance, SQL statements executed inside loops that fetch the results of other statements? Could there be a statement that is repeatedly executed when it needs to be executed only once?). Similarly, costly massive updates of one column in a table accompanied by near identical numbers of equally massive updates of other columns from the same table with similar where clauses immediately raises the question of whether a single", + "source": "The Art of SQL.pdf", + "chunk_id": 268 + }, + { + "text": "that is repeatedly executed when it needs to be executed only once?). Similarly, costly massive updates of one column in a table accompanied by near identical numbers of equally massive updates of other columns from the same table with similar where clauses immediately raises the question of whether a single pass over the table wouldn’t have been enough. Load figures must be related to SQL statements. SQL statements must be related to business activity. Business activity must be related to business requirements. www.it-ebooks.info E M P L O Y M E N T O F S P I E S 315 Checking Against Acknowledged Standards Collecting SQL statements, evaluating their cost and roughly relating them to what makes a company or agency tick is an exercise that usually points you directly to the parts of the code that require in-depth review. The questionable code may be SQL statements, algorithms, or both. But knowing what you can expect in terms of improvement or how far you could or should go is a very difficult part of the SQL expert’s craft; experience helps, but even the seasoned practitioner can be left with a degree of uncertainty. It can be useful to establish a baseline, for instance by carrying out simple insertion tests and having an idea about the rate of insertion that is sustainable on your hardware. Similarly, you should check the fetch rate that can be obtained when performing those dreaded full scans on some of the biggest tables. Comparing bare-bones rates to what some applications manage to accomplish is often illuminating: there may be an order of magnitude or more between the fetch or insert speed that the SQL engine can attain and what is achieved by application programs. Know the limits of your environment. Measure how many rows you can insert, fetch, update, or delete per unit of time on your machines. Once you have defined a few landmarks, you can identify where you will obtain the best “return on improvement,” in terms of both relevance to business activities and technical feasibility. You can then focus on those parts of your programs and get results where it matters. Some practitioners tend to think that as long as end users don’t complain about performance, there is no issue and therefore no time to waste on trying to make operations go faster. There is some wisdom in this attitude; but there is also some short- sightedness as well, for two reasons: • First, end users often have a surprisingly high level of tolerance for poor performance; or perhaps it would be more appropriate to say that their perception of slowness dif- fers widely from that of someone who has a better understanding of what happens behind the scenes. End users may complain loudly about the performance of those processes of death that cannot possibly do better, and express a mild dissatisfaction about other processes when I would have long gone ballistic. A low level of complaint doesn’t necessarily mean that everything", + "source": "The Art of SQL.pdf", + "chunk_id": 269 + }, + { + "text": "understanding of what happens behind the scenes. End users may complain loudly about the performance of those processes of death that cannot possibly do better, and express a mild dissatisfaction about other processes when I would have long gone ballistic. A low level of complaint doesn’t necessarily mean that everything is fine, nor does vocal dissatisfaction neces- sarily mean that there is anything wrong with an application except perhaps trying to do too much. www.it-ebooks.info 316 C H A P T E R T W E L V E • Second, a slight increase in the load on a server may mean that performance will dete- riorate from acceptable to unacceptable very quickly. If the environment is perfectly stable, there is indeed nothing to fear from a slight increase in load. But if your activ- ity records a very high peak during one particular month of the year, the same pro- gram that looks satisfactory for 11 months can suddenly be the reason for riots. Here the background noise matters a lot. An already overloaded machine cannot keep on providing the same level of service when activity increases. There is always a thresh- old that sees mediocre performance tumbling down all of a sudden. It is therefore important to study an entire system before a burst of activity is encountered to see whether the load can be reduced by improving the code. If improving the code isn’t enough to warrant acceptable performance, it may be time to switch to bigger iron and upgrade the hardware. Do not forget that “return on improvement” is not simply a technical matter. The perception of end users should be given the highest priority, even if it is biased and sometimes disconnected from the most severe technical issues. They have to work with the program, and ergonomics have to be taken into account. It is not unusual to meet well- meaning individuals concentrating on improving statistics rather than program throughput, let alone end-user satisfaction. These well-intentioned engineers can feel somewhat frustrated and misunderstood when end users, who only see a very local improvement, welcome the result of mighty technical efforts with lukewarm enthusiasm. An eighteenth-century author reports that somebody once said to a physician, “Well, Mr. X has died, in spite of the promise you had made to cure him.” The splendid answer from the physician was, “You were away, and didn’t check the progress of the treatment: he died cured.” A database with excellent statistics and yet unsatisfactory performance from an end-user point of view is like a patient cured of one ailment, but who died of another. Improving performance usually means both delivering a highly visible improvement to end users, even if it affects a query that is run only once a month but that is business-critical, and the more humble, longer-term work of streamlining programs, lowering the background noise, and ensuring that the server will be able to deliver that power boost when it is needed. Performance improvement as perceived by end", + "source": "The Art of SQL.pdf", + "chunk_id": 270 + }, + { + "text": "affects a query that is run only once a month but that is business-critical, and the more humble, longer-term work of streamlining programs, lowering the background noise, and ensuring that the server will be able to deliver that power boost when it is needed. Performance improvement as perceived by end users is what matters most, but never forget the narrow margin between acceptable and unacceptable performance in a loaded environment. Defining Performance Goals Performance goals are often defined in terms of elapsed time, for example, “this program must run in under 2 hours.” It is better though to define them primarily in terms of business items processed by unit of time, such as “50,000 invoices per hour” or “100 loans per minute,” for several reasons: www.it-ebooks.info E M P L O Y M E N T O F S P I E S 317 • It gives a better idea of the service actually provided by a given program. • It makes a decrease in performance more understandable to end users when it can be linked to an increase in activity. This makes meetings less stormy. • Psychologically speaking, it is slightly more exciting when trying to improve a process to boost throughput rather than diminish the elapsed time. An upward curve makes a better chart in management presentations than a downward one. More than anything else, improved performance means first, doing more work in the same time, and second, doing it in even less time. Thinking in Business Tasks Before focusing on one particular query, don’t forget its context. Queries executed in loops are a very bad indicator of the quality of code, as are program variables with no other purpose than storing information returned from the database before passing it to another query. Database accesses are costly, and should be kept to a minimum. When you consider the way some programs are written, you are left with the impression that when their authors go shopping, they jump into their car, drive to a supermarket, park their car, walk up and down the aisles, pick a few bottles of milk, head for the checkout, get in line, pay, put the milk in the car, drive home, store the milk into the fridge, then check the next item on the shopping list before returning to the supermarket. And when a spouse complains about the time spent on shopping, the excuses given are usually the dense traffic on the road, the poor signposting of the food department, and the insufficient number of cashiers. All are valid reasons in their own right that may indeed contribute to some extent to shopping time, but possibly they are not the first issues to fix. I have met developers who were genuinely persuaded that from a performance standpoint, multiplying simple queries was the proper thing to do; showing them that the opposite is true was extremely easy. I have also heard that very simple SQL statements that avoid joins make maintenance easier. The truth is", + "source": "The Art of SQL.pdf", + "chunk_id": 271 + }, + { + "text": "I have met developers who were genuinely persuaded that from a performance standpoint, multiplying simple queries was the proper thing to do; showing them that the opposite is true was extremely easy. I have also heard that very simple SQL statements that avoid joins make maintenance easier. The truth is that simplistic SQL makes it easier to use totally inexperienced (read cheaper) developers for maintenance, but that’s the only thing that can be said in defense of very elementary SQL statements. By making the most basic usage of SQL, you end up with programs full of statements that, taken one by one, look efficient, except perhaps for a handful of particularly poor performers, hastily pointed to as “the SQL statements that require tuning.” Very often, some of the statements identified as “slow” (and which may indeed be slow) are responsible for only a fraction of performance issues. www.it-ebooks.info 318 C H A P T E R T W E L V E Brilliantly tuned statements in a bad program operating against a badly designed database are no more effective than brilliant tactics at the service of a feeble strategy; all they can do is postpone the day of reckoning. You cannot design efficient programs if you don’t understand that the SQL language applies to a whole subsystem of data management, and isn’t simply a set of primitives to move data between long-term and short-term memory. Database accesses are often the most performance-critical components of a program, and must be incorporated to the overall design. In trying to make programs simpler by multiplying SQL statements, you succumb to a dangerous illusion. Complexity doesn’t originate in languages, but in business requirements. With the exclusive use of simple SQL statements, complexity doesn’t vanish, it just migrates from the SQL side to the application side, with a much increased risk of data inconsistency when the logic that should belong to the DBMS side is imbedded into the application. Moreover, it puts a significant part of processing out of reach of the DBMS optimizer. I am not advocating the indiscriminate use of long, complex SQL statements, or a “single statement” policy. For example, the following is a case where there should have been several distinct statements, and not a single one: insert into custdet (custcode, custcodedet, usr, seq, inddet) select case ? when 'GRP' then b.codgrp when 'GSR' then b.codgsr when 'NIT' then b.codnit when 'GLB' then 'GLOBAL' else b.codetb end, b.custcode, ?, ?, 'O' from edic00 a, clidet bT where ((b.codgrp = a.custcode and ? = 'GRP') or (b.codgsr = a.custcode and ? = 'GSR') or (b.codnit = a.custcode and ? = 'NIT') or (a.custcode = 'GLOBAL' and ? = 'GLB')) and a.seq = ? and b.custlvl = ? and b.histdat = ? www.it-ebooks.info E M P L O Y M E N T O F S P I E S 319 A statement where a run-time parameter is compared to a constant is usually a statement that should have been split into several", + "source": "The Art of SQL.pdf", + "chunk_id": 272 + }, + { + "text": "? and b.custlvl = ? and b.histdat = ? www.it-ebooks.info E M P L O Y M E N T O F S P I E S 319 A statement where a run-time parameter is compared to a constant is usually a statement that should have been split into several simpler statements. In the preceding example, the value that intervenes in the case construct is the same one that is successively compared to GRP, GSR, NIT, and GLB in the where clause. It makes no sense to force the SQL engine into making numerous mutually exclusive tests and sort out a situation that could have been cleared on the application side. In such a case, an if ... elsif ... elsif structure (preferably in order of decreasing probability of occurrence) and four distinct insert ... select statements would have been much better. When a complex SQL statement allows you to obtain more quickly the data you ultimately need, with a small number of accesses, the situation is completely different from the preceding case. Long, complex queries are not necessarily slow; it all depends on how they are written. A developer should obviously not exceed their personal SQL skill level, and not necessarily write 300-line statements head on; but packing as much action as possible into each SQL statement should be a prerequisite to improving individual statements. Tuning SQL statements before improving programs and minimizing database accesses means that you are ignoring some of the major means of tuning improvements. Execution Plans When our spies (whether they are users or monitoring facilities) have directed our attention to a number of SQL statements, we need to inspect these statements more closely. Scrutinizing execution plans is one of the favorite activities of many SQL tuners, if we are to believe the high number of posts in forums or mailing lists in the form of “I have a SQL query that is particularly slow; here is the execution plan....” Execution plans are usually displayed either as an indented list of the various steps involved in the processing of a (usually complex) SQL statements, or under a graphical form, as in Figure 12-1. This figure displays the execution plan for one of the queries from Chapter 7. Text execution plans are far less sexy but are easier to post on forums, which must account for the enduring popularity of such plans. Knowing how to correctly read and interpret an execution plan, whether it is represented graphically or as text, is in itself a valued skill. So far in this book, I have had very little to say on the topic of execution plans, except for a couple of examples presented here and there without any particular comment. Execution plans are tools, and different individuals have different preferences for various www.it-ebooks.info 320 C H A P T E R T W E L V E tools; you are perfectly allowed to have a different opinion, but I usually attach a secondary importance to execution plans. Some", + "source": "The Art of SQL.pdf", + "chunk_id": 273 + }, + { + "text": "particular comment. Execution plans are tools, and different individuals have different preferences for various www.it-ebooks.info 320 C H A P T E R T W E L V E tools; you are perfectly allowed to have a different opinion, but I usually attach a secondary importance to execution plans. Some developers consider execution plans as the ultimate key to the understanding of performance issues. Two real-life examples will show that one may have some reasons to be less than sanguine about using execution plans as the tool of choice for improving a query. Identifying the Fastest Execution Plan In this section, I am going to test your skills as an interpreter of execution plans. I’m going to show three execution plans and ask you to choose which is the fastest. Ready? Go, and good luck! Our contestants The following execution plans show how three variants of the same query are executed: Plan 1 Execution Plan ---------------------------------------------------------- 0 SELECT STATEMENT 1 0 SORT (ORDER BY) 2 1 CONCATENATION 3 2 NESTED LOOPS FIGURE 12-1. A DB2 execution plan www.it-ebooks.info E M P L O Y M E N T O F S P I E S 321 4 3 HASH JOIN 5 4 HASH JOIN 6 5 TABLE ACCESS (FULL) OF 'TCTRP' 7 5 TABLE ACCESS (BY INDEX ROWID) OF 'TTRAN' 8 7 INDEX (RANGE SCAN) OF 'TTRANTRADE_DATE' (NON-UNIQUE) 9 4 TABLE ACCESS (BY INDEX ROWID) OF 'TMMKT' 10 9 INDEX (RANGE SCAN) OF 'TMMKTCCY_NAME' (NON-UNIQUE) ... 11 3 TABLE ACCESS (BY INDEX ROWID) OF 'TFLOW' 12 11 INDEX (RANGE SCAN) OF 'TFLOWMAIN' (UNIQUE) 13 2 NESTED LOOPS 14 13 HASH JOIN 15 14 HASH JOIN 16 15 TABLE ACCESS (FULL) OF 'TCTRP' 17 15 TABLE ACCESS (BY INDEX ROWID) OF 'TTRAN' 18 17 INDEX (RANGE SCAN) OF 'TTRANLAST_UPDATED' (NON-UNIQUE) 19 14 TABLE ACCESS (BY INDEX ROWID) OF 'TMMKT' 20 19 INDEX (RANGE SCAN) OF 'TMMKTCCY_NAME' (NON-UNIQUE) 21 13 TABLE ACCESS (BY INDEX ROWID) OF 'TFLOW' 22 21 INDEX (RANGE SCAN) OF 'TFLOWMAIN' (UNIQUE) Plan 2 Execution Plan ---------------------------------------------------------- 0 SELECT STATEMENT 1 0 SORT (ORDER BY) 2 1 CONCATENATION 3 2 NESTED LOOPS 4 3 NESTED LOOPS 5 4 NESTED LOOPS 6 5 TABLE ACCESS (BY INDEX ROWID) OF 'TTRAN' 7 6 INDEX (RANGE SCAN) OF 'TTRANTRADE_DATE' (NON-UNIQUE) 8 5 TABLE ACCESS (BY INDEX ROWID) OF 'TMMKT' 9 8 INDEX (UNIQUE SCAN) OF 'TMMKTMAIN' (UNIQUE) 10 4 TABLE ACCESS (BY INDEX ROWID) OF 'TFLOW' 11 10 INDEX (RANGE SCAN) OF 'TFLOWMAIN' (UNIQUE) 12 3 TABLE ACCESS (BY INDEX ROWID) OF 'TCTRP' 13 12 INDEX (UNIQUE SCAN) OF 'TCTRPMAIN' (UNIQUE) 14 2 NESTED LOOPS 15 14 NESTED LOOPS 16 15 NESTED LOOPS 17 16 TABLE ACCESS (BY INDEX ROWID) OF 'TTRAN' 18 17 INDEX (RANGE SCAN) OF 'TTRANLAST_UPDATED' (NON-UNIQUE) 19 16 TABLE ACCESS (BY INDEX ROWID) OF 'TMMKT' 20 19 INDEX (UNIQUE SCAN) OF 'TMMKTMAIN' (UNIQUE) 21 15 TABLE ACCESS (BY INDEX ROWID) OF 'TFLOW' 22 21 INDEX (RANGE SCAN) OF 'TFLOWMAIN' (UNIQUE) 23 14 TABLE ACCESS (BY INDEX ROWID) OF", + "source": "The Art of SQL.pdf", + "chunk_id": 274 + }, + { + "text": "18 17 INDEX (RANGE SCAN) OF 'TTRANLAST_UPDATED' (NON-UNIQUE) 19 16 TABLE ACCESS (BY INDEX ROWID) OF 'TMMKT' 20 19 INDEX (UNIQUE SCAN) OF 'TMMKTMAIN' (UNIQUE) 21 15 TABLE ACCESS (BY INDEX ROWID) OF 'TFLOW' 22 21 INDEX (RANGE SCAN) OF 'TFLOWMAIN' (UNIQUE) 23 14 TABLE ACCESS (BY INDEX ROWID) OF 'TCTRP' 24 23 INDEX (UNIQUE SCAN) OF 'TCTRPMAIN' (UNIQUE) Plan 3 Execution Plan ---------------------------------------------------------- 0 SELECT STATEMENT 1 0 SORT (ORDER BY) 2 1 NESTED LOOPS 3 2 NESTED LOOPS 4 3 NESTED LOOPS www.it-ebooks.info 322 C H A P T E R T W E L V E 5 4 TABLE ACCESS (BY INDEX ROWID) OF 'TMMKT' 6 5 INDEX (RANGE SCAN) OF 'TMMKTCCY_NAME' (NON-UNIQUE) 7 4 TABLE ACCESS (BY INDEX ROWID) OF 'TTRAN' 8 7 INDEX (UNIQUE SCAN) OF 'TTRANMAIN' (UNIQUE) 9 3 TABLE ACCESS (BY INDEX ROWID) OF 'TCTRP' 10 9 INDEX (UNIQUE SCAN) OF 'TCTRPMAIN' (UNIQUE) 11 2 TABLE ACCESS (BY INDEX ROWID) OF 'TFLOW' 12 11 INDEX (RANGE SCAN) OF 'TFLOWMAIN' (UNIQUE) Our battle field The result set of the query consists of 860 rows, and the four following tables are involved: All tables are heavily indexed, no index was created, dropped or rebuilt, and no change was applied to the data structures. Only the text of the query changed between plans, and optimizer directives were sometimes applied. Consider the three execution plans, try to rank them in order of likely speed, and if you feel like it you may even venture an opinion about the improvement factor. And the winner is... The answer is that Plan 1 took 27 seconds, Plan 2 one second, and Plan 3 (the initial execution plan of the query) one minute and 12 seconds. You will be forgiven for choosing the wrong plan. In fact, with the information that I provided, it would be sheer luck for you to have correctly guessed at the fastest plan (or the result of a well-founded suspicion that there must be a catch somewhere). You can take note that the slowest execution plan is by far the shortest, and that it contains no reference to anything other than indexed accesses. By contrast, Plan 1 demonstrates that you can have two full scans of the same table and yet execute the query almost three times faster than a shorter, index-only plan such as Plan 3. The point of this exercise was to demonstrate that the length of an execution plan is not very meaningful, and that exclusive access to tables through indexes doesn’t guarantee that performance is the best you can achieve. True, if you have a 300-line plan for a query that returns 19 rows, then you might have a problem, but you mustn’t assume that shorter is better. Table name Row count (rounded) tctrp 18,000 ttran 1,500,000 tmmkt 1,400,000 tflow 5,400,000 www.it-ebooks.info E M P L O Y M E N T O F S P I E S 323 Forcing the Right Execution Plan The second example is the weird behavior", + "source": "The Art of SQL.pdf", + "chunk_id": 275 + }, + { + "text": "mustn’t assume that shorter is better. Table name Row count (rounded) tctrp 18,000 ttran 1,500,000 tmmkt 1,400,000 tflow 5,400,000 www.it-ebooks.info E M P L O Y M E N T O F S P I E S 323 Forcing the Right Execution Plan The second example is the weird behavior exhibited by one query issued by a commercial off-the-shelf software package. When run against one database, the query takes 4 minutes, returning 40,000 rows. Against another database, running the same version of the same DBMS, the very same query responds in 11 minutes on comparable hardware although all tables involved are much smaller. The comparison of execution plans shows that they are wildly different. Statistics are up-to-date on both databases, and the optimizer is instructed to use them everywhere. The question immediately becomes one of how to force the query to take the right execution path on the smaller database. DBAs are asked to do whatever is in their power to get the same execution plan on both databases. The vendor’s technical team works closely with the customer’s team to try to solve the problem. A stubborn query Following is the text of the query,* followed by the plan associated to the fastest execution. Take note that the good plan only accesses indexes, not tables: select o.id_outstanding, ap.cde_portfolio, ap.cde_expense, ap.branch_code, to_char(sum(ap.amt_book_round + ap.amt_book_acr_ad – ap.amt_acr_nt_pst)), to_char(sum(ap.amt_mnl_bk_adj)), o.cde_outstd_typ from accrual_port ap, accrual_cycle ac, outstanding o, deal d, facility f, branch b where ac.id_owner = o.id_outstandng and ac.id_acr_cycle = ap.id_owner and o.cde_outstd_typ in ('LOAN', 'DCTLN', 'ITRLN', 'DEPOS', 'SLOAN', 'REPOL') and d.id_deal = o.id_deal and d.acct_enabl_ind = 'Y' and (o.cde_ob_st_ctg = 'ACTUA' or o.id_outstanding in (select id_owner from subledger)) and o.id_facility = f.id_facility and f.branch_code = b.branch_code and b.cde_tme_region = 'ZONE2' group by o.id_outstanding, * Object names have been slightly changed to protect both the innocent and the culprit. www.it-ebooks.info 324 C H A P T E R T W E L V E ap.cde_portfolio, ap.cde_expense, ap.branch_code, o.cde_outstd_typ having sum(ap.amt_book_round + ap.amt_book_acr_ad – ap.amt_acr_nt_pst) <> 0 or (sum(ap.amt_mnl_bk_adj) is not null and sum(ap.amt_mnl_bk_adj) <> 0) Execution Plan ---------------------------------------------------------- 0 SELECT STATEMENT Optimizer=CHOOSE 1 0 FILTER 2 1 SORT (GROUP BY) 3 2 FILTER 4 3 HASH JOIN 5 4 HASH JOIN 6 5 HASH JOIN 7 6 INDEX (FAST FULL SCAN) OF 'XDEAUN08' (UNIQUE) 8 6 HASH JOIN 9 8 NESTED LOOPS 10 9 INDEX (FAST FULL SCAN) OF 'XBRNNN02' (NON-UNIQUE) 11 9 INDEX (RANGE SCAN) OF 'XFACNN05' (NON-UNIQUE) 12 8 INDEX (FAST FULL SCAN) OF 'XOSTNN06' (NON-UNIQUE) 13 5 INDEX (FAST FULL SCAN) OF 'XACCNN05' (NON-UNIQUE) 14 4 INDEX (FAST FULL SCAN) OF 'XAPONN05' (NON-UNIQUE) 15 3 INDEX (SKIP SCAN) OF 'XBSGNN03' (NON-UNIQUE) The addition of indexes to the smaller database leads nowhere. Existing indexes were initially identical on both databases, and creating different indexes on the smaller database brings no change to the execution plan. Three weeks after the problem was first spotted, attention is now turning to disk striping, without much hope. Constraining optimizer directives are beginning to look unpleasantly", + "source": "The Art of SQL.pdf", + "chunk_id": 276 + }, + { + "text": "nowhere. Existing indexes were initially identical on both databases, and creating different indexes on the smaller database brings no change to the execution plan. Three weeks after the problem was first spotted, attention is now turning to disk striping, without much hope. Constraining optimizer directives are beginning to look unpleasantly like the only escape route. Before using directives, it is wise to have a fair idea of the right angle of attack. Finding the proper angle, as you have seen in Chapters 4 and 6, requires an assessment of the relative precision of the various input criteria, even though in this case the reasonably large result set (of some 40,000 rows on the larger database and a little over 3,000 on the smaller database) gives us little hope of seeing one criterion coming forward as the key criterion. Study of search criteria When we use as the only criterion the condition on what looks like a time zone, the query returns 17% more rows than with all filtering conditions put together, but it does it blazingly fast: SQL> select count(*) \"FAC\" 2 from outstanding 3 where id_facility in (select f.id_facility www.it-ebooks.info E M P L O Y M E N T O F S P I E S 325 4 from facility f, 5 branch b 6 where f.branch_code = b.branch_code 7 and b.cde_tme_region = 'ZONE2'); FAC ---------- 55797 Elapsed: 00:00:00.66 The flag condition alone filters three times our number of rows, but it does it very fast, too: SQL> select count(*) \"DEA\" 2 from outstanding 3 where id_deal in (select id_deal 4 from deal 5 where acct_enabl_ind = 'Y'); DEA ---------- 123970 Elapsed: 00:00:00.63 What about our or condition on the outstanding table? Following are the results from that condition: SQL> select count(*) \"ACTUA/SUBLEDGER\" 2 from outstanding 3 where (cde_ob_st_ctg = 'ACTUA' 4 or id_outstanding in (select id_owner 5 from subledger)); ACTUA/SUBLEDGER --------------- 32757 Elapsed: 00:15:00.64 Looking at these results, it is clear that we have pinpointed the problem. This or condition causes a huge increase in the query’s execution time. The execution plan for the preceding query shows only index accesses: Execution Plan ---------------------------------------------------------- 0 SELECT STATEMENT Optimizer=CHOOSE 1 0 SORT (AGGREGATE) 2 1 FILTER 3 2 INDEX (FAST FULL SCAN) OF 'XOSTNN06' (NON-UNIQUE) 4 2 INDEX (SKIP SCAN) OF 'XBSGNN03' (NON-UNIQUE) www.it-ebooks.info 326 C H A P T E R T W E L V E Notice that both index accesses are not exactly the usual type of index descent; there is no need to get into arcane details here, but a FAST FULL SCAN is in fact the choice of using the smaller index rather than the larger associated table to perform a scan, and the choice of a SKIP SCAN comes from a similar evaluation by the optimizer. In other words, the choice of the access method is not exactly driven by the evidence of an excellent path, but proceeds from a kind of “by and large, it should be better” optimizer assessment. If the", + "source": "The Art of SQL.pdf", + "chunk_id": 277 + }, + { + "text": "choice of a SKIP SCAN comes from a similar evaluation by the optimizer. In other words, the choice of the access method is not exactly driven by the evidence of an excellent path, but proceeds from a kind of “by and large, it should be better” optimizer assessment. If the execution time is to be believed, a SKIP SCAN is not the best of choices. Let’s have a look at the indexes on outstanding (the numbers of distinct index keys and distinct column values are estimates, which accounts for the slightly inconsistent figures). Indexes in bold are the indexes that appear in the execution plan: INDEX_NAME DIST KEYS COLUMN_NAME DIST VAL --------------------- ---------- ------------------ -------- XOSTNC03 25378 ID_DEAL 1253 ID_FACILITY 1507 XOSTNN05 134875 ID_OUTSTANDING 126657 ID_DEAL 1253 IND_AUTO_EXTND 2 CDE_OUTSTD_TYP 5 ID_FACILITY 1507 UID_REC_CREATE 161 NME_ALIAS 126657 XOSTNN06 ID_OUTSTANDING 126657 CDE_OUTSTD_TYP 5 ID_DEAL 1253 CDE_OB_ST_CTG 3 ID_FACILITY 1507 XOSTUN01 (U) 121939 ID_OUTSTANDING 126657 XOSTUN02 (U) 111055 NME_ALIAS 126657 The other index (xbsgnn03) is associated with subledger: INDEX_NAME DIST KEYS COLUMN_NAME DIST VAL --------------------- ---------- ------------------ -------- XBSGNN03 101298 BRANCH_CODE 8 CDE_PORTFOLIO 5 CDE_EXPENSE 56 ID_OWNER 52664 CID_CUSTOMER 171 XBSGNN04 59542 ID_DEAL 4205 ID_FACILITY 4608 ID_OWNER 52664 XBSGNN05 49694 BRANCH_CODE 8 ID_FACILITY 4608 ID_OWNER 52664 XBSGUC02 (U) 147034 CDE_GL_ACCOUNT 9 CDE_GL_SHTNAME 9 BRANCH_CODE 8 CDE_PORTFOLIO 5 CDE_EXPENSE 56 ID_OWNER 52664 CID_CUSTOMER 171 XBSGUN01 (U) 134581 ID_SUBLEDGER 154362 www.it-ebooks.info E M P L O Y M E N T O F S P I E S 327 As is too often the case with COTS packages, we have here an excellent example of carpet-indexing. The indexes on outstanding raise a couple of questions. • Why does id_outstanding, the primary key of the outstanding table, also appears as the lead column of two other indexes? This requires some justification, and very persua- sive justification too. Even if those indexes were built with the purpose of fetching all values from them and avoiding table access, one might arguably have relegated id_ oustanding to a less prominent position; on the other hand, since few columns seem to have a high number of distinct values, the very existence of some of the indexes would need to be reassessed. • All is not quiet on the subledger front either. One of the most selective values happens to be id_owner. Why does id_owner appear in 4 of the 5 indexes, but nowhere as the lead column? Such a situation is surprising for an often referenced selective column. Incidentally, finding id_owner as the lead column of an index would have been helpful with our problem query. Modifying indexes is a delicate business that requires a careful study of all the possible side-effects. We have here a number of questionable indexes, but we also have an urgent problem to solve. Let’s therefore refrain from making any changes to the existing indexes and concentrate on the SQL code. As the numbers of distinct keys of our unique indexes show, we are not dealing here with large tables; and in fact the", + "source": "The Art of SQL.pdf", + "chunk_id": 278 + }, + { + "text": "indexes, but we also have an urgent problem to solve. Let’s therefore refrain from making any changes to the existing indexes and concentrate on the SQL code. As the numbers of distinct keys of our unique indexes show, we are not dealing here with large tables; and in fact the two other criteria we have tried to apply to outstanding both gave excellent response times, in spite of being rather weak criteria. The pathetic result we have with the or construct results from an attempt to merge data which was painfully extracted from the two indexes. Let’s try something else: SQL> select count(*) \"ACTUA/SUBLEDGER\" 2 from (select id_outstanding 3 from outstanding 4 where cde_ob_st_ctg = 'ACTUA' 5 union 6 select o.id_outstanding 7 from outstanding o, 8 subledger sl 9 where o.id_outstanding = sl.id_owner) 10 / ACTUA/SUBLEDGER --------------- 32757 Elapsed: 00:00:01.82 No change to the indexes, and yet the optimizer suddenly sees the light even if we hit the table outstanding twice. Execution is much, much faster now. www.it-ebooks.info 328 C H A P T E R T W E L V E Replacing the “problem condition” and slightly reshuffling some of the other remaining conditions, cause the query to run in 13 seconds where it used to take 4 minutes (reputedly the “good case”); and only 3.4 seconds on the other database, where it used to take 11 minutes to return 3,200 rows. A moral to the story It is likely that a more careful study and some work at the index level would allow the query to run considerably faster than 13 seconds. On the other hand, since everybody appeared to be quite happy with 4 minutes, 13 seconds is probably a good enough improvement. What is fascinating in this true story (and many examples in this book are taken from real life), is how the people involved focused (for several weeks) on the wrong issue. There was indeed a glaring problem on the smaller database. The comparison of the two different execution plans led to the immediate conclusion that the execution plan corresponding to the slower execution was wrong (true) and therefore, implicitly, that the execution plan corresponding to the faster execution was right (false). This was a major logical mistake, and it misled several people into concentrating on trying to reproduce a bad execution plan instead of improving the query. I must add a final note as a conclusion to the story. Once the query has been rewritten, the execution plan is still different on the two databases—a situation that, given the discrepancy of volumes, only proves that the optimizer is doing its job. The only yardstick of query performance is how long one takes to run, not whether the execution plan conforms to prejudices. Using Execution Plans Properly Execution plans are useful, but mostly to check that the DBMS engine is indeed proceeding as intended. The report from the field that an execution plan represents is a great tool to compare what has been realized to", + "source": "The Art of SQL.pdf", + "chunk_id": 279 + }, + { + "text": "not whether the execution plan conforms to prejudices. Using Execution Plans Properly Execution plans are useful, but mostly to check that the DBMS engine is indeed proceeding as intended. The report from the field that an execution plan represents is a great tool to compare what has been realized to the tactics that were planned, and can reveal tactical flaws or overlooked details. How Not to Execute a Query Execution plans can be useful even when one has not the slightest idea about what a proper execution plan should be. The reason is that, by definition, the execution plan of a problem query is a bad one, even if it may not look so terrible. Knowing that the plan is bad allows us to discover ways to improve the query, through the use of one of the most sophisticated tools of formal logic, the syllogism, an argument with two premises and one conclusion. www.it-ebooks.info E M P L O Y M E N T O F S P I E S 329 This reasoning is as follows: (Premise 1) The query is dreadfully slow. (Premise 2) The execution plan displays mostly one type of action—for example: full table scans, hash joins, indexed accesses, nested loops, and so forth. (Conclusion) We should rewrite the query and/or possibly change indexes so as to suggest something else to the optimizer. Coaxing the optimizer into taking a totally different course can be achieved through a number of means: • When we have few rows returned, it may be a matter of adding one index, or rebuild- ing a composite index and reversing the order of some of the columns; transforming uncorrelated subqueries into correlated ones can also be helpful. • When we have a large number of rows returned we can do the opposite, and use parentheses and subqueries in the from clause to suggest a different order when joining tables together. • In doubt, we have quite a number of options besides transforming correlated sub que- ries into uncorrelated subqueries and vice versa. We can consider operations such as factorizing queries with either a union or a with clause. The union of two complex que- ries can sometimes be transformed into a simpler union inside the from clause. Disentan- gling conditions (trying to make each condition dependent on as few other conditions as possible) is often helpful. Generally speaking, trying to remove as much as possible of whatever imposes a processing order on the query and trying to give as much free- dom as possible to the optimizer is the very first thing to do before trying to constrain it. The optimizer must be constrained only when everything else goes wrong. • As a last resort, we may remember the existence of optimizer directives and use them very carefully. Hidden Complexity Execution plans can also prove to be valuable spies in revealing hidden complexity. Queries are not always exactly what a superficial inspection shows. The participation of some database objects in a query", + "source": "The Art of SQL.pdf", + "chunk_id": 280 + }, + { + "text": "a last resort, we may remember the existence of optimizer directives and use them very carefully. Hidden Complexity Execution plans can also prove to be valuable spies in revealing hidden complexity. Queries are not always exactly what a superficial inspection shows. The participation of some database objects in a query can induce additional work that execution plans will bring to light. These database objects are chiefly: Views Queries may look deceivingly simple. But sometimes what appears to be a sim- ple table may turn out to be a view defined as a very complex query involving several other views. The names of views may not always be distinctive, and even when they are, the name by itself cannot give any indication of the complexity of the view. The execution plan will show what a casual inspection of the SQL code may have missed, and most importantly, it will also tell you if the same table is being hit repeatedly. www.it-ebooks.info 330 C H A P T E R T W E L V E Triggers Changes to the database may take an anomalous time simply because of the execution of triggers. These may be running very slow code or may even be the true reason for some locking issues. Triggers are easy to miss, execution plans will reveal them. The essential value of execution plans is to provide a starting point for performance investigations and to reveal the hidden database operations caused by complex views and triggers. What Really Matters? What really matters when trying to improve a query has been discussed in the previous chapters, namely: • The number of rows in the tables involved • The existing indexes on these tables • Storage peculiarities, such as partitioning, that can have as strong an impact as indexes on performance • The quality of the various criteria that were provided • The size of the resulting set This information provides us with a solid foundation from which to investigate query performance, and is far more valuable than an execution plan on its own. Once we know were we stand, and what we have to fight against, then we can move, and attack tables, always trying to get rid of unwanted data as quickly as we can. We must always try to leave as much freedom to the optimizer as we can by avoiding any type of intra- statement dependencies that would constrain the order in which tables must be visited. In conclusion, I would like to remind you that optimizers, which usually prove quite efficient at their job, are unable to work efficiently under the following circumstances: • If you retrieve data piecemeal through multiple statements. It is one thing for an appli- cation to issue a series of related SQL statements. However, the SQL engine can never “know” that such statements are related, and cannot optimize across statement boundaries. The SQL engine can optimize each individual statement, but it cannot optimize the overall process. • If you use, without", + "source": "The Art of SQL.pdf", + "chunk_id": 281 + }, + { + "text": "for an appli- cation to issue a series of related SQL statements. However, the SQL engine can never “know” that such statements are related, and cannot optimize across statement boundaries. The SQL engine can optimize each individual statement, but it cannot optimize the overall process. • If you use, without any care, the numerous non-relational (and sometimes quite use- ful) features provided by the various SQL dialects. www.it-ebooks.info E M P L O Y M E N T O F S P I E S 331 Remember that you should apply non-relational features last, when the bulk of data retrieval is done (in the wider acceptance of retrieval; data must be retrieved before being updated or deleted). Non-relational features operate on finite sets (in other words, arrays), not on theoretically infinite relations. There was a time when you could make a reputation as an SQL expert by identifying missing indexes and rewriting statements so as to remove functions that were applied to indexed columns. This time is, for the most part, gone. Most databases are over-indexed, although sometimes inadequately indexed. Functions applied to indexed columns are still encountered, but functional indexes provide a “quick fix” to that particular problem. However, rewriting a poorly performing query usually means more nowadays than shuffling conditions or merely making cosmetic changes. The real challenge is more and more to be able to think globally, and to acknowledge that data handling is critical in a world where the amount of stored data increases even faster than the performance of the hardware. For better or for worse, data handling spells S-Q- L. Like all languages, SQL has its idiosyncrasies, its qualities, and numerous flaws. Like all languages, mastering SQL requires time, experience—and personal talent. I hope that on that long road this book will prove helpful to you. Building optimally performing SQL can be a source of great satisfaction—enjoy! www.it-ebooks.info www.it-ebooks.info 333 P H O T O C R E D I T S C H A P T E R 1 2 All images were scanned from Mémorial de Sainte-Hélène by Comte Emmanuel de Las Cases, illustrated by Charlet (Ernest Bourdin Editeur, Paris, 1842, two volumes), with the following exceptions: • The illustration for Chapter 6 was made out of a map of the battle of Fredericksburg, found on http://www.sonofthesouth.net, and used with the permission of Paul McWhorter who runs that very rich site on the American Civil War. • The illustration for Chapter 9 comes from Notre Armée by de Lonlay, illustrated by the author (Garnier Frères, Paris, 1890, p. 931). • The illustration for Chapter 12 comes from Les Guerres de la Révolution by Camille Pel- letan, Paris, Société d’Éditions d’Art (Collection L.-Henry May, G. Mantoux), no date [end 19th–beginning 20th century; first published, Paris, Colas, 1884], p.95 (10th series), coll. Durelle-Marc, and is published courtesy of the Centre d’Histoire du Droit de l’Université Rennes 1 (http://www.chd.univ-rennes1.fr/Icono/Pelletan/Pelletan.htm). www.it-ebooks.info www.it-ebooks.info 335 I N D E X C H A P T E R 0", + "source": "The Art of SQL.pdf", + "chunk_id": 282 + }, + { + "text": "May, G. Mantoux), no date [end 19th–beginning 20th century; first published, Paris, Colas, 1884], p.95 (10th series), coll. Durelle-Marc, and is published courtesy of the Centre d’Histoire du Droit de l’Université Rennes 1 (http://www.chd.univ-rennes1.fr/Icono/Pelletan/Pelletan.htm). www.it-ebooks.info www.it-ebooks.info 335 I N D E X C H A P T E R 0 Symbols @@IDENTITY system variable (Transact- SQL), 235 10% of rows rule of thumb, 103 1NF (first normal form), 8 2NF (second normal form), 9 3NF (third normal form), 4, 9 data warehousing and, 265 5NF (fifth normal form), 5 A absence of data, result sets predicated on, 161–166 abstract layers, 202–205 accesses to the database (see database accesses) ad hoc queries, 270 addresses, 109, 300–301 atomicity and, 7 adjacency model (SQL trees), 172, 174, 197 aggregating values stored in leaf nodes, 191 computing head counts at every level, 192 bottom-up tree walk, 185 top-down walk, 178–182 aggregation consolidating multiple rows into one, 282–284 double conversion, using, 302 on dates, 156 by range (bands), 297–299 result sets obtained from, 150–156 in transformations, 268 values from trees, 190–198 propagating percentages across levels, 194–198 values stored in leaf nodes, 190–194 aggressive coding, 49 analytical functions (Oracle), 52 ANSI SQL query (example), 91 architectural solutions for contention, 242 architecture (global), choice of, 249 archival data purging, 263 putting into production, 249 archives, location of, 170, 171 array interface, communicating between program and DBMS kernel, 31 art of SQL, governing factors, 84–88 number of tables, 85 number of users, 88 result set criteria, 84 result set size, 85 total quantity of data, 84 art, science vs., xi associative table, resolving many-to-many relationship between tables, 69 asynchronous processing, 22 atomic attributes, 6, 64 atomicity, 5 business requirements and, 7 function applied to a column, 64 attributes atomic, 6, 64 dealing with varying numbers of, 281 excessive flexibility in, 18 independence of, 9 auto-incremented columns, 235 not using in order to limit contention, 245 axioms, 3 www.it-ebooks.info 3 3 6 I N D E X B backup databases, 24 bad SQL queries, 311 bands, aggregating by, 297–299 batch programs, 22 queries returning large amounts of data, 102 queries satisfied by data from an index, 111 BCNF (Boyce-Codd normal form), 5 Bill of Materials (BOM) problem, 168 bind variables, 162 bind_param( ) method, 218 binding variables, 294, 296 PHP, 218–222 bitmap indexes, 273 blanket views, performance impact on queries, 117 blocks, 108 contention for access, 88 locking, 232 minimizing accesses to, 146 pre-joined tables and, 124 BOM (Bill of Materials) problem, 168 book indexes, table of contents vs., 59 Boolean columns, qualifying, 14 bottom-up tree walk, 178, 185–189 adjacency model, 185 materialized path model, 186 nested set model, 188 performance, comparing for various models, 188 boundaries of ranges, defining well, 262 Boyce-Codd normal form (BCNF), 5 bridge tables, 270 Building the Data Warehouse, 264 business logic, mirrored by SQL statements, 42 business processes, physical design and, 109 business requirements, 2 atomicity and, 7 database modeling and, 2 business tasks, focusing on, 317–319 C C# code, 202 cache, SQL", + "source": "The Art of SQL.pdf", + "chunk_id": 283 + }, + { + "text": "defining well, 262 Boyce-Codd normal form (BCNF), 5 bridge tables, 270 Building the Data Warehouse, 264 business logic, mirrored by SQL statements, 42 business processes, physical design and, 109 business requirements, 2 atomicity and, 7 database modeling and, 2 business tasks, focusing on, 317–319 C C# code, 202 cache, SQL engine, 314 cardinality (low), 109 Cartesian joins, 285, 286 case expression, 42 case-insensitive searches with function- based index, 66 CBOs (cost-based optimizers), 36 Celko, Joe, 172, 176 centralizing data, 23 changing data, concurrency and, 231–246 contention, 240–246 architectural solutions, 242 DBA solutions for, 241 developmental solutions, 243 insertion and, 240 results from measures limiting, 243–246 locking, 232–239 committing and, 236 granularity of, 232 lock handling, 234–236 scalability and, 238 child with multiple parents, 169 classic SQL patterns, 128–166 large result set, 146 nine common situations, listed, 128 result set obtained by aggregation, 150–156 result set predicated on absence of data, 161–166 self-joins on one table, 147–150 simple or range searching on dates, 156–161 small intersection of broad criteria, 138–140 small intersection, indirect broad criteria, 140–145 small result set, direct specific criteria, 129–137 criterion indexability, 132–137 data dispersion, 130–132 index usability, 129 query efficiency and index usage, 130 small result set, indirect criteria, 137 client/server environment, database connections, 30 clustered indexes, 114, 130 drawbacks of, 114 clustering data with partitioning, 120 clustering index, 114 coalesce( ) function, 300 coarse (granularity), 34 Codd, E.F., 76 coding offensively with SQL, 48 columns, 2 auto-incremented, 235 effects on contention, 245 Boolean, qualifying, 14 www.it-ebooks.info I N D E X 337 locking, 232 rows that should have been, 281–284 single, that should have been something else, 289–294 that should have been rows, 284–289 comments, identifying programs and critical modules, 28 commercial off-the-shelf (COTS) software package, 323 commit statements, 34 committing, locking and, 236 comparisons, 43 complexity degree for the request, performance and, 230 introduced by storage options other than the default, 124 sources of hidden complexity, 329 composite primary keys, 70 order of columns in, 158 concurrency, 226–246 considering in SQL code design, 88 data modifications, 231–246 contention, 240–246 locking, 232–239 database engine as service provider, 226–231 increasing load revealing performance problems, 227 indexes, virtues of, 226 data-driven partitioning and, 119 increased, with partitioning, 115 concurrent updates, foreign key indexing for, 69 conditional logic, 42 conditions applied at the wrong place, 98 order of evaluation, 89 (see also criteria; filtering conditions) connect by operator (Oracle), 172, 178, 181 propagating percentages across different tree levels, 196 substituting materialized path model for, 189 constraints implicit, unsoundness of, 17 major impact of, 17 violation of, 50 containers, contention when trying to access, 241 content lists, indexes and, 59 contention, 88, 240–246 architectural solutions, 242 DBA solutions for, 241 developmental solutions, 243 indexing system-generated primary keys, 71 insertion and, 240 physical layout of data and, 108 results from measures limiting, 243–246 correctness of data, 6 correlated subqueries, 94, 100 determining when to use, 137 looking for rows with no matching data, 162 performance effects when processing huge numbers of", + "source": "The Art of SQL.pdf", + "chunk_id": 284 + }, + { + "text": "developmental solutions, 243 indexing system-generated primary keys, 71 insertion and, 240 physical layout of data and, 108 results from measures limiting, 243–246 correctness of data, 6 correlated subqueries, 94, 100 determining when to use, 137 looking for rows with no matching data, 162 performance effects when processing huge numbers of rows, 147 testing for existence without other search criteria, 207 un-correlating, 100, 158 volume increases and, 256–261 corruption of data, 10 (see also data corruption) cost-based optimizers (CBOs), 36 COTS (commercial off-the-shelf) software package, 323 counts redundant, 41, 49, 310 using as test for existence, 163 CPU, excessive use of, 312 CPU-intensive operations, high level of concurrency for, 88 credit card validation procedures, 200–202 criteria defining result sets, 84 dynamic search criteria, 208–223 quality of, 330 (see also classic SQL patterns; conditions; filtering conditions), 223 current table and historical table, using, 21 (see also tables) current values, 160 cursor loops, 310 customer, defining, 7 D data containers, contention when trying to access, 241 data corruption, 10 data definition language (DDL), 33 data duplication detection of duplicate primary keys, 50 minimizing with normalization, 10 data entry errors, 6 data flow, 22 data manipulation language (DML), commit statements, 34 www.it-ebooks.info 3 3 8 I N D E X data manipulation operations, ranking in terms of overall cost, 263 data modeling, 4, 25 historical data, 19 data pages accessed by database engine, keeping as low as possible, 108 number visited by DBMS during a query, 103 number you are hitting, 312 data purges, 263 data redundancy, 8 data volumes (large), coping with, 248–278 data warehousing, 264–278 increasing volumes, 248–264 partitioning as solution, 262 sensitivity of operations to, 250–261 sudden increase in volume, 309 Data Warehouse Toolkit, The, 264 data warehousing, 264–278 cautions about, 277 data extraction, 268 integrity constraints and indexes, 270 loading data, 269 querying dimensions and facts, 270–273 star transformation, 273 emulating, 274–276 transformation, 268 database access libraries, 202–205 database accesses maximizing usefulness of, 36 minimizing, 317–319 multiplying, 310 database connections, managing use of, 29 database links, 205 performance and, 205 database optimizers (see optimizers) databases conflicting goals in optimizing physical layout of data, 108 locking entire database, 232 reorganizations of, 132 SQL and, 76–79 structural types, 106–107 data-driven partitioning, 116, 262 concurrency problems and, 119 true partitioning, 118 date arithmetic, 45 date type, 64 dates and times, 64 comparing dates, 43 partitioning, 262 partitioning historical data tables by date, 116 simple or range searching on dates, 156–161 datetime type, 64 DB2 call to get new sequence value, 236 clustering index, 114 range-clustering, 118 recursive with statement, 172, 179 DBA (database administrator), solutions to contention, 241, 244 DBMS closeness to kernel, 37–40 partition, different meanings of, 115 dbms_application_info package (Oracle), 28 DDL (data definition language), 33 decision support systems (DSS) interaction with production databases, 265 query tools, 266 (see also data warehousing) declarative language, 79 declarative processing, procedural vs., 35 decode( ) function, 42 delete operations, 263 against a database, 34 ranking in terms of overall cost, 263 denormalization, 20 caused", + "source": "The Art of SQL.pdf", + "chunk_id": 285 + }, + { + "text": "DDL (data definition language), 33 decision support systems (DSS) interaction with production databases, 265 query tools, 266 (see also data warehousing) declarative language, 79 declarative processing, procedural vs., 35 decode( ) function, 42 delete operations, 263 against a database, 34 ranking in terms of overall cost, 263 denormalization, 20 caused by ready-made solutions, 32 dependencies, analyzing, 8–11 Boolean columns, 14 checking attribute independence, 9 checking dependence on whole key, 8 data replications and, 13 depth, hierarchical data, 169, 179, 294 design irredeemable failure caused by, 249 performance and, 21 developmental solutions to contention, 243 dimension tables, 265 joining dimensions to fact tables, 273 querying, 270 dimensional modeling, 265 caution with, 277 facts and dimensions, 265 querying dimensions and facts, 270–273 SQL implications, 270 (see also data warehousing) directives to the optimizer, 144–145, 305, 329 disk addresses, indexes referring to, 80 distinct, 91 avoiding at the top level, 93 implicit, 95 regular join with, 137 www.it-ebooks.info I N D E X 339 distributed systems, 205–208 DML (data manipulation language), commit statements, 34 double conversion, 302, 303 DSS (see decision support systems) duplicate data detection of duplicate primary keys, 50 minimizing with normalization, 10 duration, determining without dedicated interval data type, 66 dynamic queries, 117, 309 dynamic search criteria, 208–223 defining movie database and main query, 209–216 mistakes common in queries with, 223 redesigning main query to fit criteria tightly, 216 wrapping SQL in PHP, 217–222 E efficiency of filtering conditions, 84, 90 of searches, descriptions and, 6 use of SQL, x (see also performance) ELSE logic, obtaining, 42 encapsulation of database accesses, how not to, 202–205 entry points, identifying, 56–59 errors, data entry, 6 evaluating filtering conditions, 90–98 evolutionary database model, 107 except operator, 163, 164 exception handling cost of, 52 forcing use of procedural logic, 53 exceptions, judicious use of, 50–53 excessive flexibility, dangers of, 18 execution plans, 319–330 forcing the right plan, 323–328 identifying the fastest, 320–322 using properly, 328–330 existence test, 93 correlated subquery without other search criteria, 207 within subquery, 95 explain command, 142 explode( ) operator, 169 exploding a materialized path, 293 explosion of links, 193 expressions, complex SQL expressions, 88 extending DBMS products, 37 extraction of data, 268 F fact dimension, 281 fact tables, 265 joining to dimensions, 273 querying, 270 querying star schema through, 276 federated systems, 205 fifth normal form (5NF), 5 filtering conditions, 84, 89–103 dynamically concatenated, 216 evaluation of, 90–98 large quantities of data, 98–102 meaning of, 89 proportions of retrieved data, 103 queries returning a few rows from direct, specific criteria, 129 financial structures, risk exposure calculations, 170 fine (granularity), 34 first normal form (1NF), 8 fixed, inflexible database model, 106 flexibility (excessive), dangers of, 18 foreign keys, 17 indexes and, 67–69 integrity constraint in master/detail relationship, 169 multiple indexing of the same columns, 69 referencing underlying tables in partitioned view, 117 (see also primary keys) free lists, 242 from clauses nested queries in, 144 uncorrelated subqueries rewritten as inline views, 96 uncorrelated subquery in, 138 full table scans, 135,", + "source": "The Art of SQL.pdf", + "chunk_id": 286 + }, + { + "text": "and, 67–69 integrity constraint in master/detail relationship, 169 multiple indexing of the same columns, 69 referencing underlying tables in partitioned view, 117 (see also primary keys) free lists, 242 from clauses nested queries in, 144 uncorrelated subqueries rewritten as inline views, 96 uncorrelated subquery in, 138 full table scans, 135, 146 indexes vs., 109 on tables expected to grow, 102 functions added to DBMS products, 37 aggregate, 150–156 built-in, advantages over external functions, 37 indexes with, 62–66, 129 appropriate use of, 66 implicit conversions and, 64 OLAP, operating on sliding windows, 149–150 user-defined, 44, 146 www.it-ebooks.info 3 4 0 I N D E X G global architecture, choice of, 249 good performance, defining, 311–317 checking against standards, 315–316 knowing what you get, 312–314 knowing what you spend, 312 performance goals, 316 granularity, 34 of locking, 232, 239 required by decision support systems, 268 table of contents vs. index, 59 greatest( ) function, 43 group by clauses filtering and, 99 use in aggregate statements, 156 H hardcoding, 203, 310 hardware balancing programming mistakes with power, 248 failures of, 24 hash indexes, 72 hash joins, 140, 147, 164, 262, 272 hash-partitioning, 118 having clause filtering after group by clause, 99 use in aggregate statements, 155 head counts, modeling, 190 computing head counts at every level, 192–194 heap-organized table, index-organized table vs., 112 hierarchical data depth (see depth, hierarchical data) practical example of hierarchies, 170 suggested explode( ) operator, 169 walking a tree in SQL, 177–189 aggregating values from trees, 190–198 bottom-up walk, 185–189 top-down walk, 178–185 walking hierarchies, 173 hierarchical databases, 168 hierarchical ordering of tables with IOTs and clustered indexes, 114 hints to the optimizer, 144–145 historical data, 156–161 current values, obtaining, 160 design of tables storing, 157 difficulties of working with, 19–21 many historical values per item, 160 many items with few historical values, 157 partitioning tables by date, 116 historical table, using, 21 (see also tables) host language, SQL statements embedded in, 42 hot spot in an index tree, 72 I I/Os (increased), relieving over-worked CPUs, 88 implicit constraints on data, 17 implicit rules, 13 implicit type conversions, 129 indexes and, 64 inconsistency of data, 10, 13 incorrect results, difficulty of spotting, 93 increasing data volumes (see large data volumes, coping with) indeterminate condition, 11 index range scan, 64 indexes, 330 adding to DMBS to tune it, 248 bitmap, 273 clustered vs. non-clustered index performance, 114 completing queries with data returned from, 146 content lists and, 59 contention within, improving, 244 correlated subqueries and, 94 costs of, 56 costs of maintaining, 108 as data repositories, 109–113 full table scans vs., 109 storing maximum data possible, 110 data access at atomic level of granularity, 60 data loading and, 270 data warehousing, 276 defining row order in tables, 114 dimension tables, 271 dimensional model, 277 finding physical location of a row, 62 foreign keys and, 67–69 with functions, 62–66, 129 appropriate situations for use, 66 implicit conversions and, 64 maintenance costs, 57 www.it-ebooks.info I N D E X 341 making them", + "source": "The Art of SQL.pdf", + "chunk_id": 287 + }, + { + "text": "warehousing, 276 defining row order in tables, 114 dimension tables, 271 dimensional model, 277 finding physical location of a row, 62 foreign keys and, 67–69 with functions, 62–66, 129 appropriate situations for use, 66 implicit conversions and, 64 maintenance costs, 57 www.it-ebooks.info I N D E X 341 making them work, 60 location of rows associated with index key, 61 multiple indexing of a column, 69 necessity of using, despite costs, 59 optimizer and, 79 partitioning, 115 performance and, 309 physical design of, 109 production database tuning and, 22 proportions of retrieved data, 103 queries returning small result set from very specific criteria, 129–137 criterion indexability, 132–137 data dispersion, 130–132 index usability, 129 query efficiency and index usage, 130 reference to disk addresses, 80 reverse, 71 solving contention problems, 243 searching, 92 system-generated keys, 70 temporary tables, 34 transactional databases, 59 variability of accesses, 72 virtues of, 226 index-organized tables (IOTs), 112 drawbacks of, 114 forcing row ordering, 113 solving contention problems with, 243 indirect criterion, 140 ingredients, identifying, 170 names and proportions in various products, 195–198 inline views, 135 rewriting uncorrelated subqueries as, 96 rewriting uncorrelated subqueries as joins in, 101 Inmon, Bill, 264 inner queries, 94 input, primitive, 130 insensitivity to volume increases, 250 insert statements dynamic, built for varying table names, 117 returning ... into ... clause, 236 insertion rates obtained with contention-limiting measures, 243 regular table vs. index-organized table, 112 insertions contention and, 240 into referencing tables, preventing, 68 ranking in terms of overall cost, 263 integrity checking, taken out of DBMS kernel, 117 integrity constraints data loading and, 270 foreign key constraint in master/detail relationship, 169 partitioned view and, 117 validation performed through, 206 inter-process communications, caused by use of procedural logic, 35 inter-process database link, 205 intersect operator, 164 intersection of result sets final result set, small intersection of broad criteria, 138–140 intermediate result sets, 85 interval data type, 66 IOTs (see index-organized tables) IP address database link, 205 J joins Cartesian, 285, 286 delaying till query end, 254–256 filtering conditions, 89, 92 older join syntax, 92 overlooked join condition, hidden by distinct, 93 pattern of, deciding factor, 93 performance and, 266 pre-joining tables, 123 problems with, 86 regular join with a distinct, 137 rewriting uncorrelated subqueries as joins in inline views, 101 self-joins on one table, 147–150 of two remote tables, 208 unions of complex joins, 98 (see also hash joins; nested loops) K Kent, William, 147 key values (index), querying indexes without full key, 110 Kimball, Ralph, 264 www.it-ebooks.info 3 4 2 I N D E X L lag( ) OLAP function, 150 large data volumes, coping with, 248–278 data warehousing, 264–278 increasing volumes, 248–264 partitioning as solution, 262 sensitivity of operations to, 250–261 sudden increase in volume, 309 last_insert_id( ) (MySQL), 235 latching, 88 (see also locking) LDAP (Lightweight Directory Access Protocol), 168 leading bytes, using for index queries, 110 leaf nodes, 168 aggregation of values stored in, 190–194 computing head counts at all levels, 192–194 modeling head counts, 190 least(", + "source": "The Art of SQL.pdf", + "chunk_id": 288 + }, + { + "text": "to, 250–261 sudden increase in volume, 309 last_insert_id( ) (MySQL), 235 latching, 88 (see also locking) LDAP (Lightweight Directory Access Protocol), 168 leading bytes, using for index queries, 110 leaf nodes, 168 aggregation of values stored in, 190–194 computing head counts at all levels, 192–194 modeling head counts, 190 least( ) function, 43 legacy system data, 248 putting into production, 249 levels in trees, 171 Lightweight Directory Access Protocol (LDAP), 168 like operator, 201 limitation criteria, 80 linear sensitivity to data volume increases, 251 linked server, 205 listener program, contacting in database connections, 30 list-partitioning, 119 lists content lists, indexes and, 59 querying a list variable, 294–297 selecting rows matching several items, 301–303 load, 310 database load, main indicators of, 312 increase in server load, 316 relating to execution of SQL statements, 313 loading data, 269 locking, 88, 232–239 causing sudden localized slowness, 309 committing and, 236 granularity of, 232 lock handling, 234–236 scalability and, 238 logic, programming into queries, 42 loop-back database link, 205 loops, 42 not executing queries in, 227 SQL statements executed in, 314 low cardinality, 109 M many-to-many relationship resolving between two tables, 69 master/detail relationships, 168 matching, finding the best match, 304 materialized path model (SQL trees), 172, 175 aggregating values stored in leaf nodes, 191 computing head counts at every level, 193 bottom-up tree walk, 186 path explosion, 293 substituting for connect by or recursive with, 189 top-down walk, 182–183 materialized views, 33 pre-joining vs., 123 measurement values, storage of, 265 memory addresses for data storage, 109 corrupting by mishandling pointers, 37 merge statement, Oracle 9i, 41 merge table (MySQL), 116 meta-design, 281 mini-dimensions, 270 modeling, 4 head counts, 190 computing at every level, 192–194 (see also data modeling; dimensional model; relational model) modifying data (see changing data, concurrency and) modularity, database programming and, 45 monitoring performance, 308–331 defining good performance, 311–317 checking against standards, 315–316 defining performance goals, 316 knowing what you get, 312–314 knowing what you spend, 312 execution plans, 319–330 forcing the right plan, 323–328 identifying the fastest, 320–322 using properly, 328–330 server load, 310 www.it-ebooks.info I N D E X 343 slow database, 308–310 statements currently being executed, 42 thinking in business tasks, 317–319 what really matters in improving queries, 330 “more-flexible-than-thou” construct, 18 movie database, 209–223 designing database and main search query, 209–216 redesigning main search query for tight fit, 216 wrapping SQL in PHP, 217–222 MySQL last_insert_id( ), 235 merge table, 116 PHP, using with, 209 N nested “containers”, 170 nested interval model (SQL trees), 173 nested loops, 24, 140, 142, 144 nested queries, 135 in from clause, 144 nested set model (SQL trees), 172, 176–177 aggregating values stored in leaf nodes, 190 bottom-up tree walk, 188 top-down walk, 183 network problems, 308 insufficient speed or bandwidth, 310 “next value to use” table, 235 nine situations, 128 (see also classic SQL patterns) nodes relational view of a tree, 169 tree representing a hierarchy, 168 (see also leaf nodes) non-linear sensitivity to data volume increases, 251–254 non-relational layer of", + "source": "The Art of SQL.pdf", + "chunk_id": 289 + }, + { + "text": "walk, 183 network problems, 308 insufficient speed or bandwidth, 310 “next value to use” table, 235 nine situations, 128 (see also classic SQL patterns) nodes relational view of a tree, 169 tree representing a hierarchy, 168 (see also leaf nodes) non-linear sensitivity to data volume increases, 251–254 non-relational layer of SQL applying last, 331 limiting thickness of, 256 OLAP functions, 159 normalization, 4–11 atomicity, 5 checking attribute independence, 9 checking dependence on the whole key, 8 data warehousing and 3NF design, 265 ensuring atomicity, 5 not exists ( ), using with a correlated subquery, 162 not in ( ), using with uncorrelated subquery, 161, 165, 166 null returns, 85 null values, 11–14, 166 indicating need for subtypes, 16 numerical values, comparing, 43 O object-oriented (OO) practice, relational database processing vs., 37 offensive coding with SQL, 48 OLAP functions current value for an item at a given date, 159 operating on sliding windows, 149–150 row_number( ), 180 “one size fits all” philosophy, 223 online analytical processing (OLAP), DB2, 52 online transaction processing (OLTP), 22 operating mode, 22 operating systems, contention issues and, 108 operational data stores, 265 operations (data manipulation), ranking in terms of overall cost, 263 operations, sensitivity to data volume increases, 250–261 disentangling subqueries, 256–261 insensitivity to, 250 linear sensitivity to, 251 non-linear sensitivity to, 251–254 optimistic concurrency control method, 49 optimizers, 77, 79 causing to take a different course, 329 checking execution plan, 142 circumstances not allowing efficient working of, 330 data distributions and, 161 directives, 305 directives or hints to, 144–145 heterogeneous, on distributed systems, 207 join filtering conditions and, 89 joins and filtering conditions, 92 limits of, 83 queries and, 79 rewriting of queries, 98 views and, 87 www.it-ebooks.info 3 4 4 I N D E X Oracle connect by operator, 172, 178, 181 propagating percentages across tree levels, 196 substituting materialized path model for, 189 dbms_application_info package, 28 rownums, 80, 87 tablespace referred to as a partition, 115 Oracle 9i Database, merge statement, 41 order of evaluation, filtering conditions, 89 ordering criteria, 80 ordering information, relations vs., 81 (see also sorts) orders/order_detail relationship, 168 outer joins expressing nonexistence with, 165 using in dynamically defined search query, 216 outer queries, 94 outriggers, 270 P pages, 108 locking, 232 pre-joined tables and, 124 parallelism adjusting to solve contention problems, 243 increased, with partitioning, 115 processing very large volumes of information, 147 parallelized queries, 207 parent/child link, master/detail relationship vs., 168 parents, multiple, 169 Parkinson’s Law, 310 partition clause, 149 partition key, 118 partition pruning, 118 partitioned view, 116 partitioning, 115–123, 330 archival and data purges, 263 data distribution and, 120 data-driven, 116 determining best way to partition data, 121–123 methods of, 118 round-robin, 116 scattering or clustering data, 119 solution for contention, 242 solving data volume problems, 262 partitions, 130 varying meanings among DBMS systems, 115 Pascal, Fabian, 169 patterns (see classic SQL patterns) percentages, propagating across different tree levels, 194–198 performance bottom-up tree walk, comparing for various models, 188 clustered vs. non-clustered indexes, 114 committing and,", + "source": "The Art of SQL.pdf", + "chunk_id": 290 + }, + { + "text": "data, 119 solution for contention, 242 solving data volume problems, 262 partitions, 130 varying meanings among DBMS systems, 115 Pascal, Fabian, 169 patterns (see classic SQL patterns) percentages, propagating across different tree levels, 194–198 performance bottom-up tree walk, comparing for various models, 188 clustered vs. non-clustered indexes, 114 committing and, 237 computing head counts from aggregation of information in leaf nodes, 194 costs of excess flexibility, 18 database connections, minimizing, 29 database engine, hardware, and I/O subsystems, 230 database links, costs of, 205 design and, 21 improving queries, what really matters, 330 joins and, 266 monitoring (see monitoring performance) non-relational layer of queries, 82 procedural logic and, 36 request complexity and, 230 results from contention-limiting measures, 243–246 solving problems with, 280–306 aggregating by range (bands), 297–299 columns that should have been rows, 284–289 columns that should have been something else, 289–294 finding the best match, 304 optimizer directives, 305 querying a list variable, 294–297 rows that should have been columns, 281–284 selecting rows matching several list items, 301–303 superseding a general case, 299–301 statements arriving faster than they are serviced, 231 top-down query, comparing various models, 184 transparent references to remote data, 23 tuning vs., x www.it-ebooks.info I N D E X 345 persistency layers, abstract, 202–205 PHP, 209 wrapping SQL in, 217–222 physical layout of data conflicting goals in optimization attempts, 108 forcing row ordering, 113 index performance and, 130–132 indexes as data repositories, 109–113 partitioning, 115–123 best way to partition, determining, 121–123 data distribution and, 120 scattering or clustering data, 119 pre-joining tables, 123 process requirements and physical design, 109 sacrificing simplicity with strong structuring, 124 pivot operator, 288 pivot tables binding a list, 303 creating, 285 multiplying rows, 286 passing list of values as single string to a statement, 294–297 using values, 286 place-holders (bind variables), 162 pointers, manipulation of, 37 Practical Issues in Database Management, 169 primary key index clustering index, using as, 114 insertion rate for IOT vs. regular table, 112 limiting contention within, 245 primary keys composite, 70 order of columns, 158 defining, 7 detection of duplicates, 50 indexing of, 59 subtype relationships, 16 system-generated, indexing of, 70 values in operational database vs. dimension identifiers, 269 whole key dependence and, 8 (see also foreign keys) primitive input, 130 principles, 3 probabilistic basis, coding on, 48 problems, defining before solution, 32 procedural logic achieving in database applications, 42 exception handling and, 53 in SQL, reasons for shunning, 35 procedural processes within a business, paying too much attention to, 31 procedural processing, declarative vs., 35 processes contention between (see contention) physical database layout and, 35 spawned by listener in database connections, 30 processing flow, 22 production databases, 265 program variable for sequence value, 236 purging data, 263 Q queries, 3, 76–103 answered by returning index data, 110 blanket views, performance impact of, 117 complex, and complex views, 86 distributed, 206 distributed and parallelized, 207 dynamic, 309 for variable number of search criteria, 214–216 mistakes commonly made in, 223 estimating behavior with increased data volumes, 254 expression", + "source": "The Art of SQL.pdf", + "chunk_id": 291 + }, + { + "text": "263 Q queries, 3, 76–103 answered by returning index data, 110 blanket views, performance impact of, 117 complex, and complex views, 86 distributed, 206 distributed and parallelized, 207 dynamic, 309 for variable number of search criteria, 214–216 mistakes commonly made in, 223 estimating behavior with increased data volumes, 254 expression of, association with implicit assumptions about data, 98 functionally equivalent, comparison to synonyms, 96 identifying, 28 improving, what really matters, 330 limits of the optimizer, 83 making as fast as possible, 108 nested, 135 non-relational aspects of, 81 number of data page hits by DBMS during performance of, 103 order of evaluation of conditions, 90 performance, whole key dependence and, 9 programming logic into, 42 relational component, 80 relational layer, doing maximum work in, 82 returning a very large amount of data, 102 rewriting by the optimizer, 98 size of result set, 85 SQL expression of, 77 tuning, 22 various layers of, 78 query tools, 266 www.it-ebooks.info 3 4 6 I N D E X R random numbers, using instead of system- generated values, 243, 244 range scans, 130 on clustered data, 113 converting variable-length comparison to common case, 200–202 reverse indexes and, 71 simple or range searching on dates, 156–161 range-clustering (DB2), 118 range-partitioning, 118 ranges aggregating by range (bands), 297–299 importance of well-defined boundaries, 262 ranking functions (SQL Server), 52 recovering databases, 24 recursive with statement, 172, 179 adjacency model, top-down tree walk, 180 propagating percentages across different tree levels, 196 substituting materialized path model for, 189 redundant data, 8 reference data in dimension tables, 265 referencing tables, preventing insertions into, 68 relational databases, 76 hierarchical databases vs., 114 processing, confusing with object- oriented methods, 37 SQL and, 76 relational model, 2 coherence of, 3 flexibility of, sacrificing by strongly structured data, 125 two-valued logic, 11 view of a tree, 169 relational operations, reporting requirements vs., 78 relational theory, 77 relations, 3 associating large numbers of possible characteristics in, 11 ordering information vs., 81 remote data querying, 24 transparent references to, 23 remote data sources, 205–208 remote validation checks, 206 reorganizations of databases, 132 reporting requirements, 77 request type, partitioning by, 122 requirements, evolution of, 10 response times, 147 (see also performance) result sets criteria defining, 84 difficulty of spotting incorrect data, 93 filtering conditions, 89–103 evaluation of, 90–98 large quantities of data, 98–102 meaning of, 89 proportions of retrieved data, 103 large, 146 obtained by aggregation, 150–156 predicated on absence of data, 161–166 size of, 85, 330 small intersection of broad criteria, 138–140 small intersection, indirect broad criteria, 140–145 small result set, indirect criteria, 137 small, from direct, specific criteria, 129–137 retrieval ratios, 60 returning ... into ... clause, 236 reverse indexes, 71 solving contention problems, 243 right-padding function (rpad( )), 202 risk exposure in a financial structure, 170 round-robin partitioning, 116 row_number( ) OLAP function, 149, 159, 180 rownums (Oracle), 80, 87 rows, 330 associated with index key, physical closeness of, 61 columns that should have been rows, 284–289 emptying a table of all rows, 33", + "source": "The Art of SQL.pdf", + "chunk_id": 292 + }, + { + "text": "right-padding function (rpad( )), 202 risk exposure in a financial structure, 170 round-robin partitioning, 116 row_number( ) OLAP function, 149, 159, 180 rownums (Oracle), 80, 87 rows, 330 associated with index key, physical closeness of, 61 columns that should have been rows, 284–289 emptying a table of all rows, 33 locking, 232, 238 matching several list items, selecting, 301–303 ordering of, forcing, 113 physical location, finding with an index, 62 primary key, defining, 7 proportions of retrieved data, 103 that should have been columns, 281–284 updating and inserting, dedicated statements for, 41 rpad( ) function (right-padding), 202 www.it-ebooks.info I N D E X 347 S scalability, locking and, 238 scattering data with partitioning, 120 schemas classical order schema, 91 movie database (example), 210 (see also star schema) science, art vs., xi searches dynamically defined criteria, 208–223 designing movie database and main query, 209–216 mistakes common in queries, 223 redesigning query for tight fit with criteria, 216 wrapping SQL in PHP, 217–222 efficiency, descriptions and, 6 second normal form (2NF), 9 select distinct queries, 9 select operator, filtering conditions, 89 selectivity of an index, 61 self-joins, 147, 282 performance and, 282, 284 semantic inconsistency, 13 sensitivity of operations to volume increases, 250–261 disentangling subqueries, 256–261 insensitivity to, 250 linear sensitivity to, 251 non-linear sensitivity to, 251–254 sequences call to database for new value, 236 not using in order to limit contention, 245 server load, 310 increase in, 316 servers, 205 set operators, 163–165 assembling data from several sources, 268 getting rid of unwanted data quickly, 98 sets nested set model, SQL trees, 172, 176–177 bottom-up walk, 188 top-down walk, 183 processing in SQL, 34 relational theory and, 78 slow database, 308–310 it’s not the database, 308 particularly slow query, 309 slow performance degradation reaching a threshold, 309 sudden global sluggishness, 308 sudden localized slowness, 309 snapshots, 33, 314 solutions (ready-made), problems caused by, 32 sorts, 80 volume increases and, 251–253 delaying joins to end of query, 254–256 spreading data across many servers, 23 SQL art of, governing factors, 84–88 number of tables, 85 number of users, 88 result set criteria, 84 result set size, 85 total quantity of data, 84 classic patterns (see classic SQL patterns) efficient use of, x general characteristics of, 76–83 relational and non-relational aspects, 80 SQL and databases, 76–79 SQL and the optimizer, 79 wrapping in PHP, 217–222 SQL Communication Area (SQLCA), 41 SQL engine cache, 314 SQL Server clustered index, 114 pivot and unpivot operators, 288 recursive with statement, 172 star schema, 265 querying tables, 271 querying through facts and dimensions, 276 star transformation, 273 emulating, 274–276 statements action-packed, 35 first questions to consider when writing, 90 mirroring business logic, 42 relating load to execution of, 313 succinct, 46 statistical functions, 77 statistics, automated collection of, 34 status, partitioning by, 122 storage options other than default, introducing complexity with, 124 peculiarities in, 330 temporary, 82 stored procedures, 10, 310 www.it-ebooks.info 3 4 8 I N D E X strategy, defining tactics with, 31 strings comparing, 43", + "source": "The Art of SQL.pdf", + "chunk_id": 293 + }, + { + "text": "313 succinct, 46 statistical functions, 77 statistics, automated collection of, 34 status, partitioning by, 122 storage options other than default, introducing complexity with, 124 peculiarities in, 330 temporary, 82 stored procedures, 10, 310 www.it-ebooks.info 3 4 8 I N D E X strategy, defining tactics with, 31 strings comparing, 43 extracting individual characters and returning them on separate rows, 290–293 structural types, databases, 106–107 subqueries correlated or uncorrelated, deciding between, 137 sensitivity of operations to data volume increases, 254 use in processing massive numbers of rows, 147 value of an item on a given date, 157 where clauses, 84 (see also correlated subqueries; uncorrelated subqueries) subtypes, 15 defining to deal with varying numbers of attributes, 281 succinct statements, 46 summary tables, pre-joining vs., 123 Sybase, clustered index, 114 syllogisms, 328 synchronization of databases after recovery, 25 synchronous processing, 22 synonyms, comparison to functionally equivalent queries, 96 system changes in, causing sudden global slowness, 309 complexity of, 24 database connections, 30 distributed, 205–208 tuning, 21 system-generated keys, 70 system-generated values, 235 not using in order to prevent contention, 243, 244 T table of contents, index vs., 59 tables, 2 current table and historical table, using, 21 enabling data retrieval as with table of contents, 60 forcing row ordering, 113 improving contention within indexes, 244 index-organized table (IOT), 112 locking, 232, 238 number involved in a query, 85 partitioning, 115 physical design of, 109 pre-joining, 123 remote, joins, 208 single table in hierarchical tree structure, 168 tables that are views, 106 valuation, 19–21 tablespace (Oracle), referred to as a partition, 115 tactics, defined by strategy, 31 target volumes for database systems, 249 temporary storage, 82 temporary tables, 264 disadvantages of using, 34 third normal form (3NF), 4, 9 data warehousing and, 265 threads spawned by listener in database connections, 30 three-valued logic (implied by nulls), 12 “tight-fit” query, 216 time information, 64 timestamps, 148 reverse indexing and, 72 top-down tree walk, 178–185 adjacency model, 178–182 aggregating values stored in leaf nodes, 190 materialized path model, 182–183 nested set model, 183 performance for various models, 184 transaction space, 242 transactional databases indexes, 109 indexing requirements, 59 transactions good practices in, 234 across heterogeneous systems, 206 locking and committing, 236 Transact-SQL, @@IDENTITY system variable, 235 transformations, 268 mathematical equivalence of, 83 star transformation, 273 emulating, 274–276 transparent references to remote data, 23 tree structures, 168–171 aggregating values from trees, 190–198 propagating percentages across levels, 194–198 values stored in leaf nodes, 190–194 www.it-ebooks.info I N D E X 349 hierarchies, practical examples of, 170 of indexes, 62 master/detail relationships vs., 168 materialized path model, exploding, 293 practical implementation of trees, 174–177 adjacency model, 174 materialized path model, 175 nested set model, 176–177 representing trees in SQL database, 172–174 walking a tree in SQL, 177–189 bottom-up walk, 185–189 top-down walk, 178–185 triggers, 10, 330 index maintenance costs vs., 58 Tropashko, Vadim, 173 truncate operations, 33 delete vs., 263 truths, 3 freedom in the choice of, 4 tuning, 21 adding indexes, 248 performance vs., x two-valued logic,", + "source": "The Art of SQL.pdf", + "chunk_id": 294 + }, + { + "text": "SQL database, 172–174 walking a tree in SQL, 177–189 bottom-up walk, 185–189 top-down walk, 178–185 triggers, 10, 330 index maintenance costs vs., 58 Tropashko, Vadim, 173 truncate operations, 33 delete vs., 263 truths, 3 freedom in the choice of, 4 tuning, 21 adding indexes, 248 performance vs., x two-valued logic, 11 types implicit conversions, 129 partitioning by, 122 U uncorrelated subqueries, 94, 158 in classic style or in from clause, 138 not in ( ), using with, 162 rewriting as a join in an inline view, 101 rewriting as inline views in from clause, 96 sensitivity of operations to data volume increase, 254 testing for existence without other search criteria, 207 union operator, 164 querying large quantities of data, 98 unions of complex joins, 98 of intermediate result sets, 85 overhead of querying large union view, 117 partitioned tables, 116 uniqueness, enforcement of, 50, 117 unnecessary coding, avoiding, 41 unpivot operator, 288 update statement, returning ... into ... clause, 236 updates against a database, 34 combining multiple into one, 43 concurrent foreign key indexing, 69 costly massive updates, 314 locking and scalability, 238 making as fast as possible, 108 multiple massive updates to a table, 268 optimistic concurrency control, 49 ranking in terms of overall cost, 263 (see changing data, concurrency and) useless queries, 310 user-defined functions, 44, 146 users not complaining about performance, 315 number of, concurrency and, 88 perception of performance improvement, 316 V valuation, 19 valuation tables, 19–21 variable number of search criteria, mistakes in queries containing, 223 variables, binding, 294, 296 PHP, 218–222 views, 3, 329 complex, 86 partitioned view, 116 tables as, 106 where clauses, 84 volume of data (see large data volumes, coping with) W warehousing data (see data warehousing) where clauses in aggregate statements, 156 atomic attributes, 6, 64 filtering conditions, 84, 89 filtering conditions independent of the aggregate, 99 join operator, 89 two-valued logic, 11 with statement, recursive (see recursive with statement) X XML, 168, 290 www.it-ebooks.info www.it-ebooks.info A B O U T T H E A U T H O R S STÉPHANE FAROULT first discovered relational databases and the SQL language back in 1983. He joined Oracle France in their early days (after a brief spell with IBM and a bout of teach- ing at the University of Ottawa) and soon developed an interest in performance and tuning topics. After leaving Oracle in 1988, he briefly tried to reform and did a bit of operational research, but after one year, he succumbed again to relational databases. He has been con- tinuously performing database consultancy since then, and founded RoughSea Ltd in 1998 (http://www.roughsea.com). Stéphane Faroult has written (in French) Fortran Structuré et Méthodes Numériques (Dunod, 1986, with Didier Simon) and a number of articles in English, in magazines such as Oracle Scene (the UK Oracle user group magazine) and Select (the North American Oracle user group magazine), as well as on the Web (including the online edition of Oracle Magazine). He has also been a speaker at a number of", + "source": "The Art of SQL.pdf", + "chunk_id": 295 + }, + { + "text": "a number of articles in English, in magazines such as Oracle Scene (the UK Oracle user group magazine) and Select (the North American Oracle user group magazine), as well as on the Web (including the online edition of Oracle Magazine). He has also been a speaker at a number of user group conferences in the U.S., in the UK, and in Norway. PETER ROBSON graduated in geology from Durham University (1968), then taught at Edin- burgh University, obtaining an M.Phil in geology in 1975. After working in Greece as a geologist, he specialized in both geological and medical databases at the University of Newcastle. He has worked with databases since 1977, relational databases since 1981, and Oracle since 1985, in roles which included developer, data architect, and database administrator. In 1980, Peter joined the British Geological Survey and was influential in guiding their adoption of relational DBMS. He has specialized in aspects of the SQL system as well as data modeling from corporate architecture down to the departmental level. Peter has presented at various Oracle database conferences in the UK, Europe, and North America, and he has published in various specialist database magazines. Currently, he is a Director on the Board of the UK Oracle User Group; he can be contacted via his own domain at peter.robson@justsql.com. www.it-ebooks.info", + "source": "The Art of SQL.pdf", + "chunk_id": 296 + }, + { + "text": "1. Preface a. Why Learn SQL? b. Why Use This Book to Do It? c. Structure of This Book d. Conventions Used in This Book e. Using Code Examples f. O’Reilly Online Learning g. How to Contact Us 2. 1. A Little Background a. Introduction to Databases i. Nonrelational Database Systems ii. The Relational Model iii. Some Terminology b. What Is SQL? i. SQL Statement Classes ii. SQL: A Nonprocedural Language iii. SQL Examples c. What Is MySQL? d. SQL Unplugged e. What’s in Store 3. 2. Creating and Populating a Database a. Creating a MySQL Database b. Using the mysql Command-Line Tool c. MySQL Data Types i. Character Data ii. Numeric Data iii. Temporal Data d. Table Creation i. Step 1: Design ii. Step 2: Refinement iii. Step 3: Building SQL Schema Statements e. Populating and Modifying Tables i. Inserting Data ii. Updating Data iii. Deleting Data f. When Good Statements Go Bad i. Nonunique Primary Key ii. Nonexistent Foreign Key iii. Column Value Violations iv. Invalid Date Conversions g. The Sakila Database 4. 3. Query Primer a. Query Mechanics b. Query Clauses c. The select Clause i. Column Aliases ii. Removing Duplicates d. The from Clause i. Tables ii. Table Links iii. Defining Table Aliases e. The where Clause f. The group by and having Clauses g. The order by Clause i. Ascending Versus Descending Sort Order ii. Sorting via Numeric Placeholders h. Test Your Knowledge i. Exercise 3-1 ii. Exercise 3-2 iii. Exercise 3-3 iv. Exercise 3-4 5. 4. Filtering a. Condition Evaluation i. Using Parentheses ii. Using the not Operator b. Building a Condition c. Condition Types i. Equality Conditions ii. Range Conditions iii. Membership Conditions iv. Matching Conditions d. Null: That Four-Letter Word e. Test Your Knowledge i. Exercise 4-1 ii. Exercise 4-2 iii. Exercise 4-3 iv. Exercise 4-4 6. 5. Querying Multiple Tables a. What Is a Join? i. Cartesian Product ii. Inner Joins iii. The ANSI Join Syntax b. Joining Three or More Tables i. Using Subqueries As Tables ii. Using the Same Table Twice c. Self-Joins d. Test Your Knowledge i. Exercise 5-1 ii. Exercise 5-2 iii. Exercise 5-3 7. 6. Working with Sets a. Set Theory Primer b. Set Theory in Practice c. Set Operators i. The union Operator ii. The intersect Operator iii. The except Operator d. Set Operation Rules i. Sorting Compound Query Results ii. Set Operation Precedence e. Test Your Knowledge i. Exercise 6-1 ii. Exercise 6-2 iii. Exercise 6-3 8. 7. Data Generation, Manipulation, and Conversion a. Working with String Data i. String Generation ii. String Manipulation b. Working with Numeric Data i. Performing Arithmetic Functions ii. Controlling Number Precision iii. Handling Signed Data c. Working with Temporal Data i. Dealing with Time Zones ii. Generating Temporal Data iii. Manipulating Temporal Data d. Conversion Functions e. Test Your Knowledge i. Exercise 7-1 ii. Exercise 7-2 iii. Exercise 7-3 9. 8. Grouping and Aggregates a. Grouping Concepts b. Aggregate Functions i. Implicit Versus Explicit Groups ii. Counting Distinct", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 0 + }, + { + "text": "Temporal Data i. Dealing with Time Zones ii. Generating Temporal Data iii. Manipulating Temporal Data d. Conversion Functions e. Test Your Knowledge i. Exercise 7-1 ii. Exercise 7-2 iii. Exercise 7-3 9. 8. Grouping and Aggregates a. Grouping Concepts b. Aggregate Functions i. Implicit Versus Explicit Groups ii. Counting Distinct Values iii. Using Expressions iv. How Nulls Are Handled c. Generating Groups i. Single-Column Grouping ii. Multicolumn Grouping iii. Grouping via Expressions iv. Generating Rollups d. Group Filter Conditions e. Test Your Knowledge i. Exercise 8-1 ii. Exercise 8-2 iii. Exercise 8-3 10. 9. Subqueries a. What Is a Subquery? b. Subquery Types c. Noncorrelated Subqueries i. Multiple-Row, Single-Column Subqueries ii. Multicolumn Subqueries d. Correlated Subqueries i. The exists Operator ii. Data Manipulation Using Correlated Subqueries e. When to Use Subqueries i. Subqueries As Data Sources ii. Subqueries As Expression Generators f. Subquery Wrap-up g. Test Your Knowledge i. Exercise 9-1 ii. Exercise 9-2 iii. Exercise 9-3 11. 10. Joins Revisited a. Outer Joins i. Left Versus Right Outer Joins ii. Three-Way Outer Joins b. Cross Joins c. Natural Joins d. Test Your Knowledge i. Exercise 10-1 ii. Exercise 10-2 iii. Exercise 10-3 (Extra Credit) 12. 11. Conditional Logic a. What Is Conditional Logic? b. The Case Expression i. Searched Case Expressions ii. Simple Case Expressions c. Case Expression Examples i. Result Set Transformations ii. Checking for Existence iii. Division-by-Zero Errors iv. Conditional Updates v. Handling Null Values d. Test Your Knowledge i. Exercise 11-1 ii. Exercise 11-2 13. 12. Transactions a. Multiuser Databases i. Locking ii. Lock Granularities b. What Is a Transaction? i. Starting a Transaction ii. Ending a Transaction iii. Transaction Savepoints c. Test Your Knowledge i. Exercise 12-1 14. 13. Indexes and Constraints a. Indexes i. Index Creation ii. Types of Indexes iii. How Indexes Are Used iv. The Downside of Indexes b. Constraints i. Constraint Creation c. Test Your Knowledge i. Exercise 13-1 ii. Exercise 13-2 15. 14. Views a. What Are Views? b. Why Use Views? i. Data Security ii. Data Aggregation iii. Hiding Complexity iv. Joining Partitioned Data c. Updatable Views i. Updating Simple Views ii. Updating Complex Views d. Test Your Knowledge i. Exercise 14-1 ii. Exercise 14-2 16. 15. Metadata a. Data About Data b. Information_Schema c. Working with Metadata i. Schema Generation Scripts ii. Deployment Verification iii. Dynamic SQL Generation d. Test Your Knowledge i. Exercise 15-1 ii. Exercise 15-2 17. 16. Analytic Functions a. Analytic Function Concepts i. Data Windows ii. Localized Sorting b. Ranking i. Ranking Functions ii. Generating Multiple Rankings c. Reporting Functions i. Window Frames ii. Lag and Lead d. Test Your Knowledge i. Exercise 16-1 ii. Exercise 16-2 iii. Exercise 16-3 18. 17. Working with Large Databases a. Partitioning i. Partitioning Concepts ii. Table Partitioning iii. Index Partitioning iv. Partitioning Methods v. Partitioning Benefits b. Sharding c. Big Data i. Hadoop ii. NoSQL and Document Databases iii. Cloud Computing iv. Future of SQL 19. 18. SQL and Big Data a. Apache Drill b. Drill and", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 1 + }, + { + "text": "Large Databases a. Partitioning i. Partitioning Concepts ii. Table Partitioning iii. Index Partitioning iv. Partitioning Methods v. Partitioning Benefits b. Sharding c. Big Data i. Hadoop ii. NoSQL and Document Databases iii. Cloud Computing iv. Future of SQL 19. 18. SQL and Big Data a. Apache Drill b. Drill and MySQL c. Drill and MongoDB d. Drill with Multiple Data Sources Learning SQL THIRD EDITION Generate, Manipulate, and Retrieve Data With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles. Alan Beaulieu Learning SQL by Alan Beaulieu Copyright © 2020 Alan Beaulieu. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com/safari). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Jessica Haberman Development Editor: Jeff Bleiel Production Editor: Deborah Baker Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Rebecca Demarest May 2020: Third Edition Revision History for the Early Release 2019-12-11: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781492057611 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Learning SQL, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 978-1-492-05754-3 [LSI] Preface Programming languages come and go constantly, and very few languages in use today have roots going back more than a decade or so. Some examples are Cobol, which is still used quite heavily in mainframe environments, and C, which is still quite popular for operating system and server development and for embedded systems. In the database arena, we have SQL, whose roots go all the way back to the 1970s. SQL is the language for generating, manipulating, and retrieving data from a relational database. One of the reasons for the popularity of relational databases is that properly designed relational databases can handle huge amounts of data. When working with large data sets, SQL is akin to one of those snazzy digital cameras with the high-power zoom lens in that you can use SQL to", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 2 + }, + { + "text": "One of the reasons for the popularity of relational databases is that properly designed relational databases can handle huge amounts of data. When working with large data sets, SQL is akin to one of those snazzy digital cameras with the high-power zoom lens in that you can use SQL to look at large sets of data, or you can zoom in on individual rows (or anywhere in between). Other database management systems tend to break down under heavy loads because their focus is too narrow (the zoom lens is stuck on maximum), which is why attempts to dethrone relational databases and SQL have largely failed. Therefore, even though SQL is an old language, it is going to be around for a lot longer and has a bright future in store. Why Learn SQL? If you are going to work with a relational database, whether you are writing applications, performing administrative tasks, or generating reports, you will need to know how to interact with the data in your database. Even if you are using a tool that generates SQL for you, such as a reporting tool, there may be times when you need to bypass the automatic generation feature and write your own SQL statements. Learning SQL has the added benefit of forcing you to confront and understand the data structures used to store information about your organization. As you become comfortable with the tables in your database, you may find yourself proposing modifications or additions to your database schema. Why Use This Book to Do It? The SQL language is broken into several categories. Statements used to create database objects (tables, indexes, constraints, etc.) are collectively known as SQL schema statements. The statements used to create, manipulate, and retrieve the data stored in a database are known as the SQL data statements. If you are an administrator, you will be using both SQL schema and SQL data statements. If you are a programmer or report writer, you may only need to use (or be allowed to use) SQL data statements. While this book demonstrates many of the SQL schema statements, the main focus of this book is on programming features. With only a handful of commands, the SQL data statements look deceptively simple. In my opinion, many of the available SQL books help to foster this notion by only skimming the surface of what is possible with the language. However, if you are going to work with SQL, it behooves you to understand fully the capabilities of the language and how different features can be combined to produce powerful results. I feel that this is the only book that provides detailed coverage of the SQL language without the added benefit of doubling as a “door stop” (you know, those 1,250- page “complete references” that tend to gather dust on people’s cubicle shelves). While the examples in this book run on MySQL, Oracle Database, and SQL Server, I had to pick one of those products to host my sample database and", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 3 + }, + { + "text": "of doubling as a “door stop” (you know, those 1,250- page “complete references” that tend to gather dust on people’s cubicle shelves). While the examples in this book run on MySQL, Oracle Database, and SQL Server, I had to pick one of those products to host my sample database and to format the result sets returned by the example queries. Of the three, I chose MySQL because it is freely obtainable, easy to install, and simple to administer. For those readers using a different server, I ask that you download and install MySQL and load the sample database so that you can run the examples and experiment with the data. Structure of This Book This book is divided into 15 chapters and 3 appendixes: Chapter 1, explores the history of computerized databases, including the rise of the relational model and the SQL language. Chapter 2, demonstrates how to create a MySQL database, create the tables used for the examples in this book, and populate the tables with data. Chapter 3, introduces the selectstatement and further demonstrates the most common clauses (select, from, where). Chapter 4, demonstrates the different types of conditions that can be used in the whereclause of a select, update, or deletestatement. Chapter 5, shows how queries can utilize multiple tables via table joins. Chapter 6, is all about data sets and how they can interact within queries. Chapter 7, demonstrates several built-in functions used for manipulating or converting data. Chapter 8, shows how data can be aggregated. Chapter 9, introduces the subquery (a personal favorite) and shows how and where they can be utilized. Chapter 10, further explores the various types of table joins. Chapter 11, explores how conditional logic (i.e., if-then-else) can be utilized in select, insert, update, and deletestatements. Chapter 12, introduces transactions and shows how to use them. Chapter 13, explores indexes and constraints. Chapter 14, shows how to build an interface to shield users from data complexities. Chapter 15, demonstrates the utility of the data dictionary. Appendix A shows the database schema used for all examples in the book. Appendix B demonstrates some of the interesting non-ANSI features of MySQL’s SQL implementation. Appendix C shows solutions to the chapter exercises. Conventions Used in This Book The following typographical conventions are used in this book: Italic Used for filenames, directory names, and URLs. Also used for emphasis and to indicate the first use of a technical term. Constant width Used for code examples and to indicate SQL keywords within text. Constant width italic Used to indicate user-defined terms. plainUPPERCASE Used to indicate SQL keywords within example code. Constant width bold Indicates user input in examples showing an interaction. Also indicates emphasized code elements to which you should pay particular attention. NOTE Indicates a tip, suggestion, or general note. For example, I use notes to point you to useful new features in Oracle9i. WARNING Indicates a warning or caution. For example, I’ll tell you if a certain SQL clause might have unintended consequences if not", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 4 + }, + { + "text": "which you should pay particular attention. NOTE Indicates a tip, suggestion, or general note. For example, I use notes to point you to useful new features in Oracle9i. WARNING Indicates a warning or caution. For example, I’ll tell you if a certain SQL clause might have unintended consequences if not used carefully. Using Code Examples This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Learning SQL, Third Edition, by Alan Beaulieu. Copyright 2020 Alan Beaulieu, 978-1-492-05761-1.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. O’Reilly Online Learning NOTE For more than 40 years, O’Reilly has provided technology and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, conferences, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, please visit http://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://www.oreilly.com/catalog/9781492057611. Emails us with comments or technical questions at bookquestions@oreilly.com. For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://www.youtube.com/oreillymedia Chapter 1. A Little Background Before we roll up our sleeves and get to work, it would be helpful to survey the history of database technology in order to better understand how relational databases and the SQL language evolved. Therefore, I’d like to start by introducing some basic database concepts and looking at the history of computerized data storage and retrieval. NOTE For those readers anxious to start writing queries, feel free to skip ahead to Chapter 3, but I recommend returning later to the first two chapters in order to better understand the history and utility of", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 5 + }, + { + "text": "database concepts and looking at the history of computerized data storage and retrieval. NOTE For those readers anxious to start writing queries, feel free to skip ahead to Chapter 3, but I recommend returning later to the first two chapters in order to better understand the history and utility of the SQL language. Introduction to Databases A database is nothing more than a set of related information. A telephone book, for example, is a database of the names, phone numbers, and addresses of all people living in a particular region. While a telephone book is certainly a ubiquitous and frequently used database, it suffers from the following: Finding a person’s telephone number can be time-consuming, especially if the telephone book contains a large number of entries. A telephone book is indexed only by last/first names, so finding the names of the people living at a particular address, while possible in theory, is not a practical use for this database. From the moment the telephone book is printed, the information becomes less and less accurate as people move into or out of a region, change their telephone numbers, or move to another location within the same region. The same drawbacks attributed to telephone books can also apply to any manual data storage system, such as patient records stored in a filing cabinet. Because of the cumbersome nature of paper databases, some of the first computer applications developed were database systems, which are computerized data storage and retrieval mechanisms. Because a database system stores data electronically rather than on paper, a database system is able to retrieve data more quickly, index data in multiple ways, and deliver up-to-the-minute information to its user community. Early database systems managed data stored on magnetic tapes. Because there were generally far more tapes than tape readers, technicians were tasked with loading and unloading tapes as specific data was requested. Because the computers of that era had very little memory, multiple requests for the same data generally required the data to be read from the tape multiple times. While these database systems were a significant improvement over paper databases, they are a far cry from what is possible with today’s technology. (Modern database systems can manage petabytes of data, accessed by clusters of servers each caching tens of gigabytes of that data in high-speed memory, but I’m getting a bit ahead of myself.) Nonrelational Database Systems NOTE This section contains some background information about pre-relational database systems. For those readers eager to dive into SQL, feel free to skip ahead a couple of pages to the next section. Over the first several decades of computerized database systems, data was stored and represented to users in various ways. In a hierarchical database system, for example, data is represented as one or more tree structures. Figure 1-1 shows how data relating to George Blake’s and Sue Smith’s bank accounts might be represented via tree structures. Figure 1-1. Hierarchical view of account data George and Sue each have their own", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 6 + }, + { + "text": "In a hierarchical database system, for example, data is represented as one or more tree structures. Figure 1-1 shows how data relating to George Blake’s and Sue Smith’s bank accounts might be represented via tree structures. Figure 1-1. Hierarchical view of account data George and Sue each have their own tree containing their accounts and the transactions on those accounts. The hierarchical database system provides tools for locating a particular customer’s tree and then traversing the tree to find the desired accounts and/or transactions. Each node in the tree may have either zero or one parent and zero, one, or many children. This configuration is known as a single-parent hierarchy. Another common approach, called the network database system, exposes sets of records and sets of links that define relationships between different records. Figure 1-2 shows how George’s and Sue’s same accounts might look in such a system. Figure 1-2. Network view of account data In order to find the transactions posted to Sue’s money market account, you would need to perform the following steps: 1. Find the customer record for Sue Smith. 2. Follow the link from Sue Smith’s customer record to her list of accounts. 3. Traverse the chain of accounts until you find the money market account. 4. Follow the link from the money market record to its list of transactions. One interesting feature of network database systems is demonstrated by the set of product records on the far right of Figure 1-2. Notice that each product record (Checking, Savings, etc.) points to a list of account records that are of that product type. Account records, therefore, can be accessed from multiple places (both customer records and product records), allowing a network database to act as a multiparent hierarchy. Both hierarchical and network database systems are alive and well today, although generally in the mainframe world. Additionally, hierarchical database systems have enjoyed a rebirth in the directory services realm, such as Microsoft’s Active Directory and the open-source Apache Directory Server. Beginning in the 1970s, however, a new way of representing data began to take root, one that was more rigorous yet easy to understand and implement. The Relational Model In 1970, Dr. E. F. Codd of IBM’s research laboratory published a paper titled “A Relational Model of Data for Large Shared Data Banks” that proposed that data be represented as sets of tables. Rather than using pointers to navigate between related entities, redundant data is used to link records in different tables. Figure 1-3 shows how George’s and Sue’s account information would appear in this context. Figure 1-3. Relational view of account data There are four tables in Figure 1-3 representing the four entities discussed so far: customer, product, account, and transaction. Looking across the top of the customer table in Figure 1-3, you can see three columns: cust_id (which contains the customer’s ID number), fname (which contains the customer’s first name), and lname (which contains the customer’s last name). Looking down the side of the customer table,", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 7 + }, + { + "text": "product, account, and transaction. Looking across the top of the customer table in Figure 1-3, you can see three columns: cust_id (which contains the customer’s ID number), fname (which contains the customer’s first name), and lname (which contains the customer’s last name). Looking down the side of the customer table, you can see two rows, one containing George Blake’s data and the other containing Sue Smith’s data. The number of columns that a table may contain differs from server to server, but it is generally large enough not to be an issue (Microsoft SQL Server, for example, allows up to 1,024 columns per table). The number of rows that a table may contain is more a matter of physical limits (i.e., how much disk drive space is available) and maintainability (i.e., how large a table can get before it becomes difficult to work with) than of database server limitations. Each table in a relational database includes information that uniquely identifies a row in that table (known as the primary key), along with additional information needed to describe the entity completely. Looking again at the customer table, the cust_id column holds a different number for each customer; George Blake, for example, can be uniquely identified by customer ID #1. No other customer will ever be assigned that identifier, and no other information is needed to locate George Blake’s data in the customer table. NOTE Every database server provides a mechanism for generating unique sets of numbers to use as primary key values, so you won’t need to worry about keeping track of what numbers have been assigned. While I might have chosen to use the combination of the fname and lname columns as the primary key (a primary key consisting of two or more columns is known as a compound key), there could easily be two or more people with the same first and last names that have accounts at the bank. Therefore, I chose to include the cust_id column in the customer table specifically for use as a primary key column. NOTE In this example, choosing fname/lname as the primary key would be referred to as a natural key, whereas the choice of cust_id would be referred to as a surrogate key. The decision whether to employ natural or surrogate keys is up to the database designer, but in this particular case the choice is clear, since a person’s last name may change (such as when a person adopts a spouse’s last name), and primary key columns should never be allowed to change once a value has been assigned. Some of the tables also include information used to navigate to another table; this is where the “redundant data” mentioned earlier comes in. For example, the account table includes a column called cust_id, which contains the unique identifier of the customer who opened the account, along with a column called product_cd, which contains the unique identifier of the product to which the account will conform. These columns are known as foreign", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 8 + }, + { + "text": "comes in. For example, the account table includes a column called cust_id, which contains the unique identifier of the customer who opened the account, along with a column called product_cd, which contains the unique identifier of the product to which the account will conform. These columns are known as foreign keys, and they serve the same purpose as the lines that connect the entities in the hierarchical and network versions of the account information. If you are looking at a particular account record and want to know more information about the customer who opened the account, you would take the value of the cust_id column and use it to find the appropriate row in the customer table (this process is known, in relational database lingo, as a join; joins are introduced in Chapter 3 and probed deeply in Chapters Chapter 5 and Chapter 10). It might seem wasteful to store the same data many times, but the relational model is quite clear on what redundant data may be stored. For example, it is proper for the account table to include a column for the unique identifier of the customer who opened the account, but it is not proper to include the customer’s first and last names in the account table as well. If a customer were to change her name, for example, you want to make sure that there is only one place in the database that holds the customer’s name; otherwise, the data might be changed in one place but not another, causing the data in the database to be unreliable. The proper place for this data is the customer table, and only the cust_id values should be included in other tables. It is also not proper for a single column to contain multiple pieces of information, such as a name column that contains both a person’s first and last names, or an address column that contains street, city, state, and zip code information. The process of refining a database design to ensure that each independent piece of information is in only one place (except for foreign keys) is known as normalization. Getting back to the four tables in Figure 1-3, you may wonder how you would use these tables to find George Blake’s transactions against his checking account. First, you would find George Blake’s unique identifier in the customer table. Then, you would find the row in the account table whose cust_id column contains George’s unique identifier and whose product_cd column matches the row in the product table whose name column equals “Checking.” Finally, you would locate the rows in the transaction table whose account_id column matches the unique identifier from the account table. This might sound complicated, but you can do it in a single command, using the SQL language, as you will see shortly. Some Terminology I introduced some new terminology in the previous sections, so maybe it’s time for some formal definitions. Table 1-1 shows the terms we use for the remainder of the book", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 9 + }, + { + "text": "but you can do it in a single command, using the SQL language, as you will see shortly. Some Terminology I introduced some new terminology in the previous sections, so maybe it’s time for some formal definitions. Table 1-1 shows the terms we use for the remainder of the book along with their definitions. Table 1-1. Terms and definitions Term Definition Entity Something of interest to the database user community. Examples include customers, parts, geographic locations, etc. Column An individual piece of data stored in a table. Row A set of columns that together completely describe an entity or some action on an entity. Also called a record. Table A set of rows, held either in memory (nonpersistent) or on permanent storage (persistent). Result set Another name for a nonpersistent table, generally the result of an SQL query. Primary key One or more columns that can be used as a unique identifier for each row in a table. Foreign key One or more columns that can be used together to identify a single row in another table. What Is SQL? Along with Codd’s definition of the relational model, he proposed a language called DSL/Alpha for manipulating the data in relational tables. Shortly after Codd’s paper was released, IBM commissioned a group to build a prototype based on Codd’s ideas. This group created a simplified version of DSL/Alpha that they called SQUARE. Refinements to SQUARE led to a language called SEQUEL, which was, finally, shortened to SQL.While SQL began as a language used to manipulate data in relational databases, it has evolved (as you will see toward the end of this book) to be a language for manipulating data across various database technologies. SQL is now over 40 years old, and it has undergone a great deal of change along the way. In the mid-1980s, the American National Standards Institute (ANSI) began working on the first standard for the SQL language, which was published in 1986. Subsequent refinements led to new releases of the SQL standard in 1989, 1992, 1999, 2003, 2006, 2008, 2011, 2016. Along with refinements to the core language, new features have been added to the SQL language to incorporate object-oriented functionality, among other things. The later standards focus on the integration of related technologies, such as XML and JSON. SQL goes hand in hand with the relational model because the result of an SQL query is a table (also called, in this context, a result set). Thus, a new permanent table can be created in a relational database simply by storing the result set of a query. Similarly, a query can use both permanent tables and the result sets from other queries as inputs (we explore this in detail in Chapter 9). One final note: SQL is not an acronym for anything (although many people will insist it stands for “Structured Query Language”). When referring to the language, it is equally acceptable to say the letters individually (i.e., S. Q. L.) or to use the word sequel.", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 10 + }, + { + "text": "in detail in Chapter 9). One final note: SQL is not an acronym for anything (although many people will insist it stands for “Structured Query Language”). When referring to the language, it is equally acceptable to say the letters individually (i.e., S. Q. L.) or to use the word sequel. SQL Statement Classes The SQL language is divided into several distinct parts: the parts that we explore in this book include SQL schema statements, which are used to define the data structures stored in the database; SQL data statements, which are used to manipulate the data structures previously defined using SQL schema statements; and SQL transaction statements, which are used to begin, end, and roll back transactions (covered in Chapter 12). For example, to create a new table in your database, you would use the SQL schema statement create table, whereas the process of populating your new table with data would require the SQL data statement insert. To give you a taste of what these statements look like, here’s an SQL schema statement that creates a table called corporation: CREATE TABLE corporation (corp_id SMALLINT, name VARCHAR(30), CONSTRAINT pk_corporation PRIMARY KEY (corp_id) ); This statement creates a table with two columns, corp_id and name, with the corp_id column identified as the primary key for the table. We probe the finer details of this statement, such as the different data types available with MySQL, in Chapter 2. Next, here’s an SQL data statement that inserts a row into the corporation table for Acme Paper Corporation: INSERT INTO corporation (corp_id, name) VALUES (27, 'Acme Paper Corporation'); This statement adds a row to the corporation table with a value of 27 for the corp_id column and a value of Acme Paper Corporation for the name column. Finally, here’s a simple select statement to retrieve the data that was just created: mysql< SELECT name -> FROM corporation -> WHERE corp_id = 27; +------------------------+ | name | +------------------------+ | Acme Paper Corporation | +------------------------+ All database elements created via SQL schema statements are stored in a special set of tables called the data dictionary. This “data about the database” is known collectively as metadata and is explored in Chapter 15. Just like tables that you create yourself, data dictionary tables can be queried via a select statement, thereby allowing you to discover the current data structures deployed in the database at runtime. For example, if you are asked to write a report showing the new accounts created last month, you could either hardcode the names of the columns in the account table that were known to you when you wrote the report, or query the data dictionary to determine the current set of columns and dynamically generate the report each time it is executed. Most of this book is concerned with the data portion of the SQL language, which consists of the select, update, insert, and delete commands. SQL schema statements is demonstrated in Chapter 2, where the sample database used throughout this book is generated.", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 11 + }, + { + "text": "generate the report each time it is executed. Most of this book is concerned with the data portion of the SQL language, which consists of the select, update, insert, and delete commands. SQL schema statements is demonstrated in Chapter 2, where the sample database used throughout this book is generated. In general, SQL schema statements do not require much discussion apart from their syntax, whereas SQL data statements, while few in number, offer numerous opportunities for detailed study. Therefore, while I try to introduce you to many of the SQL schema statements, most chapters in this book concentrate on the SQL data statements. SQL: A Nonprocedural Language If you have worked with programming languages in the past, you are used to defining variables and data structures, using conditional logic (i.e., if- then-else) and looping constructs (i.e., do while … end), and breaking your code into small, reusable pieces (i.e., objects, functions, procedures). Your code is handed to a compiler, and the executable that results does exactly (well, not always exactly) what you programmed it to do. Whether you work with Java, Python, Scala, or some other procedural language, you are in complete control of what the program does. NOTE A procedural language defines both the desired results and the mechanism, or process, by which the results are generated. Nonprocedural languages also define the desired results, but the process by which the results are generated is left to an external agent. With SQL, however, you will need to give up some of the control you are used to, because SQL statements define the necessary inputs and outputs, but the manner in which a statement is executed is left to a component of your database engine known as the optimizer. The optimizer’s job is to look at your SQL statements and, taking into account how your tables are configured and what indexes are available, decide the most efficient execution path (well, not always the most efficient). Most database engines will allow you to influence the optimizer’s decisions by specifying optimizer hints, such as suggesting that a particular index be used; most SQL users, however, will never get to this level of sophistication and will leave such tweaking to their database administrator or performance expert. With SQL, therefore, you will not be able to write complete applications. Unless you are writing a simple script to manipulate certain data, you will need to integrate SQL with your favorite programming language. Some database vendors have done this for you, such as Oracle’s PL/SQL language, MySQL’s stored procedure language, and Microsoft’s Transact- SQL language. With these languages, the SQL data statements are part of the language’s grammar, allowing you to seamlessly integrate database queries with procedural commands. If you are using a non-database- specific language such as Java or Python, however, you will need to use a toolkit/API to execute SQL statements from your code. Some of these toolkits are provided by your database vendor, whereas others are created by third-party vendors or by open-source providers.", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 12 + }, + { + "text": "commands. If you are using a non-database- specific language such as Java or Python, however, you will need to use a toolkit/API to execute SQL statements from your code. Some of these toolkits are provided by your database vendor, whereas others are created by third-party vendors or by open-source providers. Table 1-2 shows some of the available options for integrating SQL into a specific language. Table 1-2. SQL integration toolkits Language Toolkit Java JDBC (Java Database Connectivity) C# ADO.NET (Microsoft) Ruby Ruby DBI Python Python DB Go Package database/sql If you only need to execute SQL commands interactively, every database vendor provides at least a simple command-line tool for submitting SQL commands to the database engine and inspecting the results. Most vendors provide a graphical tool as well that includes one window showing your SQL commands and another window showing the results from your SQL commands. Additionally, there are 3rd-party tools such as SQuirrel, which will connect via a JDBC connection to many different database servers. Since the examples in this book are executed against a MySQL database, I use the mysql command-line tool that is included as part of the MySQL installation to run the examples and format the results. SQL Examples Earlier in this chapter, I promised to show you an SQL statement that would return all the transactions against George Blake’s checking account. Without further ado, here it is: SELECT t.txn_id, t.txn_type_cd, t.txn_date, t.amount FROM individual i INNER JOIN account a ON i.cust_id = a.cust_id INNER JOIN product p ON p.product_cd = a.product_cd INNER JOIN transaction t ON t.account_id = a.account_id WHERE i.fname = 'George' AND i.lname = 'Blake' AND p.name = 'checking account'; +--------+-------------+---------------------+--------+ | txn_id | txn_type_cd | txn_date | amount | +--------+-------------+---------------------+--------+ | 11 | DBT | 2008-01-05 00:00:00 | 100.00 | +--------+-------------+---------------------+--------+ 1 row in set (0.00 sec) Without going into too much detail at this point, this query identifies the row in the individual table for George Blake and the row in the product table for the “checking” product, finds the row in the account table for this individual/product combination, and returns four columns from the transaction table for all transactions posted to this account. If you happen to know that George Blake’s customer ID is 8 and that checking accounts are designated by the code 'CHK', then you can simply find George Blake’s checking account in the account table based on the customer ID and use the account ID to find the appropriate transactions: SELECT t.txn_id, t.txn_type_cd, t.txn_date, t.amount FROM account a INNER JOIN transaction t ON t.account_id = a.account_id WHERE a.cust_id = 8 AND a.product_cd = 'CHK'; I cover all of the concepts in these queries (plus a lot more) in the following chapters, but I wanted to at least show what they would look like. The previous queries contain three different clauses: select, from, and where. Almost every query that you encounter will include at least these three clauses, although there are several more that can be used for", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 13 + }, + { + "text": "in the following chapters, but I wanted to at least show what they would look like. The previous queries contain three different clauses: select, from, and where. Almost every query that you encounter will include at least these three clauses, although there are several more that can be used for more specialized purposes. The role of each of these three clauses is demonstrated by the following: SELECT /* one or more things */ ... FROM /* one or more places */ ... WHERE /* one or more conditions apply */ ... NOTE Most SQL implementations treat any text between the /* and */ tags as comments. When constructing your query, your first task is generally to determine which table or tables will be needed and then add them to your from clause. Next, you will need to add conditions to your where clause to filter out the data from these tables that you aren’t interested in. Finally, you will decide which columns from the different tables need to be retrieved and add them to your select clause. Here’s a simple example that shows how you would find all customers with the last name “Smith”: SELECT cust_id, fname FROM individual WHERE lname = 'Smith'; This query searches the individual table for all rows whose lname column matches the string 'Smith' and returns the cust_id and fname columns from those rows. Along with querying your database, you will most likely be involved with populating and modifying the data in your database. Here’s a simple example of how you would insert a new row into the product table: INSERT INTO product (product_cd, name) VALUES ('CD', 'Certificate of Depysit') Whoops, looks like you misspelled “Deposit.” No problem. You can clean that up with an update statement: UPDATE product SET name = 'Certificate of Deposit' WHERE product_cd = 'CD'; Notice that the update statement also contains a where clause, just like the select statement. This is because an update statement must identify the rows to be modified; in this case, you are specifying that only those rows whose product_cd column matches the string 'CD' should be modified. Since the product_cd column is the primary key for the product table, you should expect your update statement to modify exactly one row (or zero, if the value doesn’t exist in the table). Whenever you execute an SQL data statement, you will receive feedback from the database engine as to how many rows were affected by your statement. If you are using an interactive tool such as the mysql command-line tool mentioned earlier, then you will receive feedback concerning how many rows were either: Returned by your select statement Created by your insert statement Modified by your update statement Removed by your delete statement If you are using a procedural language with one of the toolkits mentioned earlier, the toolkit will include a call to ask for this information after your SQL data statement has executed. In general, it’s a good idea to check this info to make", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 14 + }, + { + "text": "statement Removed by your delete statement If you are using a procedural language with one of the toolkits mentioned earlier, the toolkit will include a call to ask for this information after your SQL data statement has executed. In general, it’s a good idea to check this info to make sure your statement didn’t do something unexpected (like when you forget to put a where clause on your delete statement and delete every row in the table!). What Is MySQL? Relational databases have been available commercially for over two decades. Some of the most mature and popular commercial products include: Oracle Database from Oracle Corporation SQL Server from Microsoft DB2 Universal Database from IBM All these database servers do approximately the same thing, although some are better equipped to run very large or very-high-throughput databases. Others are better at handling objects or very large files or XML documents, and so on. Additionally, all these servers do a pretty good job of complying with the latest ANSI SQL standard. This is a good thing, and I make it a point to show you how to write SQL statements that will run on any of these platforms with little or no modification. Along with the commercial database servers, there has been quite a bit of activity in the open source community in the past two decades with the goal of creating a viable alternative to the commercial database servers. Two of the most commonly used open source database servers are PostgreSQL and MySQL. The MySQL server is available for free, and I have found it to be extremely simple to download and install. For these reasons, I have decided that all examples for this book be run against a MySQL (version 8.0) database, and that the mysql command-line tool be used to format query results. Even if you are already using another server and never plan to use MySQL, I urge you to install the latest MySQL server, load the sample schema and data, and experiment with the data and examples in this book. However, keep in mind the following caveat: This is not a book about MySQL’s SQL implementation. Rather, this book is designed to teach you how to craft SQL statements that will run on MySQL with no modifications, and will run on recent releases of Oracle Database, DB2, and SQL Server with few or no modifications. SQL Unplugged A great deal has happened in the database world during the decade between the 2nd and 3rd editions of this book. While relational databases are still heavily used and will continue to be for some time, new database technologies have emerged to meet the needs of companies like Amazon and Google. These technologies include Hadoop, Spark, NoSQL, and NewSQL, which are distributed, scalable systems typically deployed on clusters of commodity servers. While it is beyond the scope of this book to explore these technologies in detail, they do all share something in common with relational databases: SQL. Since organizations frequently store", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 15 + }, + { + "text": "These technologies include Hadoop, Spark, NoSQL, and NewSQL, which are distributed, scalable systems typically deployed on clusters of commodity servers. While it is beyond the scope of this book to explore these technologies in detail, they do all share something in common with relational databases: SQL. Since organizations frequently store data using multiple technologies, there is a need to unplug SQL from a particular database server and provide a service which can span multiple databases. For example, a report may need to bring together data stored in Oracle, Hadoop, JSON files, CSV files, and Unix log files. A new generation of tools have been built to meet this type of challenge, and one of the most promising is Apache Drill, which is an open-source query engine which allows users to write queries which can access data stored in most any database or file system. We will explore Apache Drill in Chapter 18, SQL and Big Data. What’s in Store The overall goal of the next four chapters is to introduce the SQL data statements, with a special emphasis on the three main clauses of the select statement. Additionally, you will see many examples that use the sakila schema (introduced in the next chapter), which will be used for all examples in the book. It is my hope that familiarity with a single database will allow you to get to the crux of an example without your having to stop and examine the tables being used each time. If it becomes a bit tedious working with the same set of tables, feel free to augment the sample database with additional tables, or invent your own database with which to experiment. After you have a solid grasp on the basics, the remaining chapters will drill deep into additional concepts, most of which are independent of each other. Thus, if you find yourself getting confused, you can always move ahead and come back later to revisit a chapter. When you have finished the book and worked through all of the examples, you will be well on your way to becoming a seasoned SQL practitioner. For readers interested in learning more about relational databases, the history of computerized database systems, or the SQL language than was covered in this short introduction, here are a few resources worth checking out: C.J. Date’s Database in Depth: Relational Theory for Practitioners (O’Reilly) C.J. Date’s An Introduction to Database Systems, Eighth Edition (Addison-Wesley) C.J. Date’s The Database Relational Model: A Retrospective Review and Analysis: A Historical Account and Assessment of E. F. Codd’s Contribution to the Field of Database Technology (Addison-Wesley) http://en.wikipedia.org/wiki/Database_management_system Chapter 2. Creating and Populating a Database This chapter provides you with the information you need to create your first database and to create the tables and associated data used for the examples in this book. You will also learn about various data types and see how to create tables using them. Because the examples in this book are executed against a MySQL database, this chapter", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 16 + }, + { + "text": "to create your first database and to create the tables and associated data used for the examples in this book. You will also learn about various data types and see how to create tables using them. Because the examples in this book are executed against a MySQL database, this chapter is somewhat skewed toward MySQL’s features and syntax, but most concepts are applicable to any server. Creating a MySQL Database If you want the ability to experiment with the data used for the examples in this book, you have 2 options: 1. Download and install the MySQL server version 8.0 (or later) server and load the Sakila example database from https://dev.mysql.com/doc/index-other.html 2. Go to the page for this book and click on the “Launch Sample Database Session” button. If you choose the 2nd option, you will have a session available for a period of time (most likely 1 hour), after which any changes you have made to the data will be lost. You may open a new session each time you wish to run SQL statements. This is certainly the easiest option, and I anticipate that most readers will choose this option; if this sounds good to you, feel free to skip ahead to the next section. If you prefer to have your own copy of the data and want any changes you have made to be permanent, or if you are just interested in installing the MySQL server on your own machine, you may prefer option 1. You may also opt to use a MySQL server hosted in an environment such as Amazon Web Services or Google Cloud. In either case, you will need to perform the installation/configuration yourself, as it is beyond the scope of this book. Once your database is available, you will need to follow a few steps to load the Sakila sample database. First, you will need to launch the MySQL Command Line Client and provide a password, and then perform the following steps: 1. Go to https://dev.mysql.com/doc/index-other.html and download the files for “sakila database” under the Example Databases section 2. Put the files in a local directory such as C:\\temp\\sakila-db (used for the next 2 steps, but overwrite with your directory path). 3. Type source c:\\temp\\sakila-db\\sakila- schema.sql; and press Enter. 4. Type source c:\\temp\\sakila-db\\sakila- data.sql; and press Enter. You should now have a working database populated with all the data needed for the examples in this book. NOTE The Sakila Sample Database is made available by MySQL, and is licensed via the New BSD license. Sakila contains data for a fictitious movie rental company, and includes tables such as Store, Inventory, Film, Customer, and Payment. While actual movie-rental stores are largely a thing of the past, with a little imagination we could rebrand it as a movie-streaming company by ignoring the Staff and Address tables, and renaming Store to Streaming_Service. However, the examples in this book will stick to the original script (pun intended). Using the mysql Command-Line Tool Whether you have your own", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 17 + }, + { + "text": "the past, with a little imagination we could rebrand it as a movie-streaming company by ignoring the Staff and Address tables, and renaming Store to Streaming_Service. However, the examples in this book will stick to the original script (pun intended). Using the mysql Command-Line Tool Whether you have your own database server installed or are using a temporary database session (option #2 in the previous section), you will need to start the MySQL command-line tool in order to interact with the database. To do so, you will need to open a Windows or Unix shell and execute the mysql utility. For example, if you are logging in using the root account, you would do the following: mysql -u root -p; You will then be asked for your password, after which you will see the mysql> prompt. To see all of the available databases, you can use the following command: mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | sakila | | sys | +--------------------+ 5 rows in set (0.01 sec) Since you will be using the Sakila database, you will need to specify the database you want to work with via the use command: mysql> use sakila; Database changed Whenever you invoke the mysql command-line tool, you can specify both the username and database to use, as in the following: mysql -u root -p sakila; This will save you from having to type use sakila; every time you start up the tool. Now that you have established a session and specified the database, you will be able to issue SQL statements and view the results. For example, if you want to know the current date and time, you could issue the following query: mysql> SELECT now(); +---------------------+ | now() | +---------------------+ | 2019-04-04 20:44:26 | +---------------------+ 1 row in set (0.01 sec) The now() function is a built-in MySQL function that returns the current date and time. As you can see, the mysql command-line tool formats the results of your queries within a rectangle bounded by +, -, and | characters. After the results have been exhausted (in this case, there is only a single row of results), the mysql command-line tool shows how many rows were returned, along with how long the SQL statement took to execute. ABOUT MISSING FROM CLAUSES With some database servers, you won’t be able to issue a query without a from clause that names at least one table. Oracle Database is a commonly used server for which this is true. For cases when you only need to call a function, Oracle provides a table called dual, which consists of a single column called dummy that contains a single row of data. In order to be compatible with Oracle Database, MySQL also provides a dual table. The previous query to determine the current date and time could therefore be written as: mysql> SELECT now() FROM dual; +---------------------+ | now() | +---------------------+ | 2019-04-04 20:44:26 | +---------------------+", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 18 + }, + { + "text": "contains a single row of data. In order to be compatible with Oracle Database, MySQL also provides a dual table. The previous query to determine the current date and time could therefore be written as: mysql> SELECT now() FROM dual; +---------------------+ | now() | +---------------------+ | 2019-04-04 20:44:26 | +---------------------+ 1 row in set (0.01 sec) If you are not using Oracle and have no need to be compatible with Oracle, you can ignore the dual table altogether and use just a select clause without a from clause. When you are done with the mysql command-line tool, simply type quit; or exit; to return to the Unix or Windows command shell. MySQL Data Types In general, all the popular database servers have the capacity to store the same types of data, such as strings, dates, and numbers. Where they typically differ is in the specialty data types, such as XML and JSON documents or spatial data. Since this is an introductory book on SQL, and since 98% of the columns you encounter will be simple data types, this chapter covers only the character, date (a.k.a. temporal), and numeric data types. The use of SQL to query JSON documents will be explored in the chapter on SQL and Big Data. Character Data Character data can be stored as either fixed-length or variable-length strings; the difference is that fixed-length strings are right-padded with spaces and always consume the same number of bytes, and variable-length strings are not right-padded with spaces and don’t always consume the same number of bytes. When defining a character column, you must specify the maximum size of any string to be stored in the column. For example, if you want to store strings up to 20 characters in length, you could use either of the following definitions: char(20) /* fixed-length */ varchar(20) /* variable-length */ The maximum length for char columns is currently 255 bytes, whereas varchar columns can be up to 65,535 bytes. If you need to store longer strings (such as emails, XML documents, etc.), then you will want to use one of the text types (mediumtext and longtext), which I cover later in this section. In general, you should use the char type when all strings to be stored in the column are of the same length, such as state abbreviations, and the varchar type when strings to be stored in the column are of varying lengths. Both char and varchar are used in a similar fashion in all the major database servers. NOTE Oracle Database is an exception when it comes to the use of varchar. Oracle users should use the varchar2 type when defining variable-length character columns. CHARACTER SETS For languages that use the Latin alphabet, such as English, there is a sufficiently small number of characters such that only a single byte is needed to store each character. Other languages, such as Japanese and Korean, contain large numbers of characters, thus requiring multiple bytes of storage for each character. Such character sets", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 19 + }, + { + "text": "the Latin alphabet, such as English, there is a sufficiently small number of characters such that only a single byte is needed to store each character. Other languages, such as Japanese and Korean, contain large numbers of characters, thus requiring multiple bytes of storage for each character. Such character sets are therefore called multibyte character sets. MySQL can store data using various character sets, both single- and multibyte. To view the supported character sets in your server, you can use the show command, as in: mysql> SHOW CHARACTER SET; +----------+---------------------------------+------------------ ---+--------+ | Charset | Description | Default collation | Maxlen | +----------+---------------------------------+------------------ ---+--------+ | armscii8 | ARMSCII-8 Armenian | armscii8_general_ci | 1 | | ascii | US ASCII | ascii_general_ci | 1 | | big5 | Big5 Traditional Chinese | big5_chinese_ci | 2 | | binary | Binary pseudo charset | binary | 1 | | cp1250 | Windows Central European | cp1250_general_ci | 1 | | cp1251 | Windows Cyrillic | cp1251_general_ci | 1 | | cp1256 | Windows Arabic | cp1256_general_ci | 1 | | cp1257 | Windows Baltic | cp1257_general_ci | 1 | | cp850 | DOS West European | cp850_general_ci | 1 | | cp852 | DOS Central European | cp852_general_ci | 1 | | cp866 | DOS Russian | cp866_general_ci | 1 | | cp932 | SJIS for Windows Japanese | cp932_japanese_ci | 2 | | dec8 | DEC West European | dec8_swedish_ci | 1 | | eucjpms | UJIS for Windows Japanese | eucjpms_japanese_ci | 3 | | euckr | EUC-KR Korean | euckr_korean_ci | 2 | | gb18030 | China National Standard GB18030 | gb18030_chinese_ci | 4 | | gb2312 | GB2312 Simplified Chinese | gb2312_chinese_ci | 2 | | gbk | GBK Simplified Chinese | gbk_chinese_ci | 2 | | geostd8 | GEOSTD8 Georgian | geostd8_general_ci | 1 | | greek | ISO 8859-7 Greek | greek_general_ci | 1 | | hebrew | ISO 8859-8 Hebrew | hebrew_general_ci | 1 | | hp8 | HP West European | hp8_english_ci | 1 | | keybcs2 | DOS Kamenicky Czech-Slovak | keybcs2_general_ci | 1 | | koi8r | KOI8-R Relcom Russian | koi8r_general_ci | 1 | | koi8u | KOI8-U Ukrainian | koi8u_general_ci | 1 | | latin1 | cp1252 West European | latin1_swedish_ci | 1 | | latin2 | ISO 8859-2 Central European | latin2_general_ci | 1 | | latin5 | ISO 8859-9 Turkish | latin5_turkish_ci | 1 | | latin7 | ISO 8859-13 Baltic | latin7_general_ci | 1 | | macce | Mac Central European | macce_general_ci | 1 | | macroman | Mac West European | macroman_general_ci | 1 | | sjis | Shift-JIS Japanese | sjis_japanese_ci | 2 | | swe7 | 7bit Swedish | swe7_swedish_ci | 1 | | tis620 | TIS620 Thai | tis620_thai_ci | 1 | | ucs2 | UCS-2 Unicode | ucs2_general_ci | 2 | | ujis | EUC-JP Japanese | ujis_japanese_ci | 3 | | utf16 | UTF-16 Unicode | utf16_general_ci | 4 | |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 20 + }, + { + "text": "swe7 | 7bit Swedish | swe7_swedish_ci | 1 | | tis620 | TIS620 Thai | tis620_thai_ci | 1 | | ucs2 | UCS-2 Unicode | ucs2_general_ci | 2 | | ujis | EUC-JP Japanese | ujis_japanese_ci | 3 | | utf16 | UTF-16 Unicode | utf16_general_ci | 4 | | utf16le | UTF-16LE Unicode | utf16le_general_ci | 4 | | utf32 | UTF-32 Unicode | utf32_general_ci | 4 | | utf8 | UTF-8 Unicode | utf8_general_ci | 3 | | utf8mb4 | UTF-8 Unicode | utf8mb4_0900_ai_ci | 4 | +----------+---------------------------------+------------------ ---+--------+ 41 rows in set (0.04 sec) If the value in the fourth column, maxlen, is greater than 1, then the character set is a multibyte character set. In prior versions of the MySQL server, the latin1 character set was automatically chosen as the default character set, but version 8 defaults to utf8mb4. However, you may choose to use a different character set for each character column in your database, and you can even store different character sets within the same table. To choose a character set other than the default when defining a column, simply name one of the supported character sets after the type definition, as in: varchar(20) character set latin1 With MySQL, you may also set the default character set for your entire database: create database european_sales character set latin1; While this is as much information regarding character sets as is appropriate for an introductory book, there is a great deal more to the topic of internationalization than what is shown here. If you plan to deal with multiple or unfamiliar character sets, you may want to pick up a book such as Jukka Korpela’s Unicode Explained: Internationalize Documents, Programs, and Web Sites (O’Reilly). TEXT DATA If you need to store data that might exceed the 64 KB limit for varchar columns, you will need to use one of the text types. Table 2-1 shows the available text types and their maximum sizes. Table 2-1. MySQL text types Text type Maximum number of bytes Tinytext 255 Text 65,535 Mediumtext 16,777,215 Longtext 4,294,967,295 When choosing to use one of the text types, you should be aware of the following: If the data being loaded into a text column exceeds the maximum size for that type, the data will be truncated. Trailing spaces will not be removed when data is loaded into the column. When using text columns for sorting or grouping, only the first 1,024 bytes are used, although this limit may be increased if necessary. The different text types are unique to MySQL. SQL Server has a single text type for large character data, whereas DB2 and Oracle use a data type called clob, for Character Large Object. Now that MySQL allows up to 65,535 bytes for varchar columns (it was limited to 255 bytes in version 4), there isn’t any particular need to use the tinytext or text type. If you are creating a column for free-form data entry, such as a notes column to hold", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 21 + }, + { + "text": "Object. Now that MySQL allows up to 65,535 bytes for varchar columns (it was limited to 255 bytes in version 4), there isn’t any particular need to use the tinytext or text type. If you are creating a column for free-form data entry, such as a notes column to hold data about customer interactions with your company’s customer service department, then varchar will probably be adequate. If you are storing documents, however, you should choose either the mediumtext or longtext type. NOTE Oracle Database allows up to 2,000 bytes for char columns and 4,000 bytes for varchar2 columns. For larger documents you may use the CLOB type. SQL Server can handle up to 8,000 bytes for both char and varchar data, but you can store up to 2GB of data in a column defined as varchar(max). Numeric Data Although it might seem reasonable to have a single numeric data type called “numeric,” there are actually several different numeric data types that reflect the various ways in which numbers are used, as illustrated here: A column indicating whether a customer order has been shipped This type of column, referred to as a Boolean, would contain a 0 to indicate false and a 1 to indicate true. A system-generated primary key for a transaction table This data would generally start at 1 and increase in increments of one up to a potentially very large number. An item number for a customer’s electronic shopping basket The values for this type of column would be positive whole numbers between 1 and, perhaps, 200 (for shopaholics). Positional data for a circuit board drill machine High-precision scientific or manufacturing data often requires accuracy to eight decimal points. To handle these types of data (and more), MySQL has several different numeric data types. The most commonly used numeric types are those used to store whole numbers, or integers. When specifying one of these types, you may also specify that the data is unsigned, which tells the server that all data stored in the column will be greater than or equal to zero. Table 2-2 shows the five different data types used to store whole- number integers. Table 2-2. MySQL integer types Type Signed range Unsigned range Tinyint −128 to 127 0 to 255 Smallint −32,768 to 32,767 0 to 65,535 Mediumint −8,388,608 to 8,388,607 0 to 16,777,215 Int −2,147,483,648 to 2,147,483,647 0 to 4,294,967,295 Bigint −2^63 to 2^63 - 1 0 to 2^64 - 1 When you create a column using one of the integer types, MySQL will allocate an appropriate amount of space to store the data, which ranges from one byte for a tinyint to eight bytes for a bigint. Therefore, you should try to choose a type that will be large enough to hold the biggest number you can envision being stored in the column without needlessly wasting storage space. For floating-point numbers (such as 3.1415927), you may choose from the numeric types shown in Table 2-3. Table 2-3. MySQL floating-point types Type Numeric", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 22 + }, + { + "text": "choose a type that will be large enough to hold the biggest number you can envision being stored in the column without needlessly wasting storage space. For floating-point numbers (such as 3.1415927), you may choose from the numeric types shown in Table 2-3. Table 2-3. MySQL floating-point types Type Numeric range Float( p , s ) −3.402823466E+38 to −1.175494351E-38 and 1.175494351E-38 to 3.402823466E+38 Double( p , s ) −1.7976931348623157E+308 to −2.2250738585072014E-308 and 2.2250738585072014E-308 to 1.7976931348623157E+308 When using a floating-point type, you can specify a precision (the total number of allowable digits both to the left and to the right of the decimal point) and a scale (the number of allowable digits to the right of the decimal point), but they are not required. These values are represented in Table 2-3 as p and s. If you specify a precision and scale for your floating-point column, remember that the data stored in the column will be rounded if the number of digits exceeds the scale and/or precision of the column. For example, a column defined as float(4,2) will store a total of four digits, two to the left of the decimal and two to the right of the decimal. Therefore, such a column would handle the numbers 27.44 and 8.19 just fine, but the number 17.8675 would be rounded to 17.87, and attempting to store the number 178.375 in your float(4,2) column would generate an error. Like the integer types, floating-point columns can be defined as unsigned, but this designation only prevents negative numbers from being stored in the column rather than altering the range of data that may be stored in the column. Temporal Data Along with strings and numbers, you will almost certainly be working with information about dates and/or times. This type of data is referred to as temporal, and some examples of temporal data in a database include: The future date that a particular event is expected to happen, such as shipping a customer’s order The date that a customer’s order was shipped The date and time that a user modified a particular row in a table An employee’s birth date The year corresponding to a row in a yearly_sales fact table in a data warehouse The elapsed time needed to complete a wiring harness on an automobile assembly line MySQL includes data types to handle all of these situations. Table 2-4 shows the temporal data types supported by MySQL. Table 2-4. MySQL temporal types Type Default format Allowable values Date YYYY-MM-DD 1000-01-01 to 9999-12-31 Datetime YYYY-MM-DD HH:MI:SS 1000-01-01 00:00:00.000000 to 9999-12-31 23:59:59.999999 Timestamp YYYY-MM-DD HH:MI:SS 1970-01-01 00:00:00.000000 to 2038-01-18 22:14:07.999999 Year YYYY 1901 to 2155 Time HHH:MI:SS −838:59:59.000000 to 838:59:59.000000 While database servers store temporal data in various ways, the purpose of a format string (second column of Table 2-4) is to show how the data will be represented when retrieved, along with how a date string should be constructed when inserting or updating a temporal column. Thus, if you wanted to insert the date March", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 23 + }, + { + "text": "in various ways, the purpose of a format string (second column of Table 2-4) is to show how the data will be represented when retrieved, along with how a date string should be constructed when inserting or updating a temporal column. Thus, if you wanted to insert the date March 23, 2020 into a date column using the default format YYYY-MM-DD, you would use the string '2020-03-23'. The datetime, timestamp, and time types also allow fractional seconds of up to 6 decimal places (microseconds). When defining columns using one of these data types, you may supply a value from 0 to 6; for example, specifying datetime(2) would allow your time values to include hundredths of a second. NOTE Each database server allows a different range of dates for temporal columns. Oracle Database accepts dates ranging from 4712 BC to 9999 AD, while SQL Server only handles dates ranging from 1753 AD to 9999 AD (unless you are using SQL Server 2008’s datetime2 data type, which allows for dates ranging from 1 AD to 9999 AD). MySQL falls in between Oracle and SQL Server and can store dates from 1000 AD to 9999 AD. Although this might not make any difference for most systems that track current and future events, it is important to keep in mind if you are storing historical dates. Table 2-5 describes the various components of the date formats shown in Table 2-4. Table 2-5. Date format components Component Definition Range YYYY Year, including century 1000 to 9999 MM Month 01 (January) to 12 (December) DD Day 01 to 31 HH Hour 00 to 23 HHH Hours (elapsed) −838 to 838 MI Minute 00 to 59 SS Second 00 to 59 Here’s how the various temporal types would be used to implement the examples shown earlier: Columns to hold the expected future shipping date of a customer order and an employee’s birth date would use the date type, since it is unnecessary to know at what time a person was born and unrealistic to schedule a future shipment down to the second. A column to hold information about when a customer order was actually shipped would use the datetime type, since it is important to track not only the date that the shipment occurred but the time as well. A column that tracks when a user last modified a particular row in a table would use the timestamp type. The timestamp type holds the same information as the datetime type (year, month, day, hour, minute, second), but a timestamp column will automatically be populated with the current date/time by the MySQL server when a row is added to a table or when a row is later modified. A column holding just year data would use the year type. Columns that hold data regarding the length of time needed to complete a task would use the time type. For this type of data, it would be unnecessary and confusing to store a date component, since you are interested", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 24 + }, + { + "text": "A column holding just year data would use the year type. Columns that hold data regarding the length of time needed to complete a task would use the time type. For this type of data, it would be unnecessary and confusing to store a date component, since you are interested only in the number of hours/minutes/seconds needed to complete the task. This information could be derived using two datetime columns (one for the task start date/time and the other for the task completion date/time) and subtracting one from the other, but it is simpler to use a single time column. Table Creation Now that you have a firm grasp on what data types may be stored in a MySQL database, it’s time to see how to use these types in table definitions. Let’s start by defining a table to hold information about a person. Step 1: Design A good way to start designing a table is to do a bit of brainstorming to see what kind of information would be helpful to include. Here’s what I came up with after thinking for a short time about the types of information that describe a person: Name Eye color Birth date Address Favorite foods This is certainly not an exhaustive list, but it’s good enough for now. The next step is to assign column names and data types. Table 2-6 shows my initial attempt. Table 2-6. Person table, first pass Column Type Allowable values Name Varchar(40) Eye_color Char(2) BL, BR, GR Birth_date Date Address Varchar(100) Favorite_foods Varchar(200) The name, address, and favorite_foods columns are of type varchar and allow for free-form data entry. The eye_color column allows 2 characters which should equal only BR, BL, or GR. The birth_date column is of type date, since a time component is not needed. Step 2: Refinement In Chapter 1, you were introduced to the concept of normalization, which is the process of ensuring that there are no duplicate (other than foreign keys) or compound columns in your database design. In looking at the columns in the person table a second time, the following issues arise: The name column is actually a compound object consisting of a first name and a last name. Since multiple people can have the same name, eye color, birth date, and so forth, there are no columns in the person table that guarantee uniqueness. The address column is also a compound object consisting of street, city, state/province, country, and postal code. The favorite_foods column is a list containing 0, 1, or more independent items. It would be best to create a separate table for this data that includes a foreign key to the person table so that you know to which person a particular food may be attributed. After taking these issues into consideration, Table 2-7 gives a normalized version of the person table. Table 2-7. Person table, second pass Column Type Allowable values Person_id Smallint (unsigned) First_name Varchar(20) Last_name Varchar(20) Eye_color Char(2) BR, BL, GR Birth_date Date Street Varchar(30)", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 25 + }, + { + "text": "which person a particular food may be attributed. After taking these issues into consideration, Table 2-7 gives a normalized version of the person table. Table 2-7. Person table, second pass Column Type Allowable values Person_id Smallint (unsigned) First_name Varchar(20) Last_name Varchar(20) Eye_color Char(2) BR, BL, GR Birth_date Date Street Varchar(30) City Varchar(20) State Varchar(20) Country Varchar(20) Postal_code Varchar(20) Now that the person table has a primary key (person_id) to guarantee uniqueness, the next step is to build a favorite_food table that includes a foreign key to the person table. Table 2-8 shows the result. Table 2-8. Favorite food table Table 2 8. Favorite_food table Column Type Person_id Smallint (unsigned) Food Varchar(20) The person_id and food columns comprise the primary key of the favorite_food table, and the person_id column is also a foreign key to the person table. HOW MUCH IS ENOUGH? Moving the favorite_foods column out of the person table was definitely a good idea, but are we done yet? What happens, for example, if one person lists “pasta” as a favorite food while another person lists “spaghetti”? Are they the same thing? In order to prevent this problem, you might decide that you want people to choose their favorite foods from a list of options, in which case you should create a food table with food_id and food_name columns, and then change the favorite_food table to contain a foreign key to the food table. While this design would be fully normalized, you might decide that you simply want to store the values that the user has entered, in which case you may leave the table as is. Step 3: Building SQL Schema Statements Now that the design is complete for the two tables holding information about people and their favorite foods, the next step is to generate SQL statements to create the tables in the database. Here is the statement to create the person table: CREATE TABLE person (person_id SMALLINT UNSIGNED, fname VARCHAR(20), lname VARCHAR(20), eye_color CHAR(2), birth_date DATE, street VARCHAR(30), city VARCHAR(20), state VARCHAR(20), country VARCHAR(20), postal_code VARCHAR(20), CONSTRAINT pk_person PRIMARY KEY (person_id) ); Everything in this statement should be fairly self-explanatory except for the last item; when you define your table, you need to tell the database server what column or columns will serve as the primary key for the table. You do this by creating a constraint on the table. You can add several types of constraints to a table definition. This constraint is a primary key constraint. It is created on the person_id column and given the name pk_person. While on the topic of constraints, there is another type of constraint that would be useful for the person table. In Table 2-6, I added a third column to show the allowable values for certain columns (such as 'BR' and 'BL' for the eye_color column). Another type of constraint called a check constraint constrains the allowable values for a particular column. MySQL allows a check constraint to be attached to a column definition, as in the", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 26 + }, + { + "text": "third column to show the allowable values for certain columns (such as 'BR' and 'BL' for the eye_color column). Another type of constraint called a check constraint constrains the allowable values for a particular column. MySQL allows a check constraint to be attached to a column definition, as in the following: eye_color CHAR(2) CHECK (eye_color IN ('BR','BL','GR')), While check constraints operate as expected on most database servers, the MySQL server allows check constraints to be defined but does not enforce them. However, MySQL does provide another character data type called enum that merges the check constraint into the data type definition. Here’s what it would look like for the eye_color column definition: eye_color ENUM('BR','BL','GR'), Here’s how the person table definition looks with an enum data type for the eye_color column: CREATE TABLE person (person_id SMALLINT UNSIGNED, fname VARCHAR(20), lname VARCHAR(20), eye_color ENUM('BR','BL','GR'), birth_date DATE, street VARCHAR(30), city VARCHAR(20), state VARCHAR(20), country VARCHAR(20), postal_code VARCHAR(20), CONSTRAINT pk_person PRIMARY KEY (person_id) ); Later in this chapter, you will see what happens if you try to add data to a column that violates its check constraint (or, in the case of MySQL, its enumeration values). You are now ready to run the create table statement using the mysql command-line tool. Here’s what it looks like: mysql> CREATE TABLE person -> (person_id SMALLINT UNSIGNED, -> fname VARCHAR(20), -> lname VARCHAR(20), -> eye_color ENUM('BR','BL','GR'), -> birth_date DATE, -> street VARCHAR(30), -> city VARCHAR(20), -> state VARCHAR(20), -> country VARCHAR(20), -> postal_code VARCHAR(20), -> CONSTRAINT pk_person PRIMARY KEY (person_id) -> ); Query OK, 0 rows affected (0.37 sec) After processing the create table statement, the MySQL server returns the message “Query OK, 0 rows affected,” which tells me that the statement had no syntax errors. If you want to make sure that the person table does, in fact, exist, you can use the describe command (or desc for short) to look at the table definition: mysql> desc person; +-------------+----------------------+------+-----+---------+--- ----+ | Field | Type | Null | Key | Default | Extra | +-------------+----------------------+------+-----+---------+--- ----+ | person_id | smallint(5) unsigned | NO | PRI | NULL | | | fname | varchar(20) | YES | | NULL | | | lname | varchar(20) | YES | | NULL | | | eye_color | enum('BR','BL','GR') | YES | | NULL | | | birth_date | date | YES | | NULL | | | street | varchar(30) | YES | | NULL | | | city | varchar(20) | YES | | NULL | | | state | varchar(20) | YES | | NULL | | | country | varchar(20) | YES | | NULL | | | postal_code | varchar(20) | YES | | NULL | | +-------------+----------------------+------+-----+---------+--- ----+ 10 rows in set (0.00 sec) Columns 1 and 2 of the describe output are self-explanatory. Column 3 shows whether a particular column can be omitted when data is inserted into the table. I purposefully left this topic out of the discussion for now (see the sidebar", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 27 + }, + { + "text": "| +-------------+----------------------+------+-----+---------+--- ----+ 10 rows in set (0.00 sec) Columns 1 and 2 of the describe output are self-explanatory. Column 3 shows whether a particular column can be omitted when data is inserted into the table. I purposefully left this topic out of the discussion for now (see the sidebar “What Is Null?” for a short discourse), but we explore it fully in Chapter 4. The fourth column shows whether a column takes part in any keys (primary or foreign); in this case, the person_id column is marked as the primary key. Column 5 shows whether a particular column will be populated with a default value if you omit the column when inserting data into the table. The sixth column (called “Extra”) shows any other pertinent information that might apply to a column. WHAT IS NULL? In some cases, it is not possible or applicable to provide a value for a particular column in your table. For example, when adding data about a new customer order, the ship_date column cannot yet be determined. In this case, the column is said to be null (note that I do not say that it equals null), which indicates the absence of a value. Null is used for various cases where a value cannot be supplied, such as: Not applicable Unknown Empty set When designing a table, you may specify which columns are allowed to be null (the default), and which columns are not allowed to be null (designated by adding the keywords not null after the type definition). Now that you’ve created the person table, your next step is to create the favorite_food table: mysql> CREATE TABLE favorite_food -> (person_id SMALLINT UNSIGNED, -> food VARCHAR(20), -> CONSTRAINT pk_favorite_food PRIMARY KEY (person_id, food), -> CONSTRAINT fk_fav_food_person_id FOREIGN KEY (person_id) -> REFERENCES person (person_id) -> ); Query OK, 0 rows affected (0.10 sec) This should look very similar to the create table statement for the person table, with the following exceptions: Since a person can have more than one favorite food (which is the reason this table was created in the first place), it takes more than just the person_id column to guarantee uniqueness in the table. This table, therefore, has a two-column primary key: person_id and food. The favorite_food table contains another type of constraint called a foreign key constraint. This constrains the values of the person_id column in the favorite_food table to include only values found in the person table. With this constraint in place, I will not be able to add a row to the favorite_food table indicating that person_id 27 likes pizza if there isn’t already a row in the person table having a person_id of 27. NOTE If you forget to create the foreign key constraint when you first create the table, you can add it later via the alter table statement. Describe shows the following after executing the create table statement: mysql> desc favorite_food; +-----------+----------------------+------+-----+---------+----- --+ | Field | Type | Null | Key | Default | Extra |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 28 + }, + { + "text": "to create the foreign key constraint when you first create the table, you can add it later via the alter table statement. Describe shows the following after executing the create table statement: mysql> desc favorite_food; +-----------+----------------------+------+-----+---------+----- --+ | Field | Type | Null | Key | Default | Extra | +-----------+----------------------+------+-----+---------+----- --+ | person_id | smallint(5) unsigned | NO | PRI | NULL | | | food | varchar(20) | NO | PRI | NULL | | +-----------+----------------------+------+-----+---------+----- --+ 2 rows in set (0.00 sec) Now that the tables are in place, the next logical step is to add some data. Populating and Modifying Tables With the person and favorite_food tables in place, you can now begin to explore the four SQL data statements: insert, update, delete, and select. Inserting Data Since there is not yet any data in the person and favorite_food tables, the first of the four SQL data statements to be explored will be the insert statement. There are three main components to an insert statement: The name of the table into which to add the data The names of the columns in the table to be populated The values with which to populate the columns You are not required to provide data for every column in the table (unless all the columns in the table have been defined as not null). In some cases, those columns that are not included in the initial insert statement will be given a value later via an update statement. In other cases, a column may never receive a value for a particular row of data (such as a customer order that is canceled before being shipped, thus rendering the ship_date column inapplicable). GENERATING NUMERIC KEY DATA Before inserting data into the person table, it would be useful to discuss how values are generated for numeric primary keys. Other than picking a number out of thin air, you have a couple of options: Look at the largest value currently in the table and add one. Let the database server provide the value for you. Although the first option may seem valid, it proves problematic in a multiuser environment, since two users might look at the table at the same time and generate the same value for the primary key. Instead, all database servers on the market today provide a safe, robust method for generating numeric keys. In some servers, such as the Oracle Database, a separate schema object is used (called a sequence); in the case of MySQL, however, you simply need to turn on the auto-increment feature for your primary key column. Normally, you would do this at table creation, but doing it now provides the opportunity to learn another SQL schema statement, alter table, which is used to modify the definition of an existing table: ALTER TABLE person MODIFY person_id SMALLINT UNSIGNED AUTO_INCREMENT; This statement essentially redefines the person_id column in the person table. If you describe the table, you will now see the auto- increment feature listed under", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 29 + }, + { + "text": "SQL schema statement, alter table, which is used to modify the definition of an existing table: ALTER TABLE person MODIFY person_id SMALLINT UNSIGNED AUTO_INCREMENT; This statement essentially redefines the person_id column in the person table. If you describe the table, you will now see the auto- increment feature listed under the “Extra” column for person_id: mysql> DESC person; +-------------+----------------------------+------+-----+------- --+-----------------+ | Field | Type | Null | Key | Default | Extra | +-------------+----------------------------+------+-----+------- --+-----------------+ | person_id | smallint(5) unsigned | NO | PRI | NULL | auto_increment | | . | | | | | | | . | | | | | | | . | | | | | | When you insert data into the person table, simply provide a null value for the person_id column, and MySQL will populate the column with the next available number (by default, MySQL starts at 1 for auto- increment columns). THE INSERT STATEMENT Now that all the pieces are in place, it’s time to add some data. The following statement creates a row in the person table for William Turner: mysql> INSERT INTO person -> (person_id, fname, lname, eye_color, birth_date) -> VALUES (null, 'William','Turner', 'BR', '1972-05-27'); Query OK, 1 row affected (0.22 sec) The feedback (“Query OK, 1 row affected”) tells you that your statement syntax was proper, and that one row was added to the database (since it was an insert statement). You can look at the data just added to the table by issuing a select statement: mysql> SELECT person_id, fname, lname, birth_date -> FROM person; +-----------+---------+--------+------------+ | person_id | fname | lname | birth_date | +-----------+---------+--------+------------+ | 1 | William | Turner | 1972-05-27 | +-----------+---------+--------+------------+ 1 row in set (0.06 sec) As you can see, the MySQL server generated a value of 1 for the primary key. Since there is only a single row in the person table, I neglected to specify which row I am interested in and simply retrieved all the rows in the table. If there were more than one row in the table, however, I could add a where clause to specify that I want to retrieve data only for the row having a value of 1 for the person_id column: mysql> SELECT person_id, fname, lname, birth_date -> FROM person -> WHERE person_id = 1; +-----------+---------+--------+------------+ | person_id | fname | lname | birth_date | +-----------+---------+--------+------------+ | 1 | William | Turner | 1972-05-27 | +-----------+---------+--------+------------+ 1 row in set (0.00 sec) While this query specifies a particular primary key value, you can use any column in the table to search for rows, as shown by the following query, which finds all rows with a value of 'Turner' for the lname column: mysql> SELECT person_id, fname, lname, birth_date -> FROM person -> WHERE lname = 'Turner'; +-----------+---------+--------+------------+ | person_id | fname | lname | birth_date | +-----------+---------+--------+------------+ | 1 | William | Turner | 1972-05-27 | +-----------+---------+--------+------------+ 1 row in set (0.00 sec) Before moving on, a couple of things about", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 30 + }, + { + "text": "column: mysql> SELECT person_id, fname, lname, birth_date -> FROM person -> WHERE lname = 'Turner'; +-----------+---------+--------+------------+ | person_id | fname | lname | birth_date | +-----------+---------+--------+------------+ | 1 | William | Turner | 1972-05-27 | +-----------+---------+--------+------------+ 1 row in set (0.00 sec) Before moving on, a couple of things about the earlier insert statement are worth mentioning: Values were not provided for any of the address columns. This is fine, since nulls are allowed for those columns. The value provided for the birth_date column was a string. As long as you match the required format shown in Table 2-4, MySQL will convert the string to a date for you. The column names and the values provided must correspond in number and type. If you name seven columns and provide only six values, or if you provide values that cannot be converted to the appropriate data type for the corresponding column, you will receive an error. William Turner has also provided information about his favorite three foods, so here are three insert statements to store his food preferences: mysql> INSERT INTO favorite_food (person_id, food) -> VALUES (1, 'pizza'); Query OK, 1 row affected (0.01 sec) mysql> INSERT INTO favorite_food (person_id, food) -> VALUES (1, 'cookies'); Query OK, 1 row affected (0.00 sec) mysql> INSERT INTO favorite_food (person_id, food) -> VALUES (1, 'nachos'); Query OK, 1 row affected (0.01 sec) Here’s a query that retrieves William’s favorite foods in alphabetical order using an order by clause: mysql> SELECT food -> FROM favorite_food -> WHERE person_id = 1 -> ORDER BY food; +---------+ | food | +---------+ | cookies | | nachos | | pizza | +---------+ 3 rows in set (0.02 sec) The order by clause tells the server how to sort the data returned by the query. Without the order by clause, there is no guarantee that the data in the table will be retrieved in any particular order. So that William doesn’t get lonely, you can execute another insert statement to add Susan Smith to the person table: mysql> INSERT INTO person -> (person_id, fname, lname, eye_color, birth_date, -> street, city, state, country, postal_code) -> VALUES (null, 'Susan','Smith', 'BL', '1975-11-02', -> '23 Maple St.', 'Arlington', 'VA', 'USA', '20220'); Query OK, 1 row affected (0.01 sec) Since Susan was kind enough to provide her address, we included five more columns than when William’s data was inserted. If you query the table again, you will see that Susan’s row has been assigned the value 2 for its primary key value: mysql> SELECT person_id, fname, lname, birth_date -> FROM person; +-----------+---------+--------+------------+ | person_id | fname | lname | birth_date | +-----------+---------+--------+------------+ | 1 | William | Turner | 1972-05-27 | | 2 | Susan | Smith | 1975-11-02 | +-----------+---------+--------+------------+ 2 rows in set (0.00 sec) CAN I GET THAT IN XML? If you will be working with XML data, you will be happy to know that most database servers provide a simple way to generate XML output from a query. With MySQL,", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 31 + }, + { + "text": "Susan | Smith | 1975-11-02 | +-----------+---------+--------+------------+ 2 rows in set (0.00 sec) CAN I GET THAT IN XML? If you will be working with XML data, you will be happy to know that most database servers provide a simple way to generate XML output from a query. With MySQL, for example, you can use the --xml option when invoking the mysql tool, and all your output will automatically be formatted using XML. Here’s what the favorite-food data looks like as an XML document: C:\\database> mysql -u lrngsql -p --xml bank Enter password: xxxxxx Welcome to the MySQL Monitor... Mysql> SELECT * FROM favorite_food; 1 cookies 1 nachos 1 pizza 3 rows in set (0.00 sec) With SQL Server, you don’t need to configure your command-line tool; you just need to add the for xml clause to the end of your query, as in: SELECT * FROM favorite_food FOR XML AUTO, ELEMENTS Updating Data When the data for William Turner was initially added to the table, data for the various address columns was not included the insert statement. The next statement shows how these columns can be populated at a later time via an update statement: mysql> UPDATE person -> SET street = '1225 Tremont St.', -> city = 'Boston', -> state = 'MA', -> country = 'USA', -> postal_code = '02138' -> WHERE person_id = 1; Query OK, 1 row affected (0.04 sec) Rows matched: 1 Changed: 1 Warnings: 0 The server responded with a two-line message: the “Rows matched: 1” item tells you that the condition in the where clause matched a single row in the table, and the “Changed: 1” item tells you that a single row in the table has been modified. Since the where clause specifies the primary key of William’s row, this is exactly what you would expect to have happen. Depending on the conditions in your where clause, it is also possible to modify more than one row using a single statement. Consider, for example, what would happen if your where clause looked as follows: WHERE person_id < 10 Since both William and Susan have a person_id value less than 10, both of their rows would be modified. If you leave off the where clause altogether, your update statement will modify every row in the table. Deleting Data It seems that William and Susan aren’t getting along very well together, so one of them has got to go. Since William was there first, Susan will get the boot courtesy of the delete statement: mysql> DELETE FROM person -> WHERE person_id = 2; Query OK, 1 row affected (0.01 sec) Again, the primary key is being used to isolate the row of interest, so a single row is deleted from the table. Similar to the update statement, more than one row can be deleted depending on the conditions in your where clause, and", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 32 + }, + { + "text": "Query OK, 1 row affected (0.01 sec) Again, the primary key is being used to isolate the row of interest, so a single row is deleted from the table. Similar to the update statement, more than one row can be deleted depending on the conditions in your where clause, and all rows will be deleted if the where clause is omitted. When Good Statements Go Bad So far, all of the SQL data statements shown in this chapter have been well formed and have played by the rules. Based on the table definitions for the person and favorite_food tables, however, there are lots of ways that you can run afoul when inserting or modifying data. This section shows you some of the common mistakes that you might come across and how the MySQL server will respond. Nonunique Primary Key Because the table definitions include the creation of primary key constraints, MySQL will make sure that duplicate key values are not inserted into the tables. The next statement attempts to bypass the auto- increment feature of the person_id column and create another row in the person table with a person_id of 1: mysql> INSERT INTO person -> (person_id, fname, lname, eye_color, birth_date) -> VALUES (1, 'Charles','Fulton', 'GR', '1968-01-15'); ERROR 1062 (23000): Duplicate entry '1' for key 'PRIMARY' There is nothing stopping you (with the current schema objects, at least) from creating two rows with identical names, addresses, birth dates, and so on, as long as they have different values for the person_id column. Nonexistent Foreign Key The table definition for the favorite_food table includes the creation of a foreign key constraint on the person_id column. This constraint ensures that all values of person_id entered into the favorite_food table exist in the person table. Here’s what would happen if you tried to create a row that violates this constraint: mysql> INSERT INTO favorite_food (person_id, food) -> VALUES (999, 'lasagna'); ERROR 1452 (23000): Cannot add or update a child row: a foreign key constraint fails ('bank'.'favorite_food', CONSTRAINT 'fk_fav_food_person_id' FOREIGN KEY ('person_id') REFERENCES 'person' ('person_id')) In this case, the favorite_food table is considered the child and the person table is considered the parent, since the favorite_food table is dependent on the person table for some of its data. If you plan to enter data into both tables, you will need to create a row in parent before you can enter data into favorite_food. NOTE Foreign key constraints are enforced only if your tables are created using the InnoDB storage engine. We discuss MySQL’s storage engines in Chapter 12. Column Value Violations The eye_colorcolumn in the person table is restricted to the values 'BR' for brown, 'BL' for blue, and 'GR' for green. If you mistakenly attempt to set the value of the column to any other value, you will receive the following response: mysql> UPDATE person -> SET eye_color = 'ZZ' -> WHERE person_id = 1; ERROR 1265 (01000): Data truncated for column 'eye_color' at row 1 The error message is a bit confusing,", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 33 + }, + { + "text": "mistakenly attempt to set the value of the column to any other value, you will receive the following response: mysql> UPDATE person -> SET eye_color = 'ZZ' -> WHERE person_id = 1; ERROR 1265 (01000): Data truncated for column 'eye_color' at row 1 The error message is a bit confusing, but it gives you the general idea that the server is unhappy about the value provided for the eye_color column. Invalid Date Conversions If you construct a string with which to populate a date column, and that string does not match the expected format, you will receive another error. Here’s an example that uses a date format that does not match the default date format of “YYYY-MM-DD”: mysql> UPDATE person -> SET birth_date = 'DEC-21-1980' -> WHERE person_id = 1; ERROR 1292 (22007): Incorrect date value: 'DEC-21-1980' for column 'birth_date' at row 1 In general, it is always a good idea to explicitly specify the format string rather than relying on the default format. Here’s another version of the statement that uses the str_to_date function to specify which format string to use: mysql> UPDATE person -> SET birth_date = str_to_date('DEC-21-1980' , '%b-%d-%Y') -> WHERE person_id = 1; Query OK, 1 row affected (0.12 sec) Rows matched: 1 Changed: 1 Warnings: 0 Not only is the database server happy, but William is happy as well (we just made him eight years younger, without the need for expensive cosmetic surgery!). NOTE Earlier in the chapter, when I discussed the various temporal data types, I showed date- formatting strings such as “YYYY-MM-DD”. While many database servers use this style of formatting, MySQL uses %Y to indicate a four-character year. Here are a few more formatters that you might need when converting strings to datetimes in MySQL: %a The short weekday name, such as Sun, Mon, ... %b The short month name, such as Jan, Feb, ... %c The numeric month (0..12) %d The numeric day of the month (00..31) %f The number of microseconds (000000..999999) %H The hour of the day, in 24-hour format (00..23) %h The hour of the day, in 12-hour format (01..12) %i The minutes within the hour (00..59) %j The day of year (001..366) %M The full month name (January..December) %m The numeric month %p AM or PM %s The number of seconds (00..59) %W The full weekday name (Sunday..Saturday) %w The numeric day of the week (0=Sunday..6=Saturday) %Y The four-digit year The Sakila Database For the remainder of the book, most examples will use a sample database called Sakila, which is made available by the nice people at MySQL. This database models a chain of DVD rental stores, which is a bit outdated, but with a bit of imagination it can be rebranded as a video-streaming company. Some of the tables include Customer, Film, Actor, Payment, Rental, and Category. The entire schema and example data should have been created when you followed the final steps at the beginning of the chapter for loading the MySQL server and generating", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 34 + }, + { + "text": "imagination it can be rebranded as a video-streaming company. Some of the tables include Customer, Film, Actor, Payment, Rental, and Category. The entire schema and example data should have been created when you followed the final steps at the beginning of the chapter for loading the MySQL server and generating the sample data. To see a diagram of the tables and their columns and relationships, see Appendix A. Table 2-9 shows some of the tables used in the sakila schema along with short definitions. Table 2-9. Sakila schema definitions Table name Definition Film A movie which has been released and can be rented Actor A person who plays acts in films Customer A person who watches films Category A genre of films Payment A rental of a film by a customer Language A language spoken by the actors of a film Film_Actor An actor in a film Inventory A film available for rental Feel free to experiment with the tables as much as you want, including adding your own tables to expand the business functions. You can always drop the database and re-create it from the downloaded file if you want to make sure your sample data is intact. If you are using the temporary session, any changes you make will be lost when the session closes, so you may want to keep a script of your changes so you can recreate any changes you have made. If you want to see the tables available in your database, you can use the show tables command, as in: mysql> show tables; +----------------------------+ | Tables_in_sakila | +----------------------------+ | actor | | actor_info | | address | | category | | city | | country | | customer | | customer_list | | film | | film_actor | | film_category | | film_list | | film_text | | inventory | | language | | nicer_but_slower_film_list | | payment | | rental | | sales_by_film_category | | sales_by_store | | staff | | staff_list | | store | +----------------------------+ 23 rows in set (0.02 sec) Along with the 23 tables in the sakila schema, your table listing may also include the two tables created in this chapter: person and favorite_food. These tables will not be used in later chapters, so feel free to drop them by issuing the following commands: mysql> DROP TABLE favorite_food; Query OK, 0 rows affected (0.56 sec) mysql> DROP TABLE person; Query OK, 0 rows affected (0.05 sec) If you want to look at the columns in a table, you can use the describe command. Here’s an example of the describe output for the customer table: mysql> desc customer; +-------------+----------------------+------+-----+------------- ------+-------------------------------+ | Field | Type | Null | Key | Default | Extra | +-------------+----------------------+------+-----+------------- ------+-------------------------------+ | customer_id | smallint(5) unsigned | NO | PRI | NULL | auto_increment | | store_id | tinyint(3) unsigned | NO | MUL | NULL | | | first_name | varchar(45) | NO | | NULL | | | last_name | varchar(45) | NO", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 35 + }, + { + "text": "Default | Extra | +-------------+----------------------+------+-----+------------- ------+-------------------------------+ | customer_id | smallint(5) unsigned | NO | PRI | NULL | auto_increment | | store_id | tinyint(3) unsigned | NO | MUL | NULL | | | first_name | varchar(45) | NO | | NULL | | | last_name | varchar(45) | NO | MUL | NULL | | | email | varchar(50) | YES | | NULL | | | address_id | smallint(5) unsigned | NO | MUL | NULL | | | active | tinyint(1) | NO | | 1 | | | create_date | datetime | NO | | NULL | | | last_update | timestamp | YES | | CURRENT_TIMESTAMP | DEFAULT_GENERATED | | | on update CURRENT_TIMESTAMP | +-------------+----------------------+------+-----+------------- ------+-------------------------------+ The more comfortable you are with the example database, the better you will understand the examples and, consequently, the concepts in the following chapters. Chapter 3. Query Primer So far, you have seen a few examples of database queries (a.k.a. select statements) sprinkled throughout the first two chapters. Now it’s time to take a closer look at the different parts of the select statement and how they interact. After finishing this chapter, you should have a basic understanding of how data is retrieved, joined, filtered, grouped, and sorted; these topics will be covered in detail in chapters 4 through 10. Query Mechanics Before dissecting the select statement, it might be interesting to look at how queries are executed by the MySQL server (or, for that matter, any database server). If you are using the mysql command-line tool (which I assume you are), then you have already logged in to the MySQL server by providing your username and password (and possibly a hostname if the MySQL server is running on a different computer). Once the server has verified that your username and password are correct, a database connection is generated for you to use. This connection is held by the application that requested it (which, in this case, is the mysql tool) until the application releases the connection (i.e., as a result of your typing quit) or the server closes the connection (i.e., when the server is shut down). Each connection to the MySQL server is assigned an identifier, which is shown to you when you first log in: Welcome to the MySQL monitor. Commands end with ; or \\g. Your MySQL connection id is 11 Server version: 8.0.15 MySQL Community Server - GPL Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\\h' for help. Type '\\c' to clear the buffer. In this case, my connection ID is 11. This information might be useful to your database administrator if something goes awry, such as a malformed query that runs for hours, so you might want to jot it down. Once the server has verified your username and password and issued you a connection,", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 36 + }, + { + "text": "case, my connection ID is 11. This information might be useful to your database administrator if something goes awry, such as a malformed query that runs for hours, so you might want to jot it down. Once the server has verified your username and password and issued you a connection, you are ready to execute queries (along with other SQL statements). Each time a query is sent to the server, the server checks the following things prior to statement execution: Do you have permission to execute the statement? Do you have permission to access the desired data? Is your statement syntax correct? If your statement passes these three tests, then your query is handed to the query optimizer, whose job it is to determine the most efficient way to execute your query. The optimizer will look at such things as the order in which to join the tables named in your from clause and what indexes are available, and then picks an execution plan, which the server uses to execute your query. NOTE Understanding and influencing how your database server chooses execution plans is a fascinating topic that many of you will wish to explore. For those readers using MySQL, you might consider reading Baron Schwartz et al.’s High Performance MySQL (O’Reilly). Among other things, you will learn how to generate indexes, analyze execution plans, influence the optimizer via query hints, and tune your server’s startup parameters. If you are using Oracle Database or SQL Server, dozens of tuning books are available. Once the server has finished executing your query, the result set is returned to the calling application (which is, once again, the mysql tool). As I mentioned in Chapter 1, a result set is just another table containing rows and columns. If your query fails to yield any results, the mysql tool will show you the message found at the end of the following example: mysql> SELECT first_name, last_name -> FROM customer -> WHERE last_name = 'ZIEGLER'; Empty set (0.02 sec) If the query returns one or more rows, the mysql tool will format the results by adding column headers and by constructing boxes around the columns using the -, |, and + symbols, as shown in the next example: mysql> SELECT * -> FROM category; +-------------+-------------+---------------------+ | category_id | name | last_update | +-------------+-------------+---------------------+ | 1 | Action | 2006-02-15 04:46:27 | | 2 | Animation | 2006-02-15 04:46:27 | | 3 | Children | 2006-02-15 04:46:27 | | 4 | Classics | 2006-02-15 04:46:27 | | 5 | Comedy | 2006-02-15 04:46:27 | | 6 | Documentary | 2006-02-15 04:46:27 | | 7 | Drama | 2006-02-15 04:46:27 | | 8 | Family | 2006-02-15 04:46:27 | | 9 | Foreign | 2006-02-15 04:46:27 | | 10 | Games | 2006-02-15 04:46:27 | | 11 | Horror | 2006-02-15 04:46:27 | | 12 | Music | 2006-02-15 04:46:27 | | 13 | New | 2006-02-15 04:46:27 | | 14 | Sci-Fi | 2006-02-15 04:46:27 | |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 37 + }, + { + "text": "| | 9 | Foreign | 2006-02-15 04:46:27 | | 10 | Games | 2006-02-15 04:46:27 | | 11 | Horror | 2006-02-15 04:46:27 | | 12 | Music | 2006-02-15 04:46:27 | | 13 | New | 2006-02-15 04:46:27 | | 14 | Sci-Fi | 2006-02-15 04:46:27 | | 15 | Sports | 2006-02-15 04:46:27 | | 16 | Travel | 2006-02-15 04:46:27 | +-------------+-------------+---------------------+ 16 rows in set (0.02 sec) This query returns all three columns for of all the rows in the category table. After the last row of data is displayed, the mysql tool displays a message telling you how many rows were returned, which, in this case, is 16. Query Clauses Several components or clauses make up the select statement. While only one of them is mandatory when using MySQL (the select clause), you will usually include at least two or three of the six available clauses. Table 3-1 shows the different clauses and their purposes. Table 3-1. Query clauses Clause name Purpose Select Determines which columns to include in the query’s result set From Identifies the tables from which to retrieve data and how the tables should be joined Where Filters out unwanted data Group by Used to group rows together by common column values Having Filters out unwanted groups Order by Sorts the rows of the final result set by one or more columns All of the clauses shown in Table 3-1 are included in the ANSI specification; additionally, several other clauses are unique to MySQL. The following sections delve into the uses of the six major query clauses. The select Clause Even though the select clause is the first clause of a select statement, it is one of the last clauses that the database server evaluates. The reason for this is that before you can determine what to include in the final result set, you need to know all of the possible columns that could be included in the final result set. In order to fully understand the role of the select clause, therefore, you will need to understand a bit about the from clause. Here’s a query to get started: mysql> SELECT * -> FROM language; +-------------+----------+---------------------+ | language_id | name | last_update | +-------------+----------+---------------------+ | 1 | English | 2006-02-15 05:02:19 | | 2 | Italian | 2006-02-15 05:02:19 | | 3 | Japanese | 2006-02-15 05:02:19 | | 4 | Mandarin | 2006-02-15 05:02:19 | | 5 | French | 2006-02-15 05:02:19 | | 6 | German | 2006-02-15 05:02:19 | +-------------+----------+---------------------+ 6 rows in set (0.03 sec) In this query, the from clause lists a single table (language), and the select clause indicates that all columns (designated by *) in the language table should be included in the result set. This query could be described in English as follows: Show me all the columns and all the rows in the language table. In addition to specifying all the columns via the asterisk character, you can explicitly name the", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 38 + }, + { + "text": "*) in the language table should be included in the result set. This query could be described in English as follows: Show me all the columns and all the rows in the language table. In addition to specifying all the columns via the asterisk character, you can explicitly name the columns you are interested in, such as: mysql> SELECT language_id, name, last_update -> FROM language; +-------------+----------+---------------------+ | language_id | name | last_update | +-------------+----------+---------------------+ | 1 | English | 2006-02-15 05:02:19 | | 2 | Italian | 2006-02-15 05:02:19 | | 3 | Japanese | 2006-02-15 05:02:19 | | 4 | Mandarin | 2006-02-15 05:02:19 | | 5 | French | 2006-02-15 05:02:19 | | 6 | German | 2006-02-15 05:02:19 | +-------------+----------+---------------------+ 6 rows in set (0.00 sec) The results are identical to the first query, since all the columns in the language table (language_id, name, and last_update ) are named in the select clause. You can choose to include only a subset of the columns in the language table as well: mysql> SELECT name -> FROM language; +----------+ | name | +----------+ | English | | Italian | | Japanese | | Mandarin | | French | | German | +----------+ 6 rows in set (0.00 sec) The job of the select clause, therefore, is the following: The select clause determines which of all possible columns should be included in the query’s result set. If you were limited to including only columns from the table or tables named in the from clause, things would be rather dull. However, you can spice things up by including in your select clause such things as: Literals, such as numbers or strings Expressions, such as transaction.amount * −1 Built-in function calls, such as ROUND(transaction.amount, 2) User-defined function calls The next query demonstrates the use of a table column, a literal, an expression, and a built-in function call in a single query against the employee table: mysql> SELECT language_id, -> 'COMMON' language_usage, -> language_id * 3.1415927 lang_pi_value, -> upper(name) language_name -> FROM language; +-------------+----------------+---------------+---------------+ | language_id | language_usage | lang_pi_value | language_name | +-------------+----------------+---------------+---------------+ | 1 | COMMON | 3.1415927 | ENGLISH | | 2 | COMMON | 6.2831854 | ITALIAN | | 3 | COMMON | 9.4247781 | JAPANESE | | 4 | COMMON | 12.5663708 | MANDARIN | | 5 | COMMON | 15.7079635 | FRENCH | | 6 | COMMON | 18.8495562 | GERMAN | +-------------+----------------+---------------+---------------+ 6 rows in set (0.04 sec) We cover expressions and built-in functions in detail later, but I wanted to give you a feel for what kinds of things can be included in the select clause. If you only need to execute a built-in function or evaluate a simple expression, you can skip the from clause entirely. Here’s an example: mysql> SELECT version(), -> user(), -> database(); +-----------+----------------+------------+ | version() | user() | database() | +-----------+----------------+------------+ | 8.0.15 | root@localhost | sakila | +-----------+----------------+------------+ 1 row in set (0.00 sec) Since this query simply calls", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 39 + }, + { + "text": "or evaluate a simple expression, you can skip the from clause entirely. Here’s an example: mysql> SELECT version(), -> user(), -> database(); +-----------+----------------+------------+ | version() | user() | database() | +-----------+----------------+------------+ | 8.0.15 | root@localhost | sakila | +-----------+----------------+------------+ 1 row in set (0.00 sec) Since this query simply calls three built-in functions and doesn’t retrieve data from any tables, there is no need for a from clause. Column Aliases Although the mysql tool will generate labels for the columns returned by your queries, you may want to assign your own labels. While you might want to assign a new label to a column from a table (if it is poorly or ambiguously named), you will almost certainly want to assign your own labels to those columns in your result set that are generated by expressions or built-in function calls. You can do so by adding a column alias after each element of your select clause. Here’s the previous query against the language table, which included column aliases for three of the columns: mysql> SELECT language_id, -> 'COMMON' language_usage, -> language_id * 3.1415927 lang_pi_value, -> upper(name) language_name -> FROM language; +-------------+----------------+---------------+---------------+ | language_id | language_usage | lang_pi_value | language_name | +-------------+----------------+---------------+---------------+ | 1 | COMMON | 3.1415927 | ENGLISH | | 2 | COMMON | 6.2831854 | ITALIAN | | 3 | COMMON | 9.4247781 | JAPANESE | | 4 | COMMON | 12.5663708 | MANDARIN | | 5 | COMMON | 15.7079635 | FRENCH | | 6 | COMMON | 18.8495562 | GERMAN | +-------------+----------------+---------------+---------------+ 6 rows in set (0.04 sec) If you look at the select clause, you can see how the column aliases language_usage, lang_pi_value, and language_name are added after the second, third, and fourth columns. I think you will agree that the output is easier to understand with column aliases in place, and it would be easier to work with programmatically if you were issuing the query from within Java or Python rather than interactively via the mysql tool. In order to make your column aliases stand out even more, you also have the option of using the as keyword before the alias name, as in: mysql> SELECT language_id, -> 'COMMON' AS language_usage, -> language_id * 3.1415927 AS lang_pi_value, -> upper(name) AS language_name -> FROM language; Many people feel that including the optional as keyword improves readability, although I have chosen not to use it for the examples in this book. Removing Duplicates In some cases, a query might return duplicate rows of data. For example, if you were to retrieve the IDs of all actors who appeared in a film, you would see the following: mysql> SELECT actor_id FROM film_actor ORDER BY actor_id; +----------+ | actor_id | +----------+ | 1 | | 1 | | 1 | | 1 | | 1 | | 1 | | 1 | | 1 | | 1 | | 1 | ... | 200 | | 200 | | 200 | | 200 | | 200 | |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 40 + }, + { + "text": "actor_id | +----------+ | 1 | | 1 | | 1 | | 1 | | 1 | | 1 | | 1 | | 1 | | 1 | | 1 | ... | 200 | | 200 | | 200 | | 200 | | 200 | | 200 | | 200 | | 200 | | 200 | +----------+ 5462 rows in set (0.01 sec) Since some actors appeared in more than one film, you will see the same actor ID multiple times. What you probably want in this case is the distinct set of actors, instead of seeing the actor IDs repeated for each film in which they appeared. You can achieve this by adding the keyword distinct directly after the select keyword, as demonstrated by the following: mysql> SELECT DISTINCT actor_id FROM film_actor ORDER BY actor_id; +----------+ | actor_id | +----------+ | 1 | | 2 | | 3 | | 4 | | 5 | | 6 | | 7 | | 8 | | 9 | | 10 | ... | 192 | | 193 | | 194 | | 195 | | 196 | | 197 | | 198 | | 199 | | 200 | +----------+ 200 rows in set (0.01 sec) The result set now contains 200 rows, one for each distinct actor, rather than 5,462 rows, one for each film appearance by an actor. NOTE If you simply want a list of all actors, you can query the Actor table rather than reading through all the rows in Film_Actor and removing duplicates. If you do not want the server to remove duplicate data, or you are sure there will be no duplicates in your result set, you can specify the ALL keyword instead of specifying DISTINCT. However, the ALL keyword is the default and never needs to be explicitly named, so most programmers do not include ALL in their queries. WARNING Keep in mind that generating a distinct set of results requires the data to be sorted, which can be time-consuming for large result sets. Don’t fall into the trap of using DISTINCT just to be sure there are no duplicates; instead, take the time to understand the data you are working with so that you will know whether duplicates are possible. The from Clause Thus far, you have seen queries whose from clauses contain a single table. Although most SQL books will define the from clause as simply a list of one or more tables, I would like to broaden the definition as follows: The from clause defines the tables used by a query, along with the means of linking the tables together. This definition is composed of two separate but related concepts, which we explore in the following sections. Tables When confronted with the term table, most people think of a set of related rows stored in a database. While this does describe one type of table, I would like to use the word in a more", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 41 + }, + { + "text": "two separate but related concepts, which we explore in the following sections. Tables When confronted with the term table, most people think of a set of related rows stored in a database. While this does describe one type of table, I would like to use the word in a more general way by removing any notion of how the data might be stored and concentrating on just the set of related rows. Three different types of tables meet this relaxed definition: Permanent tables (i.e., created using the create table statement) Derived tables (i.e., rows returned by a subquery and held in memory) Temporary tables (i.e. volatile data held in memory) Virtual tables (i.e., created using the create view statement) Each of these table types may be included in a query’s from clause. By now, you should be comfortable with including a permanent table in a from clause, so I will briefly describe the other types of tables that can be referenced in a from clause. DERIVED (SUBQUERY-GENERATED) TABLES A subquery is a query contained within another query. Subqueries are surrounded by parentheses and can be found in various parts of a select statement; within the from clause, however, a subquery serves the role of generating a derived table that is visible from all other query clauses and can interact with other tables named in the from clause. Here’s a simple example: mysql> SELECT concat(cust.last_name, ', ', cust.first_name) full_name -> FROM -> (SELECT first_name, last_name, email -> FROM customer -> WHERE first_name = 'JESSIE' -> ) cust; +---------------+ | full_name | +---------------+ | BANKS, JESSIE | | MILAM, JESSIE | +---------------+ 2 rows in set (0.00 sec) In this example, a subquery against the employee table returns three columns, and the containing query references two of the three available columns. The subquery is referenced by the containing query via its alias, which, in this case, is cust. The data in cust is held in memory for the duration of the query and is then discarded. This is a simplistic and not particularly useful example of a subquery in a from clause; you will find detailed coverage of subqueries in Chapter 9. TEMPORARY TABLES Although the implementations differ, every relational database allows the ability to define volatile, or temporary, tables. These tables look just like permanent tables, but any data inserted into a temporary table will disappear at some point (generally at the end of a transaction or when your database session is closed). Here’s a simple example showing how actors whose last names start with J can be created: mysql> CREATE TEMPORARY TABLE actors_j -> (actor_id smallint(5), -> first_name varchar(45), -> last_name varchar(45) -> ); Query OK, 0 rows affected (0.00 sec) mysql> INSERT INTO actors_j -> SELECT actor_id, first_name, last_name -> FROM actor -> WHERE last_name LIKE 'J%'; Query OK, 7 rows affected (0.03 sec) Records: 7 Duplicates: 0 Warnings: 0 mysql> SELECT * FROM actors_j; +----------+------------+-----------+ | actor_id | first_name | last_name | +----------+------------+-----------+ | 119 | WARREN |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 42 + }, + { + "text": "sec) mysql> INSERT INTO actors_j -> SELECT actor_id, first_name, last_name -> FROM actor -> WHERE last_name LIKE 'J%'; Query OK, 7 rows affected (0.03 sec) Records: 7 Duplicates: 0 Warnings: 0 mysql> SELECT * FROM actors_j; +----------+------------+-----------+ | actor_id | first_name | last_name | +----------+------------+-----------+ | 119 | WARREN | JACKMAN | | 131 | JANE | JACKMAN | | 8 | MATTHEW | JOHANSSON | | 64 | RAY | JOHANSSON | | 146 | ALBERT | JOHANSSON | | 82 | WOODY | JOLIE | | 43 | KIRK | JOVOVICH | +----------+------------+-----------+ 7 rows in set (0.00 sec) These seven rows are held in memory temporarily and will disappear after your session is closed. NOTE Most database servers also drop the temporary table when the session ends. The exception is Oracle Database, which keeps the definition of the temporary table available for future sessions. VIEWS A view is a query that is stored in the data dictionary. It looks and acts like a table, but there is no data associated with a view (this is why I call it a virtual table). When you issue a query against a view, your query is merged with the view definition to create a final query to be executed. To demonstrate, here’s a view definition that queries the employee table and includes a call to a built-in function: mysql> CREATE VIEW cust_vw AS -> SELECT customer_id, first_name, last_name, active -> FROM customer; Query OK, 0 rows affected (0.12 sec) When the view is created, no additional data is generated or stored: the server simply tucks away the select statement for future use. Now that the view exists, you can issue queries against it, as in: mysql> SELECT first_name, last_name -> FROM cust_vw -> WHERE active = 0; +------------+-----------+ | first_name | last_name | +------------+-----------+ | SANDRA | MARTIN | | JUDITH | COX | | SHEILA | WELLS | | ERICA | MATTHEWS | | HEIDI | LARSON | | PENNY | NEAL | | KENNETH | GOODEN | | HARRY | ARCE | | NATHAN | RUNYON | | THEODORE | CULP | | MAURICE | CRAWLEY | | BEN | EASTER | | CHRISTIAN | JUNG | | JIMMIE | EGGLESTON | | TERRANCE | ROUSH | +------------+-----------+ 15 rows in set (0.00 sec) Views are created for various reasons, including to hide columns from users and to simplify complex database designs. Table Links The second deviation from the simple from clause definition is the mandate that if more than one table appears in the from clause, the conditions used to link the tables must be included as well. This is not a requirement of MySQL or any other database server, but it is the ANSI- approved method of joining multiple tables, and it is the most portable across the various database servers. We explore joining multiple tables in depth in Chapters Chapter 5 and Chapter 10, but here’s a simple example in case I have piqued your", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 43 + }, + { + "text": "other database server, but it is the ANSI- approved method of joining multiple tables, and it is the most portable across the various database servers. We explore joining multiple tables in depth in Chapters Chapter 5 and Chapter 10, but here’s a simple example in case I have piqued your curiosity: mysql> SELECT customer.first_name, customer.last_name, -> time(rental.rental_date) rental_time -> FROM customer -> INNER JOIN rental -> ON customer.customer_id = rental.customer_id -> WHERE date(rental.rental_date) = '2005-06-14'; +------------+-----------+-------------+ | first_name | last_name | rental_time | +------------+-----------+-------------+ | JEFFERY | PINSON | 22:53:33 | | ELMER | NOE | 22:55:13 | | MINNIE | ROMERO | 23:00:34 | | MIRIAM | MCKINNEY | 23:07:08 | | DANIEL | CABRAL | 23:09:38 | | TERRANCE | ROUSH | 23:12:46 | | JOYCE | EDWARDS | 23:16:26 | | GWENDOLYN | MAY | 23:16:27 | | CATHERINE | CAMPBELL | 23:17:03 | | MATTHEW | MAHAN | 23:25:58 | | HERMAN | DEVORE | 23:35:09 | | AMBER | DIXON | 23:42:56 | | TERRENCE | GUNDERSON | 23:47:35 | | SONIA | GREGORY | 23:50:11 | | CHARLES | KOWALSKI | 23:54:34 | | JEANETTE | GREENE | 23:54:46 | +------------+-----------+-------------+ 16 rows in set (0.01 sec) The previous query displays data from both the customer table (first_name, last_name) and the rental table (rental_date), so both tables are included in the from clause. The mechanism for linking the two tables (referred to as a join) is the customer ID stored in both the customer and rental tables. Thus, the database server is instructed to use the value of the customer_id column in the customer table to find all of the customer’s rentals in the rental table. Join conditions for the two tables are found in the on subclause of the from clause; in this case, the join condition is ON customer.customer_id = rental.customer_id. The where clause is not part of the join, and is only included to keep the result set fairly small, since there are over 16,000 rows in the rental table. Again, please refer to Chapter 5 for a thorough discussion of joining multiple tables. Defining Table Aliases When multiple tables are joined in a single query, you need a way to identify which table you are referring to when you reference columns in the select, where, group by, having, and order by clauses. You have two choices when referencing a table outside the from clause: Use the entire table name, such as employee.emp_id. Assign each table an alias and use the alias throughout the query. In the previous query, I chose to use the entire table name in the select and on clauses. Here’s what the same query looks like using table aliases: SELECT c.first_name, c.last_name, time(r.rental_date) rental_time FROM customer c INNER JOIN rental r ON c.customer_id = r.customer_id WHERE date(r.rental_date) = '2005-06-14'; If you look closely at the from clause, you will see that the customer table is assigned the alias c, and the rental table is assigned the alias", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 44 + }, + { + "text": "table aliases: SELECT c.first_name, c.last_name, time(r.rental_date) rental_time FROM customer c INNER JOIN rental r ON c.customer_id = r.customer_id WHERE date(r.rental_date) = '2005-06-14'; If you look closely at the from clause, you will see that the customer table is assigned the alias c, and the rental table is assigned the alias r. These aliases are then used in the on clause when defining the join condition as well as in the select clause when specifying the columns to include in the result set. I hope you will agree that using aliases makes for a more compact statement without causing confusion (as long as your choices for alias names are reasonable). Additionally, you may use the as keyword with your table aliases, similar to what was demonstrated earlier for column aliases: SELECT c.first_name, c.last_name, time(r.rental_date) rental_time FROM customer AS c INNER JOIN rental AS r ON c.customer_id = r.customer_id WHERE date(r.rental_date) = '2005-06-14'; I have found that roughly half of the database developers I have worked with use the as keyword with their column and table aliases, and half do not. The where Clause In some cases, you may want to retrieve all rows from a table, especially for small tables such as language. Most of the time, however, you will not wish to retrieve every row from a table but will want a way to filter out those rows that are not of interest. This is a job for the where clause. The where clause is the mechanism for filtering out unwanted rows from your result set. For example, perhaps you are interested in renting a film, but you are only interested in movies rated G that can be kept for at least a week. The following query employs a where clause to retrieve only the films meeting these criteria: mysql> SELECT title -> FROM film -> WHERE rating = 'G' AND rental_duration >= 7; +-------------------------+ | title | +-------------------------+ | BLANKET BEVERLY | | BORROWERS BEDAZZLED | | BRIDE INTRIGUE | | CATCH AMISTAD | | CITIZEN SHREK | | COLDBLOODED DARLING | | CONTROL ANTHEM | | CRUELTY UNFORGIVEN | | DARN FORRESTER | | DESPERATE TRAINSPOTTING | | DIARY PANIC | | DRACULA CRYSTAL | | EMPIRE MALKOVICH | | FIREHOUSE VIETNAM | | GILBERT PELICAN | | GRADUATE LORD | | GREASE YOUTH | | GUN BONNIE | | HOOK CHARIOTS | | MARRIED GO | | MENAGERIE RUSHMORE | | MUSCLE BRIGHT | | OPERATION OPERATION | | PRIMARY GLASS | | REBEL AIRPORT | | SPIKING ELEMENT | | TRUMAN CRAZY | | WAKE JAWS | | WAR NOTTING | +-------------------------+ 29 rows in set (0.00 sec) In this case, the where clause filtered out 971 of the 1000 rows in the film table. This where clause contains two filter conditions, but you can include as many conditions as required; individual conditions are separated using operators such as and, or, and not (see Chapter 4 for a complete discussion of the where clause and filter", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 45 + }, + { + "text": "971 of the 1000 rows in the film table. This where clause contains two filter conditions, but you can include as many conditions as required; individual conditions are separated using operators such as and, or, and not (see Chapter 4 for a complete discussion of the where clause and filter conditions). Let’s see what would happen if you change the operator separating the two conditions from and to or: mysql> SELECT title -> FROM film -> WHERE rating = 'G' OR rental_duration >= 7; +---------------------------+ | title | +---------------------------+ | ACE GOLDFINGER | | ADAPTATION HOLES | | AFFAIR PREJUDICE | | AFRICAN EGG | | ALAMO VIDEOTAPE | | AMISTAD MIDSUMMER | | ANGELS LIFE | | ANNIE IDENTITY | |... | | WATERSHIP FRONTIER | | WEREWOLF LOLA | | WEST LION | | WESTWARD SEABISCUIT | | WOLVES DESIRE | | WON DARES | | WORKER TARZAN | | YOUNG LANGUAGE | +---------------------------+ 340 rows in set (0.00 sec) When you separate conditions using the and operator, all conditions must evaluate to true to be included in the result set; when you use or, however, only one of the conditions needs to evaluate to true for a row to be included, which explains why the size of the result set has jumped from 29 to 340 rows. So, what should you do if you need to use both and and or operators in your where clause? Glad you asked. You should use parentheses to group conditions together. The next query specifies that only those films which are rated G and are available for 7 or more days, or are rated PG-13 and are available 3 or fewer days be included in the result set: mysql> SELECT title, rating, rental_duration -> FROM film -> WHERE (rating = 'G' AND rental_duration >= 7) -> OR (rating = 'PG-13' AND rental_duration < 4); +-------------------------+--------+-----------------+ | title | rating | rental_duration | +-------------------------+--------+-----------------+ | ALABAMA DEVIL | PG-13 | 3 | | BACKLASH UNDEFEATED | PG-13 | 3 | | BILKO ANONYMOUS | PG-13 | 3 | | BLANKET BEVERLY | G | 7 | | BORROWERS BEDAZZLED | G | 7 | | BRIDE INTRIGUE | G | 7 | | CASPER DRAGONFLY | PG-13 | 3 | | CATCH AMISTAD | G | 7 | | CITIZEN SHREK | G | 7 | | COLDBLOODED DARLING | G | 7 | |... | | TREASURE COMMAND | PG-13 | 3 | | TRUMAN CRAZY | G | 7 | | WAIT CIDER | PG-13 | 3 | | WAKE JAWS | G | 7 | | WAR NOTTING | G | 7 | | WORLD LEATHERNECKS | PG-13 | 3 | +-------------------------+--------+-----------------+ 68 rows in set (0.00 sec) You should always use parentheses to separate groups of conditions when mixing different operators so that you, the database server, and anyone who comes along later to modify your code will be on the same page. The group by and having", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 46 + }, + { + "text": "3 | +-------------------------+--------+-----------------+ 68 rows in set (0.00 sec) You should always use parentheses to separate groups of conditions when mixing different operators so that you, the database server, and anyone who comes along later to modify your code will be on the same page. The group by and having Clauses All the queries thus far have retrieved raw data without any manipulation. Sometimes, however, you will want to find trends in your data that will require the database server to cook the data a bit before you retrieve your result set. One such mechanism is the group by clause, which is used to group data by column values. For example, let’s say you wanted to find all of the customers who have rented 40 or more films. Rather than looking through all 16,044 rows in the rental table, you can write a query which instructs the server to group all rentals by customer, count the number of rentals for each customer, and then return only those customers whose rental count is at least 40. When using the group by clause to generate groups of rows, you may also use the having clause, which allows you to filter grouped data in the same way the where clause lets you filter raw data. Here’s what the query looks like: mysql> SELECT c.first_name, c.last_name, count(*) -> FROM customer c -> INNER JOIN rental r -> ON c.customer_id = r.customer_id -> GROUP BY c.first_name, c.last_name -> HAVING count(*) >= 40; +------------+-----------+----------+ | first_name | last_name | count(*) | +------------+-----------+----------+ | TAMMY | SANDERS | 41 | | CLARA | SHAW | 42 | | ELEANOR | HUNT | 46 | | SUE | PETERS | 40 | | MARCIA | DEAN | 42 | | WESLEY | BULL | 40 | | KARL | SEAL | 45 | +------------+-----------+----------+ 7 rows in set (0.03 sec) I wanted to briefly mention these two clauses so that they don’t catch you by surprise later in the book, but they are a bit more advanced than the other four select clauses. Therefore, I ask that you wait until Chapter 8 for a full description of how and when to use group by and having. The order by Clause In general, the rows in a result set returned from a query are not in any particular order. If you want your result set to be sorted, you will need to instruct the server to sort the results using the order by clause: The order by clause is the mechanism for sorting your result set using either raw column data or expressions based on column data. For example, here’s another look at an earlier query which returns all customers who rented a film on June 14th, 2005: mysql> SELECT c.first_name, c.last_name, -> time(r.rental_date) rental_time -> FROM customer c -> INNER JOIN rental r -> ON c.customer_id = r.customer_id -> WHERE date(r.rental_date) = '2005-06-14'; +------------+-----------+-------------+ | first_name | last_name | rental_time | +------------+-----------+-------------+ | JEFFERY | PINSON | 22:53:33", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 47 + }, + { + "text": "customers who rented a film on June 14th, 2005: mysql> SELECT c.first_name, c.last_name, -> time(r.rental_date) rental_time -> FROM customer c -> INNER JOIN rental r -> ON c.customer_id = r.customer_id -> WHERE date(r.rental_date) = '2005-06-14'; +------------+-----------+-------------+ | first_name | last_name | rental_time | +------------+-----------+-------------+ | JEFFERY | PINSON | 22:53:33 | | ELMER | NOE | 22:55:13 | | MINNIE | ROMERO | 23:00:34 | | MIRIAM | MCKINNEY | 23:07:08 | | DANIEL | CABRAL | 23:09:38 | | TERRANCE | ROUSH | 23:12:46 | | JOYCE | EDWARDS | 23:16:26 | | GWENDOLYN | MAY | 23:16:27 | | CATHERINE | CAMPBELL | 23:17:03 | | MATTHEW | MAHAN | 23:25:58 | | HERMAN | DEVORE | 23:35:09 | | AMBER | DIXON | 23:42:56 | | TERRENCE | GUNDERSON | 23:47:35 | | SONIA | GREGORY | 23:50:11 | | CHARLES | KOWALSKI | 23:54:34 | | JEANETTE | GREENE | 23:54:46 | +------------+-----------+-------------+ 16 rows in set (0.01 sec) If you would like the results to be in alphabetical order by last name, you can add the last_name column to the order by clause: mysql> SELECT c.first_name, c.last_name, -> time(r.rental_date) rental_time -> FROM customer c -> INNER JOIN rental r -> ON c.customer_id = r.customer_id -> WHERE date(r.rental_date) = '2005-06-14' -> ORDER BY c.last_name; +------------+-----------+-------------+ | first_name | last_name | rental_time | +------------+-----------+-------------+ | DANIEL | CABRAL | 23:09:38 | | CATHERINE | CAMPBELL | 23:17:03 | | HERMAN | DEVORE | 23:35:09 | | AMBER | DIXON | 23:42:56 | | JOYCE | EDWARDS | 23:16:26 | | JEANETTE | GREENE | 23:54:46 | | SONIA | GREGORY | 23:50:11 | | TERRENCE | GUNDERSON | 23:47:35 | | CHARLES | KOWALSKI | 23:54:34 | | MATTHEW | MAHAN | 23:25:58 | | GWENDOLYN | MAY | 23:16:27 | | MIRIAM | MCKINNEY | 23:07:08 | | ELMER | NOE | 22:55:13 | | JEFFERY | PINSON | 22:53:33 | | MINNIE | ROMERO | 23:00:34 | | TERRANCE | ROUSH | 23:12:46 | +------------+-----------+-------------+ 16 rows in set (0.01 sec) While it is not the case in this example, large customer lists will often contain multiple people having the same last name, so you may want to extend the sort criteria to include the person’s first name as well; you can accomplish this by adding the first_name column after the last_name column in the order by clause: mysql> SELECT c.first_name, c.last_name, -> time(r.rental_date) rental_time -> FROM customer c -> INNER JOIN rental r -> ON c.customer_id = r.customer_id -> WHERE date(r.rental_date) = '2005-06-14' -> ORDER BY c.last_name, c.first_name; +------------+-----------+-------------+ | first_name | last_name | rental_time | +------------+-----------+-------------+ | DANIEL | CABRAL | 23:09:38 | | CATHERINE | CAMPBELL | 23:17:03 | | HERMAN | DEVORE | 23:35:09 | | AMBER | DIXON | 23:42:56 | | JOYCE | EDWARDS | 23:16:26 | | JEANETTE | GREENE | 23:54:46 | | SONIA | GREGORY | 23:50:11 | | TERRENCE | GUNDERSON | 23:47:35 | |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 48 + }, + { + "text": "| CATHERINE | CAMPBELL | 23:17:03 | | HERMAN | DEVORE | 23:35:09 | | AMBER | DIXON | 23:42:56 | | JOYCE | EDWARDS | 23:16:26 | | JEANETTE | GREENE | 23:54:46 | | SONIA | GREGORY | 23:50:11 | | TERRENCE | GUNDERSON | 23:47:35 | | CHARLES | KOWALSKI | 23:54:34 | | MATTHEW | MAHAN | 23:25:58 | | GWENDOLYN | MAY | 23:16:27 | | MIRIAM | MCKINNEY | 23:07:08 | | ELMER | NOE | 22:55:13 | | JEFFERY | PINSON | 22:53:33 | | MINNIE | ROMERO | 23:00:34 | | TERRANCE | ROUSH | 23:12:46 | +------------+-----------+-------------+ 16 rows in set (0.01 sec) The order in which columns appear in your order by clause does make a difference when you include more than one column. If you were to switch the order of the two columns in the order by clause, Amber Dixon would appear first in the result set. Ascending Versus Descending Sort Order When sorting, you have the option of specifying ascending or descending order via the asc and desc keywords. The default is ascending, so you will need to add the desc keyword, only if you want to use a descending sort. For example, the following query shows all customers who rented films on June 14th 2005 in descending order of rental time: mysql> SELECT c.first_name, c.last_name, -> time(r.rental_date) rental_time -> FROM customer c -> INNER JOIN rental r -> ON c.customer_id = r.customer_id -> WHERE date(r.rental_date) = '2005-06-14' -> ORDER BY time(r.rental_date) desc; +------------+-----------+-------------+ | first_name | last_name | rental_time | +------------+-----------+-------------+ | JEANETTE | GREENE | 23:54:46 | | CHARLES | KOWALSKI | 23:54:34 | | SONIA | GREGORY | 23:50:11 | | TERRENCE | GUNDERSON | 23:47:35 | | AMBER | DIXON | 23:42:56 | | HERMAN | DEVORE | 23:35:09 | | MATTHEW | MAHAN | 23:25:58 | | CATHERINE | CAMPBELL | 23:17:03 | | GWENDOLYN | MAY | 23:16:27 | | JOYCE | EDWARDS | 23:16:26 | | TERRANCE | ROUSH | 23:12:46 | | DANIEL | CABRAL | 23:09:38 | | MIRIAM | MCKINNEY | 23:07:08 | | MINNIE | ROMERO | 23:00:34 | | ELMER | NOE | 22:55:13 | | JEFFERY | PINSON | 22:53:33 | +------------+-----------+-------------+ 16 rows in set (0.01 sec) Descending sorts are commonly used for ranking queries, such as “show me the top five account balances.” MySQL includes a limit clause that allows you to sort your data and then discard all but the first X rows. Sorting via Numeric Placeholders If you are sorting using the columns in your select clause, you can opt to reference the columns by their position in the select clause rather than by name. This can be especially helpful if you are sorting on an expression, such as in the previous example. Here’s the previous example one last time, with an order by clause specifying a descending sort using the 3rd element in the select clause: mysql> SELECT c.first_name,", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 49 + }, + { + "text": "clause rather than by name. This can be especially helpful if you are sorting on an expression, such as in the previous example. Here’s the previous example one last time, with an order by clause specifying a descending sort using the 3rd element in the select clause: mysql> SELECT c.first_name, c.last_name, -> time(r.rental_date) rental_time -> FROM customer c -> INNER JOIN rental r -> ON c.customer_id = r.customer_id -> WHERE date(r.rental_date) = '2005-06-14' -> ORDER BY 3 desc; +------------+-----------+-------------+ | first_name | last_name | rental_time | +------------+-----------+-------------+ | JEANETTE | GREENE | 23:54:46 | | CHARLES | KOWALSKI | 23:54:34 | | SONIA | GREGORY | 23:50:11 | | TERRENCE | GUNDERSON | 23:47:35 | | AMBER | DIXON | 23:42:56 | | HERMAN | DEVORE | 23:35:09 | | MATTHEW | MAHAN | 23:25:58 | | CATHERINE | CAMPBELL | 23:17:03 | | GWENDOLYN | MAY | 23:16:27 | | JOYCE | EDWARDS | 23:16:26 | | TERRANCE | ROUSH | 23:12:46 | | DANIEL | CABRAL | 23:09:38 | | MIRIAM | MCKINNEY | 23:07:08 | | MINNIE | ROMERO | 23:00:34 | | ELMER | NOE | 22:55:13 | | JEFFERY | PINSON | 22:53:33 | +------------+-----------+-------------+ 16 rows in set (0.01 sec) You might want to use this feature sparingly, since adding a column to the select clause without changing the numbers in the order by clause can lead to unexpected results. Personally, I may reference columns positionally when writing ad hoc queries, but I always reference columns by name when writing code. Test Your Knowledge The following exercises are designed to strengthen your understanding of the select statement and its various clauses. Please see Appendix C. Exercise 3-1 Retrieve the actor ID, first name, and last name for all actors. Sort by last name and then by first name. Exercise 3-2 Retrieve the actor ID, first name, and last name for all actors whose last name equals 'WILLIAMS' or .'DAVIS' Exercise 3-3 Write a query against the rental table that returns the IDs of the customers who rented a film on July 5th 2005 (use the rental.rental_date column, and you can use the date() function to ignore the time component). Include a single row for each distinct customer ID. Exercise 3-4 Fill in the blanks (denoted by <#>) for this multi-table query to achieve the results shown below. mysql> SELECT c.email, r.return_date -> FROM customer c -> INNER JOIN rental <1> -> ON c.customer_id = <2> -> WHERE date(r.rental_date) = '2005-06-14' -> ORDER BY <3>, <4>; +---------------------------------------+---------------------+ | email | return_date | +---------------------------------------+---------------------+ | DANIEL.CABRAL@sakilacustomer.org | 2005-06-23 22:00:38 | | TERRANCE.ROUSH@sakilacustomer.org | 2005-06-23 21:53:46 | | MIRIAM.MCKINNEY@sakilacustomer.org | 2005-06-21 17:12:08 | | GWENDOLYN.MAY@sakilacustomer.org | 2005-06-20 02:40:27 | | JEANETTE.GREENE@sakilacustomer.org | 2005-06-19 23:26:46 | | HERMAN.DEVORE@sakilacustomer.org | 2005-06-19 03:20:09 | | JEFFERY.PINSON@sakilacustomer.org | 2005-06-18 21:37:33 | | MATTHEW.MAHAN@sakilacustomer.org | 2005-06-18 05:18:58 | | MINNIE.ROMERO@sakilacustomer.org | 2005-06-18 01:58:34 | | SONIA.GREGORY@sakilacustomer.org | 2005-06-17 21:44:11 | | TERRENCE.GUNDERSON@sakilacustomer.org | 2005-06-17 05:28:35 | | ELMER.NOE@sakilacustomer.org | 2005-06-17 02:11:13 |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 50 + }, + { + "text": "02:40:27 | | JEANETTE.GREENE@sakilacustomer.org | 2005-06-19 23:26:46 | | HERMAN.DEVORE@sakilacustomer.org | 2005-06-19 03:20:09 | | JEFFERY.PINSON@sakilacustomer.org | 2005-06-18 21:37:33 | | MATTHEW.MAHAN@sakilacustomer.org | 2005-06-18 05:18:58 | | MINNIE.ROMERO@sakilacustomer.org | 2005-06-18 01:58:34 | | SONIA.GREGORY@sakilacustomer.org | 2005-06-17 21:44:11 | | TERRENCE.GUNDERSON@sakilacustomer.org | 2005-06-17 05:28:35 | | ELMER.NOE@sakilacustomer.org | 2005-06-17 02:11:13 | | JOYCE.EDWARDS@sakilacustomer.org | 2005-06-16 21:00:26 | | AMBER.DIXON@sakilacustomer.org | 2005-06-16 04:02:56 | | CHARLES.KOWALSKI@sakilacustomer.org | 2005-06-16 02:26:34 | | CATHERINE.CAMPBELL@sakilacustomer.org | 2005-06-15 20:43:03 | +---------------------------------------+---------------------+ 16 rows in set (0.03 sec) Chapter 4. Filtering Sometimes you will want to work with every row in a table, such as: Purging all data from a table used to stage new data warehouse feeds Modifying all rows in a table after a new column has been added Retrieving all rows from a message queue table In cases like these, your SQL statements won’t need to have a where clause, since you don’t need to exclude any rows from consideration. Most of the time, however, you will want to narrow your focus to a subset of a table’s rows. Therefore, all the SQL data statements (except the insert statement) include an optional where clause containing one or more filter conditions used to restrict the number of rows acted on by the SQL statement. Additionally, the select statement includes a having clause in which filter conditions pertaining to grouped data may be included. This chapter explores the various types of filter conditions that you can employ in the where clauses of select, update, and delete statements; I demonstrate the use of filter conditions in the having clause of a select statement in Chapter 8. Condition Evaluation A where clause may contain one or more conditions, separated by the operators and and or. If multiple conditions are separated only by the and operator, then all the conditions must evaluate to true for the row to be included in the result set. Consider the following where clause: WHERE first_name = 'STEVEN' AND create_date > '2006-01-01' Given these two conditions, only rows where the first name is Steven and the creation date was after January 1st 2006 will be included in the result set. Although this example uses only two conditions, no matter how many conditions are in your where clause, if they are separated by the and operator they must all evaluate to true for the row to be included in the result set. If all conditions in the where clause are separated by the or operator, however, only one of the conditions must evaluate to true for the row to be included in the result set. Consider the following two conditions: WHERE first_name = 'STEVEN' OR create_date > '2006-01-01' There are now various ways for a given row to be included in the result set: The first name is Steven and the creation date was after January 1, 2006 The first name is Steven and the creation date was on or before January 1, 2006 The first name is anything other than Steven but the creation", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 51 + }, + { + "text": "given row to be included in the result set: The first name is Steven and the creation date was after January 1, 2006 The first name is Steven and the creation date was on or before January 1, 2006 The first name is anything other than Steven but the creation date was after January 1, 2006 Table 4-1 shows the possible outcomes for a where clause containing two conditions separated by the or operator. Table 4-1. Two-condition evaluation using or Intermediate result Final result WHERE true OR true True WHERE true OR false True WHERE false OR true True WHERE false OR false False In the case of the preceding example, the only way for a row to be excluded from the result set is if the person’s first name was not Steven and the creation date was on or before January 1, 2006. Using Parentheses If your where clause includes three or more conditions using both the and and or operators, you should use parentheses to make your intent clear, both to the database server and to anyone else reading your code. Here’s a where clause that extends the previous example by checking to make sure that the first name is Steven or the last name is Young, and the creation date is after January 1st 2006: WHERE (first_name = 'STEVEN' OR last_name = 'YOUNG') AND create_date > '2006-01-01' There are now three conditions; for a row to make it to the final result set, either the first or second conditions (or both) must evaluate to true , and the third condition must evaluate to true . Table 4-2 shows the possible outcomes for this where clause. Table 4-2. Three-condition evaluation using and, or Intermediate result Final result WHERE (true OR true) AND true True WHERE (true OR false) AND true True WHERE (false OR true) AND true True WHERE (false OR false) AND true False WHERE (true OR true) AND false False WHERE (true OR false) AND false False WHERE (false OR true) AND false False WHERE (false OR false) AND false False As you can see, the more conditions you have in your where clause, the more combinations there are for the server to evaluate. In this case, only three of the eight combinations yield a final result of true. Using the not Operator Hopefully, the previous three-condition example is fairly easy to understand. Consider the following modification, however: WHERE NOT (first_name = 'STEVEN' OR last_name = 'YOUNG') AND create_date > '2006-01-01' Did you spot the change from the previous example? I added the not operator before the first set of conditions. Now, instead of looking for people with the first name of Steven or the last name of Young whose record was created after January 1st 2006, I am retrieving only rows where the first name is not Steven or the last name is not Young whose record was created after January 1st 2006. Table 4-3 shows the possible outcomes for this example. Table 4-3. Three-condition", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 52 + }, + { + "text": "name of Young whose record was created after January 1st 2006, I am retrieving only rows where the first name is not Steven or the last name is not Young whose record was created after January 1st 2006. Table 4-3 shows the possible outcomes for this example. Table 4-3. Three-condition evaluation using and, or, and not Intermediate result Final result WHERE NOT (true OR true) AND true False WHERE NOT (true OR false) AND true False WHERE NOT (false OR true) AND true False WHERE NOT (false OR false) AND true True WHERE NOT (true OR true) AND false False WHERE NOT (true OR false) AND false False WHERE NOT (false OR true) AND false False WHERE NOT (false OR false) AND false False While it is easy for the database server to handle, it is typically difficult for a person to evaluate a where clause that includes the not operator, which is why you won’t encounter it very often. In this case, you can rewrite the where clause to avoid using the not operator: WHERE first_name <> 'STEVEN' AND last_name <> 'YOUNG' AND create_date > '2006-01-01' While I’m sure that the server doesn’t have a preference, you probably have an easier time understanding this version of the where clause. Building a Condition Now that you have seen how the server evaluates multiple conditions, let’s take a step back and look at what comprises a single condition. A condition is made up of one or more expressions combined with one or more operators. An expression can be any of the following: A number A column in a table or view A string literal, such as 'Maple Street' A built-in function, such as concat('Learning', ' ', 'SQL') A subquery A list of expressions, such as ('Boston', 'New York', 'Chicago') The operators used within conditions include: Comparison operators, such as =, !=, <, >, <>, LIKE, IN, and BETWEEN Arithmetic operators, such as +, −, *, and / The following section demonstrates how you can combine these expressions and operators to manufacture the various types of conditions. Condition Types There are many different ways to filter out unwanted data. You can look for specific values, sets of values, or ranges of values to include or exclude, or you can use various pattern-searching techniques to look for partial matches when dealing with string data. The next four subsections explore each of these condition types in detail. Equality Conditions A large percentage of the filter conditions that you write or come across will be of the form 'column = expression' as in: title = 'RIVER OUTLAW' fed_id = '111-11-1111' amount = 375.25 film_id = (SELECT film_id FROM film WHERE title = 'RIVER OUTLAW') Conditions such as these are called equality conditions because they equate one expression to another. The first three examples equate a column to a literal (two strings and a number), and the fourth example equates a column to the value returned from a subquery. The following query uses two equality conditions;", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 53 + }, + { + "text": "such as these are called equality conditions because they equate one expression to another. The first three examples equate a column to a literal (two strings and a number), and the fourth example equates a column to the value returned from a subquery. The following query uses two equality conditions; one in the on clause (a join condition), and the other in the where clause (a filter condition): mysql> SELECT c.email -> FROM customer c -> INNER JOIN rental r -> ON c.customer_id = r.customer_id -> WHERE date(r.rental_date) = '2005-06-14'; +---------------------------------------+ | email | +---------------------------------------+ | CATHERINE.CAMPBELL@sakilacustomer.org | | JOYCE.EDWARDS@sakilacustomer.org | | AMBER.DIXON@sakilacustomer.org | | JEANETTE.GREENE@sakilacustomer.org | | MINNIE.ROMERO@sakilacustomer.org | | GWENDOLYN.MAY@sakilacustomer.org | | SONIA.GREGORY@sakilacustomer.org | | MIRIAM.MCKINNEY@sakilacustomer.org | | CHARLES.KOWALSKI@sakilacustomer.org | | DANIEL.CABRAL@sakilacustomer.org | | MATTHEW.MAHAN@sakilacustomer.org | | JEFFERY.PINSON@sakilacustomer.org | | HERMAN.DEVORE@sakilacustomer.org | | ELMER.NOE@sakilacustomer.org | | TERRANCE.ROUSH@sakilacustomer.org | | TERRENCE.GUNDERSON@sakilacustomer.org | +---------------------------------------+ 16 rows in set (0.03 sec) This query shows all email addresses of every customer who rented a film on June 14 2005. INEQUALITY CONDITIONS Another fairly common type of condition is the inequality condition, which asserts that two expressions are not equal. Here’s the previous query with the filter condition in the where clause changed to an inequality condition: mysql> SELECT c.email -> FROM customer c -> INNER JOIN rental r -> ON c.customer_id = r.customer_id -> WHERE date(r.rental_date) <> '2005-06-14'; +-----------------------------------+ | email | +-----------------------------------+ | MARY.SMITH@sakilacustomer.org | | MARY.SMITH@sakilacustomer.org | | MARY.SMITH@sakilacustomer.org | | MARY.SMITH@sakilacustomer.org | | MARY.SMITH@sakilacustomer.org | | MARY.SMITH@sakilacustomer.org | | MARY.SMITH@sakilacustomer.org | | MARY.SMITH@sakilacustomer.org | | MARY.SMITH@sakilacustomer.org | | MARY.SMITH@sakilacustomer.org | ... | AUSTIN.CINTRON@sakilacustomer.org | | AUSTIN.CINTRON@sakilacustomer.org | | AUSTIN.CINTRON@sakilacustomer.org | | AUSTIN.CINTRON@sakilacustomer.org | | AUSTIN.CINTRON@sakilacustomer.org | | AUSTIN.CINTRON@sakilacustomer.org | | AUSTIN.CINTRON@sakilacustomer.org | | AUSTIN.CINTRON@sakilacustomer.org | +-----------------------------------+ 16028 rows in set (0.03 sec) This query returns all email addresses for films rented on any other date than June 14 2005. When building inequality conditions, you may choose to use either the != or <> operator. DATA MODIFICATION USING EQUALITY CONDITIONS Equality/inequality conditions are commonly used when modifying data. For example, let’s say that the movie rental company has a policy of removing old account rows once per year. Your task is to remove rows from the rental table where the rental date was in 2004. Here’s one way to tackle it: DELETE FROM rental WHERE year(rental_date) = 2004; This statement includes a single equality condition; here’s an example which uses two inequality conditions to remove any rows where the rental date was not in 2005 or 2006: DELETE FROM rental WHERE year(rental_date) <> 2005 AND year(rental_date) <> 2006; NOTE When crafting examples of delete and update statements, I try to write each statement such that no rows are modified. That way, when you execute the statements, your data will remain unchanged, and your output from select statements will always match that shown in this book. Since MySQL sessions are in auto-commit mode by default (see <>), you would not be able to roll back (undo) any changes made to the", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 54 + }, + { + "text": "way, when you execute the statements, your data will remain unchanged, and your output from select statements will always match that shown in this book. Since MySQL sessions are in auto-commit mode by default (see <>), you would not be able to roll back (undo) any changes made to the example data if one of my statements modified the data. You may, of course, do whatever you want with the example data, including wiping it clean and rerunning the scripts to populate the tables, but I try to leave it intact. Range Conditions Along with checking that an expression is equal to (or not equal to) another expression, you can build conditions that check whether an expression falls within a certain range. This type of condition is common when working with numeric or temporal data. Consider the following query: mysql> SELECT customer_id, rental_date -> FROM rental -> WHERE rental_date < '2005-05-25'; +-------------+---------------------+ | customer_id | rental_date | +-------------+---------------------+ | 130 | 2005-05-24 22:53:30 | | 459 | 2005-05-24 22:54:33 | | 408 | 2005-05-24 23:03:39 | | 333 | 2005-05-24 23:04:41 | | 222 | 2005-05-24 23:05:21 | | 549 | 2005-05-24 23:08:07 | | 269 | 2005-05-24 23:11:53 | | 239 | 2005-05-24 23:31:46 | +-------------+---------------------+ 8 rows in set (0.00 sec) This query finds all film rentals prior to May 25 2005. Along with specifying an upper limit for the rental date, you may also want to specify a lower range as well: mysql> SELECT customer_id, rental_date -> FROM rental -> WHERE rental_date <= '2005-06-16' -> AND rental_date >= '2005-06-14'; +-------------+---------------------+ | customer_id | rental_date | +-------------+---------------------+ | 416 | 2005-06-14 22:53:33 | | 516 | 2005-06-14 22:55:13 | | 239 | 2005-06-14 23:00:34 | | 285 | 2005-06-14 23:07:08 | | 310 | 2005-06-14 23:09:38 | | 592 | 2005-06-14 23:12:46 | ... | 148 | 2005-06-15 23:20:26 | | 237 | 2005-06-15 23:36:37 | | 155 | 2005-06-15 23:55:27 | | 341 | 2005-06-15 23:57:20 | | 149 | 2005-06-15 23:58:53 | +-------------+---------------------+ 364 rows in set (0.00 sec) This version of the query retrieves all films rented on June 14 or 15 of 2005. THE BETWEEN OPERATOR When you have both an upper and lower limit for your range, you may choose to use a single condition that utilizes the between operator rather than using two separate conditions, as in: mysql> SELECT customer_id, rental_date -> FROM rental -> WHERE rental_date BETWEEN '2005-06-14' AND '2005-06-16'; +-------------+---------------------+ | customer_id | rental_date | +-------------+---------------------+ | 416 | 2005-06-14 22:53:33 | | 516 | 2005-06-14 22:55:13 | | 239 | 2005-06-14 23:00:34 | | 285 | 2005-06-14 23:07:08 | | 310 | 2005-06-14 23:09:38 | | 592 | 2005-06-14 23:12:46 | … | 148 | 2005-06-15 23:20:26 | | 237 | 2005-06-15 23:36:37 | | 155 | 2005-06-15 23:55:27 | | 341 | 2005-06-15 23:57:20 | | 149 | 2005-06-15 23:58:53 | +-------------+---------------------+ 364 rows in set (0.00 sec) When using the between operator, there are a couple of", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 55 + }, + { + "text": "23:12:46 | … | 148 | 2005-06-15 23:20:26 | | 237 | 2005-06-15 23:36:37 | | 155 | 2005-06-15 23:55:27 | | 341 | 2005-06-15 23:57:20 | | 149 | 2005-06-15 23:58:53 | +-------------+---------------------+ 364 rows in set (0.00 sec) When using the between operator, there are a couple of things to keep in mind. You should always specify the lower limit of the range first (after between) and the upper limit of the range second (after and). Here’s what happens if you mistakenly specify the upper limit first: mysql> SELECT customer_id, rental_date -> FROM rental -> WHERE rental_date BETWEEN '2005-06-16' AND '2005-06-14'; Empty set (0.00 sec) As you can see, no data is returned. This is because the server is, in effect, generating two conditions from your single condition using the <= and >= operators, as in: SELECT customer_id, rental_date -> FROM rental -> WHERE rental_date >= '2005-06-16' -> AND rental_date <= '2005-06-14' Empty set (0.00 sec) Since it is impossible to have a date that is both greater than June 16 2005 and less than June 14 2005, the query returns an empty set. This brings me to the second pitfall when using between, which is to remember that your upper and lower limits are inclusive, meaning that the values you provide are included in the range limits. In this case, I want to return any films rented on the 14th or 15th of June, so I specify 2005-06-14 as the lower end of the range and 2005-06-16as the upper end. Since I am not specifying the time component of the date, the time defaults to midnight, so the effective range is 2005-06-14 00:00:00 to 2005-06-16 00:00:00, which will include any rentals made on the 14th or 15th. Along with dates, you can also build conditions to specify ranges of numbers. Numeric ranges are fairly easy to grasp, as demonstrated by the following: mysql> SELECT customer_id, payment_date, amount -> FROM payment -> WHERE amount BETWEEN 10.0 AND 11.99; +-------------+---------------------+--------+ | customer_id | payment_date | amount | +-------------+---------------------+--------+ | 2 | 2005-07-30 13:47:43 | 10.99 | | 3 | 2005-07-27 20:23:12 | 10.99 | | 12 | 2005-08-01 06:50:26 | 10.99 | | 13 | 2005-07-29 22:37:41 | 11.99 | | 21 | 2005-06-21 01:04:35 | 10.99 | | 29 | 2005-07-09 21:55:19 | 10.99 | ... | 571 | 2005-06-20 08:15:27 | 10.99 | | 572 | 2005-06-17 04:05:12 | 10.99 | | 573 | 2005-07-31 12:14:19 | 10.99 | | 591 | 2005-07-07 20:45:51 | 11.99 | | 592 | 2005-07-06 22:58:31 | 11.99 | | 595 | 2005-07-31 11:51:46 | 10.99 | +-------------+---------------------+--------+ 114 rows in set (0.01 sec) All payments between $10 and $11.99 are returned. Again, make sure that you specify the lower amount first. STRING RANGES While ranges of dates and numbers are easy to understand, you can also build conditions that search for ranges of strings, which are a bit harder to visualize. Say, for example, you are searching for customers whose", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 56 + }, + { + "text": "returned. Again, make sure that you specify the lower amount first. STRING RANGES While ranges of dates and numbers are easy to understand, you can also build conditions that search for ranges of strings, which are a bit harder to visualize. Say, for example, you are searching for customers whose last name falls within a range. Here’s a query which returns customers whose last name falls between FA and FR: mysql> SELECT last_name, first_name -> FROM customer -> WHERE last_name BETWEEN 'FA' AND 'FR'; +------------+------------+ | last_name | first_name | +------------+------------+ | FARNSWORTH | JOHN | | FENNELL | ALEXANDER | | FERGUSON | BERTHA | | FERNANDEZ | MELINDA | | FIELDS | VICKI | | FISHER | CINDY | | FLEMING | MYRTLE | | FLETCHER | MAE | | FLORES | JULIA | | FORD | CRYSTAL | | FORMAN | MICHEAL | | FORSYTHE | ENRIQUE | | FORTIER | RAUL | | FORTNER | HOWARD | | FOSTER | PHYLLIS | | FOUST | JACK | | FOWLER | JO | | FOX | HOLLY | +------------+------------+ 18 rows in set (0.00 sec) While there are 5 customers whose last name starts with FR, they are not included in the results, since a name like FRANKLIN is outside of the range. However, we can pick up 4 of the 5 customers by extending the right-hand range to be FRB: mysql> SELECT last_name, first_name -> FROM customer -> WHERE last_name BETWEEN 'FA' AND 'FRB'; +------------+------------+ | last_name | first_name | +------------+------------+ | FARNSWORTH | JOHN | | FENNELL | ALEXANDER | | FERGUSON | BERTHA | | FERNANDEZ | MELINDA | | FIELDS | VICKI | | FISHER | CINDY | | FLEMING | MYRTLE | | FLETCHER | MAE | | FLORES | JULIA | | FORD | CRYSTAL | | FORMAN | MICHEAL | | FORSYTHE | ENRIQUE | | FORTIER | RAUL | | FORTNER | HOWARD | | FOSTER | PHYLLIS | | FOUST | JACK | | FOWLER | JO | | FOX | HOLLY | | FRALEY | JUAN | | FRANCISCO | JOEL | | FRANKLIN | BETH | | FRAZIER | GLENDA | +------------+------------+ 22 rows in set (0.00 sec) To work with string ranges, you need to know the order of the characters within your character set (the order in which the characters within a character set are sorted is called a collation). Membership Conditions In some cases, you will not be restricting an expression to a single value or range of values, but rather to a finite set of values. For example, you might want to locate all films which have a rating of either 'G' or 'PG': mysql> SELECT title, rating -> FROM film -> WHERE rating = 'G' OR rating = 'PG'; +---------------------------+--------+ | title | rating | +---------------------------+--------+ | ACADEMY DINOSAUR | PG | | ACE GOLDFINGER | G | | AFFAIR PREJUDICE | G | | AFRICAN EGG | G | |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 57 + }, + { + "text": "or 'PG': mysql> SELECT title, rating -> FROM film -> WHERE rating = 'G' OR rating = 'PG'; +---------------------------+--------+ | title | rating | +---------------------------+--------+ | ACADEMY DINOSAUR | PG | | ACE GOLDFINGER | G | | AFFAIR PREJUDICE | G | | AFRICAN EGG | G | | AGENT TRUMAN | PG | | ALAMO VIDEOTAPE | G | | ALASKA PHANTOM | PG | | ALI FOREVER | PG | | AMADEUS HOLY | PG | ... | WEDDING APOLLO | PG | | WEREWOLF LOLA | G | | WEST LION | G | | WIZARD COLDBLOODED | PG | | WON DARES | PG | | WONDERLAND CHRISTMAS | PG | | WORDS HUNTER | PG | | WORST BANGER | PG | | YOUNG LANGUAGE | G | +---------------------------+--------+ 372 rows in set (0.00 sec) While this where clause (two conditions or‘d together) wasn’t too tedious to generate, imagine if the set of expressions contained 10 or 20 members. For these situations, you can use the in operator instead: SELECT title, rating FROM film WHERE rating IN ('G','PG'); With the in operator, you can write a single condition no matter how many expressions are in the set. USING SUBQUERIES Along with writing your own set of expressions, such as ('G','PG'), you can use a subquery to generate a set for you on the fly. For example, if you can assume that any film whose title includes the string 'PET' would be safe for family viewing, you could execute a subquery against the film table to retrieve all ratings associated with these films, and then retrieve all films having any of these ratings: mysql> SELECT title, rating -> FROM film -> WHERE rating IN (SELECT rating FROM film WHERE title LIKE '%PET%'); +---------------------------+--------+ | title | rating | +---------------------------+--------+ | ACADEMY DINOSAUR | PG | | ACE GOLDFINGER | G | | AFFAIR PREJUDICE | G | | AFRICAN EGG | G | | AGENT TRUMAN | PG | | ALAMO VIDEOTAPE | G | | ALASKA PHANTOM | PG | | ALI FOREVER | PG | | AMADEUS HOLY | PG | ... | WEDDING APOLLO | PG | | WEREWOLF LOLA | G | | WEST LION | G | | WIZARD COLDBLOODED | PG | | WON DARES | PG | | WONDERLAND CHRISTMAS | PG | | WORDS HUNTER | PG | | WORST BANGER | PG | | YOUNG LANGUAGE | G | +---------------------------+--------+ 372 rows in set (0.00 sec) The subquery returns the set 'G' and 'PG', and the main query checks to see whether the value of the rating column can be found in the set returned by the subquery. USING NOT IN Sometimes you want to see whether a particular expression exists within a set of expressions, and sometimes you want to see whether the expression does not exist within the set. For these situations, you can use the not in operator: SELECT title, rating FROM film", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 58 + }, + { + "text": "the subquery. USING NOT IN Sometimes you want to see whether a particular expression exists within a set of expressions, and sometimes you want to see whether the expression does not exist within the set. For these situations, you can use the not in operator: SELECT title, rating FROM film WHERE rating NOT IN ('PG-13','R', 'NC-17'); This query finds all accounts that are not rated 'PG-13' ,'R', or 'NC- 17', which will return the same set of 372 rows as the previous queries. Matching Conditions So far, you have been introduced to conditions that identify an exact string, a range of strings, or a set of strings; the final condition type deals with partial string matches. You may, for example, want to find all customers whose last name begins with Q. You could use a built-in function to strip off the first letter of the last_name column, as in: mysql> SELECT last_name, first_name -> FROM customer -> WHERE left(last_name, 1) = 'Q'; +-------------+------------+ | last_name | first_name | +-------------+------------+ | QUALLS | STEPHEN | | QUINTANILLA | ROGER | | QUIGLEY | TROY | +-------------+------------+ 3 rows in set (0.00 sec) While the built-in function left() does the job, it doesn’t give you much flexibility. Instead, you can use wildcard characters to build search expressions, as demonstrated in the next section. USING WILDCARDS When searching for partial string matches, you might be interested in: Strings beginning/ending with a certain character Strings beginning/ending with a substring Strings containing a certain character anywhere within the string Strings containing a substring anywhere within the string Strings with a specific format, regardless of individual characters You can build search expressions to identify these and many other partial string matches by using the wildcard characters shown in Table 4-4. Table 4-4. Wildcard characters Wildcard character Matches _ Exactly one character % Any number of characters (including 0) The underscore character takes the place of a single character, while the percent sign can take the place of a variable number of characters. When building conditions that utilize search expressions, you use the like operator, as in: mysql> SELECT last_name, first_name -> FROM customer -> WHERE last_name LIKE '_A_T%S'; +-----------+------------+ | last_name | first_name | +-----------+------------+ | MATTHEWS | ERICA | | WALTERS | CASSANDRA | | WATTS | SHELLY | +-----------+------------+ 3 rows in set (0.00 sec) The search expression in the previous example specifies strings containing an A in the second position and a T in the fourth position, followed by any number of characters and ending in S. Table 4-5 shows some more search expressions and their interpretations. Table 4-5. Sample search expressions Search expression Interpretation F% Strings beginning with F %t Strings ending with t %bas% Strings containing the substring 'bas' _ _t_ Four-character strings with a t in the third position _ _ _-_ _-_ _ _ _ 11-character strings with dashes in the fourth and seventh positions The wildcard characters work fine for building simple search expressions; if your needs are", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 59 + }, + { + "text": "with t %bas% Strings containing the substring 'bas' _ _t_ Four-character strings with a t in the third position _ _ _-_ _-_ _ _ _ 11-character strings with dashes in the fourth and seventh positions The wildcard characters work fine for building simple search expressions; if your needs are a bit more sophisticated, however, you can use multiple search expressions, as demonstrated by the following: mysql> SELECT last_name, first_name -> FROM customer -> WHERE last_name LIKE 'Q%' OR last_name LIKE 'Y%'; +-------------+------------+ | last_name | first_name | +-------------+------------+ | QUALLS | STEPHEN | | QUIGLEY | TROY | | QUINTANILLA | ROGER | | YANEZ | LUIS | | YEE | MARVIN | | YOUNG | CYNTHIA | +-------------+------------+ 6 rows in set (0.00 sec) This query finds all customers whose last name begins with Q or Y. USING REGULAR EXPRESSIONS If you find that the wildcard characters don’t provide enough flexibility, you can use regular expressions to build search expressions. A regular expression is, in essence, a search expression on steroids. If you are new to SQL but have coded using programming languages such as Perl, then you might already be intimately familiar with regular expressions. If you have never used regular expressions, then you may want to consult Jeffrey E.F. Friedl’s Mastering Regular Expressions (O’Reilly), since it is far too large a topic to try to cover in this book. Here’s what the previous query (find all customers whose last name starts with Q or Y) would look like using the MySQL implementation of regular expressions: mysql> SELECT last_name, first_name -> FROM customer -> WHERE last_name REGEXP '^[QY]'; +-------------+------------+ | last_name | first_name | +-------------+------------+ | YOUNG | CYNTHIA | | QUALLS | STEPHEN | | QUINTANILLA | ROGER | | YANEZ | LUIS | | YEE | MARVIN | | QUIGLEY | TROY | +-------------+------------+ 6 rows in set (0.16 sec) The regexp operator takes a regular expression ('^[QY]' in this example) and applies it to the expression on the left-hand side of the condition (the column last_name). The query now contains a single condition using a regular expression rather than two conditions using wildcard characters. Oracle Database and Microsoft SQL Server also support regular expressions. With Oracle Database, you would use the regexp_like function instead of the regexp operator shown in the previous example, whereas SQL Server allows regular expressions to be used with the like operator. Null: That Four-Letter Word I put it off as long as I could, but it’s time to broach a topic that tends to be met with fear, uncertainty, and dread: the null value. Null is the absence of a value; before an employee is terminated, for example, her end_date column in the employee table should be null. There is no value that can be assigned to the end_date column that would make sense in this situation. Null is a bit slippery, however, as there are various flavors of null: Not applicable Such as the employee ID column for", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 60 + }, + { + "text": "end_date column in the employee table should be null. There is no value that can be assigned to the end_date column that would make sense in this situation. Null is a bit slippery, however, as there are various flavors of null: Not applicable Such as the employee ID column for a transaction that took place at an ATM machine Value not yet known Such as when the federal ID is not known at the time a customer row is created Value undefined Such as when an account is created for a product that has not yet been added to the database NOTE Some theorists argue that there should be a different expression to cover each of these (and more) situations, but most practitioners would agree that having multiple null values would be far too confusing. When working with null, you should remember: An expression can be null, but it can never equal null. Two nulls are never equal to each other. To test whether an expression is null, you need to use the is null operator, as demonstrated by the following: mysql> SELECT rental_id, customer_id -> FROM rental -> WHERE return_date IS NULL; +-----------+-------------+ | rental_id | customer_id | +-----------+-------------+ | 11496 | 155 | | 11541 | 335 | | 11563 | 83 | | 11577 | 219 | | 11593 | 99 | ... | 15867 | 505 | | 15875 | 41 | | 15894 | 168 | | 15966 | 374 | +-----------+-------------+ 183 rows in set (0.01 sec) This query finds all film rentals which were never returned. Here’s the same query using = null instead of is null: mysql> SELECT rental_id, customer_id -> FROM rental -> WHERE return_date = NULL; Empty set (0.01 sec) As you can see, the query parses and executes but does not return any rows. This is a common mistake made by inexperienced SQL programmers, and the database server will not alert you to your error, so be careful when constructing conditions that test for null. If you want to see whether a value has been assigned to a column, you can use the is not null operator, as in: mysql> SELECT rental_id, customer_id, return_date -> FROM rental -> WHERE return_date IS NOT NULL -> limit 20; +-----------+-------------+---------------------+ | rental_id | customer_id | return_date | +-----------+-------------+---------------------+ | 1 | 130 | 2005-05-26 22:04:30 | | 2 | 459 | 2005-05-28 19:40:33 | | 3 | 408 | 2005-06-01 22:12:39 | | 4 | 333 | 2005-06-03 01:43:41 | | 5 | 222 | 2005-06-02 04:33:21 | | 6 | 549 | 2005-05-27 01:32:07 | | 7 | 269 | 2005-05-29 20:34:53 | ... | 16043 | 526 | 2005-08-31 03:09:03 | | 16044 | 468 | 2005-08-25 04:08:39 | | 16045 | 14 | 2005-08-25 23:54:26 | | 16046 | 74 | 2005-08-27 18:02:47 | | 16047 | 114 | 2005-08-25 02:48:48 | | 16048 | 103 | 2005-08-31 21:33:07 | | 16049 | 393 | 2005-08-30 01:01:12 | +-----------+-------------+---------------------+", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 61 + }, + { + "text": "| | 16044 | 468 | 2005-08-25 04:08:39 | | 16045 | 14 | 2005-08-25 23:54:26 | | 16046 | 74 | 2005-08-27 18:02:47 | | 16047 | 114 | 2005-08-25 02:48:48 | | 16048 | 103 | 2005-08-31 21:33:07 | | 16049 | 393 | 2005-08-30 01:01:12 | +-----------+-------------+---------------------+ 15861 rows in set (0.02 sec) This version of the query returns all rentals which were returned, which is the majority of the rows in the table (15,861 out of 16,044). Before putting null aside for a while, it would be helpful to investigate one more potential pitfall. Suppose that you have been asked to find all rentals which were not returned during May through August of 2005. Your first instinct might be to do the following: mysql> SELECT rental_id, customer_id, return_date -> FROM rental -> WHERE return_date NOT BETWEEN '2005-05-01' AND '2005-09- 01'; +-----------+-------------+---------------------+ | rental_id | customer_id | return_date | +-----------+-------------+---------------------+ | 15365 | 327 | 2005-09-01 03:14:17 | | 15388 | 50 | 2005-09-01 03:50:23 | | 15392 | 410 | 2005-09-01 01:14:15 | | 15401 | 103 | 2005-09-01 03:44:10 | | 15415 | 204 | 2005-09-01 02:05:56 | ... | 15977 | 550 | 2005-09-01 22:12:10 | | 15982 | 370 | 2005-09-01 21:51:31 | | 16005 | 466 | 2005-09-02 02:35:22 | | 16020 | 311 | 2005-09-01 18:17:33 | | 16033 | 226 | 2005-09-01 02:36:15 | | 16037 | 45 | 2005-09-01 02:48:04 | | 16040 | 195 | 2005-09-02 02:19:33 | +-----------+-------------+---------------------+ 62 rows in set (0.01 sec) While it is true that these 62 rentals were returned outside of the May to August window, if you look carefully at the data, you will see that all of the rows returned have a non-Null return date. But what about the 183 rentals which were never returned? One might argue that these 183 rows were also not returned between May and August, so they should also be included in the result set. To answer the question correctly, therefore, you need to account for the possibility that some rows might contain a null in the return_date column: mysql> SELECT rental_id, customer_id, return_date -> FROM rental -> WHERE return_date IS NULL -> OR return_date NOT BETWEEN '2005-05-01' AND '2005-09- 01'; +-----------+-------------+---------------------+ | rental_id | customer_id | return_date | +-----------+-------------+---------------------+ | 11496 | 155 | NULL | | 11541 | 335 | NULL | | 11563 | 83 | NULL | | 11577 | 219 | NULL | | 11593 | 99 | NULL | ... | 15939 | 382 | 2005-09-01 17:25:21 | | 15942 | 210 | 2005-09-01 18:39:40 | | 15966 | 374 | NULL | | 15971 | 187 | 2005-09-02 01:28:33 | | 15973 | 343 | 2005-09-01 20:08:41 | | 15977 | 550 | 2005-09-01 22:12:10 | | 15982 | 370 | 2005-09-01 21:51:31 | | 16005 | 466 | 2005-09-02 02:35:22 | | 16020 | 311 | 2005-09-01 18:17:33 | | 16033 | 226 | 2005-09-01 02:36:15 |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 62 + }, + { + "text": "01:28:33 | | 15973 | 343 | 2005-09-01 20:08:41 | | 15977 | 550 | 2005-09-01 22:12:10 | | 15982 | 370 | 2005-09-01 21:51:31 | | 16005 | 466 | 2005-09-02 02:35:22 | | 16020 | 311 | 2005-09-01 18:17:33 | | 16033 | 226 | 2005-09-01 02:36:15 | | 16037 | 45 | 2005-09-01 02:48:04 | | 16040 | 195 | 2005-09-02 02:19:33 | +-----------+-------------+---------------------+ 245 rows in set (0.01 sec) The result set now includes the 62 rentals which were returned outside of the May to August window, along with the 183 rentals which were never returned, for a total of 245 rows. When working with a database that you are not familiar with, it is a good idea to find out which columns in a table allow nulls so that you can take appropriate measures with your filter conditions to keep data from slipping through the cracks. Test Your Knowledge The following exercises test your understanding of filter conditions. Please see Appendix C for solutions. The following subset of rows from the Payment table are used for the first two exercises: +------------+-------------+--------+--------------------+ | payment_id | customer_id | amount | date(payment_date) | +------------+-------------+--------+--------------------+ | 101 | 4 | 8.99 | 2005-08-18 | | 102 | 4 | 1.99 | 2005-08-19 | | 103 | 4 | 2.99 | 2005-08-20 | | 104 | 4 | 6.99 | 2005-08-20 | | 105 | 4 | 4.99 | 2005-08-21 | | 106 | 4 | 2.99 | 2005-08-22 | | 107 | 4 | 1.99 | 2005-08-23 | | 108 | 5 | 0.99 | 2005-05-29 | | 109 | 5 | 6.99 | 2005-05-31 | | 110 | 5 | 1.99 | 2005-05-31 | | 111 | 5 | 3.99 | 2005-06-15 | | 112 | 5 | 2.99 | 2005-06-16 | | 113 | 5 | 4.99 | 2005-06-17 | | 114 | 5 | 2.99 | 2005-06-19 | | 115 | 5 | 4.99 | 2005-06-20 | | 116 | 5 | 4.99 | 2005-07-06 | | 117 | 5 | 2.99 | 2005-07-08 | | 118 | 5 | 4.99 | 2005-07-09 | | 119 | 5 | 5.99 | 2005-07-09 | | 120 | 5 | 1.99 | 2005-07-09 | +------------+-------------+--------+--------------------+ Exercise 4-1 Which of the payment IDs would be returned by the following filter conditions? customer_id <> 5 AND (amount > 8 OR date(payment_date) = '2005- 08-23') Exercise 4-2 Which of the payment IDs would be returned by the following filter conditions? customer_id = 5 AND NOT (amount > 6 OR date(payment_date) = '2005-06-19') Exercise 4-3 Construct a query that retrieves all rows from the Payment table where the amount is either 1.98, 7.98, or 9.98. Exercise 4-4 Construct a query that finds all customers whose last name contains an A in the second position and a W anywhere after the A. Chapter 5. Querying Multiple Tables Back in Chapter 2, I demonstrated how related concepts are broken into separate pieces through a", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 63 + }, + { + "text": "7.98, or 9.98. Exercise 4-4 Construct a query that finds all customers whose last name contains an A in the second position and a W anywhere after the A. Chapter 5. Querying Multiple Tables Back in Chapter 2, I demonstrated how related concepts are broken into separate pieces through a process known as normalization. The end result of this exercise was two tables: person and favorite_food. If, however, you want to generate a single report showing a person’s name, address, and favorite foods, you will need a mechanism to bring the data from these two tables back together again; this mechanism is known as a join, and this chapter concentrates on the simplest and most common join, the inner join. Chapter 10 demonstrates all of the different join types. What Is a Join? Queries against a single table are certainly not rare, but you will find that most of your queries will require two, three, or even more tables. To illustrate, let’s look at the definitions for the customer and address tables and then define a query that retrieves data from both tables: mysql> desc customer; +-------------+----------------------+------+-----+------------- ------+ | Field | Type | Null | Key | Default | +-------------+----------------------+------+-----+------------- ------+ | customer_id | smallint(5) unsigned | NO | PRI | NULL | | store_id | tinyint(3) unsigned | NO | MUL | NULL | | first_name | varchar(45) | NO | | NULL | | last_name | varchar(45) | NO | MUL | NULL | | email | varchar(50) | YES | | NULL | | address_id | smallint(5) unsigned | NO | MUL | NULL | | active | tinyint(1) | NO | | 1 | | create_date | datetime | NO | | NULL | | last_update | timestamp | YES | | CURRENT_TIMESTAMP | +-------------+----------------------+------+-----+------------- ------+ mysql> desc address; +-------------+----------------------+------+-----+------------- ------+ | Field | Type | Null | Key | Default | +-------------+----------------------+------+-----+------------- ------+ | address_id | smallint(5) unsigned | NO | PRI | NULL | | address | varchar(50) | NO | | NULL | | address2 | varchar(50) | YES | | NULL | | district | varchar(20) | NO | | NULL | | city_id | smallint(5) unsigned | NO | MUL | NULL | | postal_code | varchar(10) | YES | | NULL | | phone | varchar(20) | NO | | NULL | | location | geometry | NO | MUL | NULL | | last_update | timestamp | NO | | CURRENT_TIMESTAMP | +-------------+----------------------+------+-----+------------- ------+ Let’s say you want to retrieve the first and last names of each customer, along with their street address. Your query will therefore need to retrieve the customer.first_name, customer.last_name, and address.address columns. But how can you retrieve data from both tables in the same query? The answer lies in the customer.address_id column, which holds the ID of the customer’s record in the address table (in more formal terms, the customer.address_id column is the foreign key to the address table). The query, which you will see", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 64 + }, + { + "text": "you retrieve data from both tables in the same query? The answer lies in the customer.address_id column, which holds the ID of the customer’s record in the address table (in more formal terms, the customer.address_id column is the foreign key to the address table). The query, which you will see shortly, instructs the server to use the customer.address_id column as the transportation between the customer and address tables, thereby allowing columns from both tables to be included in the query’s result set. This type of operation is known as a join. NOTE A foreign key constraint can optionally be created to verify that the values in one table exist in another table. For the previous example, a foreign key constraint could be created on the customer table to ensure that any values inserted into the customer.address_id column can be found in the address.address_id column. Please note that it is not necessary to have a foreign key constraint in place in order to join two tables. Cartesian Product The easiest way to start is to put the customer and address tables into the from clause of a query and see what happens. Here’s a query that retrieves the customer’s first and last names along with the street address, with a from clause naming both tables separated by the join keyword: mysql> SELECT c.first_name, c.last_name, a.address -> FROM customer c JOIN address; +------------+-----------+----------------------+ | first_name | last_name | address | +------------+-----------+----------------------+ | MARY | SMITH | 47 MySakila Drive | | PATRICIA | JOHNSON | 47 MySakila Drive | | LINDA | WILLIAMS | 47 MySakila Drive | | BARBARA | JONES | 47 MySakila Drive | | ELIZABETH | BROWN | 47 MySakila Drive | | JENNIFER | DAVIS | 47 MySakila Drive | | MARIA | MILLER | 47 MySakila Drive | | SUSAN | WILSON | 47 MySakila Drive | ... | SETH | HANNON | 1325 Fukuyama Street | | KENT | ARSENAULT | 1325 Fukuyama Street | | TERRANCE | ROUSH | 1325 Fukuyama Street | | RENE | MCALISTER | 1325 Fukuyama Street | | EDUARDO | HIATT | 1325 Fukuyama Street | | TERRENCE | GUNDERSON | 1325 Fukuyama Street | | ENRIQUE | FORSYTHE | 1325 Fukuyama Street | | FREDDIE | DUGGAN | 1325 Fukuyama Street | | WADE | DELVALLE | 1325 Fukuyama Street | | AUSTIN | CINTRON | 1325 Fukuyama Street | +------------+-----------+----------------------+ 361197 rows in set (0.03 sec) Hmmm…there are only 599 customers and 603 rows in the address table, so how did the result set end up with 361,197 rows? Looking more closely, you can see that many of the customers seem to have the same street address. Because the query didn’t specify how the two tables should be joined, the database server generated the Cartesian product, which is every permutation of the two tables (599 customers x 603 addresses = 361,197 permutations). This type of join is known as a cross join, and it is rarely used", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 65 + }, + { + "text": "Because the query didn’t specify how the two tables should be joined, the database server generated the Cartesian product, which is every permutation of the two tables (599 customers x 603 addresses = 361,197 permutations). This type of join is known as a cross join, and it is rarely used (on purpose, at least). Cross joins are one of the join types that we study in Chapter 10. Inner Joins To modify the previous query so that only a single row is returned for each customer, you need to describe how the two tables are related. Earlier, I showed that the customer.address_id column serves as the link between the two tables, so this information needs to be added to the on subclause of the from clause: mysql> SELECT c.first_name, c.last_name, a.address -> FROM customer c JOIN address a -> ON c.address_id = a.address_id; +-------------+--------------+---------------------------------- ------+ | first_name | last_name | address | +-------------+--------------+---------------------------------- ------+ | MARY | SMITH | 1913 Hanoi Way | | PATRICIA | JOHNSON | 1121 Loja Avenue | | LINDA | WILLIAMS | 692 Joliet Street | | BARBARA | JONES | 1566 Inegl Manor | | ELIZABETH | BROWN | 53 Idfu Parkway | | JENNIFER | DAVIS | 1795 Santiago de Compostela Way | | MARIA | MILLER | 900 Santiago de Compostela Parkway | | SUSAN | WILSON | 478 Joliet Way | | MARGARET | MOORE | 613 Korolev Drive | ... | TERRANCE | ROUSH | 42 Fontana Avenue | | RENE | MCALISTER | 1895 Zhezqazghan Drive | | EDUARDO | HIATT | 1837 Kaduna Parkway | | TERRENCE | GUNDERSON | 844 Bucuresti Place | | ENRIQUE | FORSYTHE | 1101 Bucuresti Boulevard | | FREDDIE | DUGGAN | 1103 Quilmes Boulevard | | WADE | DELVALLE | 1331 Usak Boulevard | | AUSTIN | CINTRON | 1325 Fukuyama Street | +-------------+--------------+---------------------------------- ------+ 599 rows in set (0.00 sec) Instead of 361,197 rows, you now have the expected 599 rows due to the addition of the on subclause, which instructs the server to join the customer and address tables by using the address_id column to traverse from one table to the other. For example, Mary Smith’s row in the customer table contains a value of 5 in the address_id column (not shown in the example). The server uses this value to look up the row in the address table having a value of 5 in its address_id column and then retrieves the value '1913 Hanoi Way' from the address column in that row. If a value exists for the address_id column in one table but not the other, then the join fails for the rows containing that value and those rows are excluded from the result set. This type of join is known as an inner join, and it is the most commonly used type of join. To clarify, if a row in the customer table has the value 999 in the address_id column, and there’s no row in", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 66 + }, + { + "text": "those rows are excluded from the result set. This type of join is known as an inner join, and it is the most commonly used type of join. To clarify, if a row in the customer table has the value 999 in the address_id column, and there’s no row in the address table with a value of 999 in the address_id column, then that customer row would not be included in the result set. If you want to include all rows from one table or the other regardless of whether a match exists, you need to specify an outer join, but we cover this later in the book. In the previous example, I did not specify in the from clause which type of join to use. However, when you wish to join two tables using an inner join, you should explicitly specify this in your from clause; here’s the same example, with the addition of the join type (note the keyword INNER): SELECT c.first_name, c.last_name, a.address FROM customer c INNER JOIN address a ON c.address_id = a.address_id; If you do not specify the type of join, then the server will do an inner join by default. As you will see later in the book, however, there are several types of joins, so you should get in the habit of specifying the exact type of join that you require, especially for the benefit of any other people who might use/maintain your queries in the future. If the names of the columns used to join the two tables are identical, which is true in the previous query, you can use the using subclause instead of the on subclause, as in: SELECT c.first_name, c.last_name, a.address FROM customer c INNER JOIN address a USING (address_id); Since using is a shorthand notation that you can use in only a specific situation, I prefer always to use the on subclause to avoid confusion. The ANSI Join Syntax The notation used throughout this book for joining tables was introduced in the SQL92 version of the ANSI SQL standard. All the major databases (Oracle Database, Microsoft SQL Server, MySQL, IBM DB2 Universal Database, and Sybase Adaptive Server) have adopted the SQL92 join syntax. Because most of these servers have been around since before the release of the SQL92 specification, they all include an older join syntax as well. For example, all these servers would understand the following variation of the previous query: mysql> SELECT c.first_name, c.last_name, a.address -> FROM customer c, address a -> WHERE c.address_id = a.address_id; +------------+------------+------------------------------------+ | first_name | last_name | address | +------------+------------+------------------------------------+ | MARY | SMITH | 1913 Hanoi Way | | PATRICIA | JOHNSON | 1121 Loja Avenue | | LINDA | WILLIAMS | 692 Joliet Street | | BARBARA | JONES | 1566 Inegl Manor | | ELIZABETH | BROWN | 53 Idfu Parkway | | JENNIFER | DAVIS | 1795 Santiago de Compostela Way | | MARIA | MILLER | 900 Santiago de Compostela Parkway | | SUSAN | WILSON", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 67 + }, + { + "text": "WILLIAMS | 692 Joliet Street | | BARBARA | JONES | 1566 Inegl Manor | | ELIZABETH | BROWN | 53 Idfu Parkway | | JENNIFER | DAVIS | 1795 Santiago de Compostela Way | | MARIA | MILLER | 900 Santiago de Compostela Parkway | | SUSAN | WILSON | 478 Joliet Way | | MARGARET | MOORE | 613 Korolev Drive | ... | TERRANCE | ROUSH | 42 Fontana Avenue | | RENE | MCALISTER | 1895 Zhezqazghan Drive | | EDUARDO | HIATT | 1837 Kaduna Parkway | | TERRENCE | GUNDERSON | 844 Bucuresti Place | | ENRIQUE | FORSYTHE | 1101 Bucuresti Boulevard | | FREDDIE | DUGGAN | 1103 Quilmes Boulevard | | WADE | DELVALLE | 1331 Usak Boulevard | | AUSTIN | CINTRON | 1325 Fukuyama Street | +------------+------------+------------------------------------+ 599 rows in set (0.00 sec) This older method of specifying joins does not include the on subclause; instead, tables are named in the from clause separated by commas, and join conditions are included in the where clause. While you may decide to ignore the SQL92 syntax in favor of the older join syntax, the ANSI join syntax has the following advantages: Join conditions and filter conditions are separated into two different clauses (the on subclause and the where clause, respectively), making a query easier to understand. The join conditions for each pair of tables are contained in their own on clause, making it less likely that part of a join will be mistakenly omitted. Queries that use the SQL92 join syntax are portable across database servers, whereas the older syntax is slightly different across the different servers. The benefits of the SQL92 join syntax are easier to identify for complex queries that include both join and filter conditions. Consider the following query, which returns only those customers whose postal code is 52137: mysql> SELECT c.first_name, c.last_name, a.address -> FROM customer c, address a -> WHERE c.address_id = a.address_id -> AND a.postal_code = 52137; +------------+-----------+------------------------+ | first_name | last_name | address | +------------+-----------+------------------------+ | JAMES | GANNON | 1635 Kuwana Boulevard | | FREDDIE | DUGGAN | 1103 Quilmes Boulevard | +------------+-----------+------------------------+ 2 rows in set (0.01 sec) At first glance, it is not so easy to determine which conditions in the where clause are join conditions and which are filter conditions. It is also not readily apparent which type of join is being employed (to identify the type of join, you would need to look closely at the join conditions in the where clause to see whether any special characters are employed), nor is it easy to determine whether any join conditions have been mistakenly left out. Here’s the same query using the SQL92 join syntax: mysql> SELECT c.first_name, c.last_name, a.address -> FROM customer c INNER JOIN address a -> ON c.address_id = a.address_id -> WHERE a.postal_code = 52137; +------------+-----------+------------------------+ | first_name | last_name | address | +------------+-----------+------------------------+ | JAMES | GANNON | 1635 Kuwana Boulevard | | FREDDIE | DUGGAN", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 68 + }, + { + "text": "using the SQL92 join syntax: mysql> SELECT c.first_name, c.last_name, a.address -> FROM customer c INNER JOIN address a -> ON c.address_id = a.address_id -> WHERE a.postal_code = 52137; +------------+-----------+------------------------+ | first_name | last_name | address | +------------+-----------+------------------------+ | JAMES | GANNON | 1635 Kuwana Boulevard | | FREDDIE | DUGGAN | 1103 Quilmes Boulevard | +------------+-----------+------------------------+ 2 rows in set (0.00 sec) With this version, it is clear which condition is used for the join, and which condition is used for filtering. Hopefully, you will agree that the version using SQL92 join syntax is easier to understand. Joining Three or More Tables Joining three tables is similar to joining two tables, but with one slight wrinkle. With a two-table join, there are two tables and one join type in the from clause, and a single on subclause to define how the tables are joined. With a three-table join, there are three tables and two join types in the from clause, and two on subclauses. To illustrate, let’s change the previous query to return the customer’s city rather than their street address. The city name, however, is not stored in the address table, but is accessed via a foreign key to the city table. Here are the table definitions: mysql> desc address; +-------------+----------------------+------+-----+------------- ------+ | Field | Type | Null | Key | Default | +-------------+----------------------+------+-----+------------- ------+ | address_id | smallint(5) unsigned | NO | PRI | NULL | | address | varchar(50) | NO | | NULL | | address2 | varchar(50) | YES | | NULL | | district | varchar(20) | NO | | NULL | | city_id | smallint(5) unsigned | NO | MUL | NULL | | postal_code | varchar(10) | YES | | NULL | | phone | varchar(20) | NO | | NULL | | location | geometry | NO | MUL | NULL | | last_update | timestamp | NO | | CURRENT_TIMESTAMP | +-------------+----------------------+------+-----+------------- ------+ mysql> desc city; +-------------+----------------------+------+-----+------------- ------+ | Field | Type | Null | Key | Default | +-------------+----------------------+------+-----+------------- ------+ | city_id | smallint(5) unsigned | NO | PRI | NULL | | city | varchar(50) | NO | | NULL | | country_id | smallint(5) unsigned | NO | MUL | NULL | | last_update | timestamp | NO | | CURRENT_TIMESTAMP | +-------------+----------------------+------+-----+------------- ------+ In order to show each customer’s city, you will need to traverse from the customer table to the address table using the address_id column, and then from the address table to the city table using the city_id column. Here’s what the query would look like: mysql> SELECT c.first_name, c.last_name, ct.city -> FROM customer c -> INNER JOIN address a -> ON c.address_id = a.address_id -> INNER JOIN city ct -> ON a.city_id = ct.city_id; +-------------+--------------+----------------------------+ | first_name | last_name | city | +-------------+--------------+----------------------------+ | JULIE | SANCHEZ | A Corua (La Corua) | | PEGGY | MYERS | Abha | | TOM | MILNER | Abu Dhabi | | GLEN | TALBERT |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 69 + }, + { + "text": "a.address_id -> INNER JOIN city ct -> ON a.city_id = ct.city_id; +-------------+--------------+----------------------------+ | first_name | last_name | city | +-------------+--------------+----------------------------+ | JULIE | SANCHEZ | A Corua (La Corua) | | PEGGY | MYERS | Abha | | TOM | MILNER | Abu Dhabi | | GLEN | TALBERT | Acua | | LARRY | THRASHER | Adana | | SEAN | DOUGLASS | Addis Abeba | ... | MICHELE | GRANT | Yuncheng | | GARY | COY | Yuzhou | | PHYLLIS | FOSTER | Zalantun | | CHARLENE | ALVAREZ | Zanzibar | | FRANKLIN | TROUTMAN | Zaoyang | | FLOYD | GANDY | Zapopan | | CONSTANCE | REID | Zaria | | JACK | FOUST | Zeleznogorsk | | BYRON | BOX | Zhezqazghan | | GUY | BROWNLEE | Zhoushan | | RONNIE | RICKETTS | Ziguinchor | +-------------+--------------+----------------------------+ 599 rows in set (0.03 sec) For this query, there are three tables, two join types, and two on subclauses in the from clause, so things have gotten quite a bit busier. At first glance, it might seem like the order in which the tables appear in the from clause is important, but if you switch the table order you will get the exact same results. All three of these variations return the same results: SELECT c.first_name, c.last_name, ct.city FROM customer c INNER JOIN address a ON c.address_id = a.address_id INNER JOIN city ct ON a.city_id = ct.city_id; SELECT c.first_name, c.last_name, ct.city FROM city ct INNER JOIN address a ON a.city_id = ct.city_id INNER JOIN customer c ON c.address_id = a.address_id; SELECT c.first_name, c.last_name, ct.city FROM address a INNER JOIN city ct ON a.city_id = ct.city_id INNER JOIN customer c ON c.address_id = a.address_id; The only difference you may see would be the order in which the rows are returned, since there is no order by clause to specify how the results should be ordered. DOES JOIN ORDER MATTER? If you are confused about why all three versions of the customer/address/city query yield the same results, keep in mind that SQL is a nonprocedural language, meaning that you describe what you want to retrieve and which database objects need to be involved, but it is up to the database server to determine how best to execute your query. Using statistics gathered from your database objects, the server must pick one of three tables as a starting point (the chosen table is thereafter known as the driving table), and then decide in which order to join the remaining tables. Therefore, the order in which tables appear in your from clause is not significant. If, however, you believe that the tables in your query should always be joined in a particular order, you can place the tables in the desired order and then specify the keyword STRAIGHT_JOIN in MySQL, request the FORCE ORDER option in SQL Server, or use either the ORDERED or the LEADING optimizer hint in Oracle Database. For example, to tell the MySQL", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 70 + }, + { + "text": "be joined in a particular order, you can place the tables in the desired order and then specify the keyword STRAIGHT_JOIN in MySQL, request the FORCE ORDER option in SQL Server, or use either the ORDERED or the LEADING optimizer hint in Oracle Database. For example, to tell the MySQL server to use the city table as the driving table and to then join the address and customer tables, you could do the following: SELECT STRAIGHT_JOIN c.first_name, c.last_name, ct.city FROM city ct INNER JOIN address a ON a.city_id = ct.city_id INNER JOIN customer c ON c.address_id = a.address_id Using Subqueries As Tables You have already seen several examples of queries that include multiple tables, but there is one variation worth mentioning: what to do if some of the data sets are generated by subqueries. Subqueries are the focus of Chapter 9, but I already introduced the concept of a subquery in the from clause in the previous chapter. Here’s a query which joins the customer table to a subquery against the address and city tables: mysql> SELECT c.first_name, c.last_name, addr.address, addr.city -> FROM customer c -> INNER JOIN -> (SELECT a.address_id, a.address, ct.city -> FROM address a -> INNER JOIN city ct -> ON a.city_id = ct.city_id -> WHERE a.district = 'California' -> ) addr -> ON c.address_id = addr.address_id; +------------+-----------+------------------------+------------- ---+ | first_name | last_name | address | city | +------------+-----------+------------------------+------------- ---+ | PATRICIA | JOHNSON | 1121 Loja Avenue | San Bernardino | | BETTY | WHITE | 770 Bydgoszcz Avenue | Citrus Heights | | ALICE | STEWART | 1135 Izumisano Parkway | Fontana | | ROSA | REYNOLDS | 793 Cam Ranh Avenue | Lancaster | | RENEE | LANE | 533 al-Ayn Boulevard | Compton | | KRISTIN | JOHNSTON | 226 Brest Manor | Sunnyvale | | CASSANDRA | WALTERS | 920 Kumbakonam Loop | Salinas | | JACOB | LANCE | 1866 al-Qatif Avenue | El Monte | | RENE | MCALISTER | 1895 Zhezqazghan Drive | Garden Grove | +------------+-----------+------------------------+------------- ---+ 9 rows in set (0.00 sec) The subquery, which starts on line 4 and is given the alias addr, finds all addresses which are in California. The outer query joins the subquery results to the customer table to return the first name, last name, street address, and city of all customers who live in California. While this query could have been written without the use of a subquery by simply joining the three tables, it can sometimes be advantageous from a performance and/or readability aspect to use one or more subqueries. One way to visualize what is going on is to run the subquery by itself and look at the results. Here are the results of the subquery from the prior example: mysql> SELECT a.address_id, a.address, ct.city -> FROM address a -> INNER JOIN city ct -> ON a.city_id = ct.city_id -> WHERE a.district = 'California'; +------------+------------------------+----------------+ | address_id | address | city | +------------+------------------------+----------------+ | 6 | 1121 Loja Avenue", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 71 + }, + { + "text": "Here are the results of the subquery from the prior example: mysql> SELECT a.address_id, a.address, ct.city -> FROM address a -> INNER JOIN city ct -> ON a.city_id = ct.city_id -> WHERE a.district = 'California'; +------------+------------------------+----------------+ | address_id | address | city | +------------+------------------------+----------------+ | 6 | 1121 Loja Avenue | San Bernardino | | 18 | 770 Bydgoszcz Avenue | Citrus Heights | | 55 | 1135 Izumisano Parkway | Fontana | | 116 | 793 Cam Ranh Avenue | Lancaster | | 186 | 533 al-Ayn Boulevard | Compton | | 218 | 226 Brest Manor | Sunnyvale | | 274 | 920 Kumbakonam Loop | Salinas | | 425 | 1866 al-Qatif Avenue | El Monte | | 599 | 1895 Zhezqazghan Drive | Garden Grove | +------------+------------------------+----------------+ 9 rows in set (0.00 sec) This result set consists of all 9 California addresses. When joined to the customer table via the address_id column, your result set will contain information about the customers assigned to these addresses. Using the Same Table Twice If you are joining multiple tables, you might find that you need to join the same table more than once. In the sample database, for example, actors are related to the films in which they appeared via the film_actor table. If you want to find all of the films in which two specific actors appear, you could write a query such as this one, which joins the film table to the film_actor table to the actor table: mysql> SELECT f.title -> FROM film f -> INNER JOIN film_actor fa -> ON f.film_id = fa.film_id -> INNER JOIN actor a -> ON fa.actor_id = a.actor_id -> WHERE ((a.first_name = 'CATE' AND a.last_name = 'MCQUEEN') -> OR (a.first_name = 'CUBA' AND a.last_name = 'BIRCH')); +----------------------+ | title | +----------------------+ | ATLANTIS CAUSE | | BLOOD ARGONAUTS | | COMMANDMENTS EXPRESS | | DYNAMITE TARZAN | | EDGE KISSING | ... | TOWERS HURRICANE | | TROJAN TOMORROW | | VIRGIN DAISY | | VOLCANO TEXAS | | WATERSHIP FRONTIER | +----------------------+ 54 rows in set (0.00 sec) This query returns all movies in which either Cate McQueen or Cuba Birch appeared. However, let’s say that you want to retrieve only those films in which both of these actors appeared. To accomplish this, you will need to find all rows in the film table which have two rows in the film_actor table, one of which is associated with Cate McQueen, and the other one associated with Cuba Birch. Therefore, you will need to include the film_actor and actor tables twice, each with a different alias so that the server knows which one you are referring to in the various clauses: mysql> SELECT f.title -> FROM film f -> INNER JOIN film_actor fa1 -> ON f.film_id = fa1.film_id -> INNER JOIN actor a1 -> ON fa1.actor_id = a1.actor_id -> INNER JOIN film_actor fa2 -> ON f.film_id = fa2.film_id -> INNER JOIN actor a2 -> ON fa2.actor_id = a2.actor_id -> WHERE", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 72 + }, + { + "text": "clauses: mysql> SELECT f.title -> FROM film f -> INNER JOIN film_actor fa1 -> ON f.film_id = fa1.film_id -> INNER JOIN actor a1 -> ON fa1.actor_id = a1.actor_id -> INNER JOIN film_actor fa2 -> ON f.film_id = fa2.film_id -> INNER JOIN actor a2 -> ON fa2.actor_id = a2.actor_id -> WHERE (a1.first_name = 'CATE' AND a1.last_name = 'MCQUEEN') -> AND (a2.first_name = 'CUBA' AND a2.last_name = 'BIRCH'); +------------------+ | title | +------------------+ | BLOOD ARGONAUTS | | TOWERS HURRICANE | +------------------+ 2 rows in set (0.00 sec) Between them, the two actors appeared in 52 different films, but there are only 2 films in which both actors appeared. This is one example of a query that requires the use of table aliases, since the same tables are used multiple times. Self-Joins Not only can you include the same table more than once in the same query, but you can actually join a table to itself. This might seem like a strange thing to do at first, but there are valid reasons for doing so. Some tables include a self-referencing foreign key, which means that it includes a column which points to the primary key within the same table. While the sample database doesn’t include such a relationship, let’s imagine that the film table includes the column prequel_film_id, which points to the film’s parent (e.g. the film “Fiddler Lost II” would use this column to point to the parent film “Fiddler Lost”). Here’s what the table would look like if we were to add this additional column: mysql> desc film; +----------------------+---------------------------------------- --+------+-----+-------------------+ | Field | Type | Null | Key | Default | +----------------------+---------------------------------------- --+------+-----+-------------------+ | film_id | smallint(5) unsigned | NO | PRI | NULL | | title | varchar(255) | NO | MUL | NULL | | description | text | YES | | NULL | | release_year | year(4) | YES | | NULL | | language_id | tinyint(3) unsigned | NO | MUL | NULL | | original_language_id | tinyint(3) unsigned | YES | MUL | NULL | | rental_duration | tinyint(3) unsigned | NO | | 3 | | rental_rate | decimal(4,2) | NO | | 4.99 | | length | smallint(5) unsigned | YES | | NULL | | replacement_cost | decimal(5,2) | NO | | 19.99 | | rating | enum('G','PG','PG-13','R','NC- 17') | YES | | G | | special_features | set('Trailers',...,'Behind the Scenes') | YES | | NULL | | last_update | timestamp | NO | | CURRENT_TIMESTAMP | | prequel_film_id | smallint(5) unsigned | YES | MUL | NULL | +----------------------+---------------------------------------- --+------+-----+-------------------+ Using a self-join, you can write a query that lists every film which has a prequel, along with the prequel’s title: mysql> SELECT f.title, f_prnt.title prequel -> FROM film f -> INNER JOIN film f_prnt -> ON f_prnt.film_id = f.prequel_film_id -> WHERE f.prequel_film_id IS NOT NULL; +-----------------+--------------+ | title | prequel | +-----------------+--------------+ | FIDDLER LOST II | FIDDLER LOST | +-----------------+--------------+ 1 row in set (0.00 sec) This query joins", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 73 + }, + { + "text": "mysql> SELECT f.title, f_prnt.title prequel -> FROM film f -> INNER JOIN film f_prnt -> ON f_prnt.film_id = f.prequel_film_id -> WHERE f.prequel_film_id IS NOT NULL; +-----------------+--------------+ | title | prequel | +-----------------+--------------+ | FIDDLER LOST II | FIDDLER LOST | +-----------------+--------------+ 1 row in set (0.00 sec) This query joins the film table to itself using the prequel_film_id foreign key, and the table aliases f and f_prnt are assigned in order to make it clear which table is used for which purpose. Test Your Knowledge The following exercises are designed to test your understanding of inner joins. Please see Appendix C for the solutions to these exercises. Exercise 5-1 Fill in the blanks (denoted by <#>) for the following query to obtain the results that follow: mysql> SELECT c.first_name, c.last_name, a.address, ct.city -> FROM customer c -> INNER JOIN address <1> -> ON c.address_id = a.address_id -> INNER JOIN city ct -> ON a.city_id = <2> -> WHERE a.district = 'California'; +------------+-----------+------------------------+------------- ---+ | first_name | last_name | address | city | +------------+-----------+------------------------+------------- ---+ | PATRICIA | JOHNSON | 1121 Loja Avenue | San Bernardino | | BETTY | WHITE | 770 Bydgoszcz Avenue | Citrus Heights | | ALICE | STEWART | 1135 Izumisano Parkway | Fontana | | ROSA | REYNOLDS | 793 Cam Ranh Avenue | Lancaster | | RENEE | LANE | 533 al-Ayn Boulevard | Compton | | KRISTIN | JOHNSTON | 226 Brest Manor | Sunnyvale | | CASSANDRA | WALTERS | 920 Kumbakonam Loop | Salinas | | JACOB | LANCE | 1866 al-Qatif Avenue | El Monte | | RENE | MCALISTER | 1895 Zhezqazghan Drive | Garden Grove | +------------+-----------+------------------------+------------- ---+ 9 rows in set (0.00 sec) Exercise 5-2 Write a query that returns the title of every film in which an actor with the first name JOHN appeared. Exercise 5-3 Construct a query that finds returns all addresses which are in the same city. You will need to join the address table to itself, and each row should include 2 different addresses. Chapter 6. Working with Sets Although you can interact with the data in a database one row at a time, relational databases are really all about sets. This chapter explores how you can combine multiple result sets using various set operators. After a quick overview of set theory, I’ll demonstrate how to use the set operators union , intersect , and except to blend multiple data sets together. Set Theory Primer In many parts of the world, basic set theory is included in elementary-level math curriculums. Perhaps you recall looking at something like what is shown in Figure 6-1. Figure 6-1. The union operation The shaded area in Figure 6-1 represents the union of sets A and B, which is the combination of the two sets (with any overlapping regions included only once). Is this starting to look familiar? If so, then you’ll finally get a chance to put that knowledge to use; if not, don’t worry, because", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 74 + }, + { + "text": "Figure 6-1 represents the union of sets A and B, which is the combination of the two sets (with any overlapping regions included only once). Is this starting to look familiar? If so, then you’ll finally get a chance to put that knowledge to use; if not, don’t worry, because it’s easy to visualize using a couple of diagrams. Using circles to represent two data sets (A and B), imagine a subset of data that is common to both sets; this common data is represented by the overlapping area shown in Figure 6-1. Since set theory is rather uninteresting without an overlap between data sets, I use the same diagram to illustrate each set operation. There is another set operation that is concerned only with the overlap between two data sets; this operation is known as the intersection and is demonstrated in Figure 6-2. Figure 6-2. The intersection operation The data set generated by the intersection of sets A and B is just the area of overlap between the two sets. If the two sets have no overlap, then the intersection operation yields the empty set. The third and final set operation, which is demonstrated in Figure 6-3, is known as the except operation. Figure 6-3. The except operation Figure 6-3 shows the results of A except B, which is the whole of set A minus any overlap with set B. If the two sets have no overlap, then the operation A except B yields the whole of set A. Using these three operations, or by combining different operations together, you can generate whatever results you need. For example, imagine that you want to build a set demonstrated by Figure 6-4. Figure 6-4. Mystery data set The data set you are looking for includes all of sets A and B without the overlapping region. You can’t achieve this outcome with just one of the three operations shown earlier; instead, you will need to first build a data set that encompasses all of sets A and B, and then utilize a second operation to remove the overlapping region. If the combined set is described as A union B, and the overlapping region is described as A intersect B, then the operation needed to generate the data set represented by Figure 6-4 would look as follows: (A union B) except (A intersect B) Of course, there are often multiple ways to achieve the same results; you could reach a similar outcome using the following operation: (A except B) union (B except A) While these concepts are fairly easy to understand using diagrams, the next sections show you how these concepts are applied to a relational database using the SQL set operators. Set Theory in Practice The circles used in the previous section’s diagrams to represent data sets don’t convey anything about what the data sets comprise. When dealing with actual data, however, there is a need to describe the composition of the data sets involved if they are to be combined. Imagine,", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 75 + }, + { + "text": "Theory in Practice The circles used in the previous section’s diagrams to represent data sets don’t convey anything about what the data sets comprise. When dealing with actual data, however, there is a need to describe the composition of the data sets involved if they are to be combined. Imagine, for example, what would happen if you tried to generate the union of the customer table and the city table, whose definitions are as follows: mysql> desc customer; +-------------+----------------------+------+-----+------------- ------+ | Field | Type | Null | Key | Default | +-------------+----------------------+------+-----+------------- ------+ | customer_id | smallint(5) unsigned | NO | PRI | NULL | | store_id | tinyint(3) unsigned | NO | MUL | NULL | | first_name | varchar(45) | NO | | NULL | | last_name | varchar(45) | NO | MUL | NULL | | email | varchar(50) | YES | | NULL | | address_id | smallint(5) unsigned | NO | MUL | NULL | | active | tinyint(1) | NO | | 1 | | create_date | datetime | NO | | NULL | | last_update | timestamp | YES | | CURRENT_TIMESTAMP | +-------------+----------------------+------+-----+------------- ------+ mysql> desc city; +-------------+----------------------+------+-----+------------- ------+ | Field | Type | Null | Key | Default | +-------------+----------------------+------+-----+------------- ------+ | city_id | smallint(5) unsigned | NO | PRI | NULL | | city | varchar(50) | NO | | NULL | | country_id | smallint(5) unsigned | NO | MUL | NULL | | last_update | timestamp | NO | | CURRENT_TIMESTAMP | +-------------+----------------------+------+-----+------------- ------+ When combined, the first column in the result set would include both the customer.customer_id and city.city_id columns, the second column would be the combination of the customer.store_id and city.city columns, and so forth. While some of the column pairs are easy to combine (e.g., two numeric columns), it is unclear how other column pairs should be combined, such as a numeric column with a string column or a string column with a date column. Additionally, the fifth through ninth columns of the combined tables would include data from only the customer table’s fifth through ninth columns, since the city table has only four columns. Clearly, there needs to be some commonality between two data sets that you wish to combine. Therefore, when performing set operations on two data sets, the following guidelines must apply: Both data sets must have the same number of columns. The data types of each column across the two data sets must be the same (or the server must be able to convert one to the other). With these rules in place, it is easier to envision what “overlapping data” means in practice; each column pair from the two sets being combined must contain the same string, number, or date for rows in the two tables to be considered the same. You perform a set operation by placing a set operator between two select statements, as demonstrated by the following: mysql> SELECT 1 num, 'abc' str -> UNION -> SELECT", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 76 + }, + { + "text": "being combined must contain the same string, number, or date for rows in the two tables to be considered the same. You perform a set operation by placing a set operator between two select statements, as demonstrated by the following: mysql> SELECT 1 num, 'abc' str -> UNION -> SELECT 9 num, 'xyz' str; +-----+-----+ | num | str | +-----+-----+ | 1 | abc | | 9 | xyz | +-----+-----+ 2 rows in set (0.02 sec) Each of the individual queries yields a data set consisting of a single row having a numeric column and a string column. The set operator, which in this case is union, tells the database server to combine all rows from the two sets. Thus, the final set includes two rows of two columns. This query is known as a compound query because it comprises multiple, otherwise- independent queries. As you will see later, compound queries may include more than two queries if multiple set operations are needed to attain the final results. Set Operators The SQL language includes three set operators that allow you to perform each of the various set operations described earlier in the chapter. Additionally, each set operator has two flavors, one that includes duplicates and another that removes duplicates (but not necessarily all of the duplicates). The following subsections define each operator and demonstrate how they are used. The union Operator The union and union all operators allow you to combine multiple data sets. The difference between the two is that union sorts the combined set and removes duplicates, whereas union all does not. With union all, the number of rows in the final data set will always equal the sum of the number of rows in the sets being combined. This operation is the simplest set operation to perform (from the server’s point of view), since there is no need for the server to check for overlapping data. The following example demonstrates how you can use the union all operator to generate a set of first and last names from multiple tables: mysql> SELECT 'CUST' typ, c.first_name, c.last_name -> FROM customer c -> UNION ALL -> SELECT 'ACTR' typ, a.first_name, a.last_name -> FROM actor a; +------+------------+-------------+ | typ | first_name | last_name | +------+------------+-------------+ | CUST | MARY | SMITH | | CUST | PATRICIA | JOHNSON | | CUST | LINDA | WILLIAMS | | CUST | BARBARA | JONES | | CUST | ELIZABETH | BROWN | | CUST | JENNIFER | DAVIS | | CUST | MARIA | MILLER | | CUST | SUSAN | WILSON | | CUST | MARGARET | MOORE | | CUST | DOROTHY | TAYLOR | | CUST | LISA | ANDERSON | | CUST | NANCY | THOMAS | | CUST | KAREN | JACKSON | ... | ACTR | BURT | TEMPLE | | ACTR | MERYL | ALLEN | | ACTR | JAYNE | SILVERSTONE | | ACTR | BELA | WALKEN | | ACTR |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 77 + }, + { + "text": "LISA | ANDERSON | | CUST | NANCY | THOMAS | | CUST | KAREN | JACKSON | ... | ACTR | BURT | TEMPLE | | ACTR | MERYL | ALLEN | | ACTR | JAYNE | SILVERSTONE | | ACTR | BELA | WALKEN | | ACTR | REESE | WEST | | ACTR | MARY | KEITEL | | ACTR | JULIA | FAWCETT | | ACTR | THORA | TEMPLE | +------+------------+-------------+ 799 rows in set (0.00 sec) The query returns 799 names, with 599 rows coming from the customer table and the other 200 coming from the actor table. The first column, which has the alias typ, is not necessary, but was added to show the source of each name returned by the query. Just to drive home the point that the union all operator doesn’t remove duplicates, here’s another version of the previous example, but with two identical queries against the actor table: mysql> SELECT 'ACTR' typ, a.first_name, a.last_name -> FROM actor a -> UNION ALL -> SELECT 'ACTR' typ, a.first_name, a.last_name -> FROM actor a; +------+-------------+--------------+ | typ | first_name | last_name | +------+-------------+--------------+ | ACTR | PENELOPE | GUINESS | | ACTR | NICK | WAHLBERG | | ACTR | ED | CHASE | | ACTR | JENNIFER | DAVIS | | ACTR | JOHNNY | LOLLOBRIGIDA | | ACTR | BETTE | NICHOLSON | | ACTR | GRACE | MOSTEL | ... | ACTR | BURT | TEMPLE | | ACTR | MERYL | ALLEN | | ACTR | JAYNE | SILVERSTONE | | ACTR | BELA | WALKEN | | ACTR | REESE | WEST | | ACTR | MARY | KEITEL | | ACTR | JULIA | FAWCETT | | ACTR | THORA | TEMPLE | +------+-------------+--------------+ 400 rows in set (0.00 sec) As you can see by the results, the 200 rows from the actor table are included twice, for a total of 400 rows. While you are unlikely to repeat the same query twice in a compound query, here is another compound query that returns duplicate data: mysql> SELECT c.first_name, c.last_name -> FROM customer c -> WHERE c.first_name LIKE 'J%' AND c.last_name LIKE 'D%' -> UNION ALL -> SELECT a.first_name, a.last_name -> FROM actor a -> WHERE a.first_name LIKE 'J%' AND a.last_name LIKE 'D%'; +------------+-----------+ | first_name | last_name | +------------+-----------+ | JENNIFER | DAVIS | | JENNIFER | DAVIS | | JUDY | DEAN | | JODIE | DEGENERES | | JULIANNE | DENCH | +------------+-----------+ 5 rows in set (0.00 sec) Both queries return the names of people having the initials “JD”. Of the five rows in the result set, one of them is a duplicate (Jennifer Davis). If you would like your combined table to exclude duplicate rows, you need to use the union operator instead of union all: mysql> SELECT c.first_name, c.last_name -> FROM customer c -> WHERE c.first_name LIKE 'J%' AND c.last_name LIKE 'D%' -> UNION -> SELECT a.first_name, a.last_name -> FROM", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 78 + }, + { + "text": "duplicate (Jennifer Davis). If you would like your combined table to exclude duplicate rows, you need to use the union operator instead of union all: mysql> SELECT c.first_name, c.last_name -> FROM customer c -> WHERE c.first_name LIKE 'J%' AND c.last_name LIKE 'D%' -> UNION -> SELECT a.first_name, a.last_name -> FROM actor a -> WHERE a.first_name LIKE 'J%' AND a.last_name LIKE 'D%'; +------------+-----------+ | first_name | last_name | +------------+-----------+ | JENNIFER | DAVIS | | JUDY | DEAN | | JODIE | DEGENERES | | JULIANNE | DENCH | +------------+-----------+ 4 rows in set (0.00 sec) For this version of the query, only the four distinct names are included in the result set, rather than the five rows returned when using union all. The intersect Operator The ANSI SQL specification includes the intersect operator for performing intersections. Unfortunately, version 8.0 of MySQL does not implement the intersect operator. If you are using Oracle or SQL Server 2008, you will be able to use intersect; since I am using MySQL for all examples in this book, however, the result sets for the example queries in this section are fabricated and cannot be executed with any versions up to and including version 8.0. I also refrain from showing the MySQL prompt (mysql>), since the statements are not being executed by the MySQL server. If the two queries in a compound query return nonoverlapping data sets, then the intersection will be an empty set. Consider the following query: SELECT c.first_name, c.last_name FROM customer c WHERE c.first_name LIKE 'D%' AND c.last_name LIKE 'T%' INTERSECT SELECT a.first_name, a.last_name FROM actor a WHERE a.first_name LIKE 'D%' AND a.last_name LIKE 'T%'; Empty set (0.04 sec) While there are both actors and customers having the initials “DT”, these sets are completely nonoverlapping, so the intersection of the two sets yields the empty set. If we switch back to the initials “JD”, however, the intersection will yield a single row: SELECT c.first_name, c.last_name FROM customer c WHERE c.first_name LIKE 'J%' AND c.last_name LIKE 'D%' INTERSECT SELECT a.first_name, a.last_name FROM actor a WHERE a.first_name LIKE 'J%' AND a.last_name LIKE 'D%'; +------------+-----------+ | first_name | last_name | +------------+-----------+ | JENNIFER | DAVIS | +------------+-----------+ 1 row in set (0.00 sec) The intersection of these two queries yields Jennifer Davis, which is the only name found in both queries’ result sets. Along with the intersect operator, which removes any duplicate rows found in the overlapping region, the ANSI SQL specification calls for an intersect all operator, which does not remove duplicates. The only database server that currently implements the intersect all operator is IBM’s DB2 Universal Server. The except Operator The ANSI SQL specification includes the except operator for performing the except operation. Once again, unfortunately, version 8.0 of MySQL does not implement the except operator, so the same rules apply for this section as for the previous section. NOTE If you are using Oracle Database, you will need to use the non-ANSI-compliant minus operator instead. The except operator returns the first result", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 79 + }, + { + "text": "Once again, unfortunately, version 8.0 of MySQL does not implement the except operator, so the same rules apply for this section as for the previous section. NOTE If you are using Oracle Database, you will need to use the non-ANSI-compliant minus operator instead. The except operator returns the first result set minus any overlap with the second result set. Here’s the example from the previous section, but using except instead of intersect, and with the order of the queries reversed: SELECT a.first_name, a.last_name FROM actor a WHERE a.first_name LIKE 'J%' AND a.last_name LIKE 'D%' EXCEPT SELECT c.first_name, c.last_name FROM customer c WHERE c.first_name LIKE 'J%' AND c.last_name LIKE 'D%'; +------------+-----------+ | first_name | last_name | +------------+-----------+ | JUDY | DEAN | | JODIE | DEGENERES | | JULIANNE | DENCH | +------------+-----------+ 3 rows in set (0.00 sec) In this version of the query, the result set consists of the three rows from the first query minus Jennifer Davis, who is found in the result sets from both queries. There is also an except all operator specified in the ANSI SQL specification, but once again, only IBM’s DB2 Universal Server has implemented the except all operator. The except all operator is a bit tricky, so here’s an example to demonstrate how duplicate data is handled. Let’s say you have two data sets that look as follows: Set A +----------+ | actor_id | +----------+ | 10 | | 11 | | 12 | | 10 | | 10 | +----------+ Set B +----------+ | actor_id | +----------+ | 10 | | 10 | +----------+ The operation A except B yields the following: +----------+ | actor_id | +----------+ | 11 | | 12 | +----------+ If you change the operation to A except all B, you will see the following: +----------+ | actor_id | +----------+ | 10 | | 11 | | 12 | +----------+ Therefore, the difference between the two operations is that except removes all occurrences of duplicate data from set A, whereas except all only removes one occurrence of duplicate data from set A for every occurrence in set B. Set Operation Rules The following sections outline some rules that you must follow when working with compound queries. Sorting Compound Query Results If you want the results of your compound query to be sorted, you can add an order by clause after the last query. When specifying column names in the order by clause, you will need to choose from the column names in the first query of the compound query. Frequently, the column names are the same for both queries in a compound query, but this does not need to be the case, as demonstrated by the following: mysql> SELECT a.first_name fname, a.last_name lname -> FROM actor a -> WHERE a.first_name LIKE 'J%' AND a.last_name LIKE 'D%' -> UNION ALL -> SELECT c.first_name, c.last_name -> FROM customer c -> WHERE c.first_name LIKE 'J%' AND c.last_name LIKE 'D%' -> ORDER BY lname, fname; +----------+-----------+ | fname | lname |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 80 + }, + { + "text": "mysql> SELECT a.first_name fname, a.last_name lname -> FROM actor a -> WHERE a.first_name LIKE 'J%' AND a.last_name LIKE 'D%' -> UNION ALL -> SELECT c.first_name, c.last_name -> FROM customer c -> WHERE c.first_name LIKE 'J%' AND c.last_name LIKE 'D%' -> ORDER BY lname, fname; +----------+-----------+ | fname | lname | +----------+-----------+ | JENNIFER | DAVIS | | JENNIFER | DAVIS | | JUDY | DEAN | | JODIE | DEGENERES | | JULIANNE | DENCH | +----------+-----------+ 5 rows in set (0.00 sec) The column names specified in the two queries are different in this example. If you specify a column name from the second query in your order by clause, you will see the following error: mysql> SELECT a.first_name fname, a.last_name lname -> FROM actor a -> WHERE a.first_name LIKE 'J%' AND a.last_name LIKE 'D%' -> UNION ALL -> SELECT c.first_name, c.last_name -> FROM customer c -> WHERE c.first_name LIKE 'J%' AND c.last_name LIKE 'D%' -> ORDER BY last_name, first_name; ERROR 1054 (42S22): Unknown column 'last_name' in 'order clause' I recommend giving the columns in both queries identical column aliases in order to avoid this issue. Set Operation Precedence If your compound query contains more than two queries using different set operators, you need to think about the order in which to place the queries in your compound statement to achieve the desired results. Consider the following three-query compound statement: mysql> SELECT a.first_name, a.last_name -> FROM actor a -> WHERE a.first_name LIKE 'J%' AND a.last_name LIKE 'D%' -> UNION ALL -> SELECT a.first_name, a.last_name -> FROM actor a -> WHERE a.first_name LIKE 'M%' AND a.last_name LIKE 'T%' -> UNION -> SELECT c.first_name, c.last_name -> FROM customer c -> WHERE c.first_name LIKE 'J%' AND c.last_name LIKE 'D%'; +------------+-----------+ | first_name | last_name | +------------+-----------+ | JENNIFER | DAVIS | | JUDY | DEAN | | JODIE | DEGENERES | | JULIANNE | DENCH | | MARY | TANDY | | MENA | TEMPLE | +------------+-----------+ 6 rows in set (0.00 sec) This compound query includes three queries that return sets of nonunique names; the first and second queries are separated with the union all operator, while the second and third queries are separated with the union operator. While it might not seem to make much difference where the union and union all operators are placed, it does, in fact, make a difference. Here’s the same compound query with the set operators reversed: mysql> SELECT a.first_name, a.last_name -> FROM actor a -> WHERE a.first_name LIKE 'J%' AND a.last_name LIKE 'D%' -> UNION -> SELECT a.first_name, a.last_name -> FROM actor a -> WHERE a.first_name LIKE 'M%' AND a.last_name LIKE 'T%' -> UNION ALL -> SELECT c.first_name, c.last_name -> FROM customer c -> WHERE c.first_name LIKE 'J%' AND c.last_name LIKE 'D%'; +------------+-----------+ | first_name | last_name | +------------+-----------+ | JENNIFER | DAVIS | | JUDY | DEAN | | JODIE | DEGENERES | | JULIANNE | DENCH | | MARY | TANDY | | MENA | TEMPLE | | JENNIFER | DAVIS |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 81 + }, + { + "text": "WHERE c.first_name LIKE 'J%' AND c.last_name LIKE 'D%'; +------------+-----------+ | first_name | last_name | +------------+-----------+ | JENNIFER | DAVIS | | JUDY | DEAN | | JODIE | DEGENERES | | JULIANNE | DENCH | | MARY | TANDY | | MENA | TEMPLE | | JENNIFER | DAVIS | +------------+-----------+ 7 rows in set (0.00 sec) Looking at the results, it’s obvious that it does make a difference how the compound query is arranged when using different set operators. In general, compound queries containing three or more queries are evaluated in order from top to bottom, but with the following caveats: The ANSI SQL specification calls for the intersect operator to have precedence over the other set operators. You may dictate the order in which queries are combined by enclosing multiple queries in parentheses. MySQL does not yet allow parentheses in compound queries, but if you are using a different database server, you can wrap adjoining queries in parentheses to override the default top-to-bottom processing of compound queries, as in: SELECT a.first_name, a.last_name FROM actor a WHERE a.first_name LIKE 'J%' AND a.last_name LIKE 'D%' UNION (SELECT a.first_name, a.last_name FROM actor a WHERE a.first_name LIKE 'M%' AND a.last_name LIKE 'T%' UNION ALL SELECT c.first_name, c.last_name FROM customer c WHERE c.first_name LIKE 'J%' AND c.last_name LIKE 'D%' ) For this compound query, the second and third queries would be combined using the union all operator, then the results would be combined with the first query using the union operator. Test Your Knowledge The following exercises are designed to test your understanding of set operations. See Appendix C for answers to these exercises. Exercise 6-1 If set A = {L M N O P} and set B = {P Q R S T}, what sets are generated by the following operations? A union B A union all B A intersect B A except B Exercise 6-2 Write a compound query that finds the first and last names of all Actors and Customers whose last name starts with L. Exercise 6-3 Sort the results from Exercise 6-2 by the last_name column. Chapter 7. Data Generation, Manipulation, and Conversion As I mentioned in the Preface, this book strives to teach generic SQL techniques that can be applied across multiple database servers. This chapter, however, deals with the generation, conversion, and manipulation of string, numeric, and temporal data, and the SQL language does not include commands covering this functionality. Rather, built-in functions are used to facilitate data generation, conversion, and manipulation, and while the SQL standard does specify some functions, the database vendors often do not comply with the function specifications. Therefore, my approach for this chapter is to show you some of the common ways in which data is generated and manipulated within SQL statements, and then demonstrate some of the built-in functions implemented by Microsoft SQL Server, Oracle Database, and MySQL. Along with reading this chapter, I strongly recommend you download a reference guide covering all the functions implemented by your server. If", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 82 + }, + { + "text": "ways in which data is generated and manipulated within SQL statements, and then demonstrate some of the built-in functions implemented by Microsoft SQL Server, Oracle Database, and MySQL. Along with reading this chapter, I strongly recommend you download a reference guide covering all the functions implemented by your server. If you work with more than one database server, there are several reference guides that cover multiple servers, such as Kevin Kline et al.’s SQL in a Nutshell and Jonathan Gennick’s SQL Pocket Guide, both from O’Reilly. Working with String Data When working with string data, you will be using one of the following character data types: CHAR Holds fixed-length, blank-padded strings. MySQL allows CHAR values up to 255 characters in length, Oracle Database permits up to 2,000 characters, and SQL Server allows up to 8,000 characters. varchar Holds variable-length strings. MySQL permits up to 65,535 characters in a varchar column, Oracle Database (via the varchar2 type) allows up to 4,000 characters, and SQL Server allows up to 8,000 characters. text (MySQL and SQL Server) or CLOB (Character Large Object; Oracle Database) Holds very large variable-length strings (generally referred to as documents in this context). MySQL has multiple text types (tinytext, text, mediumtext, and longtext) for documents up to 4 GB in size. SQL Server has a single text type for documents up to 2 GB in size, and Oracle Database includes the CLOB data type, which can hold documents up to a whopping 128 TB. SQL Server 2005 also includes the varchar(max) data type and recommends its use instead of the text type, which will be removed from the server in some future release. To demonstrate how you can use these various types, I use the following table for some of the examples in this section: CREATE TABLE string_tbl (char_fld CHAR(30), vchar_fld VARCHAR(30), text_fld TEXT ); The next two subsections show how you can generate and manipulate string data. String Generation The simplest way to populate a character column is to enclose a string in quotes, as in: mysql> INSERT INTO string_tbl (char_fld, vchar_fld, text_fld) -> VALUES ('This is char data', -> 'This is varchar data', -> 'This is text data'); Query OK, 1 row affected (0.00 sec) When inserting string data into a table, remember that if the length of the string exceeds the maximum size for the character column (either the designated maximum or the maximum allowed for the data type), the server will throw an exception. Although this is the default behavior for all three servers, you can configure MySQL and SQL Server to silently truncate the string instead of throwing an exception. To demonstrate how MySQL handles this situation, the following update statement attempts to modify the vchar_fld column, whose maximum length is defined as 30, with a string that is 46 characters in length: mysql> UPDATE string_tbl -> SET vchar_fld = 'This is a piece of extremely long varchar data'; ERROR 1406 (22001): Data too long for column 'vchar_fld' at row 1 Since MySQL 6.0,", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 83 + }, + { + "text": "the vchar_fld column, whose maximum length is defined as 30, with a string that is 46 characters in length: mysql> UPDATE string_tbl -> SET vchar_fld = 'This is a piece of extremely long varchar data'; ERROR 1406 (22001): Data too long for column 'vchar_fld' at row 1 Since MySQL 6.0, the default behavior is now “strict” mode, which means that exceptions are thrown when problems arise, whereas in older versions of the server the string would have been truncated and a warning issued. If you would rather have the engine truncate the string and issue a warning instead of raising an exception, you can opt to be in ANSI mode. The following example shows how to check which mode you are in, and then how to change the mode using the SET command: mysql> SELECT @@session.sql_mode; +--------------------------------------------------------------- -+ | @@session.sql_mode | +--------------------------------------------------------------- -+ | STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION | +--------------------------------------------------------------- -+ 1 row in set (0.00 sec) mysql> SET sql_mode='ansi'; Query OK, 0 rows affected (0.08 sec) mysql> SELECT @@session.sql_mode; +--------------------------------------------------------------- -----------------+ | @@session.sql_mode | +--------------------------------------------------------------- -----------------+ | REAL_AS_FLOAT,PIPES_AS_CONCAT,ANSI_QUOTES,IGNORE_SPACE,ONLY_FULL _GROUP_BY,ANSI | +--------------------------------------------------------------- -----------------+ 1 row in set (0.00 sec) If you rerun the previous UPDATE statement, you will find that the column has been modified, but the following warning is generated: mysql> SHOW WARNINGS; +---------+------+---------------------------------------------- --+ | Level | Code | Message | +---------+------+---------------------------------------------- --+ | Warning | 1265 | Data truncated for column 'vchar_fld' at row 1 | +---------+------+---------------------------------------------- --+ 1 row in set (0.00 sec) If you retrieve the vchar_fld column, you will see that the string has indeed been truncated: mysql> SELECT vchar_fld -> FROM string_tbl; +--------------------------------+ | vchar_fld | +--------------------------------+ | This is a piece of extremely l | +--------------------------------+ 1 row in set (0.05 sec) As you can see, only the first 30 characters of the 46-character string made it into the vchar_fld column. The best way to avoid string truncation (or exceptions, in the case of Oracle Database or MySQL in strict mode) when working with varchar columns is to set the upper limit of a column to a high enough value to handle the longest strings that might be stored in the column (keeping in mind that the server allocates only enough space to store the string, so it is not wasteful to set a high upper limit for varchar columns). INCLUDING SINGLE QUOTES Since strings are demarcated by single quotes, you will need to be alert for strings that include single quotes or apostrophes. For example, you won’t be able to insert the following string because the server will think that the apostrophe in the word doesn’t marks the end of the string: UPDATE string_tbl SET text_fld = 'This string doesn't work'; To make the server ignore the apostrophe in the word doesn’t, you will need to add an escape to the string so that the server treats the apostrophe like any other character in the string. All three servers allow you to escape a single quote by adding another single quote directly before, as", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 84 + }, + { + "text": "server ignore the apostrophe in the word doesn’t, you will need to add an escape to the string so that the server treats the apostrophe like any other character in the string. All three servers allow you to escape a single quote by adding another single quote directly before, as in: mysql> UPDATE string_tbl -> SET text_fld = 'This string didn''t work, but it does now'; Query OK, 1 row affected (0.01 sec) Rows matched: 1 Changed: 1 Warnings: 0 NOTE Oracle Database and MySQL users may also choose to escape a single quote by adding a backslash character immediately before, as in: UPDATE string_tbl SET text_fld = 'This string didn\\'t work, but it does now' If you retrieve a string for use in a screen or report field, you don’t need to do anything special to handle embedded quotes: mysql> SELECT text_fld -> FROM string_tbl; +------------------------------------------+ | text_fld | +------------------------------------------+ | This string didn't work, but it does now | +------------------------------------------+ 1 row in set (0.00 sec) However, if you are retrieving the string to add to a file that another program will read, you may want to include the escape as part of the retrieved string. If you are using MySQL, you can use the built-in function quote(), which places quotes around the entire string and adds escapes to any single quotes/apostrophes within the string. Here’s what our string looks like when retrieved via the quote() function: mysql> SELECT quote(text_fld) -> FROM string_tbl; +---------------------------------------------+ | QUOTE(text_fld) | +---------------------------------------------+ | 'This string didn\\'t work, but it does now' | +---------------------------------------------+ 1 row in set (0.04 sec) When retrieving data for data export, you may want to use the quote() function for all non-system-generated character columns, such as a customer_notes column. INCLUDING SPECIAL CHARACTERS If your application is multinational in scope, you might find yourself working with strings that include characters that do not appear on your keyboard. When working with the French and German languages, for example, you might need to include accented characters such as é and ö. The SQL Server and MySQL servers include the built-in function char() so that you can build strings from any of the 255 characters in the ASCII character set (Oracle Database users can use the chr() function). To demonstrate, the next example retrieves a typed string and its equivalent built via individual characters: mysql> SELECT 'abcdefg', CHAR(97,98,99,100,101,102,103); +---------+--------------------------------+ | abcdefg | CHAR(97,98,99,100,101,102,103) | +---------+--------------------------------+ | abcdefg | abcdefg | +---------+--------------------------------+ 1 row in set (0.01 sec) Thus, the 97 character in the ASCII character set is the letter a. While the characters shown in the preceding example are not special, the following examples show the location of the accented characters along with other special characters, such as currency symbols: mysql> SELECT CHAR(128,129,130,131,132,133,134,135,136,137); +-----------------------------------------------+ | CHAR(128,129,130,131,132,133,134,135,136,137) | +-----------------------------------------------+ | Çüéâäàåçêë | +-----------------------------------------------+ 1 row in set (0.01 sec) th mysql> SELECT CHAR(138,139,140,141,142,143,144,145,146,147); +-----------------------------------------------+ | CHAR(138,139,140,141,142,143,144,145,146,147) | +-----------------------------------------------+ | èïîìÄÅÉæÆô | +-----------------------------------------------+ 1 row in set (0.01 sec) mysql> SELECT CHAR(148,149,150,151,152,153,154,155,156,157); +-----------------------------------------------+ |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 85 + }, + { + "text": "with other special characters, such as currency symbols: mysql> SELECT CHAR(128,129,130,131,132,133,134,135,136,137); +-----------------------------------------------+ | CHAR(128,129,130,131,132,133,134,135,136,137) | +-----------------------------------------------+ | Çüéâäàåçêë | +-----------------------------------------------+ 1 row in set (0.01 sec) th mysql> SELECT CHAR(138,139,140,141,142,143,144,145,146,147); +-----------------------------------------------+ | CHAR(138,139,140,141,142,143,144,145,146,147) | +-----------------------------------------------+ | èïîìÄÅÉæÆô | +-----------------------------------------------+ 1 row in set (0.01 sec) mysql> SELECT CHAR(148,149,150,151,152,153,154,155,156,157); +-----------------------------------------------+ | CHAR(148,149,150,151,152,153,154,155,156,157) | +-----------------------------------------------+ | öòûùÿÖÜø£Ø | +-----------------------------------------------+ 1 row in set (0.00 sec) mysql> SELECT CHAR(158,159,160,161,162,163,164,165); +---------------------------------------+ | CHAR(158,159,160,161,162,163,164,165) | +---------------------------------------+ | ׃áíóúñÑ | +---------------------------------------+ 1 row in set (0.01 sec) NOTE I am using the utf8mb4 character set for the examples in this section. If your session is configured for a different character set, you will see a different set of characters than what is shown here. The same concepts apply, but you will need to familiarize yourself with the layout of your character set to locate specific characters. Building strings character by character can be quite tedious, especially if only a few of the characters in the string are accented. Fortunately, you can use the concat() function to concatenate individual strings, some of which you can type while others you can generate via the char() function. For example, the following shows how to build the phrase danke schön using the concat() and char() functions: mysql> SELECT CONCAT('danke sch', CHAR(148), 'n'); +-------------------------------------+ | CONCAT('danke sch', CHAR(148), 'n') | +-------------------------------------+ | danke schön | +-------------------------------------+ 1 row in set (0.00 sec) NOTE Oracle Database users can use the concatenation operator (||) instead of the concat() function, as in: SELECT 'danke sch' || CHR(148) || 'n' FROM dual; SQL Server does not include a concat() function, so you will need to use the concatenation operator (+), as in: SELECT 'danke sch' + CHAR(148) + 'n' If you have a character and need to find its ASCII equivalent, you can use the ascii() function, which takes the leftmost character in the string and returns a number: mysql> SELECT ASCII('ö'); +------------+ | ASCII('ö') | +------------+ | 148 | +------------+ 1 row in set (0.00 sec) Using the char(), ascii(), and concat() functions (or concatenation operators), you should be able to work with any Roman language even if you are using a keyboard that does not include accented or special characters. String Manipulation Each database server includes many built-in functions for manipulating strings. This section explores two types of string functions: those that return numbers and those that return strings. Before I begin, however, I reset the data in the string_tbl table to the following: mysql> DELETE FROM string_tbl; Query OK, 1 row affected (0.02 sec) mysql> INSERT INTO string_tbl (char_fld, vchar_fld, text_fld) -> VALUES ('This string is 28 characters', -> 'This string is 28 characters', -> 'This string is 28 characters'); Query OK, 1 row affected (0.00 sec) STRING FUNCTIONS THAT RETURN NUMBERS Of the string functions that return numbers, one of the most commonly used is the length() function, which returns the number of characters in the string (SQL Server users will need to use the len() function). The following query applies", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 86 + }, + { + "text": "1 row affected (0.00 sec) STRING FUNCTIONS THAT RETURN NUMBERS Of the string functions that return numbers, one of the most commonly used is the length() function, which returns the number of characters in the string (SQL Server users will need to use the len() function). The following query applies the length() function to each column in the string_tbl table: mysql> SELECT LENGTH(char_fld) char_length, -> LENGTH(vchar_fld) varchar_length, -> LENGTH(text_fld) text_length -> FROM string_tbl; +-------------+----------------+-------------+ | char_length | varchar_length | text_length | +-------------+----------------+-------------+ | 28 | 28 | 28 | +-------------+----------------+-------------+ 1 row in set (0.00 sec) While the lengths of the varchar and text columns are as expected, you might have expected the length of the char column to be 30, since I told you that strings stored in char columns are right-padded with spaces. The MySQL server removes trailing spaces from char data when it is retrieved, however, so you will see the same results from all string functions regardless of the type of column in which the strings are stored. Along with finding the length of a string, you might want to find the location of a substring within a string. For example, if you want to find the position at which the string 'characters' appears in the vchar_fld column, you could use the position() function, as demonstrated by the following: mysql> SELECT POSITION('characters' IN vchar_fld) -> FROM string_tbl; +-------------------------------------+ | POSITION('characters' IN vchar_fld) | +-------------------------------------+ | 19 | +-------------------------------------+ 1 row in set (0.12 sec) If the substring cannot be found, the position() function returns 0. WARNING For those of you who program in a language such as C or C++, where the first element of an array is at position 0, remember when working with databases that the first character in a string is at position 1. A return value of 0 from instr() indicates that the substring could not be found, not that the substring was found at the first position in the string. If you want to start your search at something other than the first character of your target string, you will need to use the locate() function, which is similar to the position() function except that it allows an optional third parameter, which is used to define the search’s start position. The locate() function is also proprietary, whereas the position() function is part of the SQL:2003 standard. Here’s an example asking for the position of the string 'is' starting at the fifth character in the vchar_fld column: mysql> SELECT LOCATE('is', vchar_fld, 5) -> FROM string_tbl; +----------------------------+ | LOCATE('is', vchar_fld, 5) | +----------------------------+ | 13 | +----------------------------+ 1 row in set (0.02 sec) NOTE Oracle Database does not include the position() or locate() function, but it does include the instr() function, which mimics the position() function when provided with two arguments and mimics the locate() function when provided with three arguments. SQL Server also doesn’t include a position() or locate() function, but it does include the charindx() function, which also accepts either two", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 87 + }, + { + "text": "function, but it does include the instr() function, which mimics the position() function when provided with two arguments and mimics the locate() function when provided with three arguments. SQL Server also doesn’t include a position() or locate() function, but it does include the charindx() function, which also accepts either two or three arguments similar to Oracle’s instr() function. Another function that takes strings as arguments and returns numbers is the string comparison function strcmp(). Strcmp(), which is implemented only by MySQL and has no analog in Oracle Database or SQL Server, takes two strings as arguments, and returns one of the following: −1 if the first string comes before the second string in sort order 0 if the strings are identical 1 if the first string comes after the second string in sort order To illustrate how the function works, I first show the sort order of five strings using a query, and then show how the strings compare to one another using strcmp(). Here are the five strings that I insert into the string_tbl table: mysql> DELETE FROM string_tbl; Query OK, 1 row affected (0.00 sec) mysql> INSERT INTO string_tbl(vchar_fld) -> VALUES ('abcd'), -> ('xyz'), -> ('QRSTUV'), -> ('qrstuv'), -> ('12345'); Query OK, 5 rows affected (0.05 sec) Records: 5 Duplicates: 0 Warnings: 0 Here are the five strings in their sort order: mysql> SELECT vchar_fld -> FROM string_tbl -> ORDER BY vchar_fld; +-----------+ | vchar_fld | +-----------+ | 12345 | | abcd | | QRSTUV | | qrstuv | | xyz | +-----------+ 5 rows in set (0.00 sec) The next query makes six comparisons among the five different strings: mysql> SELECT STRCMP('12345','12345') 12345_12345, -> STRCMP('abcd','xyz') abcd_xyz, -> STRCMP('abcd','QRSTUV') abcd_QRSTUV, -> STRCMP('qrstuv','QRSTUV') qrstuv_QRSTUV, -> STRCMP('12345','xyz') 12345_xyz, -> STRCMP('xyz','qrstuv') xyz_qrstuv; +-------------+----------+-------------+---------------+-------- ---+------------+ | 12345_12345 | abcd_xyz | abcd_QRSTUV | qrstuv_QRSTUV | 12345_xyz | xyz_qrstuv | +-------------+----------+-------------+---------------+-------- ---+------------+ | 0 | −1 | −1 | 0 | −1 | 1 | +-------------+----------+-------------+---------------+-------- ---+------------+ 1 row in set (0.00 sec) The first comparison yields 0, which is to be expected since I compared a string to itself. The fourth comparison also yields 0, which is a bit surprising, since the strings are composed of the same letters, with one string all uppercase and the other all lowercase. The reason for this result is that MySQL’s strcmp() function is case-insensitive, which is something to remember when using the function. The other four comparisons yield either −1 or 1 depending on whether the first string comes before or after the second string in sort order. For example, strcmp('abcd','xyz') yields −1, since the string 'abcd' comes before the string 'xyz'. Along with the strcmp() function, MySQL also allows you to use the like and regexp operators to compare strings in the select clause. Such comparisons will yield 1 (for true) or 0 (for false). Therefore, these operators allow you to build expressions that return a number, much like the functions described in this section. Here’s an example using like: mysql> SELECT name, name LIKE", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 88 + }, + { + "text": "regexp operators to compare strings in the select clause. Such comparisons will yield 1 (for true) or 0 (for false). Therefore, these operators allow you to build expressions that return a number, much like the functions described in this section. Here’s an example using like: mysql> SELECT name, name LIKE '%y' ends_in_y -> FROM category; +-------------+-----------+ | name | ends_in_y | +-------------+-----------+ | Action | 0 | | Animation | 0 | | Children | 0 | | Classics | 0 | | Comedy | 1 | | Documentary | 1 | | Drama | 0 | | Family | 1 | | Foreign | 0 | | Games | 0 | | Horror | 0 | | Music | 0 | | New | 0 | | Sci-Fi | 0 | | Sports | 0 | | Travel | 0 | +-------------+-----------+ 16 rows in set (0.00 sec) This example retrieves all the category names, along with an expression that returns 1 if the name ends in “y” or 0 otherwise. If you want to perform more complex pattern matches, you can use the regexp operator, as demonstrated by the following: mysql> SELECT name, name REGEXP 'y$' ends_in_y -> FROM category; +-------------+-----------+ | name | ends_in_y | +-------------+-----------+ | Action | 0 | | Animation | 0 | | Children | 0 | | Classics | 0 | | Comedy | 1 | | Documentary | 1 | | Drama | 0 | | Family | 1 | | Foreign | 0 | | Games | 0 | | Horror | 0 | | Music | 0 | | New | 0 | | Sci-Fi | 0 | | Sports | 0 | | Travel | 0 | +-------------+-----------+ 16 rows in set (0.00 sec) The second column of this query returns 1 if the value stored in the name column matches the given regular expression. NOTE SQL Server and Oracle Database users can achieve similar results by building case expressions, which I describe in detail in Chapter 11. STRING FUNCTIONS THAT RETURN STRINGS In some cases, you will need to modify existing strings, either by extracting part of the string or by adding additional text to the string. Every database server includes multiple functions to help with these tasks. Before I begin, I once again reset the data in the string_tbl table: mysql> DELETE FROM string_tbl; Query OK, 5 rows affected (0.00 sec) mysql> INSERT INTO string_tbl (text_fld) -> VALUES ('This string was 29 characters'); Query OK, 1 row affected (0.01 sec) Earlier in the chapter, I demonstrated the use of the concat() function to help build words that include accented characters. The concat() function is useful in many other situations, including when you need to append additional characters to a stored string. For instance, the following example modifies the string stored in the text_fld column by tacking an additional phrase on the end: mysql> UPDATE string_tbl -> SET text_fld = CONCAT(text_fld, ', but now it is longer');", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 89 + }, + { + "text": "many other situations, including when you need to append additional characters to a stored string. For instance, the following example modifies the string stored in the text_fld column by tacking an additional phrase on the end: mysql> UPDATE string_tbl -> SET text_fld = CONCAT(text_fld, ', but now it is longer'); Query OK, 1 row affected (0.03 sec) Rows matched: 1 Changed: 1 Warnings: 0 The contents of the text_fld column are now as follows: mysql> SELECT text_fld -> FROM string_tbl; +-----------------------------------------------------+ | text_fld | +-----------------------------------------------------+ | This string was 29 characters, but now it is longer | +-----------------------------------------------------+ 1 row in set (0.00 sec) Thus, like all functions that return a string, you can use concat() to replace the data stored in a character column. Another common use for the concat() function is to build a string from individual pieces of data. For example, the following query generates a narrative string for each customer: mysql> SELECT concat(first_name, ' ', last_name, -> ' has been a customer since ', date(create_date)) cust_narrative -> FROM customer; +---------------------------------------------------------+ | cust_narrative | +---------------------------------------------------------+ | MARY SMITH has been a customer since 2006-02-14 | | PATRICIA JOHNSON has been a customer since 2006-02-14 | | LINDA WILLIAMS has been a customer since 2006-02-14 | | BARBARA JONES has been a customer since 2006-02-14 | | ELIZABETH BROWN has been a customer since 2006-02-14 | | JENNIFER DAVIS has been a customer since 2006-02-14 | | MARIA MILLER has been a customer since 2006-02-14 | | SUSAN WILSON has been a customer since 2006-02-14 | | MARGARET MOORE has been a customer since 2006-02-14 | | DOROTHY TAYLOR has been a customer since 2006-02-14 | ... | RENE MCALISTER has been a customer since 2006-02-14 | | EDUARDO HIATT has been a customer since 2006-02-14 | | TERRENCE GUNDERSON has been a customer since 2006-02-14 | | ENRIQUE FORSYTHE has been a customer since 2006-02-14 | | FREDDIE DUGGAN has been a customer since 2006-02-14 | | WADE DELVALLE has been a customer since 2006-02-14 | | AUSTIN CINTRON has been a customer since 2006-02-14 | +---------------------------------------------------------+ 599 rows in set (0.00 sec) The concat() function can handle any expression that returns a string, and will even convert numbers and dates to string format, as evidenced by the date column (create_date) used as an argument. Although Oracle Database includes the concat() function, it will accept only two string arguments, so the previous query will not work on Oracle. Instead, you would need to use the concatenation operator (||) rather than a function call, as in: SELECT first_name || ' ' || last_name || ' has been a customer since ' || date(create_date)) cust_narrative FROM customer; SQL Server does not include a concat() function, so you would need to use the same approach as the previous query, except that you would use SQL Server’s concatenation operator (+) instead of ||. While concat() is useful for adding characters to the beginning or end of a string, you may also", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 90 + }, + { + "text": "does not include a concat() function, so you would need to use the same approach as the previous query, except that you would use SQL Server’s concatenation operator (+) instead of ||. While concat() is useful for adding characters to the beginning or end of a string, you may also have a need to add or replace characters in the middle of a string. All three database servers provide functions for this purpose, but all of them are different, so I demonstrate the MySQL function and then show the functions from the other two servers. MySQL includes the insert() function, which takes four arguments: the original string, the position at which to start, the number of characters to replace, and the replacement string. Depending on the value of the third argument, the function may be used to either insert or replace characters in a string. With a value of 0 for the third argument, the replacement string is inserted and any trailing characters are pushed to the right, as in: mysql> SELECT INSERT('goodbye world', 9, 0, 'cruel ') string; +---------------------+ | string | +---------------------+ | goodbye cruel world | +---------------------+ 1 row in set (0.00 sec) In this example, all characters starting from position 9 are pushed to the right and the string 'cruel' is inserted. If the third argument is greater than zero, then that number of characters is replaced with the replacement string, as in: mysql> SELECT INSERT('goodbye world', 1, 7, 'hello') string; +-------------+ | string | +-------------+ | hello world | +-------------+ 1 row in set (0.00 sec) For this example, the first seven characters are replaced with the string 'hello'. Oracle Database does not provide a single function with the flexibility of MySQL’s insert() function, but Oracle does provide the replace() function, which is useful for replacing one substring with another. Here’s the previous example reworked to use replace(): SELECT REPLACE('goodbye world', 'goodbye', 'hello') FROM dual; All instances of the string 'goodbye' will be replaced with the string 'hello', resulting in the string 'hello world'. The replace() function will replace every instance of the search string with the replacement string, so you need to be careful that you don’t end up with more replacements than you anticipated. SQL Server also includes a replace() function with the same functionality as Oracle’s, but SQL Server also includes a function called stuff() with similar functionality to MySQL’s insert() function. Here’s an example: SELECT STUFF('hello world', 1, 5, 'goodbye cruel') When executed, five characters are removed starting at position 1, and then the string 'goodbye cruel' is inserted at the starting position, resulting in the string 'goodbye cruel world'. Along with inserting characters into a string, you may have a need to extract a substring from a string. For this purpose, all three servers include the substring() function (although Oracle Database’s version is called substr()), which extracts a specified number of characters starting at a specified position. The following example extracts five characters from a string starting at the ninth", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 91 + }, + { + "text": "need to extract a substring from a string. For this purpose, all three servers include the substring() function (although Oracle Database’s version is called substr()), which extracts a specified number of characters starting at a specified position. The following example extracts five characters from a string starting at the ninth position: mysql> SELECT SUBSTRING('goodbye cruel world', 9, 5); +----------------------------------------+ | SUBSTRING('goodbye cruel world', 9, 5) | +----------------------------------------+ | cruel | +----------------------------------------+ 1 row in set (0.00 sec) Along with the functions demonstrated here, all three servers include many more built-in functions for manipulating string data. While many of them are designed for very specific purposes, such as generating the string equivalent of octal or hexadecimal numbers, there are many other general- purpose functions as well, such as functions that remove or add trailing spaces. For more information, consult your server’s SQL reference guide, or a general-purpose SQL reference guide such as SQL in a Nutshell (O’Reilly). Working with Numeric Data Unlike string data (and temporal data, as you will see shortly), numeric data generation is quite straightforward. You can type a number, retrieve it from another column, or generate it via a calculation. All the usual arithmetic operators (+, -, *, /) are available for performing calculations, and parentheses may be used to dictate precedence, as in: mysql> SELECT (37 * 59) / (78 - (8 * 6)); +----------------------------+ | (37 * 59) / (78 - (8 * 6)) | +----------------------------+ | 72.77 | +----------------------------+ 1 row in set (0.00 sec) As I mentioned in Chapter 2, the main concern when storing numeric data is that numbers might be rounded if they are larger than the specified size for a numeric column. For example, the number 9.96 will be rounded to 10.0 if stored in a column defined as float(3,1). Performing Arithmetic Functions Most of the built-in numeric functions are used for specific arithmetic purposes, such as determining the square root of a number. Table 7-1 lists some of the common numeric functions that take a single numeric argument and return a number. Table 7-1. Single-argument numeric functions Function name Description Acos( x ) Calculates the arc cosine of x Asin( x ) Calculates the arc sine of x Atan( x ) Calculates the arc tangent of x Cos( x ) Calculates the cosine of x Cot( x ) Calculates the cotangent of x Exp( x ) Calculates e Ln( x ) Calculates the natural log of x Sin( x ) Calculates the sine of x Sqrt( x ) Calculates the square root of x Tan( x ) Calculates the tangent of x x These functions perform very specific tasks, and I refrain from showing examples for these functions (if you don’t recognize a function by name or description, then you probably don’t need it). Other numeric functions used for calculations, however, are a bit more flexible and deserve some explanation. For example, the modulo operator, which calculates the remainder when one number is divided into another number, is implemented", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 92 + }, + { + "text": "you don’t recognize a function by name or description, then you probably don’t need it). Other numeric functions used for calculations, however, are a bit more flexible and deserve some explanation. For example, the modulo operator, which calculates the remainder when one number is divided into another number, is implemented in MySQL and Oracle Database via the mod() function. The following example calculates the remainder when 4 is divided into 10: mysql> SELECT MOD(10,4); +-----------+ | MOD(10,4) | +-----------+ | 2 | +-----------+ 1 row in set (0.02 sec) While the mod() function is typically used with integer arguments, with MySQL you can also use real numbers, as in: mysql> SELECT MOD(22.75, 5); +---------------+ | MOD(22.75, 5) | +---------------+ | 2.75 | +---------------+ 1 row in set (0.02 sec) NOTE SQL Server does not have a mod() function. Instead, the operator % is used for finding remainders. The expression 10 % 4 will therefore yield the value 2. Another numeric function that takes two numeric arguments is the pow() function (or power() if you are using Oracle Database or SQL Server), which returns one number raised to the power of a second number, as in: mysql> SELECT POW(2,8); +----------+ | POW(2,8) | +----------+ | 256 | +----------+ 1 row in set (0.03 sec) Thus, pow(2,8) is the MySQL equivalent of specifying 2 . Since computer memory is allocated in chunks of 2 bytes, the pow() function can be a handy way to determine the exact number of bytes in a certain amount of memory: mysql> SELECT POW(2,10) kilobyte, POW(2,20) megabyte, -> POW(2,30) gigabyte, POW(2,40) terabyte; +----------+----------+------------+---------------+ | kilobyte | megabyte | gigabyte | terabyte | +----------+----------+------------+---------------+ | 1024 | 1048576 | 1073741824 | 1099511627776 | +----------+----------+------------+---------------+ 1 row in set (0.00 sec) I don’t know about you, but I find it easier to remember that a gigabyte is 2 bytes than to remember the number 1,073,741,824. Controlling Number Precision When working with floating-point numbers, you may not always want to interact with or display a number with its full precision. For example, you may store monetary transaction data with a precision to six decimal places, but you might want to round to the nearest hundredth for display purposes. 8 x 30 Four functions are useful when limiting the precision of floating-point numbers: ceil(), floor(), round(), and truncate(). All three servers include these functions, although Oracle Database includes trunc() instead of truncate(), and SQL Server includes ceiling() instead of ceil(). The ceil() and floor() functions are used to round either up or down to the closest integer, as demonstrated by the following: mysql> SELECT CEIL(72.445), FLOOR(72.445); +--------------+---------------+ | CEIL(72.445) | FLOOR(72.445) | +--------------+---------------+ | 73 | 72 | +--------------+---------------+ 1 row in set (0.06 sec) Thus, any number between 72 and 73 will be evaluated as 73 by the ceil() function and 72 by the floor() function. Remember that ceil() will round up even if the decimal portion of a number is very small, and floor() will round down even if", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 93 + }, + { + "text": "in set (0.06 sec) Thus, any number between 72 and 73 will be evaluated as 73 by the ceil() function and 72 by the floor() function. Remember that ceil() will round up even if the decimal portion of a number is very small, and floor() will round down even if the decimal portion is quite significant, as in: mysql> SELECT CEIL(72.000000001), FLOOR(72.999999999); +--------------------+---------------------+ | CEIL(72.000000001) | FLOOR(72.999999999) | +--------------------+---------------------+ | 73 | 72 | +--------------------+---------------------+ 1 row in set (0.00 sec) If this is a bit too severe for your application, you can use the round() function to round up or down from the midpoint between two integers, as in: mysql> SELECT ROUND(72.49999), ROUND(72.5), ROUND(72.50001); +-----------------+-------------+-----------------+ | ROUND(72.49999) | ROUND(72.5) | ROUND(72.50001) | +-----------------+-------------+-----------------+ | 72 | 73 | 73 | +-----------------+-------------+-----------------+ 1 row in set (0.00 sec) Using round(), any number whose decimal portion is halfway or more between two integers will be rounded up, whereas the number will be rounded down if the decimal portion is anything less than halfway between the two integers. Most of the time, you will want to keep at least some part of the decimal portion of a number rather than rounding to the nearest integer; the round() function allows an optional second argument to specify how many digits to the right of the decimal place to round to. The next example shows how you can use the second argument to round the number 72.0909 to one, two, and three decimal places: mysql> SELECT ROUND(72.0909, 1), ROUND(72.0909, 2), ROUND(72.0909, 3); +-------------------+-------------------+-------------------+ | ROUND(72.0909, 1) | ROUND(72.0909, 2) | ROUND(72.0909, 3) | +-------------------+-------------------+-------------------+ | 72.1 | 72.09 | 72.091 | +-------------------+-------------------+-------------------+ 1 row in set (0.00 sec) Like the round() function, the truncate() function allows an optional second argument to specify the number of digits to the right of the decimal, but truncate() simply discards the unwanted digits without rounding. The next example shows how the number 72.0909 would be truncated to one, two, and three decimal places: mysql> SELECT TRUNCATE(72.0909, 1), TRUNCATE(72.0909, 2), -> TRUNCATE(72.0909, 3); +----------------------+----------------------+----------------- -----+ | TRUNCATE(72.0909, 1) | TRUNCATE(72.0909, 2) | TRUNCATE(72.0909, 3) | +----------------------+----------------------+----------------- -----+ | 72.0 | 72.09 | 72.090 | +----------------------+----------------------+----------------- -----+ 1 row in set (0.00 sec) NOTE SQL Server does not include a truncate() function. Instead, the round() function allows for an optional third argument which, if present and nonzero, calls for the number to be truncated rather than rounded. Both truncate() and round() also allow a negative value for the second argument, meaning that numbers to the left of the decimal place are truncated or rounded. This might seem like a strange thing to do at first, but there are valid applications. For example, you might sell a product that can be purchased only in units of 10. If a customer were to order 17 units, you could choose from one of the following methods to modify the customer’s order quantity: mysql> SELECT ROUND(17, −1), TRUNCATE(17, −1); +---------------+------------------+ | ROUND(17, −1) | TRUNCATE(17,", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 94 + }, + { + "text": "example, you might sell a product that can be purchased only in units of 10. If a customer were to order 17 units, you could choose from one of the following methods to modify the customer’s order quantity: mysql> SELECT ROUND(17, −1), TRUNCATE(17, −1); +---------------+------------------+ | ROUND(17, −1) | TRUNCATE(17, −1) | +---------------+------------------+ | 20 | 10 | +---------------+------------------+ 1 row in set (0.00 sec) If the product in question is thumbtacks, then it might not make much difference to your bottom line whether you sold the customer 10 or 20 thumbtacks when only 17 were requested; if you are selling Rolex watches, however, your business may fare better by rounding. Handling Signed Data If you are working with numeric columns that allow negative values (in Chapter 2, I showed how a numeric column may be labeled unsigned, meaning that only positive numbers are allowed), several numeric functions might be of use. Let’s say, for example, that you are asked to generate a report showing the current status of a set of bank accounts using the following data from the account table: +------------+--------------+---------+ | account_id | acct_type | balance | +------------+--------------+---------+ | 123 | MONEY MARKET | 785.22 | | 456 | SAVINGS | 0.00 | | 789 | CHECKING | -324.22 | +------------+--------------+---------+ The following query returns three columns useful for generating the report: mysql> SELECT account_id, SIGN(balance), ABS(balance) -> FROM account; +------------+---------------+--------------+ | account_id | SIGN(balance) | ABS(balance) | +------------+---------------+--------------+ | 123 | 1 | 785.22 | | 456 | 0 | 0.00 | | 789 | -1 | 324.22 | +------------+---------------+--------------+ 3 rows in set (0.00 sec) The second column uses the sign() function to return −1 if the account balance is negative, 0 if the account balance is zero, and 1 if the account balance is positive. The third column returns the absolute value of the account balance via the abs() function. Working with Temporal Data Of the three types of data discussed in this chapter (character, numeric, and temporal), temporal data is the most involved when it comes to data generation and manipulation. Some of the complexity of temporal data is caused by the myriad ways in which a single date and time can be described. For example, the date on which I wrote this paragraph can be described in all the following ways: Wednesday, June 5, 2019 6/05/2019 2:14:56 P.M. EST 6/05/2019 19:14:56 GMT 1562019 (Julian format) Star date [−4] 97026.79 14:14:56 (Star Trek format) While some of these differences are purely a matter of formatting, most of the complexity has to do with your frame of reference, which we explore in the next section. Dealing with Time Zones Because people around the world prefer that noon coincides roughly with the sun’s peak at their location, there has never been a serious attempt to coerce everyone to use a universal clock. Instead, the world has been sliced into 24 imaginary sections, called time zones; within a particular time zone, everyone agrees on the current", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 95 + }, + { + "text": "prefer that noon coincides roughly with the sun’s peak at their location, there has never been a serious attempt to coerce everyone to use a universal clock. Instead, the world has been sliced into 24 imaginary sections, called time zones; within a particular time zone, everyone agrees on the current time, whereas people in different time zones do not. While this seems simple enough, some geographic regions shift their time by one hour twice a year (implementing what is known as daylight saving time) and some do not, so the time difference between two points on Earth might be four hours for one half of the year and five hours for the other half of the year. Even within a single time zone, different regions may or may not adhere to daylight saving time, causing different clocks in the same time zone to agree for one half of the year but be one hour different for the rest of the year. While the computer age has exacerbated the issue, people have been dealing with time zone differences since the early days of naval exploration. To ensure a common point of reference for timekeeping, fifteenth-century navigators set their clocks to the time of day in Greenwich, England. This became known as Greenwich Mean Time, or GMT. All other time zones can be described by the number of hours’ difference from GMT; for example, the time zone for the Eastern United States, known as Eastern Standard Time, can be described as GMT −5:00, or five hours earlier than GMT. Today, we use a variation of GMT called Coordinated Universal Time, or UTC, which is based on an atomic clock (or, to be more precise, the average time of 200 atomic clocks in 50 locations worldwide, which is referred to as Universal Time). Both SQL Server and MySQL provide functions that will return the current UTC timestamp (getutcdate() for SQL Server and utc_timestamp() for MySQL). Most database servers default to the time zone setting of the server on which it resides and provide tools for modifying the time zone if needed. For example, a database used to store stock exchange transactions from around the world would generally be configured to use UTC time, whereas a database used to store transactions at a particular retail establishment might use the server’s time zone. MySQL keeps two different time zone settings: a global time zone, and a session time zone, which may be different for each user logged in to a database. You can see both settings via the following query: mysql> SELECT @@global.time_zone, @@session.time_zone; +--------------------+---------------------+ | @@global.time_zone | @@session.time_zone | +--------------------+---------------------+ | SYSTEM | SYSTEM | +--------------------+---------------------+ 1 row in set (0.00 sec) A value of system tells you that the server is using the time zone setting from the server on which the database resides. If you are sitting at a computer in Zurich, Switzerland, and you open a session across the network to a MySQL server situated in New York, you may want", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 96 + }, + { + "text": "of system tells you that the server is using the time zone setting from the server on which the database resides. If you are sitting at a computer in Zurich, Switzerland, and you open a session across the network to a MySQL server situated in New York, you may want to change the time zone setting for your session, which you can do via the following command: mysql> SET time_zone = 'Europe/Zurich'; Query OK, 0 rows affected (0.18 sec) If you check the time zone settings again, you will see the following: mysql> SELECT @@global.time_zone, @@session.time_zone; +--------------------+---------------------+ | @@global.time_zone | @@session.time_zone | +--------------------+---------------------+ | SYSTEM | Europe/Zurich | +--------------------+---------------------+ 1 row in set (0.00 sec) All dates displayed in your session will now conform to Zurich time. NOTE Oracle Database users can change the time zone setting for a session via the following command: ALTER SESSION TIMEZONE = 'Europe/Zurich' Generating Temporal Data You can generate temporal data via any of the following means: Copying data from an existing date, datetime, or time column Executing a built-in function that returns a date, datetime, or time Building a string representation of the temporal data to be evaluated by the server To use the last method, you will need to understand the various components used in formatting dates. STRING REPRESENTATIONS OF TEMPORAL DATA Table 2-4 in Chapter 2 presented the more popular date components; to refresh your memory, Table 7-2 shows these same components. Table 7-2. Date format components Component Definition Range YYYY Year, including century 1000 to 9999 MM Month 01 (January) to 12 (December) DD Day 01 to 31 HH Hour 00 to 23 HHH Hours (elapsed) −838 to 838 MI Minute 00 to 59 SS Second 00 to 59 To build a string that the server can interpret as a date, datetime, or time, you need to put the various components together in the order shown in Table 7-3. Table 7-3. Required date components Type Default format Date YYYY-MM-DD Datetime YYYY-MM-DD HH:MI:SS Timestamp YYYY-MM-DD HH:MI:SS Time HHH:MI:SS Thus, to populate a datetime column with 3:30 P.M. on September 17, 2019, you will need to build the following string: '2019-09-17 15:30:00' If the server is expecting a datetime value, such as when updating a datetime column or when calling a built-in function that takes a datetime argument, you can provide a properly formatted string with the required date components, and the server will do the conversion for you. For example, here’s a statement used to modify the return date of a film rental: UPDATE rental SET return_date = '2019-09-17 15:30:00' WHERE rental_id = 99999; The server determines that the string provided in the set clause must be a datetime value, since the string is being used to populate a datetime column. Therefore, the server will attempt to convert the string for you by parsing the string into the six components (year, month, day, hour, minute, second) included in the default datetime format. STRING-TO-DATE CONVERSIONS If the server is not expecting a", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 97 + }, + { + "text": "the string is being used to populate a datetime column. Therefore, the server will attempt to convert the string for you by parsing the string into the six components (year, month, day, hour, minute, second) included in the default datetime format. STRING-TO-DATE CONVERSIONS If the server is not expecting a datetime value, or if you would like to represent the datetime using a nondefault format, you will need to tell the server to convert the string to a datetime. For example, here is a simple query that returns a datetime value using the cast() function: mysql> SELECT CAST('2019-09-17 15:30:00' AS DATETIME); +-----------------------------------------+ | CAST('2019-09-17 15:30:00' AS DATETIME) | +-----------------------------------------+ | 2019-09-17 15:30:00 | +-----------------------------------------+ 1 row in set (0.00 sec) We cover the cast() function at the end of this chapter. While this example demonstrates how to build datetime values, the same logic applies to the date and time types as well. The following query uses the cast() function to generate a date value and a time value: mysql> SELECT CAST('2019-09-17' AS DATE) date_field, -> CAST('108:17:57' AS TIME) time_field; +------------+------------+ | date_field | time_field | +------------+------------+ | 2019-09-17 | 108:17:57 | +------------+------------+ 1 row in set (0.00 sec) You may, of course, explicitly convert your strings even when the server is expecting a date, datetime, or time value, rather than letting the server do an implicit conversion. When strings are converted to temporal values—whether explicitly or implicitly—you must provide all the date components in the required order. While some servers are quite strict regarding the date format, the MySQL server is quite lenient about the separators used between the components. For example, MySQL will accept all of the following strings as valid representations of 3:30 P.M. on September 17, 2019: '2019-09-17 15:30:00' '2019/09/17 15:30:00' '2019,09,17,15,30,00' '20190917153000' Although this gives you a bit more flexibility, you may find yourself trying to generate a temporal value without the default date components; the next section demonstrates a built-in function that is far more flexible than the cast() function. FUNCTIONS FOR GENERATING DATES If you need to generate temporal data from a string, and the string is not in the proper form to use the cast() function, you can use a built-in function that allows you to provide a format string along with the date string. MySQL includes the str_to_date() function for this purpose. Say, for example, that you pull the string 'September 17, 2019' from a file and need to use it to update a date column. Since the string is not in the required YYYY-MM-DD format, you can use str_to_date() instead of reformatting the string so that you can use the cast() function, as in: UPDATE rental SET return_date = STR_TO_DATE('September 17, 2019', '%M %d, %Y') WHERE rental_id = 99999; The second argument in the call to str_to_date() defines the format of the date string, with, in this case, a month name (%M), a numeric day (%d), and a four-digit numeric year (%Y). While there are over 30 recognized format components, Table", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 98 + }, + { + "text": "2019', '%M %d, %Y') WHERE rental_id = 99999; The second argument in the call to str_to_date() defines the format of the date string, with, in this case, a month name (%M), a numeric day (%d), and a four-digit numeric year (%Y). While there are over 30 recognized format components, Table 7-4 defines the dozen or so most commonly used components. Table 7-4. Date format components Format component Description %M Month name (January to December) %m Month numeric (01 to 12) %d Day numeric (01 to 31) %j Day of year (001 to 366) %W Weekday name (Sunday to Saturday) %Y Year, four-digit numeric %y Year, two-digit numeric %H Hour (00 to 23) %h Hour (01 to 12) %i Minutes (00 to 59) %s Seconds (00 to 59) %f Microseconds (000000 to 999999) %p A.M. or P.M. The str_to_date() function returns a datetime, date, or time value depending on the contents of the format string. For example, if the format string includes only %H, %i, and %s, then a time value will be returned. NOTE Oracle Database users can use the to_date() function in the same manner as MySQL’s str_to_date() function. SQL Server includes a convert() function that is not quite as flexible as MySQL and Oracle Database; rather than supplying a custom format string, your date string must conform to one of 21 predefined formats. If you are trying to generate the current date/time, then you won’t need to build a string, because the following built-in functions will access the system clock and return the current date and/or time as a string for you: mysql> SELECT CURRENT_DATE(), CURRENT_TIME(), CURRENT_TIMESTAMP(); +----------------+----------------+---------------------+ | CURRENT_DATE() | CURRENT_TIME() | CURRENT_TIMESTAMP() | +----------------+----------------+---------------------+ | 2019-06-05 | 16:54:36 | 2019-06-05 16:54:36 | +----------------+----------------+---------------------+ 1 row in set (0.12 sec) The values returned by these functions are in the default format for the temporal type being returned. Oracle Database includes current_date() and current_timestamp() but not current_time(), and SQL Server includes only the current_timestamp() function. Manipulating Temporal Data This section explores the built-in functions that take date arguments and return dates, strings, or numbers. TEMPORAL FUNCTIONS THAT RETURN DATES Many of the built-in temporal functions take one date as an argument and return another date. MySQL’s date_add() function, for example, allows you to add any kind of interval (e.g., days, months, years) to a specified date to generate another date. Here’s an example that demonstrates how to add five days to the current date: mysql> SELECT DATE_ADD(CURRENT_DATE(), INTERVAL 5 DAY); +------------------------------------------+ | DATE_ADD(CURRENT_DATE(), INTERVAL 5 DAY) | +------------------------------------------+ | 2019-06-10 | +------------------------------------------+ 1 row in set (0.06 sec) The second argument is composed of three elements: the interval keyword, the desired quantity, and the type of interval. Table 7-5 shows some of the commonly used interval types. Table 7-5. Common interval types Interval name Description Second Number of seconds Minute Number of minutes Hour Number of hours Day Number of days Month Number of months Year Number of years Minute_second Number of minutes and seconds, separated by “:”", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 99 + }, + { + "text": "7-5 shows some of the commonly used interval types. Table 7-5. Common interval types Interval name Description Second Number of seconds Minute Number of minutes Hour Number of hours Day Number of days Month Number of months Year Number of years Minute_second Number of minutes and seconds, separated by “:” Hour_second Number of hours, minutes, and seconds, separated by “:” Year_month Number of years and months, separated by “-” While the first six types listed in Table 7-5 are pretty straightforward, the last three types require a bit more explanation since they have multiple elements. For example, if you are told that a film was actually returned 3 hours, 27 minutes, and 11 seconds later than what was originally specified, you can fix it via the following: UPDATE rental SET return_date = DATE_ADD(return_date, INTERVAL '3:27:11' HOUR_SECOND) WHERE rental_id = 99999; In this example, the date_add() function takes the value in the return_date column, adds 3 hours, 27 minutes, and 11 seconds to it, and uses the value that results to modify the return_date column. Or, if you work in HR and found out that employee ID 4789 claimed to be older than he actually is, you could add 9 years and 11 months to his birth date, as in: UPDATE employee SET birth_date = DATE_ADD(birth_date, INTERVAL '9-11' YEAR_MONTH) WHERE emp_id = 4789; NOTE SQL Server users can accomplish the previous example using the dateadd() function: UPDATE employee SET birth_date = DATEADD(MONTH, 119, birth_date) WHERE emp_id = 4789 SQL Server doesn’t have combined intervals (i.e., year_month), so I converted 9 years, 11 months to 119 months. Oracle Database users can use the add_months() function for this example, as in: UPDATE employee SET birth_date = ADD_MONTHS(birth_date, 119) WHERE emp_id = 4789; There are some cases where you want to add an interval to a date, and you know where you want to arrive but not how many days it takes to get there. For example, let’s say that a bank customer logs on to the online banking system and schedules a transfer for the end of the month. Rather than writing some code that figures out the current month and then looks up the number of days in that month, you can call the last_day() function, which does the work for you (both MySQL and Oracle Database include the last_day() function; SQL Server has no comparable function). If the customer asks for the transfer on September 17, 2019, you could find the last day of September via the following: mysql> SELECT LAST_DAY('2019-09-17'); +------------------------+ | LAST_DAY('2019-09-17') | +------------------------+ | 2019-09-30 | +------------------------+ 1 row in set (0.10 sec) Whether you provide a date or datetime value, the last_day() function always returns a date. Although this function may not seem like an enormous timesaver, the underlying logic can be tricky if you’re trying to find the last day of February and need to figure out whether the current year is a leap year. TEMPORAL FUNCTIONS THAT RETURN STRINGS Most of the temporal functions that", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 100 + }, + { + "text": "Although this function may not seem like an enormous timesaver, the underlying logic can be tricky if you’re trying to find the last day of February and need to figure out whether the current year is a leap year. TEMPORAL FUNCTIONS THAT RETURN STRINGS Most of the temporal functions that return string values are used to extract a portion of a date or time. For example, MySQL includes the dayname() function to determine which day of the week a certain date falls on, as in: mysql> SELECT DAYNAME('2019-09-18'); +-----------------------+ | DAYNAME('2019-09-18') | +-----------------------+ | Wednesday | +-----------------------+ 1 row in set (0.00 sec) Many such functions are included with MySQL for extracting information from date values, but I recommend that you use the extract() function instead, since it’s easier to remember a few variations of one function than to remember a dozen different functions. Additionally, the extract() function is part of the SQL:2003 standard and has been implemented by Oracle Database as well as MySQL. The extract() function uses the same interval types as the date_add() function (see Table 7-5) to define which element of the date interests you. For example, if you want to extract just the year portion of a datetime value, you can do the following: mysql> SELECT EXTRACT(YEAR FROM '2019-09-18 22:19:05'); +------------------------------------------+ | EXTRACT(YEAR FROM '2019-09-18 22:19:05') | +------------------------------------------+ | 2019 | +------------------------------------------+ 1 row in set (0.00 sec) NOTE SQL Server doesn’t include an implementation of extract(), but it does include the datepart() function. Here’s how you would extract the year from a datetime value using datepart(): SELECT DATEPART(YEAR, GETDATE()) TEMPORAL FUNCTIONS THAT RETURN NUMBERS Earlier in this chapter, I showed you a function used to add a given interval to a date value, thus generating another date value. Another common activity when working with dates is to take two date values and determine the number of intervals (days, weeks, years) between the two dates. For this purpose, MySQL includes the function datediff(), which returns the number of full days between two dates. For example, if I want to know the number of days that my kids will be out of school this summer, I can do the following: mysql> SELECT DATEDIFF('2019-09-03', '2019-06-21'); +--------------------------------------+ | DATEDIFF('2019-09-03', '2019-06-21') | +--------------------------------------+ | 74 | +--------------------------------------+ 1 row in set (0.00 sec) Thus, I will have to endure 74 days of poison ivy, mosquito bites, and scraped knees before the kids are safely back at school. The datediff() function ignores the time of day in its arguments. Even if I include a time-of-day, setting it to one second until midnight for the first date and to one second after midnight for the second date, those times will have no effect on the calculation: mysql> SELECT DATEDIFF('2019-09-03 23:59:59', '2019-06-21 00:00:01'); +--------------------------------------------------------+ | DATEDIFF('2019-09-03 23:59:59', '2019-06-21 00:00:01') | +--------------------------------------------------------+ | 74 | +--------------------------------------------------------+ 1 row in set (0.00 sec) If I switch the arguments and have the earlier date first, datediff() will return a negative number, as in: mysql> SELECT", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 101 + }, + { + "text": "no effect on the calculation: mysql> SELECT DATEDIFF('2019-09-03 23:59:59', '2019-06-21 00:00:01'); +--------------------------------------------------------+ | DATEDIFF('2019-09-03 23:59:59', '2019-06-21 00:00:01') | +--------------------------------------------------------+ | 74 | +--------------------------------------------------------+ 1 row in set (0.00 sec) If I switch the arguments and have the earlier date first, datediff() will return a negative number, as in: mysql> SELECT DATEDIFF('2019-06-21', '2019-09-03'); +--------------------------------------+ | DATEDIFF('2019-06-21', '2019-09-03') | +--------------------------------------+ | -74 | +--------------------------------------+ 1 row in set (0.00 sec) NOTE SQL Server also includes the datediff() function, but it is more flexible than the MySQL implementation in that you can specify the interval type (i.e., year, month, day, hour) instead of counting only the number of days between two dates. Here’s how SQL Server would accomplish the previous example: SELECT DATEDIFF(DAY, '2019-06-21', '2019-09-03') Oracle Database allows you to determine the number of days between two dates simply by subtracting one date from another. Conversion Functions Earlier in this chapter, I showed you how to use the cast() function to convert a string to a datetime value. While every database server includes a number of proprietary functions used to convert data from one type to another, I recommend using the cast() function, which is included in the SQL:2003 standard and has been implemented by MySQL, Oracle Database, and Microsoft SQL Server. To use cast(), you provide a value or expression, the as keyword, and the type to which you want the value converted. Here’s an example that converts a string to an integer: mysql> SELECT CAST('1456328' AS SIGNED INTEGER); +-----------------------------------+ | CAST('1456328' AS SIGNED INTEGER) | +-----------------------------------+ | 1456328 | +-----------------------------------+ 1 row in set (0.01 sec) When converting a string to a number, the cast() function will attempt to convert the entire string from left to right; if any non-numeric characters are found in the string, the conversion halts without an error. Consider the following example: mysql> SELECT CAST('999ABC111' AS UNSIGNED INTEGER); +---------------------------------------+ | CAST('999ABC111' AS UNSIGNED INTEGER) | +---------------------------------------+ | 999 | +---------------------------------------+ 1 row in set, 1 warning (0.08 sec) mysql> show warnings; +---------+------+---------------------------------------------- --+ | Level | Code | Message | +---------+------+---------------------------------------------- --+ | Warning | 1292 | Truncated incorrect INTEGER value: '999ABC111' | +---------+------+---------------------------------------------- --+ 1 row in set (0.07 sec) In this case, the first three digits of the string are converted, whereas the rest of the string is discarded, resulting in a value of 999. The server did, however, issue a warning to let you know that not all the string was converted. If you are converting a string to a date, time, or datetime value, then you will need to stick with the default formats for each type, since you can’t provide the cast() function with a format string. If your date string is not in the default format (i.e., YYYY-MM-DD HH:MI:SS for datetime types), then you will need to resort to using another function, such as MySQL’s str_to_date() function described earlier in the chapter. Test Your Knowledge These exercises are designed to test your understanding of some of the built-in functions shown in", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 102 + }, + { + "text": "not in the default format (i.e., YYYY-MM-DD HH:MI:SS for datetime types), then you will need to resort to using another function, such as MySQL’s str_to_date() function described earlier in the chapter. Test Your Knowledge These exercises are designed to test your understanding of some of the built-in functions shown in this chapter. See Appendix C for the answers. Exercise 7-1 Write a query that returns the 17 through 25 characters of the string 'Please find the substring in this string'. Exercise 7-2 Write a query that returns the absolute value and sign (−1, 0, or 1) of the number −25.76823. Also return the number rounded to the nearest hundredth. Exercise 7-3 Write a query to return just the month portion of the current date. th th Chapter 8. Grouping and Aggregates Data is generally stored at the lowest level of granularity needed by any of a database’s users; if Chuck in accounting needs to look at individual customer transactions, then there needs to be a table in the database that stores individual transactions. That doesn’t mean, however, that all users must deal with the data as it is stored in the database. The focus of this chapter is on how data can be grouped and aggregated to allow users to interact with it at some higher level of granularity than what is stored in the database. Grouping Concepts Sometimes you will want to find trends in your data that will require the database server to cook the data a bit before you can generate the results you are looking for. For example, let’s say that you are in charge of sending coupons for free rentals to your best customers. You could issue a simple query to look at the raw data: mysql> SELECT customer_id FROM rental; +-------------+ | customer_id | +-------------+ | 1 | | 1 | | 1 | | 1 | | 1 | | 1 | | 1 | ... | 599 | | 599 | | 599 | | 599 | | 599 | | 599 | +-------------+ 16044 rows in set (0.01 sec) With 599 customers spanning over 16,000 rental records, it isn’t feasible to determine which customers have rented the most films by looking at the raw data. Instead, you can ask the database server to group the data for you by using the group by clause. Here’s the same query but employing a group by clause to group the rental data by customer ID: mysql> SELECT customer_id -> FROM rental -> GROUP BY customer_id; +-------------+ | customer_id | +-------------+ | 1 | | 2 | | 3 | | 4 | | 5 | | 6 | ... | 594 | | 595 | | 596 | | 597 | | 598 | | 599 | +-------------+ 599 rows in set (0.00 sec) The result set contains one row for each distinct value in the customer_id column, resulting in 599 rows instead of the full 16,044 rows. The reason for the smaller result", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 103 + }, + { + "text": "| | 596 | | 597 | | 598 | | 599 | +-------------+ 599 rows in set (0.00 sec) The result set contains one row for each distinct value in the customer_id column, resulting in 599 rows instead of the full 16,044 rows. The reason for the smaller result set is that some of the customers rented more than one film. To see how many films each customer opened, you can use an aggregate function in the select clause to count the number of rows in each group: mysql> SELECT customer_id, count(*) -> FROM rental -> GROUP BY customer_id; +-------------+----------+ | customer_id | count(*) | +-------------+----------+ | 1 | 32 | | 2 | 27 | | 3 | 26 | | 4 | 22 | | 5 | 38 | | 6 | 28 | ... | 594 | 27 | | 595 | 30 | | 596 | 28 | | 597 | 25 | | 598 | 22 | | 599 | 19 | +-------------+----------+ 599 rows in set (0.01 sec) The aggregate function count() counts the number of rows in each group, and the asterisk tells the server to count everything in the group. Using the combination of a group by clause and the count() aggregate function, you are able to generate exactly the data needed to answer the business question without having to look at the raw data. Looking at the results, you can see that 32 films were rented by customer ID 1, and 25 films were rented by the customer ID 597. In order to determine which customers have rented the most films, simply add anorder by clause: mysql> SELECT customer_id, count(*) -> FROM rental -> GROUP BY customer_id -> ORDER BY 2 DESC; +-------------+----------+ | customer_id | count(*) | +-------------+----------+ | 148 | 46 | | 526 | 45 | | 236 | 42 | | 144 | 42 | | 75 | 41 | ... | 248 | 15 | | 110 | 14 | | 281 | 14 | | 61 | 14 | | 318 | 12 | +-------------+----------+ 599 rows in set (0.01 sec) Now that the results are sorted, you can easily see that customer ID 148 has rented the most films (46), while customer ID 318 has rented the fewest films (12). When grouping data, you may need to filter out undesired data from your result set based on groups of data rather than based on the raw data. Since the group by clause runs after the where clause has been evaluated, you cannot add filter conditions to your where clause for this purpose. For example, here’s an attempt to filter out any customers who have rented fewer than 40 films: mysql> SELECT customer_id, count(*) -> FROM rental -> WHERE count(*) >= 40 -> GROUP BY customer_id; ERROR 1111 (HY000): Invalid use of group function You cannot refer to the aggregate function count(*) in your where clause, because the groups have not yet been generated", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 104 + }, + { + "text": "have rented fewer than 40 films: mysql> SELECT customer_id, count(*) -> FROM rental -> WHERE count(*) >= 40 -> GROUP BY customer_id; ERROR 1111 (HY000): Invalid use of group function You cannot refer to the aggregate function count(*) in your where clause, because the groups have not yet been generated at the time the where clause is evaluated. Instead, you must put your group filter conditions in the having clause. Here’s what the query would look like using having: mysql> SELECT customer_id, count(*) -> FROM rental -> GROUP BY customer_id -> HAVING count(*) >= 40; +-------------+----------+ | customer_id | count(*) | +-------------+----------+ | 75 | 41 | | 144 | 42 | | 148 | 46 | | 197 | 40 | | 236 | 42 | | 469 | 40 | | 526 | 45 | +-------------+----------+ 7 rows in set (0.01 sec) Because those groups containing fewer than 40 members have been filtered out via the having clause, the result set now contains only those customers who have rented 40 or more films. Aggregate Functions Aggregate functions perform a specific operation over all rows in a group. Although every database server has its own set of specialty aggregate functions, the common aggregate functions implemented by all major servers include: Max() Returns the maximum value within a set Min() Returns the minimum value within a set Avg() Returns the average value across a set Sum() Returns the sum of the values across a set Count() Returns the number of values in a set Here’s a query that uses all of the common aggregate functions to analyze the data on film rental payments: mysql> SELECT MAX(amount) max_amt, -> MIN(amount) min_amt, -> AVG(amount) avg_amt, -> SUM(amount) tot_amt, -> COUNT(*) num_payments -> FROM payment; +---------+---------+----------+----------+--------------+ | max_amt | min_amt | avg_amt | tot_amt | num_payments | +---------+---------+----------+----------+--------------+ | 11.99 | 0.00 | 4.200667 | 67416.51 | 16049 | +---------+---------+----------+----------+--------------+ 1 row in set (0.09 sec) The results from this query tell you that, across the 16,049 rows in the payment table, the maximum amount paid to rent a film was $11.99, the minimum amount was $0, the average payment was $4.20, and the total of all rental payments was $67,416.51. Hopefully, this gives you an appreciation for the role of these aggregate functions; the next subsections further clarify how you can utilize these functions. Implicit Versus Explicit Groups In the previous example, every value returned by the query is generated by an aggregate function. Since there is no group by clause, there is a single, implicit group (all rows in the payment table). In most cases, however, you will want to retrieve additional columns along with columns generated by aggregate functions. What if, for example, you wanted to extend the previous query to execute the same five aggregate functions for each customer, instead of across all customers? For this query, you would want to retrieve the customer_id column along with the five aggregate functions, as in: SELECT customer_id, MAX(amount) max_amt, MIN(amount) min_amt,", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 105 + }, + { + "text": "if, for example, you wanted to extend the previous query to execute the same five aggregate functions for each customer, instead of across all customers? For this query, you would want to retrieve the customer_id column along with the five aggregate functions, as in: SELECT customer_id, MAX(amount) max_amt, MIN(amount) min_amt, AVG(amount) avg_amt, SUM(amount) tot_amt, COUNT(*) num_payments FROM payment; However, if you try to execute the query, you will receive the following error: ERROR 1140 (42000): In aggregated query without GROUP BY, expression #1 of SELECT list contains nonaggregated column 'sakila.payment.customer_id'; this is incompatible with sql_mode=only_full_group_by While it may be obvious to you that you want the aggregate functions applied to each customer found in the payment table, this query fails because you have not explicitly specified how the data should be grouped. Therefore, you will need to add a group by clause to specify over which group of rows the aggregate functions should be applied: mysql> SELECT customer_id, -> MAX(amount) max_amt, -> MIN(amount) min_amt, -> AVG(amount) avg_amt, -> SUM(amount) tot_amt, -> COUNT(*) num_payments -> FROM payment -> GROUP BY customer_id; +-------------+---------+---------+----------+---------+-------- ------+ | customer_id | max_amt | min_amt | avg_amt | tot_amt | num_payments | +-------------+---------+---------+----------+---------+-------- ------+ | 1 | 9.99 | 0.99 | 3.708750 | 118.68 | 32 | | 2 | 10.99 | 0.99 | 4.767778 | 128.73 | 27 | | 3 | 10.99 | 0.99 | 5.220769 | 135.74 | 26 | | 4 | 8.99 | 0.99 | 3.717273 | 81.78 | 22 | | 5 | 9.99 | 0.99 | 3.805789 | 144.62 | 38 | | 6 | 7.99 | 0.99 | 3.347143 | 93.72 | 28 | ... | 594 | 8.99 | 0.99 | 4.841852 | 130.73 | 27 | | 595 | 10.99 | 0.99 | 3.923333 | 117.70 | 30 | | 596 | 6.99 | 0.99 | 3.454286 | 96.72 | 28 | | 597 | 8.99 | 0.99 | 3.990000 | 99.75 | 25 | | 598 | 7.99 | 0.99 | 3.808182 | 83.78 | 22 | | 599 | 9.99 | 0.99 | 4.411053 | 83.81 | 19 | +-------------+---------+---------+----------+---------+-------- ------+ 599 rows in set (0.04 sec) With the inclusion of the group by clause, the server knows to group together rows having the same value in the customer_id column first and then to apply the five aggregate functions to each of the 599 groups. Counting Distinct Values When using the count() function to determine the number of members in each group, you have your choice of counting all members in the group, or counting only the distinct values for a column across all members of the group. For example, consider the following query, which uses the count() function with the customer_id column in two different ways: mysql> SELECT COUNT(customer_id) num_rows, -> COUNT(DISTINCT customer_id) num_customers -> FROM payment; +----------+---------------+ | num_rows | num_customers | +----------+---------------+ | 16049 | 599 | +----------+---------------+ 1 row in set (0.01 sec) The first column in the query simply counts", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 106 + }, + { + "text": "uses the count() function with the customer_id column in two different ways: mysql> SELECT COUNT(customer_id) num_rows, -> COUNT(DISTINCT customer_id) num_customers -> FROM payment; +----------+---------------+ | num_rows | num_customers | +----------+---------------+ | 16049 | 599 | +----------+---------------+ 1 row in set (0.01 sec) The first column in the query simply counts the number of rows in the payment table, whereas the second column examines the values in the customer_id column and counts only the number of unique values. By specifying distinct, therefore, the count() function examines the values of a column for each member of the group in order to find and remove duplicates, rather than simply counting the number of values in the group. Using Expressions Along with using columns as arguments to aggregate functions, you can use expressions as well. For example, you may want to find the maximum number of days between when a film was rented and subsequently returned. You can achieve this via the following query: mysql> SELECT MAX(datediff(return_date,rental_date)) -> FROM rental; +----------------------------------------+ | MAX(datediff(return_date,rental_date)) | +----------------------------------------+ | 33 | +----------------------------------------+ 1 row in set (0.01 sec) The datediff function is used to compute the number of days between the return date and the rental date for every rental, and the max function returns the highest value, which in this case is 33 days. While this example uses a fairly simple expression, expressions used as arguments to aggregate functions can be as complex as needed, as long as they return a number, string, or date. In Chapter 11, I show you how you can use case expressions with aggregate functions to determine whether a particular row should or should not be included in an aggregation. How Nulls Are Handled When performing aggregations, or, indeed, any type of numeric calculation, you should always consider how null values might affect the outcome of your calculation. To illustrate, I will build a simple table to hold numeric data and populate it with the set {1, 3, 5}: mysql> CREATE TABLE number_tbl -> (val SMALLINT); Query OK, 0 rows affected (0.01 sec) mysql> INSERT INTO number_tbl VALUES (1); Query OK, 1 row affected (0.00 sec) mysql> INSERT INTO number_tbl VALUES (3); Query OK, 1 row affected (0.00 sec) mysql> INSERT INTO number_tbl VALUES (5); Query OK, 1 row affected (0.00 sec) Consider the following query, which performs five aggregate functions on the set of numbers: mysql> SELECT COUNT(*) num_rows, -> COUNT(val) num_vals, -> SUM(val) total, -> MAX(val) max_val, -> AVG(val) avg_val -> FROM number_tbl; +----------+----------+-------+---------+---------+ | num_rows | num_vals | total | max_val | avg_val | +----------+----------+-------+---------+---------+ | 3 | 3 | 9 | 5 | 3.0000 | +----------+----------+-------+---------+---------+ 1 row in set (0.08 sec) The results are as you would expect: both count(*) and count(val) return the value 3, sum(val) returns the value 9, max(val) returns 5, and avg(val) returns 3. Next, I will add a null value to the number_tbl table and run the query again: mysql> INSERT INTO number_tbl VALUES (NULL); Query OK, 1 row affected (0.01", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 107 + }, + { + "text": "would expect: both count(*) and count(val) return the value 3, sum(val) returns the value 9, max(val) returns 5, and avg(val) returns 3. Next, I will add a null value to the number_tbl table and run the query again: mysql> INSERT INTO number_tbl VALUES (NULL); Query OK, 1 row affected (0.01 sec) mysql> SELECT COUNT(*) num_rows, -> COUNT(val) num_vals, -> SUM(val) total, -> MAX(val) max_val, -> AVG(val) avg_val -> FROM number_tbl; +----------+----------+-------+---------+---------+ | num_rows | num_vals | total | max_val | avg_val | +----------+----------+-------+---------+---------+ | 4 | 3 | 9 | 5 | 3.0000 | +----------+----------+-------+---------+---------+ 1 row in set (0.00 sec) Even with the addition of the null value to the table, the sum(), max(), and avg() functions all return the same values, indicating that they ignore any null values encountered. The count(*) function now returns the value 4, which is valid since the number_tbl table contains four rows, while the count(val) function still returns the value 3. The difference is that count(*) counts the number of rows, whereas count(val) counts the number of values contained in the val column and ignores any null values encountered. Generating Groups People are rarely interested in looking at raw data; instead, people engaging in data analysis will want to manipulate the raw data to better suit their needs. Examples of common data manipulations include: Generating totals for a geographic region, such as total European sales Finding outliers, such as the top salesperson for 2020 Determining frequencies, such as the number of films rented in each month To answer these types of queries, you will need to ask the database server to group rows together by one or more columns or expressions. As you have seen already in several examples, the group by clause is the mechanism for grouping data within a query. In this section, you will see how to group data by one or more columns, how to group data using expressions, and how to generate rollups within groups. Single-Column Grouping Single-column groups are the simplest and most-often-used type of grouping. If you want to find the number of films associated with each actor, for example, you need only group on the film_actor.actor_id column, as in: mysql> SELECT actor_id, count(*) -> FROM film_actor -> GROUP BY actor_id; +----------+----------+ | actor_id | count(*) | +----------+----------+ | 1 | 19 | | 2 | 25 | | 3 | 22 | | 4 | 22 | ... | 197 | 33 | | 198 | 40 | | 199 | 15 | | 200 | 20 | +----------+----------+ 200 rows in set (0.11 sec) This query generates 200 groups, one for each actor, and then sums the number of films for each member of the group. Multicolumn Grouping In some cases, you may want to generate groups that span more than one column. Expanding on the previous example, imagine that you want to find the total number of films for each film rating (G, PG, …) for each actor. The following example shows how", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 108 + }, + { + "text": "the group. Multicolumn Grouping In some cases, you may want to generate groups that span more than one column. Expanding on the previous example, imagine that you want to find the total number of films for each film rating (G, PG, …) for each actor. The following example shows how you can accomplish this: mysql> SELECT fa.actor_id, f.rating, count(*) -> FROM film_actor fa -> INNER JOIN film f -> ON fa.film_id = f.film_id -> GROUP BY fa.actor_id, f.rating -> ORDER BY 1,2; +----------+--------+----------+ | actor_id | rating | count(*) | +----------+--------+----------+ | 1 | G | 4 | | 1 | PG | 6 | | 1 | PG-13 | 1 | | 1 | R | 3 | | 1 | NC-17 | 5 | | 2 | G | 7 | | 2 | PG | 6 | | 2 | PG-13 | 2 | | 2 | R | 2 | | 2 | NC-17 | 8 | ... | 199 | G | 3 | | 199 | PG | 4 | | 199 | PG-13 | 4 | | 199 | R | 2 | | 199 | NC-17 | 2 | | 200 | G | 5 | | 200 | PG | 3 | | 200 | PG-13 | 2 | | 200 | R | 6 | | 200 | NC-17 | 4 | +----------+--------+----------+ 996 rows in set (0.01 sec) This version of the query generates 996 groups, one for each combination of actor and film rating found by joining the film_actor table with the film table. Along with adding the rating column to the select clause, I also added it to the group by clause, since rating is retrieved from a table and is not generated via an aggregate function such as max or count. Grouping via Expressions Along with using columns to group data, you can build groups based on the values generated by expressions. Consider the following query, which groups rentals by year: mysql> SELECT extract(YEAR FROM rental_date) year, -> COUNT(*) how_many -> FROM rental -> GROUP BY extract(YEAR FROM rental_date); +------+----------+ | year | how_many | +------+----------+ | 2005 | 15862 | | 2006 | 182 | +------+----------+ 2 rows in set (0.01 sec) This query employs a fairly simple expression, which uses the extract() function to return only the year portion of a date, to group the rows in the rental table. Generating Rollups In “Multicolumn Grouping”, I showed an example that counts the number films for each actor and film rating. Let’s say, however, that along with the total count for each actor/rating combination, you also want total counts for each distinct actor. You could run an additional query and merge the results, you could load the results of the query into a spreadsheet, or you could build a Python script, Java program, or some other mechanism to take that data and perform the additional calculations. Better yet, you could use the with rollup option", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 109 + }, + { + "text": "run an additional query and merge the results, you could load the results of the query into a spreadsheet, or you could build a Python script, Java program, or some other mechanism to take that data and perform the additional calculations. Better yet, you could use the with rollup option to have the database server do the work for you. Here’s the revised query using with rollup in the group by clause: mysql> SELECT fa.actor_id, f.rating, count(*) -> FROM film_actor fa -> INNER JOIN film f -> ON fa.film_id = f.film_id -> GROUP BY fa.actor_id, f.rating WITH ROLLUP -> ORDER BY 1,2; +----------+--------+----------+ | actor_id | rating | count(*) | +----------+--------+----------+ | NULL | NULL | 5462 | | 1 | NULL | 19 | | 1 | G | 4 | | 1 | PG | 6 | | 1 | PG-13 | 1 | | 1 | R | 3 | | 1 | NC-17 | 5 | | 2 | NULL | 25 | | 2 | G | 7 | | 2 | PG | 6 | | 2 | PG-13 | 2 | | 2 | R | 2 | | 2 | NC-17 | 8 | ... | 199 | NULL | 15 | | 199 | G | 3 | | 199 | PG | 4 | | 199 | PG-13 | 4 | | 199 | R | 2 | | 199 | NC-17 | 2 | | 200 | NULL | 20 | | 200 | G | 5 | | 200 | PG | 3 | | 200 | PG-13 | 2 | | 200 | R | 6 | | 200 | NC-17 | 4 | +----------+--------+----------+ 1197 rows in set (0.07 sec) There are now 201 additional rows in the result set, one for each of the 200 distinct actors and one for the grand total (all actors combined). For the 200 actor rollups, a null value is provided for the rating column, since the rollup is being performed across all ratings. Looking at the first line for actor_id 200, for example, you will see that a total of 20 films are associated with the actor; this equals the sum of the counts for each rating (4 NC-17 + 6 R + 2 PG-13 + 3 PG + 5 G). For the grand total row in the first line of the output, a null value is provided for both the actor_id and rating columns; the total for the first line of output equals 5,462, which is equal to the number of rows in the film_actor table. NOTE If you are using Oracle Database, you need to use a slightly different syntax to indicate that you want a rollup performed. The group by clause for the previous query would look as follows when using Oracle: GROUP BY ROLLUP(fa.actor_id, f.rating) The advantage of this syntax is that it allows you to perform rollups on a subset of the columns in", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 110 + }, + { + "text": "slightly different syntax to indicate that you want a rollup performed. The group by clause for the previous query would look as follows when using Oracle: GROUP BY ROLLUP(fa.actor_id, f.rating) The advantage of this syntax is that it allows you to perform rollups on a subset of the columns in the group_by clause. If you are grouping by columns a, b, and c, for example, you could indicate that the server should perform rollups on only b and c via the following: GROUP BY a, ROLLUP(b, c) If, along with totals by actor, you also want to calculate totals per rating, then you can use the with cube option, which generates summary rows for all possible combinations of the grouping columns. Unfortunately, with cube is not available in version 8.0 of MySQL, but it is available with SQL Server and Oracle Database. Group Filter Conditions In Chapter 4, I introduced you to various types of filter conditions and showed how you can use them in the where clause. When grouping data, you also can apply filter conditions to the data after the groups have been generated. The having clause is where you should place these types of filter conditions. Consider the following example: mysql> SELECT fa.actor_id, f.rating, count(*) -> FROM film_actor fa -> INNER JOIN film f -> ON fa.film_id = f.film_id -> WHERE f.rating IN ('G','PG') -> GROUP BY fa.actor_id, f.rating -> HAVING count(*) > 9; +----------+--------+----------+ | actor_id | rating | count(*) | +----------+--------+----------+ | 137 | PG | 10 | | 37 | PG | 12 | | 180 | PG | 12 | | 7 | G | 10 | | 83 | G | 14 | | 129 | G | 12 | | 111 | PG | 15 | | 44 | PG | 12 | | 26 | PG | 11 | | 92 | PG | 12 | | 17 | G | 12 | | 158 | PG | 10 | | 147 | PG | 10 | | 14 | G | 10 | | 102 | PG | 11 | | 133 | PG | 10 | +----------+--------+----------+ 16 rows in set (0.01 sec) This query has two filter conditions: one in the where clause, which filters out any films rated something other than G or PG, and another in the having clause, which filters out any actors who appeared in less than 10 films. Thus, one of the filters acts on data before it is grouped, and the other filter acts on data after the groups have been created. If you mistakenly put both filters in the where clause, you will see the following error: mysql> SELECT fa.actor_id, f.rating, count(*) -> FROM film_actor fa -> INNER JOIN film f -> ON fa.film_id = f.film_id -> WHERE f.rating IN ('G','PG') -> AND count(*) > 9 -> GROUP BY fa.actor_id, f.rating; ERROR 1111 (HY000): Invalid use of group function This query fails because you cannot include an aggregate function in", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 111 + }, + { + "text": "f.rating, count(*) -> FROM film_actor fa -> INNER JOIN film f -> ON fa.film_id = f.film_id -> WHERE f.rating IN ('G','PG') -> AND count(*) > 9 -> GROUP BY fa.actor_id, f.rating; ERROR 1111 (HY000): Invalid use of group function This query fails because you cannot include an aggregate function in a query’s where clause. This is because the filters in the where clause are evaluated before the grouping occurs, so the server can’t yet perform any functions on groups. WARNING When adding filters to a query that includes a group by clause, think carefully about whether the filter acts on raw data, in which case it belongs in the where clause, or on grouped data, in which case it belongs in the having clause. Test Your Knowledge Work through the following exercises to test your grasp of SQL’s grouping and aggregating features. Check your work with the answers in Appendix C. Exercise 8-1 Construct a query that counts the number of rows in the payment table. Exercise 8-2 Modify your query from Exercise 8-1 to count the number of payments made by each customer. Show the customer ID and the total amount paid for each customer. Exercise 8-3 Modify your query from Exercise 8-2 to include only those customers having made at least five payments. Chapter 9. Subqueries Subqueries are a powerful tool that you can use in all four SQL data statements. In this chapter, I’ll show you how subqueries can be used to filter data, generate values, and construct temporary data sets. After a little experimentation, I think you’ll agree that subqueries are one of the most powerful features of the SQL language. What Is a Subquery? A subquery is a query contained within another SQL statement (which I refer to as the containing statement for the rest of this discussion). A subquery is always enclosed within parentheses, and it is usually executed prior to the containing statement. Like any query, a subquery returns a result set that may consist of: A single row with a single column Multiple rows with a single column Multiple rows having multiple columns The type of result set returned by the subquery determines how it may be used and which operators the containing statement may use to interact with the data the subquery returns. When the containing statement has finished executing, the data returned by any subqueries is discarded, making a subquery act like a temporary table with statement scope (meaning that the server frees up any memory allocated to the subquery results after the SQL statement has finished execution). You already saw several examples of subqueries in earlier chapters, but here’s a simple example to get started: mysql> SELECT customer_id, first_name, last_name -> FROM customer -> WHERE customer_id = (SELECT MAX(customer_id) FROM customer); +-------------+------------+-----------+ | customer_id | first_name | last_name | +-------------+------------+-----------+ | 599 | AUSTIN | CINTRON | +-------------+------------+-----------+ 1 row in set (0.27 sec) In this example, the subquery returns the maximum value found in the customer_id column in", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 112 + }, + { + "text": "last_name -> FROM customer -> WHERE customer_id = (SELECT MAX(customer_id) FROM customer); +-------------+------------+-----------+ | customer_id | first_name | last_name | +-------------+------------+-----------+ | 599 | AUSTIN | CINTRON | +-------------+------------+-----------+ 1 row in set (0.27 sec) In this example, the subquery returns the maximum value found in the customer_id column in the customer table, and the containing statement then returns data about that customer. If you are ever confused about what a subquery is doing, you can run the subquery by itself (without the parentheses) to see what it returns. Here’s the subquery from the previous example: mysql> SELECT MAX(customer_id) FROM customer; +------------------+ | MAX(customer_id) | +------------------+ | 599 | +------------------+ 1 row in set (0.00 sec) The subquery returns a single row with a single column, which allows it to be used as one of the expressions in an equality condition (if the subquery returned two or more rows, it could be compared to something but could not be equal to anything, but more on this later). In this case, you can take the value the subquery returned and substitute it into the righthand expression of the filter condition in the containing query, as in: mysql> SELECT customer_id, first_name, last_name -> FROM customer -> WHERE customer_id = 599; +-------------+------------+-----------+ | customer_id | first_name | last_name | +-------------+------------+-----------+ | 599 | AUSTIN | CINTRON | +-------------+------------+-----------+ 1 row in set (0.00 sec) The subquery is useful in this case because it allows you to retrieve information about the customer with the highest ID in a single query, rather than retrieving the maximum customer_id using one query and then writing a second query to retrieve the desired data from the customer table. As you will see, subqueries are useful in many other situations as well, and may become one of the most powerful tools in your SQL toolkit. Subquery Types Along with the differences noted previously regarding the type of result set returned by a subquery (single row/column, single row/multicolumn, or multiple columns), you can use another feature to differentiate subqueries; some subqueries are completely self-contained (called noncorrelated subqueries), while others reference columns from the containing statement (called correlated subqueries). The next several sections explore these two subquery types and show the different operators that you can employ to interact with them. Noncorrelated Subqueries The example from earlier in the chapter is a noncorrelated subquery; it may be executed alone and does not reference anything from the containing statement. Most subqueries that you encounter will be of this type unless you are writing update or delete statements, which frequently make use of correlated subqueries (more on this later). Along with being noncorrelated, the example from earlier in the chapter also returns a result set containing a single row and column. This type of subquery is known as a scalar subquery and can appear on either side of a condition using the usual operators (=, <>, <, >, <=, >=). The next example shows how you can use a scalar subquery in an", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 113 + }, + { + "text": "result set containing a single row and column. This type of subquery is known as a scalar subquery and can appear on either side of a condition using the usual operators (=, <>, <, >, <=, >=). The next example shows how you can use a scalar subquery in an inequality condition: mysql> SELECT city_id, city -> FROM city -> WHERE country_id <> -> (SELECT country_id FROM country WHERE country = 'India'); +---------+----------------------------+ | city_id | city | +---------+----------------------------+ | 1 | A Corua (La Corua) | | 2 | Abha | | 3 | Abu Dhabi | | 4 | Acua | | 5 | Adana | | 6 | Addis Abeba | ... | 595 | Zapopan | | 596 | Zaria | | 597 | Zeleznogorsk | | 598 | Zhezqazghan | | 599 | Zhoushan | | 600 | Ziguinchor | +---------+----------------------------+ 540 rows in set (0.02 sec) This query returns all cities which are not in India. The subquery, which is found on the last line of the statement, returns the country ID for India, and the containing query returns all cities which do not have that country ID. While the subquery in this example is quite simple, subqueries may be as complex as you need them to be, and they may utilize any and all the available query clauses (select, from, where, group by, having, and order by). If you use a subquery in an equality condition, but the subquery returns more than one row, you will receive an error. For example, if you modify the previous query such that the subquery returns all countries except for India, you will receive the following error: mysql> SELECT city_id, city -> FROM city -> WHERE country_id <> -> (SELECT country_id FROM country WHERE country <> 'India'); ERROR 1242 (21000): Subquery returns more than 1 row If you run the subquery by itself, you will see the following results: mysql> SELECT country_id FROM country WHERE country <> 'India'; +------------+ | country_id | +------------+ | 1 | | 2 | | 3 | | 4 | ... | 106 | | 107 | | 108 | | 109 | +------------+ 108 rows in set (0.00 sec) The containing query fails because an expression (country_id) cannot be equated to a set of expressions (country_ids 1, 2, 3, …, 109). In other words, a single thing cannot be equated to a set of things. In the next section, you will see how to fix the problem by using a different operator. Multiple-Row, Single-Column Subqueries If your subquery returns more than one row, you will not be able to use it on one side of an equality condition, as the previous example demonstrated. However, there are four additional operators that you can use to build conditions with these types of subqueries. THE IN AND NOT IN OPERATORS While you can’t equate a single value to a set of values, you can check to see whether a single value can be found", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 114 + }, + { + "text": "example demonstrated. However, there are four additional operators that you can use to build conditions with these types of subqueries. THE IN AND NOT IN OPERATORS While you can’t equate a single value to a set of values, you can check to see whether a single value can be found within a set of values. The next example, while it doesn’t use a subquery, demonstrates how to build a condition that uses the in operator to search for a value within a set of values: mysql> SELECT country_id -> FROM country -> WHERE country IN ('Canada','Mexico'); +------------+ | country_id | +------------+ | 20 | | 60 | +------------+ 2 rows in set (0.00 sec) The expression on the lefthand side of the condition is the country column, while the righthand side of the condition is a set of strings. The in operator checks to see whether either of the strings can be found in the country column; if so, the condition is met and the row is added to the result set. You could achieve the same results using two equality conditions, as in: mysql> SELECT country_id -> FROM country -> WHERE country = 'Canada' OR country = 'Mexico'; +------------+ | country_id | +------------+ | 20 | | 60 | +------------+ 2 rows in set (0.00 sec) While this approach seems reasonable when the set contains only two expressions, it is easy to see why a single condition using the in operator would be preferable if the set contained dozens (or hundreds, thousands, etc.) of values. Although you will occasionally create a set of strings, dates, or numbers to use on one side of a condition, you are more likely to generate the set using a subquery that returns one or more rows. The following query uses the in operator with a subquery on the righthand side of the filter condition to return all cities which are in Canada or Mexico: mysql> SELECT city_id, city -> FROM city -> WHERE country_id IN -> (SELECT country_id -> FROM country -> WHERE country IN ('Canada','Mexico')); +---------+----------------------------+ | city_id | city | +---------+----------------------------+ | 179 | Gatineau | | 196 | Halifax | | 300 | Lethbridge | | 313 | London | | 383 | Oshawa | | 430 | Richmond Hill | | 565 | Vancouver | ... | 452 | San Juan Bautista Tuxtepec | | 541 | Torren | | 556 | Uruapan | | 563 | Valle de Santiago | | 595 | Zapopan | +---------+----------------------------+ 37 rows in set (0.00 sec) Along with seeing whether a value exists within a set of values, you can check the converse using the not in operator. Here’s another version of the previous query using not in instead of in: mysql> SELECT city_id, city -> FROM city -> WHERE country_id NOT IN -> (SELECT country_id -> FROM country -> WHERE country IN ('Canada','Mexico')); +---------+----------------------------+ | city_id | city | +---------+----------------------------+ | 1 | A Corua (La Corua) | | 2 |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 115 + }, + { + "text": "the previous query using not in instead of in: mysql> SELECT city_id, city -> FROM city -> WHERE country_id NOT IN -> (SELECT country_id -> FROM country -> WHERE country IN ('Canada','Mexico')); +---------+----------------------------+ | city_id | city | +---------+----------------------------+ | 1 | A Corua (La Corua) | | 2 | Abha | | 3 | Abu Dhabi | | 5 | Adana | | 6 | Addis Abeba | ... | 596 | Zaria | | 597 | Zeleznogorsk | | 598 | Zhezqazghan | | 599 | Zhoushan | | 600 | Ziguinchor | +---------+----------------------------+ 563 rows in set (0.00 sec) This query finds all cities which are not in Canada or Mexico. THE ALL OPERATOR While the in operator is used to see whether an expression can be found within a set of expressions, the all operator allows you to make comparisons between a single value and every value in a set. To build such a condition, you will need to use one of the comparison operators (=, <>, <, >, etc.) in conjunction with the all operator. For example, the next query finds all customers who have never gotten a free film rental: mysql> SELECT first_name, last_name -> FROM customer -> WHERE customer_id <> ALL -> (SELECT customer_id -> FROM payment -> WHERE amount = 0); +-------------+--------------+ | first_name | last_name | +-------------+--------------+ | MARY | SMITH | | PATRICIA | JOHNSON | | LINDA | WILLIAMS | | BARBARA | JONES | ... | EDUARDO | HIATT | | TERRENCE | GUNDERSON | | ENRIQUE | FORSYTHE | | FREDDIE | DUGGAN | | WADE | DELVALLE | | AUSTIN | CINTRON | +-------------+--------------+ 576 rows in set (0.01 sec) The subquery returns the set of IDs for customers who have paid $0 for a film rental, and the containing query returns the names of all customers whose ID is not in the set returned by the subquery. If this approach seems a bit clumsy to you, you are in good company; most people would prefer to phrase the query differently and avoid using the all operator. To illustrate, the previous query generates the same results as the next example, which uses the not in operator: SELECT first_name, last_name FROM customer WHERE customer_id NOT IN (SELECT customer_id FROM payment WHERE amount = 0) It’s a matter of preference, but I think that most people would find the version that uses not in to be easier to understand. NOTE When using not in or <> all to compare a value to a set of values, you must be careful to ensure that the set of values does not contain a null value, because the server equates the value on the lefthand side of the expression to each member of the set, and any attempt to equate a value to null yields unknown. Thus, the following query returns an empty set: mysql> SELECT first_name, last_name -> FROM customer -> WHERE customer_id NOT IN (122, 452, NULL); Empty set", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 116 + }, + { + "text": "value on the lefthand side of the expression to each member of the set, and any attempt to equate a value to null yields unknown. Thus, the following query returns an empty set: mysql> SELECT first_name, last_name -> FROM customer -> WHERE customer_id NOT IN (122, 452, NULL); Empty set (0.00 sec) Here’s another example using the all operator, but this time the subquery is in the having clause: mysql> SELECT customer_id, count(*) -> FROM rental -> GROUP BY customer_id -> HAVING count(*) > ALL -> (SELECT count(*) -> FROM rental r -> INNER JOIN customer c -> ON r.customer_id = c.customer_id -> INNER JOIN address a -> ON c.address_id = a.address_id -> INNER JOIN city ct -> ON a.city_id = ct.city_id -> INNER JOIN country co -> ON ct.country_id = co.country_id -> WHERE co.country IN ('United States','Mexico','Canada') -> GROUP BY r.customer_id -> ); +-------------+----------+ | customer_id | count(*) | +-------------+----------+ | 148 | 46 | +-------------+----------+ 1 row in set (0.01 sec) The subquery in this example returns the total number of film rentals for all customers in North America, and the containing query returns all customers whose total number of film rentals exceeds any of the North American customers. THE ANY OPERATOR Like the all operator, the any operator allows a value to be compared to the members of a set of values; unlike all, however, a condition using the any operator evaluates to true as soon as a single comparison is favorable. This example will find all customers whose total film rental payments exceed the total payments for all customers in Bolivia, Paraguay, or Chile: mysql> SELECT customer_id, sum(amount) -> FROM payment -> GROUP BY customer_id -> HAVING sum(amount) > ANY -> (SELECT sum(p.amount) -> FROM payment p -> INNER JOIN customer c -> ON p.customer_id = c.customer_id -> INNER JOIN address a -> ON c.address_id = a.address_id -> INNER JOIN city ct -> ON a.city_id = ct.city_id -> INNER JOIN country co -> ON ct.country_id = co.country_id -> WHERE co.country IN ('Bolivia','Paraguay','Chile') -> GROUP BY co.country -> ); +-------------+-------------+ | customer_id | sum(amount) | +-------------+-------------+ | 137 | 194.61 | | 144 | 195.58 | | 148 | 216.54 | | 178 | 194.61 | | 459 | 186.62 | | 526 | 221.55 | +-------------+-------------+ 6 rows in set (0.03 sec) The subquery returns the total film rental fees for all customers in Bolivia, Paraguay, and Chile, and the containing query returns all customers who outspent at least one of these 3 countries (if you find yourself outspending an entire country, perhaps you need to cancel your Netflix subscription and book a trip to Bolivia, Paraguay, or Chile...). NOTE Although most people prefer to use in, using = any is equivalent to using the in operator. Multicolumn Subqueries So far, all of the subquery examples in this chapter have returned a single column and one or more rows. In certain situations, however, you can use subqueries that return two or more columns. To show the", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 117 + }, + { + "text": "in, using = any is equivalent to using the in operator. Multicolumn Subqueries So far, all of the subquery examples in this chapter have returned a single column and one or more rows. In certain situations, however, you can use subqueries that return two or more columns. To show the utility of multiple-column subqueries, it might help to look first at an example that uses multiple, single-column subqueries: mysql> SELECT fa.actor_id, fa.film_id -> FROM film_actor fa -> WHERE fa.actor_id IN -> (SELECT actor_id FROM actor WHERE last_name = 'MONROE') -> AND fa.film_id IN -> (SELECT film_id FROM film WHERE rating = 'PG'); +----------+---------+ | actor_id | film_id | +----------+---------+ | 120 | 63 | | 120 | 144 | | 120 | 414 | | 120 | 590 | | 120 | 715 | | 120 | 894 | | 178 | 164 | | 178 | 194 | | 178 | 273 | | 178 | 311 | | 178 | 983 | +----------+---------+ 11 rows in set (0.00 sec) This query uses two subqueries to identify all actors with the last name Monroe and all films rated PG, and the containing query then uses this information to retrieve all cases where an actor named Monroe appeared in a PG film. However, you could merge the two single-column subqueries into one multi-column subquery, and compare the results to two columns in the film_actor table. To do so, your filter condition must name both columns from the film_actor table surrounded by parentheses and in the same order as returned by the subquery, as in: mysql> SELECT actor_id, film_id -> FROM film_actor -> WHERE (actor_id, film_id) IN -> (SELECT a.actor_id, f.film_id -> FROM actor a -> CROSS JOIN film f -> WHERE a.last_name = 'MONROE' -> AND f.rating = 'PG'); +----------+---------+ | actor_id | film_id | +----------+---------+ | 120 | 63 | | 120 | 144 | | 120 | 414 | | 120 | 590 | | 120 | 715 | | 120 | 894 | | 178 | 164 | | 178 | 194 | | 178 | 273 | | 178 | 311 | | 178 | 983 | +----------+---------+ 11 rows in set (0.00 sec) This version of the query performs the same function as the previous example, but with a single subquery that returns two columns instead of two subqueries that each return a single column. The subquery in this version uses a type of join called a Cross Join, which will be explored in the next chapter, but the basic ideas is to return all combinations of actors named Monroe (2) and all films rated PG (194) for a total of 388 rows, 11 of which can be found in the film_actor table. Correlated Subqueries All of the subqueries shown thus far have been independent of their containing statements, meaning that you can execute them by themselves and inspect the results. A correlated subquery, on the other hand, is dependent on its containing statement from", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 118 + }, + { + "text": "which can be found in the film_actor table. Correlated Subqueries All of the subqueries shown thus far have been independent of their containing statements, meaning that you can execute them by themselves and inspect the results. A correlated subquery, on the other hand, is dependent on its containing statement from which it references one or more columns. Unlike a noncorrelated subquery, a correlated subquery is not executed once prior to execution of the containing statement; instead, the correlated subquery is executed once for each candidate row (rows that might be included in the final results). For example, the following query uses a correlated subquery to count the number of film rentals for each customer, and the containing query then retrieves those customers having rented exactly twenty films: mysql> SELECT c.first_name, c.last_name -> FROM customer c -> WHERE 20 = -> (SELECT count(*) FROM rental r -> WHERE r.customer_id = c.customer_id); +------------+-------------+ | first_name | last_name | +------------+-------------+ | LAUREN | HUDSON | | JEANETTE | GREENE | | TARA | RYAN | | WILMA | RICHARDS | | JO | FOWLER | | KAY | CALDWELL | | DANIEL | CABRAL | | ANTHONY | SCHWAB | | TERRY | GRISSOM | | LUIS | YANEZ | | HERBERT | KRUGER | | OSCAR | AQUINO | | RAUL | FORTIER | | NELSON | CHRISTENSON | | ALFREDO | MCADAMS | +------------+-------------+ 15 rows in set (0.01 sec) The reference to c.customer_id at the very end of the subquery is what makes the subquery correlated; the containing query must supply values for c.customer_id for the subquery to execute. In this case, the containing query retrieves all 599 rows from the customer table and executes the subquery once for each customer, passing in the appropriate customer ID for each execution. If the subquery returns the value 20, then the filter condition is met and the row is added to the result set. NOTE One word of caution: since the correlated subquery will be executed once for each row of the containing query, the use of correlated subqueries can cause performance issues if the containing query returns a large number of rows. Along with equality conditions, you can use correlated subqueries in other types of conditions, such as the range condition illustrated here: mysql> SELECT c.first_name, c.last_name -> FROM customer c -> WHERE -> (SELECT sum(p.amount) FROM payment p -> WHERE p.customer_id = c.customer_id) -> BETWEEN 180 AND 240; +------------+-----------+ | first_name | last_name | +------------+-----------+ | RHONDA | KENNEDY | | CLARA | SHAW | | ELEANOR | HUNT | | MARION | SNYDER | | TOMMY | COLLAZO | | KARL | SEAL | +------------+-----------+ 6 rows in set (0.03 sec) This variation on the previous query finds all customers whose total payments for all film rentals lies between $180 and $240. Once again, the correlated subquery is executed 599 times (once for each customer row), and each execution of the subquery returns the total account balance for the", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 119 + }, + { + "text": "(0.03 sec) This variation on the previous query finds all customers whose total payments for all film rentals lies between $180 and $240. Once again, the correlated subquery is executed 599 times (once for each customer row), and each execution of the subquery returns the total account balance for the given customer. NOTE Another subtle difference in the previous query is that the subquery is on the lefthand side of the condition, which may look a bit odd but is perfectly valid. The exists Operator While you will often see correlated subqueries used in equality and range conditions, the most common operator used to build conditions that utilize correlated subqueries is the exists operator. You use the exists operator when you want to identify that a relationship exists without regard for the quantity; for example, the following query finds all the customers who rented at least one film prior to May 25, 2005, without regard for how many films were rented: mysql> SELECT c.first_name, c.last_name -> FROM customer c -> WHERE EXISTS -> (SELECT 1 FROM rental r -> WHERE r.customer_id = c.customer_id -> AND date(r.rental_date) < '2005-05-25'); +------------+-------------+ | first_name | last_name | +------------+-------------+ | CHARLOTTE | HUNTER | | DELORES | HANSEN | | MINNIE | ROMERO | | CASSANDRA | WALTERS | | ANDREW | PURDY | | MANUEL | MURRELL | | TOMMY | COLLAZO | | NELSON | CHRISTENSON | +------------+-------------+ 8 rows in set (0.03 sec) Using the exists operator, your subquery can return zero, one, or many rows, and the condition simply checks whether the subquery returned one or more rows. If you look at the select clause of the subquery, you will see that it consists of a single literal (1); since the condition in the containing query only needs to know how many rows have been returned, the actual data the subquery returned is irrelevant. Your subquery can return whatever strikes your fancy, as demonstrated next: mysql> SELECT c.first_name, c.last_name -> FROM customer c -> WHERE EXISTS -> (SELECT r.rental_date, r.customer_id, 'ABCD' str, 2 * 3 / 7 nmbr -> FROM rental r -> WHERE r.customer_id = c.customer_id -> AND date(r.rental_date) < '2005-05-25'); +------------+-------------+ | first_name | last_name | +------------+-------------+ | CHARLOTTE | HUNTER | | DELORES | HANSEN | | MINNIE | ROMERO | | CASSANDRA | WALTERS | | ANDREW | PURDY | | MANUEL | MURRELL | | TOMMY | COLLAZO | | NELSON | CHRISTENSON | +------------+-------------+ 8 rows in set (0.03 sec) However, the convention is to specify either select 1 or select * when using exists. You may also use not exists to check for subqueries that return no rows, as demonstrated by the following: mysql> SELECT a.first_name, a.last_name -> FROM actor a -> WHERE NOT EXISTS -> (SELECT 1 -> FROM film_actor fa -> INNER JOIN film f ON f.film_id = fa.film_id -> WHERE fa.actor_id = a.actor_id -> AND f.rating = 'R'); +------------+-----------+ | first_name | last_name | +------------+-----------+ | JANE | JACKMAN |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 120 + }, + { + "text": "mysql> SELECT a.first_name, a.last_name -> FROM actor a -> WHERE NOT EXISTS -> (SELECT 1 -> FROM film_actor fa -> INNER JOIN film f ON f.film_id = fa.film_id -> WHERE fa.actor_id = a.actor_id -> AND f.rating = 'R'); +------------+-----------+ | first_name | last_name | +------------+-----------+ | JANE | JACKMAN | +------------+-----------+ 1 row in set (0.00 sec) This query finds all actors who have never appeared in an R-rated film. Data Manipulation Using Correlated Subqueries All of the examples thus far in the chapter have been select statements, but don’t think that means that subqueries aren’t useful in other SQL statements. Subqueries are used heavily in update, delete, and insert statements as well, with correlated subqueries appearing frequently in update and delete statements. Here’s an example of a correlated subquery used to modify the last_update column in the customer table: UPDATE customer c SET c.last_update = (SELECT max(r.rental_date) FROM rental r WHERE r.customer_id = c.customer_id); This statement modifies every row in the customer table (since there is no where clause) by finding the latest rental date for each customer in the rental table. While it seems reasonable to expect that every customer will have at least one film rental, it would be best to check before attempting to update the last_update column; otherwise, the column will be set to null, since the subquery would return no rows. Here’s another version of the update statement, this time employing a where clause with a second correlated subquery: UPDATE customer c SET c.last_update = (SELECT max(r.rental_date) FROM rental r WHERE r.customer_id = c.customer_id) WHERE EXISTS (SELECT 1 FROM rental r WHERE r.customer_id = c.customer_id); The two correlated subqueries are identical except for the select clauses. The subquery in the set clause, however, executes only if the condition in the update statement’s where clause evaluates to true (meaning that at least one rental was found for the customer), thus protecting the data in the last_update column from being overwritten with a null. Correlated subqueries are also common in delete statements. For example, you may run a data maintenance script at the end of each month that removes unnecessary data. The script might include the following statement, which removes rows from the customer table where there have been no film rentals in the past year: DELETE FROM customer WHERE 365 < (SELECT datediff(now(), r.rental_date) days_since_last_rental FROM rental r WHERE r.customer_id = customer.customer_id); When using correlated subqueries with delete statements in MySQL, keep in mind that, for whatever reason, table aliases are not allowed when using delete, which is why I had to use the entire table name in the subquery. With most other database servers, you could provide an alias for the customer table, such as: DELETE FROM customer c WHERE 365 < (SELECT datediff(now(), r.rental_date) days_since_last_rental FROM rental r WHERE r.customer_id = c.customer_id); When to Use Subqueries Now that you have learned about the different types of subqueries and the different operators that you can employ to interact with the data returned by subqueries,", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 121 + }, + { + "text": "as: DELETE FROM customer c WHERE 365 < (SELECT datediff(now(), r.rental_date) days_since_last_rental FROM rental r WHERE r.customer_id = c.customer_id); When to Use Subqueries Now that you have learned about the different types of subqueries and the different operators that you can employ to interact with the data returned by subqueries, it’s time to explore the many ways in which you can use subqueries to build powerful SQL statements. The next three sections demonstrate how you may use subqueries to construct custom tables, to build conditions, and to generate column values in result sets. Subqueries As Data Sources Back in Chapter 3, I stated that the from clause of a select statement contains the tables to be used by the query. Since a subquery generates a result set containing rows and columns of data, it is perfectly valid to include subqueries in your from clause along with tables. Although it might, at first glance, seem like an interesting feature without much practical merit, using subqueries alongside tables is one of the most powerful tools available when writing queries. Here’s a simple example: mysql> SELECT c.first_name, c.last_name, -> pymnt.num_rentals, pymnt.tot_payments -> FROM customer c -> INNER JOIN -> (SELECT customer_id, -> count(*) num_rentals, sum(amount) tot_payments -> FROM payment -> GROUP BY customer_id -> ) pymnt -> ON c.customer_id = pymnt.customer_id; +-------------+--------------+-------------+--------------+ | first_name | last_name | num_rentals | tot_payments | +-------------+--------------+-------------+--------------+ | MARY | SMITH | 32 | 118.68 | | PATRICIA | JOHNSON | 27 | 128.73 | | LINDA | WILLIAMS | 26 | 135.74 | | BARBARA | JONES | 22 | 81.78 | | ELIZABETH | BROWN | 38 | 144.62 | ... | TERRENCE | GUNDERSON | 30 | 117.70 | | ENRIQUE | FORSYTHE | 28 | 96.72 | | FREDDIE | DUGGAN | 25 | 99.75 | | WADE | DELVALLE | 22 | 83.78 | | AUSTIN | CINTRON | 19 | 83.81 | +-------------+--------------+-------------+--------------+ 599 rows in set (0.03 sec) In this example, a subquery generates a list of customer IDs along with the number of film rentals and the total payments. Here’s the result set generated by the subquery: mysql> SELECT customer_id, count(*) num_rentals, sum(amount) tot_payments -> FROM payment -> GROUP BY customer_id; +-------------+-------------+--------------+ | customer_id | num_rentals | tot_payments | +-------------+-------------+--------------+ | 1 | 32 | 118.68 | | 2 | 27 | 128.73 | | 3 | 26 | 135.74 | | 4 | 22 | 81.78 | ... | 596 | 28 | 96.72 | | 597 | 25 | 99.75 | | 598 | 22 | 83.78 | | 599 | 19 | 83.81 | +-------------+-------------+--------------+ 599 rows in set (0.03 sec) The subquery is given the name pymnt and is joined to the customer table via the customer_id column. The containing query then retrieves the customer’s name from the customer table, along with the summary columns from the pymnt subquery. Subqueries used in the from clause must be noncorrelated ; they are executed first, and the data", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 122 + }, + { + "text": "pymnt and is joined to the customer table via the customer_id column. The containing query then retrieves the customer’s name from the customer table, along with the summary columns from the pymnt subquery. Subqueries used in the from clause must be noncorrelated ; they are executed first, and the data is held in memory until the containing query 1 finishes execution. Subqueries offer immense flexibility when writing queries, because you can go far beyond the set of available tables to create virtually any view of the data that you desire, and then join the results to other tables or subqueries. If you are writing reports or generating data feeds to external systems, you may be able to do things with a single query that used to demand multiple queries or a procedural language to accomplish. DATA FABRICATION Along with using subqueries to summarize existing data, you can use subqueries to generate data that doesn’t exist in any form within your database. For example, you may wish to group your customers by the amount of money spent on film rentals, but you want to use group definitions that are not stored in your database. For example, let’s say you want to sort your customers into the groups shown in Table 9-1. Table 9-1. Customer payment groups Group name Lower limit Upper limit Small Fry 0 $74.99 Average Joes $75 $149.99 Heavy Hitters $150 $9,999,999.99 To generate these groups within a single query, you will need a way to define these three groups. The first step is to define a query that generates the group definitions: mysql> SELECT 'Small Fry' name, 0 low_limit, 74.99 high_limit -> UNION ALL -> SELECT 'Average Joes' name, 75 low_limit, 149.99 high_limit -> UNION ALL -> SELECT 'Heavy Hitters' name, 150 low_limit, 9999999.99 high_limit; +---------------+-----------+------------+ | name | low_limit | high_limit | +---------------+-----------+------------+ | Small Fry | 0 | 74.99 | | Average Joes | 75 | 149.99 | | Heavy Hitters | 150 | 9999999.99 | +---------------+-----------+------------+ 3 rows in set (0.00 sec) I have used the set operator union all to merge the results from three separate queries into a single result set. Each query retrieves three literals, and the results from the three queries are put together to generate a result set with three rows and three columns. You now have a query to generate the desired groups, and you can place it into the from clause of another query to generate your customer groups: mysql> SELECT pymnt_grps.name, count(*) num_customers -> FROM -> (SELECT customer_id, -> count(*) num_rentals, sum(amount) tot_payments -> FROM payment -> GROUP BY customer_id -> ) pymnt -> INNER JOIN -> (SELECT 'Small Fry' name, 0 low_limit, 74.99 high_limit -> UNION ALL -> SELECT 'Average Joes' name, 75 low_limit, 149.99 high_limit -> UNION ALL -> SELECT 'Heavy Hitters' name, 150 low_limit, 9999999.99 high_limit -> ) pymnt_grps -> ON pymnt.tot_payments -> BETWEEN pymnt_grps.low_limit AND pymnt_grps.high_limit -> GROUP BY pymnt_grps.name; +---------------+---------------+ | name | num_customers | +---------------+---------------+ | Average Joes | 515 |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 123 + }, + { + "text": "ALL -> SELECT 'Average Joes' name, 75 low_limit, 149.99 high_limit -> UNION ALL -> SELECT 'Heavy Hitters' name, 150 low_limit, 9999999.99 high_limit -> ) pymnt_grps -> ON pymnt.tot_payments -> BETWEEN pymnt_grps.low_limit AND pymnt_grps.high_limit -> GROUP BY pymnt_grps.name; +---------------+---------------+ | name | num_customers | +---------------+---------------+ | Average Joes | 515 | | Heavy Hitters | 46 | | Small Fry | 38 | +---------------+---------------+ 3 rows in set (0.03 sec) The from clause contains two subqueries; the first subquery, named pymnt, returns the total number of film rentals and total payments for each customer, while the second subquery, named pymnt_grps, generates the three customer groupings. The two subqueries are joined by finding which of the 3 groups each customer belongs to, and the rows are then grouped by the Group Name in order to count the number of customers in each group. Of course, you could simply decide to build a permanent (or temporary) table to hold the group definitions instead of using a subquery. Using that approach, you would find your database to be littered with small special- purpose tables after awhile, and you wouldn’t remember the reason for which most of them were created. Using subqueries, however, you will be able to adhere to a policy where tables are added to a database only when there is a clear business need to store new data. TASK-ORIENTED SUBQUERIES Let’s say that you want to generate a report showing each customer’s name, along with their city, the total number of rentals, and the total payment amount. You could accomplish this by joining the payment, customer, address, and city tables, and then grouping on the customer’s first and last names: mysql> SELECT c.first_name, c.last_name, ct.city, -> sum(p.amount) tot_payments, count(*) tot_rentals -> FROM payment p -> INNER JOIN customer c -> ON p.customer_id = c.customer_id -> INNER JOIN address a -> ON c.address_id = a.address_id -> INNER JOIN city ct -> ON a.city_id = ct.city_id -> GROUP BY c.first_name, c.last_name, ct.city; +-------------+------------+-----------------+--------------+--- ----------+ | first_name | last_name | city | tot_payments | tot_rentals | +-------------+------------+-----------------+--------------+--- ----------+ | MARY | SMITH | Sasebo | 118.68 | 32 | | PATRICIA | JOHNSON | San Bernardino | 128.73 | 27 | | LINDA | WILLIAMS | Athenai | 135.74 | 26 | | BARBARA | JONES | Myingyan | 81.78 | 22 | ... | TERRENCE | GUNDERSON | Jinzhou | 117.70 | 30 | | ENRIQUE | FORSYTHE | Patras | 96.72 | 28 | | FREDDIE | DUGGAN | Sullana | 99.75 | 25 | | WADE | DELVALLE | Lausanne | 83.78 | 22 | | AUSTIN | CINTRON | Tieli | 83.81 | 19 | +-------------+------------+-----------------+--------------+--- ----------+ 599 rows in set (0.06 sec) This query returns the desired data, but if you look at the query closely, you will see that the customer, address, and city tables are needed only for display purposes, and that the payment table has everything needed to generate the groupings (customer_id and amount). Therefore, you could", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 124 + }, + { + "text": "(0.06 sec) This query returns the desired data, but if you look at the query closely, you will see that the customer, address, and city tables are needed only for display purposes, and that the payment table has everything needed to generate the groupings (customer_id and amount). Therefore, you could separate out the task of generating the groups into a subquery, and then join the other three tables to the table generated by the subquery to achieve the desired end result. Here’s the grouping subquery: mysql> SELECT customer_id, -> count(*) tot_rentals, sum(amount) tot_payments -> FROM payment -> GROUP BY customer_id; +-------------+-------------+--------------+ | customer_id | tot_rentals | tot_payments | +-------------+-------------+--------------+ | 1 | 32 | 118.68 | | 2 | 27 | 128.73 | | 3 | 26 | 135.74 | | 4 | 22 | 81.78 | ... | 595 | 30 | 117.70 | | 596 | 28 | 96.72 | | 597 | 25 | 99.75 | | 598 | 22 | 83.78 | | 599 | 19 | 83.81 | +-------------+-------------+--------------+ 599 rows in set (0.03 sec) This is the heart of the query; the other tables are needed only to provide meaningful strings in place of the customer_id value. The next query joins the previous data set to the other three tables: mysql> SELECT c.first_name, c.last_name, -> ct.city, -> pymnt.tot_payments, pymnt.tot_rentals -> FROM -> (SELECT customer_id, -> count(*) tot_rentals, sum(amount) tot_payments -> FROM payment -> GROUP BY customer_id -> ) pymnt -> INNER JOIN customer c -> ON pymnt.customer_id = c.customer_id -> INNER JOIN address a -> ON c.address_id = a.address_id -> INNER JOIN city ct -> ON a.city_id = ct.city_id; +-------------+------------+-----------------+--------------+--- ----------+ | first_name | last_name | city | tot_payments | tot_rentals | +-------------+------------+-----------------+--------------+--- ----------+ | MARY | SMITH | Sasebo | 118.68 | 32 | | PATRICIA | JOHNSON | San Bernardino | 128.73 | 27 | | LINDA | WILLIAMS | Athenai | 135.74 | 26 | | BARBARA | JONES | Myingyan | 81.78 | 22 | ... | TERRENCE | GUNDERSON | Jinzhou | 117.70 | 30 | | ENRIQUE | FORSYTHE | Patras | 96.72 | 28 | | FREDDIE | DUGGAN | Sullana | 99.75 | 25 | | WADE | DELVALLE | Lausanne | 83.78 | 22 | | AUSTIN | CINTRON | Tieli | 83.81 | 19 | +-------------+------------+-----------------+--------------+--- ----------+ 599 rows in set (0.06 sec) I realize that beauty is in the eye of the beholder, but I find this version of the query to be far more satisfying than the big, flat version. This version may execute faster as well, because the grouping is being done on a single numeric column(customer_id) instead of multiple lengthy string columns (customer.first_name, customer.last_name, city.city). COMMON TABLE EXPRESSIONS Common table expressions (a.k.a., CTEs), which are new to MySQL in version 8.0, have been available in other database servers for quite some time. A CTE is a named subquery which appears at the top of a query in a with clause,", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 125 + }, + { + "text": "string columns (customer.first_name, customer.last_name, city.city). COMMON TABLE EXPRESSIONS Common table expressions (a.k.a., CTEs), which are new to MySQL in version 8.0, have been available in other database servers for quite some time. A CTE is a named subquery which appears at the top of a query in a with clause, which can contain multiple CTEs separated by commas. Along with making queries more understandable, this feature also allows each subquery to refer to any other subquery defined previously. Here’s an example which includes three subqueries, where the second subquery refers to the 1st, and the 3rd refers to the 2nd: mysql> WITH actors_s AS -> (SELECT actor_id, first_name, last_name -> FROM actor -> WHERE last_name LIKE 'S%' -> ), -> actors_s_pg AS -> (SELECT s.actor_id, s.first_name, s.last_name, -> f.film_id, f.title -> FROM actors_s s -> INNER JOIN film_actor fa -> ON s.actor_id = fa.actor_id -> INNER JOIN film f -> ON f.film_id = fa.film_id -> WHERE f.rating = 'PG' -> ), -> actors_s_pg_revenue AS -> (SELECT spg.first_name, spg.last_name, p.amount -> FROM actors_s_pg spg -> INNER JOIN inventory i -> ON i.film_id = spg.film_id -> INNER JOIN rental r -> ON i.inventory_id = r.inventory_id -> INNER JOIN payment p -> ON r.rental_id = p.rental_id -> ) -> SELECT spg_rev.first_name, spg_rev.last_name, -> sum(spg_rev.amount) tot_revenue -> FROM actors_s_pg_revenue spg_rev -> GROUP BY spg_rev.first_name, spg_rev.last_name -> ORDER BY 3 desc; +------------+-------------+-------------+ | first_name | last_name | tot_revenue | +------------+-------------+-------------+ | NICK | STALLONE | 692.21 | | JEFF | SILVERSTONE | 652.35 | | DAN | STREEP | 509.02 | | GROUCHO | SINATRA | 457.97 | | SISSY | SOBIESKI | 379.03 | | JAYNE | SILVERSTONE | 372.18 | | CAMERON | STREEP | 361.00 | | JOHN | SUVARI | 296.36 | | JOE | SWANK | 177.52 | +------------+-------------+-------------+ 9 rows in set (0.18 sec) This query calculates the total revenues generated from PG-rated film rentals where the cast included an actor whose last name starts with S. The first subquery (actors_s) finds all actors whose last name starts with S, the second subquery (actors_s_pg) joins that data set to the film table and filters on films having a PG rating, and the third subquery (actors_s_pg_revenue) joins that data set to the revenue table to generate the amounts paid to rent any of these films. The final query simply groups the data by the actors names and sums the revenues. Subqueries As Expression Generators For this last section of the chapter, I finish where I began: with single- column, single-row scalar subqueries. Along with being used in filter conditions, scalar subqueries may be used wherever an expression can appear, including the select and order by clauses of a query and the values clause of an insert statement. In “Task-oriented subqueries”, I showed you how to use a subquery to separate out the grouping mechanism from the rest of the query. Here’s another version of the same query that uses subqueries for the same purpose, but in a different way: mysql>", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 126 + }, + { + "text": "the values clause of an insert statement. In “Task-oriented subqueries”, I showed you how to use a subquery to separate out the grouping mechanism from the rest of the query. Here’s another version of the same query that uses subqueries for the same purpose, but in a different way: mysql> SELECT -> (SELECT c.first_name FROM customer c -> WHERE c.customer_id = p.customer_id -> ) first_name, -> (SELECT c.last_name FROM customer c -> WHERE c.customer_id = p.customer_id -> ) last_name, -> (SELECT ct.city -> FROM customer c -> INNER JOIN address a -> ON c.address_id = a.address_id -> INNER JOIN city ct -> ON a.city_id = ct.city_id -> WHERE c.customer_id = p.customer_id -> ) city, -> sum(p.amount) tot_payments, -> count(*) tot_rentals -> FROM payment p -> GROUP BY p.customer_id; +-------------+------------+-----------------+--------------+--- ----------+ | first_name | last_name | city | tot_payments | tot_rentals | +-------------+------------+-----------------+--------------+--- ----------+ | MARY | SMITH | Sasebo | 118.68 | 32 | | PATRICIA | JOHNSON | San Bernardino | 128.73 | 27 | | LINDA | WILLIAMS | Athenai | 135.74 | 26 | | BARBARA | JONES | Myingyan | 81.78 | 22 | ... | TERRENCE | GUNDERSON | Jinzhou | 117.70 | 30 | | ENRIQUE | FORSYTHE | Patras | 96.72 | 28 | | FREDDIE | DUGGAN | Sullana | 99.75 | 25 | | WADE | DELVALLE | Lausanne | 83.78 | 22 | | AUSTIN | CINTRON | Tieli | 83.81 | 19 | +-------------+------------+-----------------+--------------+--- ----------+ 599 rows in set (0.06 sec) There are two main differences between this query and the earlier version using a subquery in the from clause: Instead of joining the customer, address, and city tables to the payment data, correlated scalar subqueries are used in the select clause to look up the customer’s first/last names and city. The customer table is accessed three times (once in each of the three subqueries) rather than just once. The customer table is accessed three times because scalar subqueries can only return a single column and row, so if we need three columns related to the customer, it is necessary to use three different subqueries. As previously noted, scalar subqueries can also appear in the order by clause. The following query retrieves actor’s first and last names and sorts by the number of films in which the actor appeared: mysql> SELECT a.actor_id, a.first_name, a.last_name -> FROM actor a -> ORDER BY -> (SELECT count(*) FROM film_actor fa -> WHERE fa.actor_id = a.actor_id) DESC; +----------+-------------+--------------+ | actor_id | first_name | last_name | +----------+-------------+--------------+ | 107 | GINA | DEGENERES | | 102 | WALTER | TORN | | 198 | MARY | KEITEL | | 181 | MATTHEW | CARREY | ... | 71 | ADAM | GRANT | | 186 | JULIA | ZELLWEGER | | 35 | JUDY | DEAN | | 199 | JULIA | FAWCETT | | 148 | EMILY | DEE | +----------+-------------+--------------+ 200 rows in set (0.01 sec) The query uses a correlated scalar", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 127 + }, + { + "text": "| ... | 71 | ADAM | GRANT | | 186 | JULIA | ZELLWEGER | | 35 | JUDY | DEAN | | 199 | JULIA | FAWCETT | | 148 | EMILY | DEE | +----------+-------------+--------------+ 200 rows in set (0.01 sec) The query uses a correlated scalar subquery in the order by clause to return just the number of film appearances, and this value is used solely for sorting purposes. Along with using correlated scalar subqueries in select statements, you can use noncorrelated scalar subqueries to generate values for an insert statement. For example, let’s say you are going to generate a new row in the film_actor table, and you’ve been given the following data: The first and last name of the actor The name of the film You have two choices for how to go about it: execute two queries to retrieve the primary key values from film and actor and place those values into an insert statement, or use subqueries to retrieve the two key values from within an insert statement. Here’s an example of the latter approach: INSERT INTO film_actor (actor_id, film_id, last_update) VALUES ( (SELECT actor_id FROM actor WHERE first_name = 'JENNIFER' AND last_name = 'DAVIS'), (SELECT film_id FROM film WHERE title = 'ACE GOLDFINGER'), now() ); Using a single SQL statement, you can create a row in the film_actor table and look up two foreign key column values at the same time. Subquery Wrap-up I covered a lot of ground in this chapter, so it might be a good idea to review it. The examples I used in this chapter demonstrated subqueries that: Return a single column and row, a single column with multiple rows, and multiple columns and rows Are independent of the containing statement (noncorrelated subqueries) Reference one or more columns from the containing statement (correlated subqueries) Are used in conditions that utilize comparison operators as well as the special-purpose operators in, not in, exists, and not exists Can be found in select, update, delete, and insert statements Generate result sets that can be joined to other tables (or subqueries) in a query Can be used to generate values to populate a table or to populate columns in a query’s result set Are used in the select, from, where, having, and order by clauses of queries Obviously, subqueries are a very versatile tool, so don’t feel bad if all these concepts haven’t sunk in after reading this chapter for the first time. Keep experimenting with the various uses for subqueries, and you will soon find yourself thinking about how you might utilize a subquery every time you write a nontrivial SQL statement. Test Your Knowledge These exercises are designed to test your understanding of subqueries. Please see Appendix C for the solutions. Exercise 9-1 Construct a query against the film table that uses a filter condition with a noncorrelated subquery against the category table to find all action films (category.name = 'Action'). Exercise 9-2 Rework the query from Exercise 9-1", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 128 + }, + { + "text": "test your understanding of subqueries. Please see Appendix C for the solutions. Exercise 9-1 Construct a query against the film table that uses a filter condition with a noncorrelated subquery against the category table to find all action films (category.name = 'Action'). Exercise 9-2 Rework the query from Exercise 9-1 using a correlated subquery against the category and film_category tables to achieve the same results. Exercise 9-3 Join the following query to a subquery against the film_actor table to show the level of each actor: SELECT 'Hollywood Star' level, 30 min_roles, 99999 max_roles UNION ALL SELECT 'Prolific Actor' level, 20 min_roles, 29 max_roles UNION ALL SELECT 'Newcomer' level, 1 min_roles, 19 max_roles The subquery against the film_actor table should count the number of rows for each actor using group by actor_id , and the count should be compared to the min_roles/max_roles columns to determine which level each actor belongs to. 1 Actually, depending on which database server you are using, you might be able to include correlated subqueries in your from clause by using Cross Apply or Outer Apply, but these features are beyond the scope of this book. Chapter 10. Joins Revisited By now, you should be comfortable with the concept of the inner join, which I introduced in Chapter 5. This chapter focuses on other ways in which you can join tables, including the outer join and the cross join. Outer Joins In all the examples thus far that have included multiple tables, we haven’t been concerned that the join conditions might fail to find matches for all the rows in the tables. For example, the inventory table contains a row for every film available for rental, but of the 1,000 rows in the film table, only 958 have one or more rows in the inventory table. The other 42 films are not available for rental (perhaps they are new releases due to arrive in a few days), so these film IDs cannot be found in the inventory table. Here’s a query that counts the number of available copies of each film by joining these two tables: mysql> SELECT f.film_id, f.title, count(*) num_copies -> FROM film f -> INNER JOIN inventory i -> ON f.film_id = i.film_id -> GROUP BY f.film_id, f.title; +---------+-----------------------------+------------+ | film_id | title | num_copies | +---------+-----------------------------+------------+ | 1 | ACADEMY DINOSAUR | 8 | | 2 | ACE GOLDFINGER | 3 | | 3 | ADAPTATION HOLES | 4 | | 4 | AFFAIR PREJUDICE | 7 | ... | 13 | ALI FOREVER | 4 | | 15 | ALIEN CENTER | 6 | ... | 997 | YOUTH KICK | 2 | | 998 | ZHIVAGO CORE | 2 | | 999 | ZOOLANDER FICTION | 5 | | 1000 | ZORRO ARK | 8 | +---------+-----------------------------+------------+ 958 rows in set (0.02 sec) While you may have expected 1,000 rows to be returned (one for each film), the query only returns 958 rows. This is because the query uses an inner", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 129 + }, + { + "text": "999 | ZOOLANDER FICTION | 5 | | 1000 | ZORRO ARK | 8 | +---------+-----------------------------+------------+ 958 rows in set (0.02 sec) While you may have expected 1,000 rows to be returned (one for each film), the query only returns 958 rows. This is because the query uses an inner join, which only returns rows which satisfy the join condition. The film “ALICE FANTASIA” (film_id 14) doesn’t appear in the results, for example, because it doesn’t have any rows in the inventory table. If you want the query to return all 1,000 films, regardless of whether or not there are rows in the inventory table, you can use an outer join, which essentially makes the join condition optional : mysql> SELECT f.film_id, f.title, count(i.inventory_id) num_copies -> FROM film f -> LEFT OUTER JOIN inventory i -> ON f.film_id = i.film_id -> GROUP BY f.film_id, f.title; +---------+-----------------------------+------------+ | film_id | title | num_copies | +---------+-----------------------------+------------+ | 1 | ACADEMY DINOSAUR | 8 | | 2 | ACE GOLDFINGER | 3 | | 3 | ADAPTATION HOLES | 4 | | 4 | AFFAIR PREJUDICE | 7 | ... | 13 | ALI FOREVER | 4 | | 14 | ALICE FANTASIA | 0 | | 15 | ALIEN CENTER | 6 | ... | 997 | YOUTH KICK | 2 | | 998 | ZHIVAGO CORE | 2 | | 999 | ZOOLANDER FICTION | 5 | | 1000 | ZORRO ARK | 8 | +---------+-----------------------------+------------+ 1000 rows in set (0.01 sec) As you can see, the query now returns all 1,000 rows from the film table, and 42 of the rows (including “ALICE FANTASIA”) have a value of 0 in the num_copies column, which indicates that there are no copies in inventory. Here’s a description of the changes from the prior version of the query: The join definition was changed from inner to left outer, which instructs the server to include all rows from the table on the left side of the join (film, in this case), and then include columns from the table on the right side of the join (inventory ) if the join is successful The num_copies column definition was changed from count(*) to count(i.inventory_id), which will count the number of non- NULL values of the inventory.inventory_id column Next, let’s remove the group by clause and filter out most of the rows in order to clearly see the differences between inner and outer joins. Here’s a query using an inner join and a filter condition to return rows for just a few films: mysql> SELECT f.film_id, f.title, i.inventory_id -> FROM film f -> INNER JOIN inventory i -> ON f.film_id = i.film_id -> WHERE f.film_id BETWEEN 13 AND 15; +---------+--------------+--------------+ | film_id | title | inventory_id | +---------+--------------+--------------+ | 13 | ALI FOREVER | 67 | | 13 | ALI FOREVER | 68 | | 13 | ALI FOREVER | 69 | | 13 | ALI FOREVER | 70 | | 15 | ALIEN CENTER | 71", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 130 + }, + { + "text": "AND 15; +---------+--------------+--------------+ | film_id | title | inventory_id | +---------+--------------+--------------+ | 13 | ALI FOREVER | 67 | | 13 | ALI FOREVER | 68 | | 13 | ALI FOREVER | 69 | | 13 | ALI FOREVER | 70 | | 15 | ALIEN CENTER | 71 | | 15 | ALIEN CENTER | 72 | | 15 | ALIEN CENTER | 73 | | 15 | ALIEN CENTER | 74 | | 15 | ALIEN CENTER | 75 | | 15 | ALIEN CENTER | 76 | +---------+--------------+--------------+ 10 rows in set (0.00 sec) The results show that there are 4 copies of “ALI FOREVER” and 6 copies of “ALIEN CENTER” in inventory. Here’s the same query, but using an outer join: mysql> SELECT f.film_id, f.title, i.inventory_id -> FROM film f -> LEFT OUTER JOIN inventory i -> ON f.film_id = i.film_id -> WHERE f.film_id BETWEEN 13 AND 15; +---------+----------------+--------------+ | film_id | title | inventory_id | +---------+----------------+--------------+ | 13 | ALI FOREVER | 67 | | 13 | ALI FOREVER | 68 | | 13 | ALI FOREVER | 69 | | 13 | ALI FOREVER | 70 | | 14 | ALICE FANTASIA | NULL | | 15 | ALIEN CENTER | 71 | | 15 | ALIEN CENTER | 72 | | 15 | ALIEN CENTER | 73 | | 15 | ALIEN CENTER | 74 | | 15 | ALIEN CENTER | 75 | | 15 | ALIEN CENTER | 76 | +---------+----------------+--------------+ 11 rows in set (0.00 sec) The results are the same for “ALI FOREVER” and “ALIEN CENTER”, but there’s one new row for “ALICE FANTASIA”, with a NULL value for the inventory.inventory_id column. This example illustrates how an outer join will add column values without restricting the number of rows returned by the query. Left Versus Right Outer Joins In each of the outer join examples in the previous section, I specified left outer join. The keyword left indicates that the table on the left side of the join is responsible for determining the number of rows in the result set, whereas the table on the right side is used to provide column values whenever a match is found. However, you may also specify a right outer join, in which case the table on the right side of the join is responsible for determining the number of rows in the result set, whereas the table on the left side is used to provide column values. Here’s the last query from the previous section, but rearranged to use a right outer join instead of a left outer join: mysql> SELECT f.film_id, f.title, i.inventory_id -> FROM inventory i -> RIGHT OUTER JOIN film f -> ON f.film_id = i.film_id -> WHERE f.film_id BETWEEN 13 AND 15; +---------+----------------+--------------+ | film_id | title | inventory_id | +---------+----------------+--------------+ | 13 | ALI FOREVER | 67 | | 13 | ALI FOREVER | 68 | | 13 | ALI FOREVER | 69 | | 13", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 131 + }, + { + "text": "JOIN film f -> ON f.film_id = i.film_id -> WHERE f.film_id BETWEEN 13 AND 15; +---------+----------------+--------------+ | film_id | title | inventory_id | +---------+----------------+--------------+ | 13 | ALI FOREVER | 67 | | 13 | ALI FOREVER | 68 | | 13 | ALI FOREVER | 69 | | 13 | ALI FOREVER | 70 | | 14 | ALICE FANTASIA | NULL | | 15 | ALIEN CENTER | 71 | | 15 | ALIEN CENTER | 72 | | 15 | ALIEN CENTER | 73 | | 15 | ALIEN CENTER | 74 | | 15 | ALIEN CENTER | 75 | | 15 | ALIEN CENTER | 76 | +---------+----------------+--------------+ 11 rows in set (0.00 sec) Keep in mind that both versions of the query are performing outer joins; the keywords left and right are there just to tell the server which table is allowed to have gaps in the data. If you want to outer-join tables A and B and you want all rows from A with additional columns from B whenever there is matching data, you can specify either A left outer join B or B right outer join A. Three-Way Outer Joins In some cases, you may want to outer-join one table with two other tables. For example, the query from a prior section can be expanded to include data from the rental table: mysql> SELECT f.film_id, f.title, i.inventory_id, r.rental_date -> FROM film f -> LEFT OUTER JOIN inventory i -> ON f.film_id = i.film_id -> LEFT OUTER JOIN rental r -> ON i.inventory_id = r.inventory_id -> WHERE f.film_id BETWEEN 13 AND 15; +---------+----------------+--------------+--------------------- + | film_id | title | inventory_id | rental_date | +---------+----------------+--------------+--------------------- + | 13 | ALI FOREVER | 67 | 2005-07-31 18:11:17 | | 13 | ALI FOREVER | 67 | 2005-08-22 21:59:29 | | 13 | ALI FOREVER | 68 | 2005-07-28 15:26:20 | | 13 | ALI FOREVER | 68 | 2005-08-23 05:02:31 | | 13 | ALI FOREVER | 69 | 2005-08-01 23:36:10 | | 13 | ALI FOREVER | 69 | 2005-08-22 02:12:44 | | 13 | ALI FOREVER | 70 | 2005-07-12 10:51:09 | | 13 | ALI FOREVER | 70 | 2005-07-29 01:29:51 | | 13 | ALI FOREVER | 70 | 2006-02-14 15:16:03 | | 14 | ALICE FANTASIA | NULL | NULL | | 15 | ALIEN CENTER | 71 | 2005-05-28 02:06:37 | | 15 | ALIEN CENTER | 71 | 2005-06-17 16:40:03 | | 15 | ALIEN CENTER | 71 | 2005-07-11 05:47:08 | | 15 | ALIEN CENTER | 71 | 2005-08-02 13:58:55 | | 15 | ALIEN CENTER | 71 | 2005-08-23 05:13:09 | | 15 | ALIEN CENTER | 72 | 2005-05-27 22:49:27 | | 15 | ALIEN CENTER | 72 | 2005-06-19 13:29:28 | | 15 | ALIEN CENTER | 72 | 2005-07-07 23:05:53 | | 15 | ALIEN CENTER | 72 | 2005-08-01 05:55:13 | | 15 | ALIEN CENTER | 72 | 2005-08-20 15:11:48", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 132 + }, + { + "text": "CENTER | 72 | 2005-05-27 22:49:27 | | 15 | ALIEN CENTER | 72 | 2005-06-19 13:29:28 | | 15 | ALIEN CENTER | 72 | 2005-07-07 23:05:53 | | 15 | ALIEN CENTER | 72 | 2005-08-01 05:55:13 | | 15 | ALIEN CENTER | 72 | 2005-08-20 15:11:48 | | 15 | ALIEN CENTER | 73 | 2005-07-06 15:51:58 | | 15 | ALIEN CENTER | 73 | 2005-07-30 14:48:24 | | 15 | ALIEN CENTER | 73 | 2005-08-20 22:32:11 | | 15 | ALIEN CENTER | 74 | 2005-07-27 00:15:18 | | 15 | ALIEN CENTER | 74 | 2005-08-23 19:21:22 | | 15 | ALIEN CENTER | 75 | 2005-07-09 02:58:41 | | 15 | ALIEN CENTER | 75 | 2005-07-29 23:52:01 | | 15 | ALIEN CENTER | 75 | 2005-08-18 21:55:01 | | 15 | ALIEN CENTER | 76 | 2005-06-15 08:01:29 | | 15 | ALIEN CENTER | 76 | 2005-07-07 18:31:50 | | 15 | ALIEN CENTER | 76 | 2005-08-01 01:49:36 | | 15 | ALIEN CENTER | 76 | 2005-08-17 07:26:47 | +---------+----------------+--------------+--------------------- + 32 rows in set (0.01 sec) The results include all rentals of all films in inventory, but the film “ALICE FANTASIA” has NULL values for the columns from both outer- joined tables. Cross Joins Back in Chapter 5, I introduced the concept of a Cartesian product, which is essentially the result of joining multiple tables without specifying any join conditions. Cartesian products are used fairly frequently by accident (e.g., forgetting to add the join condition to the from clause) but are not so common otherwise. If, however, you do intend to generate the Cartesian product of two tables, you should specify a cross join, as in: mysql> SELECT c.name category_name, l.name language_name -> FROM category c -> CROSS JOIN language l; +---------------+---------------+ | category_name | language_name | +---------------+---------------+ | Action | English | | Action | Italian | | Action | Japanese | | Action | Mandarin | | Action | French | | Action | German | | Animation | English | | Animation | Italian | | Animation | Japanese | | Animation | Mandarin | | Animation | French | | Animation | German | ... | Sports | English | | Sports | Italian | | Sports | Japanese | | Sports | Mandarin | | Sports | French | | Sports | German | | Travel | English | | Travel | Italian | | Travel | Japanese | | Travel | Mandarin | | Travel | French | | Travel | German | +---------------+---------------+ 96 rows in set (0.00 sec) This query generates the Cartesian product of the category and language tables, resulting in 96 rows (16 category rows × 6 language rows). But now that you know what a cross join is and how to specify it, what is it used for? Most SQL books will describe what a cross join is and then tell you that it", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 133 + }, + { + "text": "and language tables, resulting in 96 rows (16 category rows × 6 language rows). But now that you know what a cross join is and how to specify it, what is it used for? Most SQL books will describe what a cross join is and then tell you that it is seldom useful, but I would like to share with you a situation in which I find the cross join to be quite helpful. In Chapter 9, I discussed how to use subqueries to fabricate tables. The example I used showed how to build a three-row table that could be joined to other tables. Here’s the fabricated table from the example: mysql> SELECT 'Small Fry' name, 0 low_limit, 74.99 high_limit -> UNION ALL -> SELECT 'Average Joes' name, 75 low_limit, 149.99 high_limit -> UNION ALL -> SELECT 'Heavy Hitters' name, 150 low_limit, 9999999.99 high_limit; +---------------+-----------+------------+ | name | low_limit | high_limit | +---------------+-----------+------------+ | Small Fry | 0 | 74.99 | | Average Joes | 75 | 149.99 | | Heavy Hitters | 150 | 9999999.99 | +---------------+-----------+------------+ 3 rows in set (0.00 sec) While this table was exactly what was needed for placing customers into three groups based on their total film payments, this strategy of merging single-row tables using the set operator union all doesn’t work very well if you need to fabricate a large table. Say, for example, that you want to create a query that generates a row for every day in the year 2020, but you don’t have a table in your database that contains a row for every day. Using the strategy from the example in Chapter 9, you could do something like the following: SELECT '2020-01-01' dt UNION ALL SELECT '2020-01-02' dt UNION ALL SELECT '2020-01-03' dt UNION ALL ... ... ... SELECT '2020-12-29' dt UNION ALL SELECT '2020-12-30' dt UNION ALL SELECT '2020-12-31' dt Building a query that merges together the results of 366 queries is a bit tedious, so maybe a different strategy is needed. What if you generate a table with 366 rows (2020 is a leap year) with a single column containing a number between 0 and 366, and then add that number of days to January 1, 2020? Here’s one possible method to generate such a table: mysql> SELECT ones.num + tens.num + hundreds.num -> FROM -> (SELECT 0 num UNION ALL -> SELECT 1 num UNION ALL -> SELECT 2 num UNION ALL -> SELECT 3 num UNION ALL -> SELECT 4 num UNION ALL -> SELECT 5 num UNION ALL -> SELECT 6 num UNION ALL -> SELECT 7 num UNION ALL -> SELECT 8 num UNION ALL -> SELECT 9 num) ones -> CROSS JOIN -> (SELECT 0 num UNION ALL -> SELECT 10 num UNION ALL -> SELECT 20 num UNION ALL -> SELECT 30 num UNION ALL -> SELECT 40 num UNION ALL -> SELECT 50 num UNION ALL -> SELECT 60 num UNION ALL -> SELECT 70 num UNION ALL -> SELECT", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 134 + }, + { + "text": "-> (SELECT 0 num UNION ALL -> SELECT 10 num UNION ALL -> SELECT 20 num UNION ALL -> SELECT 30 num UNION ALL -> SELECT 40 num UNION ALL -> SELECT 50 num UNION ALL -> SELECT 60 num UNION ALL -> SELECT 70 num UNION ALL -> SELECT 80 num UNION ALL -> SELECT 90 num) tens -> CROSS JOIN -> (SELECT 0 num UNION ALL -> SELECT 100 num UNION ALL -> SELECT 200 num UNION ALL -> SELECT 300 num) hundreds; +------------------------------------+ | ones.num + tens.num + hundreds.num | +------------------------------------+ | 0 | | 1 | | 2 | | 3 | | 4 | | 5 | | 6 | | 7 | | 8 | | 9 | | 10 | | 11 | | 12 | ... ... ... | 391 | | 392 | | 393 | | 394 | | 395 | | 396 | | 397 | | 398 | | 399 | +------------------------------------+ 400 rows in set (0.00 sec) If you take the Cartesian product of the three sets {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}, {0, 10, 20, 30, 40, 50, 60, 70, 80, 90}, and {0, 100, 200, 300} and add the values in the three columns, you get a 400-row result set containing all numbers between 0 and 399. While this is more than the 366 rows needed to generate the set of days in 2020, it’s easy enough to get rid of the excess rows, and I’ll show you how shortly. The next step is to convert the set of numbers to a set of dates. To do this, I will use the date_add() function to add each number in the result set to January 1, 2020. Then I’ll add a filter condition to throw away any dates that venture into 2021: mysql> SELECT DATE_ADD('2020-01-01', -> INTERVAL (ones.num + tens.num + hundreds.num) DAY) dt -> FROM -> (SELECT 0 num UNION ALL -> SELECT 1 num UNION ALL -> SELECT 2 num UNION ALL -> SELECT 3 num UNION ALL -> SELECT 4 num UNION ALL -> SELECT 5 num UNION ALL -> SELECT 6 num UNION ALL -> SELECT 7 num UNION ALL -> SELECT 8 num UNION ALL -> SELECT 9 num) ones -> CROSS JOIN -> (SELECT 0 num UNION ALL -> SELECT 10 num UNION ALL -> SELECT 20 num UNION ALL -> SELECT 30 num UNION ALL -> SELECT 40 num UNION ALL -> SELECT 50 num UNION ALL -> SELECT 60 num UNION ALL -> SELECT 70 num UNION ALL -> SELECT 80 num UNION ALL -> SELECT 90 num) tens -> CROSS JOIN -> (SELECT 0 num UNION ALL -> SELECT 100 num UNION ALL -> SELECT 200 num UNION ALL -> SELECT 300 num) hundreds -> WHERE DATE_ADD('2020-01-01', -> INTERVAL (ones.num + tens.num + hundreds.num) DAY) < '2021-01-01' -> ORDER BY 1; +------------+ | dt | +------------+ | 2020-01-01 | | 2020-01-02 | |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 135 + }, + { + "text": "0 num UNION ALL -> SELECT 100 num UNION ALL -> SELECT 200 num UNION ALL -> SELECT 300 num) hundreds -> WHERE DATE_ADD('2020-01-01', -> INTERVAL (ones.num + tens.num + hundreds.num) DAY) < '2021-01-01' -> ORDER BY 1; +------------+ | dt | +------------+ | 2020-01-01 | | 2020-01-02 | | 2020-01-03 | | 2020-01-04 | | 2020-01-05 | | 2020-01-06 | | 2020-01-07 | | 2020-01-08 | ... ... ... | 2020-02-26 | | 2020-02-27 | | 2020-02-28 | | 2020-02-29 | | 2020-03-01 | | 2020-03-02 | | 2020-03-03 | ... ... ... | 2020-12-24 | | 2020-12-25 | | 2020-12-26 | | 2020-12-27 | | 2020-12-28 | | 2020-12-29 | | 2020-12-30 | | 2020-12-31 | +------------+ 366 rows in set (0.03 sec) The nice thing about this approach is that the result set automatically includes the extra leap day (February 29) without your intervention, since the database server figures it out when it adds 59 days to January 1, 2020. Now that you have a mechanism for fabricating all the days in 2020, what should you do with it? Well, you might be asked to generate a report that shows every day in 2020 along with the number of film rentals on that day. The report needs to include every day of the year, including days when no films are rented. Here’s what the query might look like, but using the year 2005 to match the data in the rental table: mysql> SELECT days.dt, COUNT(r.rental_id) num_rentals -> FROM rental r -> RIGHT OUTER JOIN -> (SELECT DATE_ADD('2005-01-01', -> INTERVAL (ones.num + tens.num + hundreds.num) DAY) dt -> FROM -> (SELECT 0 num UNION ALL -> SELECT 1 num UNION ALL -> SELECT 2 num UNION ALL -> SELECT 3 num UNION ALL -> SELECT 4 num UNION ALL -> SELECT 5 num UNION ALL -> SELECT 6 num UNION ALL -> SELECT 7 num UNION ALL -> SELECT 8 num UNION ALL -> SELECT 9 num) ones -> CROSS JOIN -> (SELECT 0 num UNION ALL -> SELECT 10 num UNION ALL -> SELECT 20 num UNION ALL -> SELECT 30 num UNION ALL -> SELECT 40 num UNION ALL -> SELECT 50 num UNION ALL -> SELECT 60 num UNION ALL -> SELECT 70 num UNION ALL -> SELECT 80 num UNION ALL -> SELECT 90 num) tens -> CROSS JOIN -> (SELECT 0 num UNION ALL -> SELECT 100 num UNION ALL -> SELECT 200 num UNION ALL -> SELECT 300 num) hundreds -> WHERE DATE_ADD('2005-01-01', -> INTERVAL (ones.num + tens.num + hundreds.num) DAY) -> < '2006-01-01' -> ) days -> ON days.dt = date(r.rental_date) -> GROUP BY days.dt -> ORDER BY 1; +------------+-------------+ | dt | num_rentals | +------------+-------------+ | 2005-01-01 | 0 | | 2005-01-02 | 0 | | 2005-01-03 | 0 | | 2005-01-04 | 0 | ... | 2005-05-23 | 0 | | 2005-05-24 | 8 | | 2005-05-25 | 137 | | 2005-05-26 | 174 | | 2005-05-27 | 166", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 136 + }, + { + "text": "dt | num_rentals | +------------+-------------+ | 2005-01-01 | 0 | | 2005-01-02 | 0 | | 2005-01-03 | 0 | | 2005-01-04 | 0 | ... | 2005-05-23 | 0 | | 2005-05-24 | 8 | | 2005-05-25 | 137 | | 2005-05-26 | 174 | | 2005-05-27 | 166 | | 2005-05-28 | 196 | | 2005-05-29 | 154 | | 2005-05-30 | 158 | | 2005-05-31 | 163 | | 2005-06-01 | 0 | ... | 2005-06-13 | 0 | | 2005-06-14 | 16 | | 2005-06-15 | 348 | | 2005-06-16 | 324 | | 2005-06-17 | 325 | | 2005-06-18 | 344 | | 2005-06-19 | 348 | | 2005-06-20 | 331 | | 2005-06-21 | 275 | | 2005-06-22 | 0 | ... | 2005-12-27 | 0 | | 2005-12-28 | 0 | | 2005-12-29 | 0 | | 2005-12-30 | 0 | | 2005-12-31 | 0 | +------------+-------------+ 365 rows in set (8.99 sec) This is one of the more interesting queries thus far in the book, in that it includes cross joins, outer joins, a date function, grouping, set operations (union all), and an aggregate function (count()). It is also not the most elegant solution to the given problem, but it should serve as an example of how, with a little creativity and a firm grasp on the language, you can make even a seldom-used feature like cross joins a potent tool in your SQL toolkit. Natural Joins If you are lazy (and aren’t we all), you can choose a join type that allows you to name the tables to be joined but lets the database server determine what the join conditions need to be. Known as the natural join, this join type relies on identical column names across multiple tables to infer the proper join conditions. For example, the rental table includes a column named customer_id, which is the foreign key to the customer table, whose primary key is also named customer_id. Thus, you could try to write a query that uses natural join to join the two tables: mysql> SELECT c.first_name, c.last_name, date(r.rental_date) -> FROM customer c -> NATURAL JOIN rental r; Empty set (0.04 sec) Because you specified a natural join, the server inspected the table definitions and added the join condition r.customer_id = c.customer_id to join the two tables. This would have worked fine, but in the Sakila schema all of the tables include the column last_update to show when each row was last modified, so the server is also adding the join condition r.last_update = c.last_update , which causes the query to return no data. The only way around this issue is to use a subquery to restrict the columns for at least one of the tables: mysql> SELECT cust.first_name, cust.last_name, date(r.rental_date) -> FROM -> (SELECT customer_id, first_name, last_name -> FROM customer -> ) cust -> NATURAL JOIN rental r; +------------+-----------+---------------------+ | first_name | last_name | date(r.rental_date) | +------------+-----------+---------------------+ | MARY | SMITH | 2005-05-25 | | MARY", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 137 + }, + { + "text": "the columns for at least one of the tables: mysql> SELECT cust.first_name, cust.last_name, date(r.rental_date) -> FROM -> (SELECT customer_id, first_name, last_name -> FROM customer -> ) cust -> NATURAL JOIN rental r; +------------+-----------+---------------------+ | first_name | last_name | date(r.rental_date) | +------------+-----------+---------------------+ | MARY | SMITH | 2005-05-25 | | MARY | SMITH | 2005-05-28 | | MARY | SMITH | 2005-06-15 | | MARY | SMITH | 2005-06-15 | | MARY | SMITH | 2005-06-15 | | MARY | SMITH | 2005-06-16 | | MARY | SMITH | 2005-06-18 | | MARY | SMITH | 2005-06-18 | ... | AUSTIN | CINTRON | 2005-08-21 | | AUSTIN | CINTRON | 2005-08-21 | | AUSTIN | CINTRON | 2005-08-21 | | AUSTIN | CINTRON | 2005-08-23 | | AUSTIN | CINTRON | 2005-08-23 | | AUSTIN | CINTRON | 2005-08-23 | +------------+-----------+---------------------+ 16044 rows in set (0.03 sec) So, is the reduced wear and tear on the old fingers from not having to type the join condition worth the trouble? Absolutely not; you should avoid this join type and use inner joins with explicit join conditions. Test Your Knowledge The following exercises test your understanding of outer and cross joins. Please see Appendix C for solutions. Exercise 10-1 Using the table definitions and data below, write a query that returns each customer name along with their total balance across all accounts. Customer: Customer_id Name ----------- --------------- 1 John Smith 2 Kathy Jones 3 Greg Oliver Account: Account_id Customer_id Account_Name Balance ---------- ----------- ------------ -------- 101 1 Checking 1044 102 3 Savings 522 103 1 Line of Credit 9995 Include all customers, even if no accounts exist for that customer. Exercise 10-2 Reformulate your query from Exercise 10-1 to use the other outer join type (e.g., if you used a left outer join in Exercise 10-1, use a right outer join this time) such that the results are identical to Exercise 10-1. Exercise 10-3 (Extra Credit) Devise a query that will generate the set {1, 2, 3,…, 99, 100}. (Hint: use a cross join with at least two from clause subqueries.) Chapter 11. Conditional Logic In certain situations, you may want your SQL logic to branch in one direction or another depending on the values of certain columns or expressions. This chapter focuses on how to write statements that can behave differently depending on the data encountered during statement execution. What Is Conditional Logic? Conditional logic is simply the ability to take one of several paths during program execution. For example, when querying customer information, you might want to include the customer.active column, which stores 1 to indicate Active and 0 to indicate Inactive. If the query results are being used to generate a report, you may want to translate the value to improve readability. While every database includes built-in functions for these types of situations, there are no standards, so you would need to remember which functions are used by which database. Fortunately, every database’s SQL implementation includes the CASE expression,", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 138 + }, + { + "text": "generate a report, you may want to translate the value to improve readability. While every database includes built-in functions for these types of situations, there are no standards, so you would need to remember which functions are used by which database. Fortunately, every database’s SQL implementation includes the CASE expression, which is useful in many situations, including simple translations : mysql> SELECT first_name, last_name, -> CASE -> WHEN active = 1 THEN 'ACTIVE' -> ELSE 'INACTIVE' -> END activity_type -> FROM customer; +-------------+--------------+---------------+ | first_name | last_name | activity_type | +-------------+--------------+---------------+ | MARY | SMITH | ACTIVE | | PATRICIA | JOHNSON | ACTIVE | | LINDA | WILLIAMS | ACTIVE | | BARBARA | JONES | ACTIVE | | ELIZABETH | BROWN | ACTIVE | | JENNIFER | DAVIS | ACTIVE | ... | KENT | ARSENAULT | ACTIVE | | TERRANCE | ROUSH | INACTIVE | | RENE | MCALISTER | ACTIVE | | EDUARDO | HIATT | ACTIVE | | TERRENCE | GUNDERSON | ACTIVE | | ENRIQUE | FORSYTHE | ACTIVE | | FREDDIE | DUGGAN | ACTIVE | | WADE | DELVALLE | ACTIVE | | AUSTIN | CINTRON | ACTIVE | +-------------+--------------+---------------+ 599 rows in set (0.00 sec) This query includes a CASE expression to generate a value for the activity_type column, which returns the strings “ACTIVE” or “INACTIVE” depending on the value of the customer.active column. The Case Expression All of the major database servers include built-in functions designed to mimic the if-then-else statement found in most programming languages (examples include Oracle’s decode() function, MySQL’s if() function, and SQL Server’s coalesce() function). Case expressions are also designed to facilitate if-then-else logic but enjoy two advantages over built-in functions: The case expression is part of the SQL standard (SQL92 release) and has been implemented by Oracle Database, SQL Server, MySQL, Sybase, PostgreSQL, IBM UDB, and others. Case expressions are built into the SQL grammar and can be included in select, insert, update, and delete statements. The next two subsections introduce the two different types of case expressions, and then I show you some examples of case expressions in action. Searched Case Expressions The case expression demonstrated earlier in the chapter is an example of a searched case expression, which has the following syntax: CASE WHEN C1 THEN E1 WHEN C2 THEN E2 ... WHEN CN THEN EN [ELSE ED] END In the previous definition, the symbols C1, C2,…, CN represent conditions, and the symbols E1, E2,…, EN represent expressions to be returned by the case expression. If the condition in a when clause evaluates to true, then the case expression returns the corresponding expression. Additionally, the ED symbol represents the default expression, which the case expression returns if none of the conditions C1, C2,…, CN evaluate to true (the else clause is optional, which is why it is enclosed in square brackets). All the expressions returned by the various when clauses must evaluate to the same type (e.g., date, number, varchar). Here’s an example", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 139 + }, + { + "text": "case expression returns if none of the conditions C1, C2,…, CN evaluate to true (the else clause is optional, which is why it is enclosed in square brackets). All the expressions returned by the various when clauses must evaluate to the same type (e.g., date, number, varchar). Here’s an example of a searched case expression: CASE WHEN category.name IN ('Children','Family','Sports','Animation') THEN 'All Ages' WHEN category.name = 'Horror' THEN 'Adult' WHEN category.name IN ('Music','Games') THEN 'Teens' ELSE 'Other' END This case expression returns a string that can be used to classify films depending on their category. When the case expression is evaluated, the when clauses are evaluated in order from top to bottom; as soon as one of the conditions in a when clause evaluates to true, the corresponding expression is returned and any remaining when clauses are ignored. If none of the when clause conditions evaluate to true, then the expression in the else clause is returned. Although the previous example returns string expressions, keep in mind that case expressions may return any type of expression, including subqueries. Here’s another version of the query from earlier in the chapter that uses a subquery to return the number of rentals, but only for Active customers: mysql> SELECT c.first_name, c.last_name, -> CASE -> WHEN active = 0 THEN 0 -> ELSE -> (SELECT count(*) FROM rental r -> WHERE r.customer_id = c.customer_id) -> END num_rentals -> FROM customer c; +-------------+--------------+-------------+ | first_name | last_name | num_rentals | +-------------+--------------+-------------+ | MARY | SMITH | 32 | | PATRICIA | JOHNSON | 27 | | LINDA | WILLIAMS | 26 | | BARBARA | JONES | 22 | | ELIZABETH | BROWN | 38 | | JENNIFER | DAVIS | 28 | ... | TERRANCE | ROUSH | 0 | | RENE | MCALISTER | 26 | | EDUARDO | HIATT | 27 | | TERRENCE | GUNDERSON | 30 | | ENRIQUE | FORSYTHE | 28 | | FREDDIE | DUGGAN | 25 | | WADE | DELVALLE | 22 | | AUSTIN | CINTRON | 19 | +-------------+--------------+-------------+ 599 rows in set (0.01 sec) This version of the query uses a correlated subquery to retrieve the number of rentals for each Active customer. Depending on the percentage of Active customers, using this approach may be more efficient than joining the customer and rental tables and grouping on the customer_id column. Simple Case Expressions The simple case expression is quite similar to the searched case expression but is a bit less flexible. Here’s the syntax: CASE V0 WHEN V1 THEN E1 WHEN V2 THEN E2 ... WHEN VN THEN EN [ELSE ED] END In the preceding definition, V0 represents a value, and the symbols V1, V2,…, VN represent values that are to be compared to V0. The symbols E1, E2,…, EN represent expressions to be returned by the case expression, and ED represents the expression to be returned if none of the values in the set V1, V2,…, VN match the V0 value.", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 140 + }, + { + "text": "symbols V1, V2,…, VN represent values that are to be compared to V0. The symbols E1, E2,…, EN represent expressions to be returned by the case expression, and ED represents the expression to be returned if none of the values in the set V1, V2,…, VN match the V0 value. Here’s an example of a simple case expression: CASE category.name WHEN 'Children' THEN 'All Ages' WHEN 'Family' THEN 'All Ages' WHEN 'Sports' THEN 'All Ages' WHEN 'Animation' THEN 'All Ages' WHEN 'Horror' THEN 'Adult' WHEN 'Music' THEN 'Teens' WHEN 'Games' THEN 'Teens' ELSE 'Other' END Simple case expressions are less flexible than searched case expressions because you can’t specify your own conditions, whereas searched case expressions may include range conditions, inequality conditions, and multipart conditions using and/or/not, so I would recommend using searched case expressions for all but the simplest logic. Case Expression Examples The following sections present a variety of examples illustrating the utility of conditional logic in SQL statements. Result Set Transformations You may have run into a situation where you are performing aggregations over a finite set of values, such as days of the week, but you want the result set to contain a single row with one column per value instead of one row per value. As an example, let’s say you have been asked to write a query that shows the number of film rentals for May, June, and July of 2005: mysql> SELECT monthname(rental_date) rental_month, -> count(*) num_rentals -> FROM rental -> WHERE rental_date BETWEEN '2005-05-01' AND '2005-08-01' -> GROUP BY monthname(rental_date); +--------------+-------------+ | rental_month | num_rentals | +--------------+-------------+ | May | 1156 | | June | 2311 | | July | 6709 | +--------------+-------------+ 3 rows in set (0.01 sec) However, you have also been instructed to return a single row of data with three columns (one for each of the three months). To transform this result set into a single row, you will need to create three columns and, within each column, sum only those rows pertaining to the month in question: mysql> SELECT -> SUM(CASE WHEN monthname(rental_date) = 'May' THEN 1 -> ELSE 0 END) May_rentals, -> SUM(CASE WHEN monthname(rental_date) = 'June' THEN 1 -> ELSE 0 END) June_rentals, -> SUM(CASE WHEN monthname(rental_date) = 'July' THEN 1 -> ELSE 0 END) July_rentals -> FROM rental -> WHERE rental_date BETWEEN '2005-05-01' AND '2005-08-01'; +-------------+--------------+--------------+ | May_rentals | June_rentals | July_rentals | +-------------+--------------+--------------+ | 1156 | 2311 | 6709 | +-------------+--------------+--------------+ 1 row in set (0.01 sec) Each of the three columns in the previous query are identical, except for the month value. When the monthname() function returns the desired value for that column, the case expression returns the value 1; otherwise, it returns a 0. When summed over all rows, each column returns the number of accounts opened for that month. Obviously, such transformations are practical for only a small number of values; generating one column for each year since 1905 would quickly become tedious. NOTE Although it is a bit advanced", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 141 + }, + { + "text": "returns a 0. When summed over all rows, each column returns the number of accounts opened for that month. Obviously, such transformations are practical for only a small number of values; generating one column for each year since 1905 would quickly become tedious. NOTE Although it is a bit advanced for this book, it is worth pointing out that both SQL Server and Oracle Database 11g include PIVOT clauses specifically for these types of queries. Checking for Existence Sometimes you will want to determine whether a relationship exists between two entities without regard for the quantity. For example, you might want to know whether an actor has appeared in G-rated films, but you only want to know if there was at least one. Here’s a query that uses multiple case expressions to generate three output columns, one to show whether the actor has appeared in G-rated films, another for PG-rated films, and a third for NC-17-rated films: mysql> SELECT a.first_name, a.last_name, -> CASE -> WHEN EXISTS (SELECT 1 FROM film_actor fa -> INNER JOIN film f ON fa.film_id = f.film_id -> WHERE fa.actor_id = a.actor_id -> AND f.rating = 'G') THEN 'Y' -> ELSE 'N' -> END g_actor, -> CASE -> WHEN EXISTS (SELECT 1 FROM film_actor fa -> INNER JOIN film f ON fa.film_id = f.film_id -> WHERE fa.actor_id = a.actor_id -> AND f.rating = 'PG') THEN 'Y' -> ELSE 'N' -> END pg_actor, -> CASE -> WHEN EXISTS (SELECT 1 FROM film_actor fa -> INNER JOIN film f ON fa.film_id = f.film_id -> WHERE fa.actor_id = a.actor_id -> AND f.rating = 'NC-17') THEN 'Y' -> ELSE 'N' -> END nc17_actor -> FROM actor a -> WHERE a.last_name LIKE 'S%' OR a.first_name LIKE 'S%'; +------------+-------------+---------+----------+------------+ | first_name | last_name | g_actor | pg_actor | nc17_actor | +------------+-------------+---------+----------+------------+ | JOE | SWANK | Y | Y | Y | | SANDRA | KILMER | Y | Y | Y | | CAMERON | STREEP | Y | Y | Y | | SANDRA | PECK | Y | Y | Y | | SISSY | SOBIESKI | Y | Y | N | | NICK | STALLONE | Y | Y | Y | | SEAN | WILLIAMS | Y | Y | Y | | GROUCHO | SINATRA | Y | Y | Y | | SCARLETT | DAMON | Y | Y | Y | | SPENCER | PECK | Y | Y | Y | | SEAN | GUINESS | Y | Y | Y | | SPENCER | DEPP | Y | Y | Y | | SUSAN | DAVIS | Y | Y | Y | | SIDNEY | CROWE | Y | Y | Y | | SYLVESTER | DERN | Y | Y | Y | | SUSAN | DAVIS | Y | Y | Y | | DAN | STREEP | Y | Y | Y | | SALMA | NOLTE | Y | N | Y | | SCARLETT | BENING", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 142 + }, + { + "text": "Y | | SYLVESTER | DERN | Y | Y | Y | | SUSAN | DAVIS | Y | Y | Y | | DAN | STREEP | Y | Y | Y | | SALMA | NOLTE | Y | N | Y | | SCARLETT | BENING | Y | Y | Y | | JEFF | SILVERSTONE | Y | Y | Y | | JOHN | SUVARI | Y | Y | Y | | JAYNE | SILVERSTONE | Y | Y | Y | +------------+-------------+---------+----------+------------+ 22 rows in set (0.00 sec) Each case expression includes a correlated subquery against the film_actor and film tables; one looks for films with a G rating, the second for films with a PG rating, and the third for films with a NC-17 rating. Since each when clause uses the exists operator, the conditions evaluate to true as long as the actor has appeared in at least one film with the proper rating. In other cases, you may care how many rows are encountered, but only up to a point. For example, the next query uses a simple case expression to count the number of copies in inventory for each film, and then returns either 'Out Of Stock', 'Scarce', 'Available', or 'Common': mysql> SELECT f.title, -> CASE (SELECT count(*) FROM inventory i -> WHERE i.film_id = f.film_id) -> WHEN 0 THEN 'Out Of Stock' -> WHEN 1 THEN 'Scarce' -> WHEN 2 THEN 'Scarce' -> WHEN 3 THEN 'Available' -> WHEN 4 THEN 'Available' -> ELSE 'Common' -> END film_availability -> FROM film f -> ; +-----------------------------+-------------------+ | title | film_availability | +-----------------------------+-------------------+ | ACADEMY DINOSAUR | Common | | ACE GOLDFINGER | Available | | ADAPTATION HOLES | Available | | AFFAIR PREJUDICE | Common | | AFRICAN EGG | Available | | AGENT TRUMAN | Common | | AIRPLANE SIERRA | Common | | AIRPORT POLLOCK | Available | | ALABAMA DEVIL | Common | | ALADDIN CALENDAR | Common | | ALAMO VIDEOTAPE | Common | | ALASKA PHANTOM | Common | | ALI FOREVER | Available | | ALICE FANTASIA | Out Of Stock | ... | YOUNG LANGUAGE | Scarce | | YOUTH KICK | Scarce | | ZHIVAGO CORE | Scarce | | ZOOLANDER FICTION | Common | | ZORRO ARK | Common | +-----------------------------+-------------------+ 1000 rows in set (0.01 sec) For this query, I stopped counting after 5, since every other number greater than 5 will be given the 'Common' label. Division-by-Zero Errors When performing calculations that include division, you should always take care to ensure that the denominators are never equal to zero. Whereas some database servers, such as Oracle Database, will throw an error when a zero denominator is encountered, MySQL simply sets the result of the calculation to null, as demonstrated by the following: mysql> SELECT 100 / 0; +---------+ | 100 / 0 | +---------+ | NULL | +---------+ 1 row in set (0.00 sec) To safeguard your", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 143 + }, + { + "text": "will throw an error when a zero denominator is encountered, MySQL simply sets the result of the calculation to null, as demonstrated by the following: mysql> SELECT 100 / 0; +---------+ | 100 / 0 | +---------+ | NULL | +---------+ 1 row in set (0.00 sec) To safeguard your calculations from encountering errors or, even worse, from being mysteriously set to null, you should wrap all denominators in conditional logic, as demonstrated by the following: mysql> SELECT c.first_name, c.last_name, -> sum(p.amount) tot_payment_amt, -> count(p.amount) num_payments, -> sum(p.amount) / -> CASE WHEN count(p.amount) = 0 THEN 1 -> ELSE count(p.amount) -> END avg_payment -> FROM customer c -> LEFT OUTER JOIN payment p -> ON c.customer_id = p.customer_id -> GROUP BY c.first_name, c.last_name; +------------+------------+-----------------+--------------+---- ---------+ | first_name | last_name | tot_payment_amt | num_payments | avg_payment | +------------+------------+-----------------+--------------+---- ---------+ | MARY | SMITH | 118.68 | 32 | 3.708750 | | PATRICIA | JOHNSON | 128.73 | 27 | 4.767778 | | LINDA | WILLIAMS | 135.74 | 26 | 5.220769 | | BARBARA | JONES | 81.78 | 22 | 3.717273 | | ELIZABETH | BROWN | 144.62 | 38 | 3.805789 | ... | EDUARDO | HIATT | 130.73 | 27 | 4.841852 | | TERRENCE | GUNDERSON | 117.70 | 30 | 3.923333 | | ENRIQUE | FORSYTHE | 96.72 | 28 | 3.454286 | | FREDDIE | DUGGAN | 99.75 | 25 | 3.990000 | | WADE | DELVALLE | 83.78 | 22 | 3.808182 | | AUSTIN | CINTRON | 83.81 | 19 | 4.411053 | +------------+------------+-----------------+--------------+---- ---------+ 599 rows in set (0.07 sec) This query computes the average payment amount for each customer. Since some customers may be new and have yet to rent a film, it is best to include the case expression to ensure that the denominator is never zero. Conditional Updates When updating rows in a table, you sometimes need conditional logic to generate a value for a column. For example, let’s say that you run a job every week that will set the customer.active column to 0 for any customers who haven’t rented a film in the last 90 days. Here’s a statement that will set the value to either 0 or 1 for every customer: UPDATE customer SET active = CASE WHEN 90 <= (SELECT datediff(now(), max(rental_date)) FROM rental r WHERE r.customer_id = customer.customer_id) THEN 0 ELSE 1 END WHERE active = 1; This statement uses a correlated subquery to determine the number of days since the last rental date for each customer, and compares the value to 90; if the number returned by the subquery is 90 or higher, the customer is marked as Inactive. Handling Null Values While null values are the appropriate thing to store in a table if the value for a column is unknown, it is not always appropriate to retrieve null values for display or to take part in expressions. For example, you might want to display the word unknown on a data entry", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 144 + }, + { + "text": "null values are the appropriate thing to store in a table if the value for a column is unknown, it is not always appropriate to retrieve null values for display or to take part in expressions. For example, you might want to display the word unknown on a data entry screen rather than leaving a field blank. When retrieving the data, you can use a case expression to substitute the string if the value is null, as in: SELECT c.first_name, c.last_name, CASE WHEN a.address IS NULL THEN 'Unknown' ELSE a.address END address, CASE WHEN ct.city IS NULL THEN 'Unknown' ELSE ct.city END city, CASE WHEN cn.country IS NULL THEN 'Unknown' ELSE cn.country END country FROM customer c LEFT OUTER JOIN address a ON c.address_id = a.address_id LEFT OUTER JOIN city ct ON a.city_id = ct.city_id LEFT OUTER JOIN country cn ON ct.country_id = cn.country_id; For calculations, null values often cause a null result, as demonstrated by the following: mysql> SELECT (7 * 5) / ((3 + 14) * null); +-----------------------------+ | (7 * 5) / ((3 + 14) * null) | +-----------------------------+ | NULL | +-----------------------------+ 1 row in set (0.08 sec) When performing calculations, case expressions are useful for translating a null value into a number (usually 0 or 1) that will allow the calculation to yield a non-null value. Test Your Knowledge Challenge your ability to work through conditional logic problems with the examples that follow. When you’re done, compare your solutions with those in Appendix C. Exercise 11-1 Rewrite the following query, which uses a simple case expression, so that the same results are achieved using a searched case expression. Try to use as few when clauses as possible. SELECT name, CASE name WHEN 'English' THEN 'latin1' WHEN 'Italian' THEN 'latin1' WHEN 'French' THEN 'latin1' WHEN 'German' THEN 'latin1' WHEN 'Japanese' THEN 'utf8' WHEN 'Mandarin' THEN 'utf8' ELSE 'Unknown' END character_set FROM language; Exercise 11-2 Rewrite the following query so that the result set contains a single row with five columns (one for each rating). Name the five columns G, PG, PG_13, R, and NC_17. mysql> SELECT rating, count(*) -> FROM film -> GROUP BY rating; +--------+----------+ | rating | count(*) | +--------+----------+ | PG | 194 | | G | 178 | | NC-17 | 210 | | PG-13 | 223 | | R | 195 | +--------+----------+ 5 rows in set (0.00 sec) Chapter 12. Transactions All of the examples thus far in this book have been individual, independent SQL statements. While this may be the norm for ad hoc reporting or data maintenance scripts, application logic will frequently include multiple SQL statements that need to execute together as a logical unit of work. This chapter explores the need and the infrastructure necessary to execute multiple SQL statements concurrently. Multiuser Databases Database management systems allow not only a single user to query and modify data, but multiple people to do so simultaneously. If every user is only executing queries, such as might be the case", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 145 + }, + { + "text": "chapter explores the need and the infrastructure necessary to execute multiple SQL statements concurrently. Multiuser Databases Database management systems allow not only a single user to query and modify data, but multiple people to do so simultaneously. If every user is only executing queries, such as might be the case with a data warehouse during normal business hours, then there are very few issues for the database server to deal with. If some of the users are adding and/or modifying data, however, the server must handle quite a bit more bookkeeping. Let’s say, for example, that you are running a report that sums up the current week’s film rental activity. At the same time you are running the report, however, the following activities are occurring: A customer rents a film. A customer returns a film after the due date and pays a late fee. Five new films are added to inventory. While your report is running, therefore, multiple users are modifying the underlying data, so what numbers should appear on the report? The answer depends somewhat on how your server handles locking, which is described in the next section. Locking Locks are the mechanism the database server uses to control simultaneous use of data resources. When some portion of the database is locked, any other users wishing to modify (or possibly read) that data must wait until the lock has been released. Most database servers use one of two locking strategies: Database writers must request and receive from the server a write lock to modify data, and database readers must request and receive from the server a read lock to query data. While multiple users can read data simultaneously, only one write lock is given out at a time for each table (or portion thereof), and read requests are blocked until the write lock is released. Database writers must request and receive from the server a write lock to modify data, but readers do not need any type of lock to query data. Instead, the server ensures that a reader sees a consistent view of the data (the data seems the same even though other users may be making modifications) from the time her query begins until her query has finished. This approach is known as versioning. There are pros and cons to both approaches. The first approach can lead to long wait times if there are many concurrent read and write requests, and the second approach can be problematic if there are long-running queries while data is being modified. Of the three servers discussed in this book, Microsoft SQL Server uses the first approach, Oracle Database uses the second approach, and MySQL uses both approaches (depending on your choice of storage engine, which we’ll discuss a bit later in the chapter). Lock Granularities There are also a number of different strategies that you may employ when deciding how to lock a resource. The server may apply a lock at one of three different levels, or granularities: Table locks Keep", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 146 + }, + { + "text": "of storage engine, which we’ll discuss a bit later in the chapter). Lock Granularities There are also a number of different strategies that you may employ when deciding how to lock a resource. The server may apply a lock at one of three different levels, or granularities: Table locks Keep multiple users from modifying data in the same table simultaneously Page locks Keep multiple users from modifying data on the same page (a page is a segment of memory generally in the range of 2 KB to 16 KB) of a table simultaneously Row locks Keep multiple users from modifying the same row in a table simultaneously Again, there are pros and cons to these approaches. It takes very little bookkeeping to lock entire tables, but this approach quickly yields unacceptable wait times as the number of users increases. On the other hand, row locking takes quite a bit more bookkeeping, but it allows many users to modify the same table as long as they are interested in different rows. Of the three servers discussed in this book, Microsoft SQL Server uses page, row, and table locking, Oracle Database uses only row locking, and MySQL uses table, page, or row locking (depending, again, on your choice of storage engine). SQL Server will, under certain circumstances, escalate locks from row to page, and from page to table, whereas Oracle Database will never escalate locks. To get back to your report, the data that appears on the pages of the report will mirror either the state of the database when your report started (if your server uses a versioning approach) or the state of the database when the server issues the reporting application a read lock (if your server uses both read and write locks). What Is a Transaction? If database servers enjoyed 100% uptime, if users always allowed programs to finish executing, and if applications always completed without encountering fatal errors that halt execution, then there would be nothing left to discuss regarding concurrent database access. However, we can rely on none of these things, so one more element is necessary to allow multiple users to access the same data. This extra piece of the concurrency puzzle is the transaction, which is a device for grouping together multiple SQL statements such that either all or none of the statements succeed (a property known as atomicity). If you attempt to transfer $500 from your savings account to your checking account, you would be a bit upset if the money were successfully withdrawn from your savings account but never made it to your checking account. Whatever the reason for the failure (the server was shut down for maintenance, the request for a page lock on the account table timed out, etc.), you want your $500 back. To protect against this kind of error, the program that handles your transfer request would first begin a transaction, then issue the SQL statements needed to move the money from your savings to your checking account, and,", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 147 + }, + { + "text": "lock on the account table timed out, etc.), you want your $500 back. To protect against this kind of error, the program that handles your transfer request would first begin a transaction, then issue the SQL statements needed to move the money from your savings to your checking account, and, if everything succeeds, end the transaction by issuing the commit command. If something unexpected happens, however, the program would issue a rollback command, which instructs the server to undo all changes made since the transaction began. The entire process might look something like the following: START TRANSACTION; /* withdraw money from first account, making sure balance is sufficient */ UPDATE account SET avail_balance = avail_balance - 500 WHERE account_id = 9988 AND avail_balance > 500; IF THEN /* deposit money into second account */ UPDATE account SET avail_balance = avail_balance + 500 WHERE account_id = 9989; IF THEN /* everything worked, make the changes permanent */ COMMIT; ELSE /* something went wrong, undo all changes in this transaction */ ROLLBACK; END IF; ELSE /* insufficient funds, or error encountered during update */ ROLLBACK; END IF; NOTE While the previous code block may look similar to one of the procedural languages provided by the major database companies, such as Oracle’s PL/SQL or Microsoft’s Transact-SQL, it is written in pseudocode and does not attempt to mimic any particular language. The previous code block begins by starting a transaction and then attempts to remove $500 from the checking account and add it to the savings account. If all goes well, the transaction is committed; if anything goes awry, however, the transaction is rolled back, meaning that all data changes since the beginning of the transaction are undone. By using a transaction, the program ensures that your $500 either stays in your savings account or moves to your checking account, without the possibility of it falling into a crack. Regardless of whether the transaction was committed or was rolled back, all resources acquired (e.g., write locks) during the execution of the transaction are released when the transaction completes. Of course, if the program manages to complete both update statements but the server shuts down before a commit or rollback can be executed, then the transaction will be rolled back when the server comes back online. (One of the tasks that a database server must complete before coming online is to find any incomplete transactions that were underway when the server shut down and roll them back.) Additionally, if your program finishes a transaction and issues a commit, but the server shuts down before the changes have been applied to permanent storage (i.e., the modified data is sitting in memory but has not been flushed to disk), then the database server must reapply the changes from your transaction when the server is restarted (a property known as durability). Starting a Transaction Database servers handle transaction creation in", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 148 + }, + { + "text": "been applied to permanent storage (i.e., the modified data is sitting in memory but has not been flushed to disk), then the database server must reapply the changes from your transaction when the server is restarted (a property known as durability). Starting a Transaction Database servers handle transaction creation in one of two ways: An active transaction is always associated with a database session, so there is no need or method to explicitly begin a transaction. When the current transaction ends, the server automatically begins a new transaction for your session. Unless you explicitly begin a transaction, individual SQL statements are automatically committed independently of one another. To begin a transaction, you must first issue a command. Of the three servers, Oracle Database takes the first approach, while Microsoft SQL Server and MySQL take the second approach. One of the advantages of Oracle’s approach to transactions is that, even if you are issuing only a single SQL command, you have the ability to roll back the changes if you don’t like the outcome or if you change your mind. Thus, if you forget to add a where clause to your delete statement, you will have the opportunity to undo the damage (assuming you’ve had your morning coffee and realize that you didn’t mean to delete all 125,000 rows in your table). With MySQL and SQL Server, however, once you press the Enter key, the changes brought about by your SQL statement will be permanent (unless your DBA can retrieve the original data from a backup or from some other means). The SQL:2003 standard includes a start transaction command to be used when you want to explicitly begin a transaction. While MySQL conforms to the standard, SQL Server users must instead issue the command begin transaction. With both servers, until you explicitly begin a transaction, you are in what is known as auto-commit mode, which means that individual statements are automatically committed by the server. You can, therefore, decide that you want to be in a transaction and issue a start/begin transaction command, or you can simply let the server commit individual statements. Both MySQL and SQL Server allow you to turn off auto-commit mode for individual sessions, in which case, the servers will act just like Oracle Database regarding transactions. With SQL Server, you issue the following command to disable auto-commit mode: SET IMPLICIT_TRANSACTIONS ON MySQL allows you to disable auto-commit mode via the following: SET AUTOCOMMIT=0 Once you have left auto-commit mode, all SQL commands take place within the scope of a transaction and must be explicitly committed or rolled back. NOTE A word of advice: shut off auto-commit mode each time you log in, and get in the habit of running all of your SQL statements within a transaction. If nothing else, it may save you the embarrassment of having to ask your DBA to reconstruct data that you have inadvertently deleted. Ending a Transaction Once a transaction has begun, whether explicitly via the start transaction command or", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 149 + }, + { + "text": "habit of running all of your SQL statements within a transaction. If nothing else, it may save you the embarrassment of having to ask your DBA to reconstruct data that you have inadvertently deleted. Ending a Transaction Once a transaction has begun, whether explicitly via the start transaction command or implicitly by the database server, you must explicitly end your transaction for your changes to become permanent. You do this by way of the commit command, which instructs the server to mark the changes as permanent and release any resources (i.e., page or row locks) used during the transaction. If you decide that you want to undo all the changes made since starting the transaction, you must issue the rollback command, which instructs the server to return the data to its pre-transaction state. After the rollback has been completed, any resources used by your session are released. Along with issuing either the commit or rollback command, there are several other scenarios by which your transaction can end, either as an indirect result of your actions or as a result of something outside your control: The server shuts down, in which case, your transaction will be rolled back automatically when the server is restarted. You issue an SQL schema statement, such as alter table, which will cause the current transaction to be committed and a new transaction to be started. You issue another start transaction command, which will cause the previous transaction to be committed. The server prematurely ends your transaction because the server detects a deadlock and decides that your transaction is the culprit. In this case, the transaction will be rolled back and you will receive an error message. Of these four scenarios, the first and third are fairly straightforward, but the other two merit some discussion. As far as the second scenario is concerned, alterations to a database, whether it be the addition of a new table or index or the removal of a column from a table, cannot be rolled back, so commands that alter your schema must take place outside a transaction. If a transaction is currently underway, therefore, the server will commit your current transaction, execute the SQL schema statement command(s), and then automatically start a new transaction for your session. The server will not inform you of what has happened, so you should be careful that the statements that comprise a unit of work are not inadvertently broken up into multiple transactions by the server. The fourth scenario deals with deadlock detection. A deadlock occurs when two different transactions are waiting for resources that the other transaction currently holds. For example, transaction A might have just updated the account table and is waiting for a write lock on the transaction table, while transaction B has inserted a row into the transaction table and is waiting for a write lock on the account table. If both transactions happen to be modifying the same page or row (depending on the lock granularity in use by the", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 150 + }, + { + "text": "a write lock on the transaction table, while transaction B has inserted a row into the transaction table and is waiting for a write lock on the account table. If both transactions happen to be modifying the same page or row (depending on the lock granularity in use by the database server), then they will each wait forever for the other transaction to finish and free up the needed resource. Database servers must always be on the lookout for these situations so that throughput doesn’t grind to a halt; when a deadlock is detected, one of the transactions is chosen (either arbitrarily or by some criteria) to be rolled back so that the other transaction may proceed. Most of the time, the terminated transaction can be restarted and will succeed without encountering another deadlock situation. Unlike the second scenario discussed earlier, the database server will raise an error to inform you that your transaction has been rolled back due to deadlock detection. With MySQL, for example, you will receive error #1213, which carries the following message: Message: Deadlock found when trying to get lock; try restarting transaction As the error message suggests, it is a reasonable practice to retry a transaction that has been rolled back due to deadlock detection. However, if deadlocks become fairly common, then you may need to modify the applications that access the database to decrease the probability of deadlocks (one common strategy is to ensure that data resources are always accessed in the same order, such as always modifying account data before inserting transaction data). Transaction Savepoints In some cases, you may encounter an issue within a transaction that requires a rollback, but you may not want to undo all of the work that has transpired. For these situations, you can establish one or more savepoints within a transaction and use them to roll back to a particular location within your transaction rather than rolling all the way back to the start of the transaction. CHOOSING A STORAGE ENGINE When using Oracle Database or Microsoft SQL Server, a single set of code is responsible for low-level database operations, such as retrieving a particular row from a table based on primary key value. The MySQL server, however, has been designed so that multiple storage engines may be utilized to provide low-level database functionality, including resource locking and transaction management. As of version 8.0, MySQL includes the following storage engines: MyISAM A nontransactional engine employing table locking MEMORY A nontransactional engine used for in-memory tables CSV A transactional engine which stores data in comma-separated files InnoDB A transactional engine employing row-level locking Merge A specialty engine used to make multiple identical MyISAM tables appear as a single table (a.k.a. table partitioning) Archive A specialty engine used to store large amounts of unindexed data, mainly for archival purposes Although you might think that you would be forced to choose a single storage engine for your database, MySQL is flexible enough to allow you to choose a storage engine", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 151 + }, + { + "text": "table (a.k.a. table partitioning) Archive A specialty engine used to store large amounts of unindexed data, mainly for archival purposes Although you might think that you would be forced to choose a single storage engine for your database, MySQL is flexible enough to allow you to choose a storage engine on a table-by-table basis. For any tables that might take part in transactions, however, you should choose the InnoDB engine, which uses row-level locking and versioning to provide the highest level of concurrency across the different storage engines. You may explicitly specify a storage engine when creating a table, or you can change an existing table to use a different engine. If you do not know what engine is assigned to a table, you can use the show table command, as demonstrated by the following: mysql> show table status like 'customer' \\G; *************************** 1. row *************************** Name: customer Engine: InnoDB Version: 10 Row_format: Dynamic Rows: 599 Avg_row_length: 136 Data_length: 81920 Max_data_length: 0 Index_length: 49152 Data_free: 0 Auto_increment: 599 Create_time: 2019-03-12 14:24:46 Update_time: NULL Check_time: NULL Collation: utf8_general_ci Checksum: NULL Create_options: Comment: 1 row in set (0.16 sec) Looking at the second item, you can see that the Customer table is already using the InnoDB engine. If it were not, you could assign the InnoDB engine to the transaction table via the following command: ALTER TABLE customer ENGINE = INNODB; All savepoints must be given a name, which allows you to have multiple savepoints within a single transaction. To create a savepoint named my_savepoint, you can do the following: SAVEPOINT my_savepoint; To roll back to a particular savepoint, you simply issue the rollback command followed by the keywords to savepoint and the name of the savepoint, as in: ROLLBACK TO SAVEPOINT my_savepoint; Here’s an example of how savepoints may be used: START TRANSACTION; UPDATE product SET date_retired = CURRENT_TIMESTAMP() WHERE product_cd = 'XYZ'; SAVEPOINT before_close_accounts; UPDATE account SET status = 'CLOSED', close_date = CURRENT_TIMESTAMP(), last_activity_date = CURRENT_TIMESTAMP() WHERE product_cd = 'XYZ'; ROLLBACK TO SAVEPOINT before_close_accounts; COMMIT; The net effect of this transaction is that the mythical XYZ product is retired but none of the accounts are closed. When using savepoints, remember the following: Despite the name, nothing is saved when you create a savepoint. You must eventually issue a commit if you want your transaction to be made permanent. If you issue a rollback without naming a savepoint, all savepoints within the transaction will be ignored and the entire transaction will be undone. If you are using SQL Server, you will need to use the proprietary command save transaction to create a savepoint and rollback transaction to roll back to a savepoint, with each command being followed by the savepoint name. Test Your Knowledge Test your understanding of transactions by working through the following exercise. When you’re done, compare your solutions with Appendix C. Exercise 12-1 Generate a unit of work to transfer $50 from account 123 to account 789. You will need to insert two rows into the transaction table", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 152 + }, + { + "text": "Test Your Knowledge Test your understanding of transactions by working through the following exercise. When you’re done, compare your solutions with Appendix C. Exercise 12-1 Generate a unit of work to transfer $50 from account 123 to account 789. You will need to insert two rows into the transaction table and update two rows in the account table. Use the following table definitions/data: Account: account_id avail_balance last_activity_date ---------- ------------- ------------------ 123 500 2019-07-10 20:53:27 789 75 2019-06-22 15:18:35 Transaction: txn_id txn_date account_id txn_type_cd amount --------- ------------ ----------- ----------- -------- 1001 2019-05-15 123 C 500 1002 2019-06-01 789 C 75 Chapter 13. Indexes and Constraints Because the focus of this book is on programming techniques, the first 12 chapters concentrated on elements of the SQL language that you can use to craft powerful select, insert, update, and delete statements. However, other database features indirectly affect the code you write. This chapter focuses on two of those features: indexes and constraints. Indexes When you insert a row into a table, the database server does not attempt to put the data in any particular location within the table. For example, if you add a row to the customer table, the server doesn’t place the row in numeric order via the customer_id column or in alphabetical order via the last_name column. Instead, the server simply places the data in the next available location within the file (the server maintains a list of free space for each table). When you query the customer table, therefore, the server will need to inspect every row of the table to answer the query. For example, let’s say that you issue the following query: mysql> SELECT first_name, last_name -> FROM customer -> WHERE last_name LIKE 'Y%'; +------------+-----------+ | first_name | last_name | +------------+-----------+ | LUIS | YANEZ | | MARVIN | YEE | | CYNTHIA | YOUNG | +------------+-----------+ 3 rows in set (0.09 sec) To find all customers whose last name begins with Y, the server must visit each row in the customer table and inspect the contents of the last_name column; if the department name begins with Y, then the row is added to the result set. This type of access is known as a table scan. While this method works fine for a table with only three rows, imagine how long it might take to answer the query if the table contains 3 million rows. At some number of rows larger than three and smaller than 3 million, a line is crossed where the server cannot answer the query within a reasonable amount of time without additional help. This help comes in the form of one or more indexes on the customer table. Even if you have never heard of a database index, you are certainly aware of what an index is (e.g., this book has one). An index is simply a mechanism for finding a specific item within a resource. Each technical publication, for example, includes an index at the end that allows you to", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 153 + }, + { + "text": "have never heard of a database index, you are certainly aware of what an index is (e.g., this book has one). An index is simply a mechanism for finding a specific item within a resource. Each technical publication, for example, includes an index at the end that allows you to locate a specific word or phrase within the publication. The index lists these words and phrases in alphabetical order, allowing the reader to move quickly to a particular letter within the index, find the desired entry, and then find the page or pages on which the word or phrase may be found. In the same way that a person uses an index to find words within a publication, a database server uses indexes to locate rows in a table. Indexes are special tables that, unlike normal data tables, are kept in a specific order. Instead of containing all of the data about an entity, however, an index contains only the column (or columns) used to locate rows in the data table, along with information describing where the rows are physically located. Therefore, the role of indexes is to facilitate the retrieval of a subset of a table’s rows and columns without the need to inspect every row in the table. Index Creation Returning to the customer table, you might decide to add an index on the email column to speed up any queries that specify a value for this column, as well as any update or delete operations that specify a customer’s email address. Here’s how you can add such an index to a MySQL database: mysql> ALTER TABLE customer -> ADD INDEX idx_email (email); Query OK, 0 rows affected (1.87 sec) Records: 0 Duplicates: 0 Warnings: 0 This statement creates an index (a B-tree index to be precise, but more on this shortly) on the customer.email column; furthermore, the index is given the name idx_email. With the index in place, the query optimizer (which we discussed in Chapter 3) can choose to use the index if it is deemed beneficial to do so. If there is more than one index on a table, the optimizer must decide which index will be the most beneficial for a particular SQL statement. NOTE MySQL treats indexes as optional components of a table, which is why in earlier versions you would use the alter table command to add or remove an index. Other database servers, including SQL Server and Oracle Database, treat indexes as independent schema objects. For both SQL Server and Oracle, therefore, you would generate an index using the create index command, as in: CREATE INDEX idx_email ON customer (email); As of MySQL version 5, a create index command command is available, although it is mapped to the alter table command. You must still use the alter table command to create primary key indexes, however. All database servers allow you to look at the available indexes. MySQL users can use the show command to see all of the indexes on a", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 154 + }, + { + "text": "available, although it is mapped to the alter table command. You must still use the alter table command to create primary key indexes, however. All database servers allow you to look at the available indexes. MySQL users can use the show command to see all of the indexes on a specific table, as in: mysql> SHOW INDEX FROM customer \\G; *************************** 1. row *************************** Table: customer Non_unique: 0 Key_name: PRIMARY Seq_in_index: 1 Column_name: customer_id Collation: A Cardinality: 599 Sub_part: NULL Packed: NULL Null: Index_type: BTREE ... *************************** 2. row *************************** Table: customer Non_unique: 1 Key_name: idx_fk_store_id Seq_in_index: 1 Column_name: store_id Collation: A Cardinality: 2 Sub_part: NULL Packed: NULL Null: Index_type: BTREE ... *************************** 3. row *************************** Table: customer Non_unique: 1 Key_name: idx_fk_address_id Seq_in_index: 1 Column_name: address_id Collation: A Cardinality: 599 Sub_part: NULL Packed: NULL Null: Index_type: BTREE ... *************************** 4. row *************************** Table: customer Non_unique: 1 Key_name: idx_last_name Seq_in_index: 1 Column_name: last_name Collation: A Cardinality: 599 Sub_part: NULL Packed: NULL Null: Index_type: BTREE ... *************************** 5. row *************************** Table: customer Non_unique: 1 Key_name: idx_email Seq_in_index: 1 Column_name: email Collation: A Cardinality: 599 Sub_part: NULL Packed: NULL Null: YES Index_type: BTREE ... 5 rows in set (0.06 sec) The output shows that there are five indexes on the customer table: one on the customer_id column called PRIMARY, and four others on the store_id, address_id, last_name, and email columns. If you are wondering where these indexes came from, I created the index on the email column, and the rest were installed as part of the sample Sakila database. Here’s the statement used to create the table: CREATE TABLE customer ( customer_id SMALLINT UNSIGNED NOT NULL AUTO_INCREMENT, store_id TINYINT UNSIGNED NOT NULL, first_name VARCHAR(45) NOT NULL, last_name VARCHAR(45) NOT NULL, email VARCHAR(50) DEFAULT NULL, address_id SMALLINT UNSIGNED NOT NULL, active BOOLEAN NOT NULL DEFAULT TRUE, create_date DATETIME NOT NULL, last_update TIMESTAMP DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (customer_id), KEY idx_fk_store_id (store_id), KEY idx_fk_address_id (address_id), KEY idx_last_name (last_name), ... When the table was created, the MySQL server automatically generated an index on the primary key column, which, in this case is customer_id, and gave the index the name PRIMARY. I cover constraints later in this chapter. If, after creating an index, you decide that the index is not proving useful, you can remove it via the following: mysql> ALTER TABLE customer -> DROP INDEX idx_email; Query OK, 0 rows affected (0.50 sec) Records: 0 Duplicates: 0 Warnings: 0 NOTE SQL Server and Oracle Database users must use the drop index command to remove an index, as in: DROP INDEX idx_email; (Oracle) DROP INDEX idx_email ON customer; (SQL Server) MySQL now also supports the drop index command, although it is also mapped to the alter table command. UNIQUE INDEXES When designing a database, it is important to consider which columns are allowed to contain duplicate data and which are not. For example, it is allowable to have two customers named John Smith in the customer table since each row will have a different identifier (customer_id), email, and", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 155 + }, + { + "text": "UNIQUE INDEXES When designing a database, it is important to consider which columns are allowed to contain duplicate data and which are not. For example, it is allowable to have two customers named John Smith in the customer table since each row will have a different identifier (customer_id), email, and address to help tell them apart. You would not, however, want to allow two different customers to have the same email address. You can enforce a rule against duplicate values by creating a unique index on the customer.email column. A unique index plays multiple roles; along with providing all the benefits of a regular index, it also serves as a mechanism for disallowing duplicate values in the indexed column. Whenever a row is inserted or when the indexed column is modified, the database server checks the unique index to see whether the value already exists in another row in the table. Here’s how you would create a unique index on the customer.email column: mysql> ALTER TABLE customer -> ADD UNIQUE idx_email (email); Query OK, 0 rows affected (0.64 sec) Records: 0 Duplicates: 0 Warnings: 0 NOTE SQL Server and Oracle Database users need only add the unique keyword when creating an index, as in: CREATE UNIQUE INDEX idx_email ON customer (email); With the index in place, you will receive an error if you try to add a new customer with an email address which already exists: mysql> INSERT INTO customer -> (store_id, first_name, last_name, email, address_id, active) -> VALUES -> (1,'ALAN','KAHN', 'ALAN.KAHN@sakilacustomer.org', 394, 1); ERROR 1062 (23000): Duplicate entry 'ALAN.KAHN@sakilacustomer.org' for key 'idx_email' You should not build unique indexes on your primary key column(s), since the server already checks uniqueness for primary key values. You may, however, create more than one unique index on the same table if you feel that it is warranted. MULTICOLUMN INDEXES Along with the single-column indexes demonstrated thus far, you may also build indexes that span multiple columns. If, for example, you find yourself searching for customers by first and last names, you can build an index on both columns together, as in: mysql> ALTER TABLE customer -> ADD INDEX idx_full_name (last_name, first_name); Query OK, 0 rows affected (0.35 sec) Records: 0 Duplicates: 0 Warnings: 0 This index will be useful for queries that specify the first and last names or just the last name, but it would not be useful for queries that specify only the customer’s first name. To understand why, consider how you would find a person’s phone number; if you know the person’s first and last names, you can use a phone book to find the number quickly, since a phone book is organized by last name and then by first name. If you know only the person’s first name, you would need to scan every entry in the phone book to find all the entries with the specified first name. When building multiple-column indexes, therefore, you should think carefully about which column to list first, which column to list second,", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 156 + }, + { + "text": "name. If you know only the person’s first name, you would need to scan every entry in the phone book to find all the entries with the specified first name. When building multiple-column indexes, therefore, you should think carefully about which column to list first, which column to list second, and so on to help make the index as useful as possible. Keep in mind, however, that there is nothing stopping you from building multiple indexes using the same set of columns but in a different order if you feel that it is needed to ensure adequate response time. Types of Indexes Indexing is a powerful tool, but since there are many different types of data, a single indexing strategy doesn’t always do the job. The following sections illustrate the different types of indexing available from various servers. B-TREE INDEXES All the indexes shown thus far are balanced-tree indexes, which are more commonly known as B-tree indexes. MySQL, Oracle Database, and SQL Server all default to B-tree indexing, so you will get a B-tree index unless you explicitly ask for another type. As you might expect, B-tree indexes are organized as trees, with one or more levels of branch nodes leading to a single level of leaf nodes. Branch nodes are used for navigating the tree, while leaf nodes hold the actual values and location information. For example, a B-tree index built on the customer.last_name column might look something like Figure 13-1. Figure 13-1. B-tree example If you were to issue a query to retrieve all customers whose last name starts with G, the server would look at the top branch node (called the root node) and follow the link to the branch node that handles last names beginning with A through M. This branch node would, in turn, direct the server to a leaf node containing last names beginning with G through I. The server then starts reading the values in the leaf node until it encounters a value that doesn’t begin with G (which, in this case, is Hawthorne). As rows are inserted, updated, and deleted from the customer table, the server will attempt to keep the tree balanced so that there aren’t far more branch/leaf nodes on one side of the root node than the other. The server can add or remove branch nodes to redistribute the values more evenly and can even add or remove an entire level of branch nodes. By keeping the tree balanced, the server is able to traverse quickly to the leaf nodes to find the desired values without having to navigate through many levels of branch nodes. BITMAP INDEXES Although B-tree indexes are great at handling columns that contain many different values, such as a customer’s first/last names, they can become unwieldy when built on a column that allows only a small number of values. For example, you may decide to generate an index on the customer.active column so that you can quickly retrieve all Active or Inactive accounts. Because there", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 157 + }, + { + "text": "values, such as a customer’s first/last names, they can become unwieldy when built on a column that allows only a small number of values. For example, you may decide to generate an index on the customer.active column so that you can quickly retrieve all Active or Inactive accounts. Because there are only two different values (stored as 1 for Active and 0 for Inactive), however, and because there are far more Active customers, it can be difficult to maintain a balanced B-tree index as the number of customers grows. For columns that contain only a small number of values across a large number of rows (known as low-cardinality data), a different indexing strategy is needed. To handle this situation more efficiently, Oracle Database includes bitmap indexes, which generate a bitmap for each value stored in the column. If you were to build a bitmap index on the customer.active column, the index would maintain two bitmaps: one for the value 0, and another for the value 1. When you write a query to retrieve all Inactive customers, the database server can traverse the 0 bitmap and quickly retrieve the desired rows. Bitmap indexes are a nice, compact indexing solution for low-cardinality data, but this indexing strategy breaks down if the number of values stored in the column climbs too high in relation to the number of rows (known as high-cardinality data), since the server would need to maintain too many bitmaps. For example, you would never build a bitmap index on your primary key column, since this represents the highest possible cardinality (a different value for every row). Oracle users can generate bitmap indexes by simply adding the bitmap keyword to the create index statement, as in: CREATE BITMAP INDEX idx_active ON customer (active); Bitmap indexes are commonly used in data warehousing environments, where large amounts of data are generally indexed on columns containing relatively few values (e.g., sales quarters, geographic regions, products, salespeople). TEXT INDEXES If your database stores documents, you may need to allow users to search for words or phrases in the documents. You certainly don’t want the server to peruse each document and scan for the desired text each time a search is requested, but traditional indexing strategies don’t work for this situation. To handle this situation, MySQL, SQL Server, and Oracle Database include specialized indexing and search mechanisms for documents; both SQL Server and MySQL include what they call full-text indexes, and Oracle Database includes a powerful set of tools known as Oracle Text. Document searches are specialized enough that I refrain from showing an example, but I wanted you to at least know what is available. How Indexes Are Used Indexes are generally used by the server to quickly locate rows in a particular table, after which the server visits the associated table to extract the additional information requested by the user. Consider the following query: mysql> SELECT customer_id, first_name, last_name -> FROM customer -> WHERE first_name LIKE 'S%' AND last_name LIKE 'P%'; +-------------+------------+-----------+ |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 158 + }, + { + "text": "the server to quickly locate rows in a particular table, after which the server visits the associated table to extract the additional information requested by the user. Consider the following query: mysql> SELECT customer_id, first_name, last_name -> FROM customer -> WHERE first_name LIKE 'S%' AND last_name LIKE 'P%'; +-------------+------------+-----------+ | customer_id | first_name | last_name | +-------------+------------+-----------+ | 84 | SARA | PERRY | | 197 | SUE | PETERS | | 167 | SALLY | PIERCE | +-------------+------------+-----------+ 3 rows in set (0.00 sec) For this query, the server can employ any of the following strategies: Scan all rows in the customer table Use the index on the last_name column to find all customers whose last name starts with P, and then visit each row of the customer table to find only rows whose first name starts with S Use the index on the last_name, first_name columns to find all customers whose last name starts with P and whose first name starts with S The third choice seems to be the best option, since the index will yield all of the rows needed for the result set, without the need to revisit the table. But how do you know which of the three options will be utilized? To see how MySQL’s query optimizer decides to execute the query, I use the explain statement to ask the server to show the execution plan for the query rather than executing the query: mysql> EXPLAIN -> SELECT customer_id, first_name, last_name -> FROM customer -> WHERE first_name LIKE 'S%' AND last_name LIKE 'P%' \\G; *************************** 1. row *************************** id: 1 select_type: SIMPLE table: customer partitions: NULL type: range possible_keys: idx_last_name,idx_full_name key: idx_full_name key_len: 274 ref: NULL rows: 28 filtered: 11.11 Extra: Using where; Using index 1 row in set, 1 warning (0.00 sec) NOTE Each database server includes tools to allow you to see how the query optimizer handles your SQL statement. SQL Server allows you to see an execution plan by issuing the statement set showplan_text on before running your SQL statement. Oracle Database includes the explain plan statement, which writes the execution plan to a special table called plan_table. Looking at the query results, the possible_keys column tells you that the server could decide to use either the idx_last_name or the idx_full_name index, and the key column tells you that the idx_full_name index was chosen. Furthermore, the type column tells you that a range-scan will be utilized, meaning that the database server will be looking for a range of values in the index, rather than expecting to retrieve a single row. NOTE The process that I just led you through is an example of query tuning. Tuning involves looking at an SQL statement and determining the resources available to the server to execute the statement. You can decide to modify the SQL statement, to adjust the database resources, or to do both in order to make a statement run more efficiently. Tuning is a detailed topic, and I strongly urge you", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 159 + }, + { + "text": "SQL statement and determining the resources available to the server to execute the statement. You can decide to modify the SQL statement, to adjust the database resources, or to do both in order to make a statement run more efficiently. Tuning is a detailed topic, and I strongly urge you to either read your server’s tuning guide or pick up a good tuning book so that you can see all the different approaches available for your server. The Downside of Indexes If indexes are so great, why not index everything? Well, the key to understanding why more indexes are not necessarily a good thing is to keep in mind that every index is a table (a special type of table, but still a table). Therefore, every time a row is added to or removed from a table, all indexes on that table must be modified. When a row is updated, any indexes on the column or columns that were affected need to be modified as well. Therefore, the more indexes you have, the more work the server needs to do to keep all schema objects up-to-date, which tends to slow things down. Indexes also require disk space as well as some amount of care from your administrators, so the best strategy is to add an index when a clear need arises. If you need an index for only special purposes, such as a monthly maintenance routine, you can always add the index, run the routine, and then drop the index until you need it again. In the case of data warehouses, where indexes are crucial during business hours as users run reports and ad hoc queries but are problematic when data is being loaded into the warehouse overnight, it is a common practice to drop the indexes before data is loaded and then re-create them before the warehouse opens for business. In general, you should strive to have neither too many indexes nor too few. If you aren’t sure how many indexes you should have, you can use this strategy as a default: Make sure all primary key columns are indexed (most servers automatically create unique indexes when you create primary key constraints). For multicolumn primary keys, consider building additional indexes on a subset of the primary key columns, or on all the primary key columns but in a different order than the primary key constraint definition. Build indexes on all columns that are referenced in foreign key constraints. Keep in mind that the server checks to make sure there are no child rows when a parent is deleted, so it must issue a query to search for a particular value in the column. If there’s no index on the column, the entire table must be scanned. Index any columns that will frequently be used to retrieve data. Most date columns are good candidates, along with short (2- to 50-character) string columns. After you have built your initial set of indexes, try to capture actual queries against your tables, look", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 160 + }, + { + "text": "column, the entire table must be scanned. Index any columns that will frequently be used to retrieve data. Most date columns are good candidates, along with short (2- to 50-character) string columns. After you have built your initial set of indexes, try to capture actual queries against your tables, look at the server’s execution plan, and modify your indexing strategy to fit the most-common access paths. Constraints A constraint is simply a restriction placed on one or more columns of a table. There are several different types of constraints, including: Primary key constraints Identify the column or columns that guarantee uniqueness within a table Foreign key constraints Restrict one or more columns to contain only values found in another table’s primary key columns, and may also restrict the allowable values in other tables if update cascade or delete cascade rules are established Unique constraints Restrict one or more columns to contain unique values within a table (primary key constraints are a special type of unique constraint) Check constraints Restrict the allowable values for a column Without constraints, a database’s consistency is suspect. For example, if the server allows you to change a customer’s ID in the customer table without changing the same customer ID in the rental table, then you will end up with rental data that no longer point to valid customer records (known as orphaned rows). With primary and foreign key constraints in place, however, the server will either raise an error if an attempt is made to modify or delete data that is referenced by other tables, or propagate the changes to other tables for you (more on this shortly). NOTE If you want to use foreign key constraints with the MySQL server, you must use the InnoDB storage engine for your tables. Constraint Creation Constraints are generally created at the same time as the associated table via the create table statement. To illustrate, here’s an example from the schema generation script for the Sakila sample database: CREATE TABLE customer ( customer_id SMALLINT UNSIGNED NOT NULL AUTO_INCREMENT, store_id TINYINT UNSIGNED NOT NULL, first_name VARCHAR(45) NOT NULL, last_name VARCHAR(45) NOT NULL, email VARCHAR(50) DEFAULT NULL, address_id SMALLINT UNSIGNED NOT NULL, active BOOLEAN NOT NULL DEFAULT TRUE, create_date DATETIME NOT NULL, last_update TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (customer_id), KEY idx_fk_store_id (store_id), KEY idx_fk_address_id (address_id), KEY idx_last_name (last_name), CONSTRAINT fk_customer_address FOREIGN KEY (address_id) REFERENCES address (address_id) ON DELETE RESTRICT ON UPDATE CASCADE, CONSTRAINT fk_customer_store FOREIGN KEY (store_id) REFERENCES store (store_id) ON DELETE RESTRICT ON UPDATE CASCADE )ENGINE=InnoDB DEFAULT CHARSET=utf8; The customer table includes three constraints: one to specify that the customer_id column serves as the primary key for the table, and two more to specify that the address_id and store_id columns serve as foreign keys to the address and store table. Alternatively, you could create the customer table without foreign key constraints, and add the foreign key constraints later via alter table statements: ALTER TABLE customer ADD CONSTRAINT fk_customer_address FOREIGN KEY (address_id) REFERENCES address (address_id) ON DELETE", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 161 + }, + { + "text": "address_id and store_id columns serve as foreign keys to the address and store table. Alternatively, you could create the customer table without foreign key constraints, and add the foreign key constraints later via alter table statements: ALTER TABLE customer ADD CONSTRAINT fk_customer_address FOREIGN KEY (address_id) REFERENCES address (address_id) ON DELETE RESTRICT ON UPDATE CASCADE; ALTER TABLE customer ADD CONSTRAINT fk_customer_store FOREIGN KEY (store_id) REFERENCES store (store_id) ON DELETE RESTRICT ON UPDATE CASCADE; Both of these statements include several ON clauses: ON DELETE RESRICT, which will cause the server to raise an error if a row is deleted in the parent table (address or store) which is referenced in the child table (customer) ON UPDATE CASCADE, which will cause the server to propagate a chance to the primary key value of a parent table (address or store) to the child table (customer) The ON DELETE RESTRICT clause protects against orphaned records when rows are deleted from the parent table. To illustrate, let’s pick a row in the address table and show the data from both the address and customer tables that share this value: mysql> SELECT c.first_name, c.last_name, c.address_id, a.address -> FROM customer c -> INNER JOIN address a -> ON c.address_id = a.address_id -> WHERE a.address_id = 123; +------------+-----------+------------+------------------------- ---------+ | first_name | last_name | address_id | address | +------------+-----------+------------+------------------------- ---------+ | SHERRY | MARSHALL | 123 | 1987 Coacalco de Berriozbal Loop | +------------+-----------+------------+------------------------- ---------+ 1 row in set (0.00 sec) The results show that there is a single customer row ( for Sherry Marshall) whose address_id column contains the value 123. Here’s what happens if you try to remove this row from the parent (address) table: mysql> DELETE FROM address WHERE address_id = 123; ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key constraint fails (`sakila`.`customer`, CONSTRAINT `fk_customer_address` FOREIGN KEY (`address_id`) REFERENCES `address` (`address_id`) ON DELETE RESTRICT ON UPDATE CASCADE) Because at least one row in the child table contains the value 123 in the address_id column, the ON DELETE RESTRICT clause of the foreign key constraint caused the statement to fail. The ON UPDATE CASCADE clause also protects against orphaned records when a primary key value is updated in the parent table, but using a different strategy. Here’s what happens if you modify a value in the address.address_id column: mysql> UPDATE address -> SET address_id = 9999 -> WHERE address_id = 123; Query OK, 1 row affected (0.37 sec) Rows matched: 1 Changed: 1 Warnings: 0 The statement executed without error, and 1 row was modified. But what happened to Sherry Marshall’s row in the customer table? Does it still point to address ID 123, which no longer exists? To find out, let’s run the last query again, but substitute the new value 9999 for the previous value of 123: mysql> SELECT c.first_name, c.last_name, c.address_id, a.address -> FROM customer c -> INNER JOIN address a -> ON c.address_id = a.address_id -> WHERE a.address_id = 9999; +------------+-----------+------------+------------------------- ---------+ | first_name | last_name | address_id", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 162 + }, + { + "text": "run the last query again, but substitute the new value 9999 for the previous value of 123: mysql> SELECT c.first_name, c.last_name, c.address_id, a.address -> FROM customer c -> INNER JOIN address a -> ON c.address_id = a.address_id -> WHERE a.address_id = 9999; +------------+-----------+------------+------------------------- ---------+ | first_name | last_name | address_id | address | +------------+-----------+------------+------------------------- ---------+ | SHERRY | MARSHALL | 9999 | 1987 Coacalco de Berriozbal Loop | +------------+-----------+------------+------------------------- ---------+ 1 row in set (0.00 sec) As you can see, the same results are returned as before (other than the new address ID value), which means that the value 9999 was automatically updated in the customer table. This is known as a cascade, and it’s the second mechanism used to protect against orphaned rows. Along with RESTRICT and CASCADE, you can also choose SET NULL, which will set the foreign key value to NULL in the child table when a row is deleted or updated in the parent table. All together, there are six different options to choose from when defining foreign key constraints: ON DELETE RESTRICT ON DELETE CASCADE ON DELETE SET NULL ON UPDATE RESTRICT ON UPDATE CASCADE ON UPDATE SET NULL These are optional, so you can choose zero, one, or two (one ON DELETE and one ON UPDATE) of these when defining your foreign key constraints. Finally, if you want to remove a primary or foreign key constraint, you can use the alter table statement again, except that you specify drop instead of add. While it is unusual to drop a primary key constraint, foreign key constraints are sometimes dropped during certain maintenance operations and then reestablished. Test Your Knowledge Work through the following exercises to test your knowledge of indexes and constraints. When you’re done, compare your solutions with those in Appendix C. Exercise 13-1 Generate an ALTER TABLE statement for the rental table so that an error will be raised if a row is deleted from the customer table having a value found in the rental.customer_id column. Exercise 13-2 Generate a multicolumn index on the payment table that could be used by both of the following queries: SELECT customer_id, payment_date, amount FROM payment WHERE payment_date > cast('2019-12-31 23:59:59' as datetime); SELECT customer_id, payment_date, amount FROM payment WHERE payment_date > cast('2019-12-31 23:59:59' as datetime) AND amount < 5; Chapter 14. Views Well-designed applications generally expose a public interface while keeping implementation details private, thereby enabling future design changes without impacting end users. When designing your database, you can achieve a similar result by keeping your tables private and allowing your users to access data only through a set of views. This chapter strives to define what views are, how they are created, and when and how you might want to use them. What Are Views? A view is simply a mechanism for querying data. Unlike tables, views do not involve data storage; you won’t need to worry about views filling up your disk space. You create a view by assigning a name to a select", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 163 + }, + { + "text": "how you might want to use them. What Are Views? A view is simply a mechanism for querying data. Unlike tables, views do not involve data storage; you won’t need to worry about views filling up your disk space. You create a view by assigning a name to a select statement, and then storing the query for others to use. Other users can then use your view to access data just as though they were querying tables directly (in fact, they may not even know they are using a view). As a simple example, let’s say that you want to partially obscure the email address in the customer table. The marketing department, for example, may need access to email addresses in order to advertise promotions, but otherwise your company’s privacy policy dictates that this data be kept secure. Therefore, instead of allowing direct access to the customer table, you define a view called customer_vw and mandate that all non- marketing personnel use it to access customer data. Here’s the view definition: CREATE VIEW customer_vw (customer_id, first_name, last_name, email ) AS SELECT customer_id, first_name, last_name, concat(substr(email,1,2), '*****', substr(email, -4)) email FROM customer; The first part of the statement lists the view’s column names, which may be different from those of the underlying table. The second part of the statement is a select statement, which must contain one expression for each column in the view. The email column is generated by taking the first two characters of the email address, concatenated with “*****”, and then concatenated with the last 4 characters of the email address. When the create view statement is executed, the database server simply stores the view definition for future use; the query is not executed, and no data is retrieved or stored. Once the view has been created, users can query it just like they would a table, as in: mysql> SELECT first_name, last_name, email -> FROM customer_vw; +-------------+--------------+-------------+ | first_name | last_name | email | +-------------+--------------+-------------+ | MARY | SMITH | MA*****.org | | PATRICIA | JOHNSON | PA*****.org | | LINDA | WILLIAMS | LI*****.org | | BARBARA | JONES | BA*****.org | | ELIZABETH | BROWN | EL*****.org | ... | ENRIQUE | FORSYTHE | EN*****.org | | FREDDIE | DUGGAN | FR*****.org | | WADE | DELVALLE | WA*****.org | | AUSTIN | CINTRON | AU*****.org | +-------------+--------------+-------------+ 599 rows in set (0.00 sec) Even though the customer_vw view definition includes four columns of the customer table, the above query retrieves only three of the four. As you’ll see later in the chapter, this is an important distinction if some of the columns in your view are attached to functions or subqueries. From the user’s standpoint, a view looks exactly like a table. If you want to know what columns are available in a view, you can use MySQL’s (or Oracle’s) describe command to examine it: mysql> describe customer_vw; +-------------+----------------------+------+-----+---------+--- ----+ | Field | Type | Null | Key | Default | Extra | +-------------+----------------------+------+-----+---------+---", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 164 + }, + { + "text": "standpoint, a view looks exactly like a table. If you want to know what columns are available in a view, you can use MySQL’s (or Oracle’s) describe command to examine it: mysql> describe customer_vw; +-------------+----------------------+------+-----+---------+--- ----+ | Field | Type | Null | Key | Default | Extra | +-------------+----------------------+------+-----+---------+--- ----+ | customer_id | smallint(5) unsigned | NO | | 0 | | | first_name | varchar(45) | NO | | NULL | | | last_name | varchar(45) | NO | | NULL | | | email | varchar(11) | YES | | NULL | | +-------------+----------------------+------+-----+---------+--- ----+ 4 rows in set (0.00 sec) You are free to use any clauses of the select statement when querying through a view, including group by, having, and order by. Here’s an example: mysql> SELECT first_name, count(*), min(last_name), max(last_name) -> FROM customer_vw -> WHERE first_name LIKE 'J%' -> GROUP BY first_name -> HAVING count(*) > 1 -> ORDER BY 1; +------------+----------+----------------+----------------+ | first_name | count(*) | min(last_name) | max(last_name) | +------------+----------+----------------+----------------+ | JAMIE | 2 | RICE | WAUGH | | JESSIE | 2 | BANKS | MILAM | +------------+----------+----------------+----------------+ 2 rows in set (0.00 sec) In addition, you can join views to other tables (or even to other views) within a query, as in: mysql> SELECT cv.first_name, cv.last_name, p.amount -> FROM customer_vw cv -> INNER JOIN payment p -> ON cv.customer_id = p.customer_id -> WHERE p.amount >= 11; +------------+-----------+--------+ | first_name | last_name | amount | +------------+-----------+--------+ | KAREN | JACKSON | 11.99 | | VICTORIA | GIBSON | 11.99 | | VANESSA | SIMS | 11.99 | | ALMA | AUSTIN | 11.99 | | ROSEMARY | SCHMIDT | 11.99 | | TANYA | GILBERT | 11.99 | | RICHARD | MCCRARY | 11.99 | | NICHOLAS | BARFIELD | 11.99 | | KENT | ARSENAULT | 11.99 | | TERRANCE | ROUSH | 11.99 | +------------+-----------+--------+ 10 rows in set (0.01 sec) This query joins the customer_vw view to the payment table in order to find customers who have paid $11 or more for a film rental. Why Use Views? In the previous section, I demonstrated a simple view whose sole purpose was to mask the contents of the customer.email column. While views are often employed for this purpose, there are many reasons for using views, as detailed in the following subsections. Data Security If you create a table and allow users to query it, they will be able to access every column and every row in the table. As I pointed out earlier, however, your table may include some columns that contain sensitive data, such as identification numbers or credit card numbers; not only is it a bad idea to expose such data to all users, but also it might violate your company’s privacy policies, or even state or federal laws, to do so. The best approach for these situations is to keep the table private (i.e., don’t grant select permission to any users) and then to create one", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 165 + }, + { + "text": "to expose such data to all users, but also it might violate your company’s privacy policies, or even state or federal laws, to do so. The best approach for these situations is to keep the table private (i.e., don’t grant select permission to any users) and then to create one or more views that either omit or obscure (such as the '*****' approach taken with the customer_vw.email column) the sensitive columns. You may also constrain which rows a set of users may access by adding a where clause to your view definition. For example, the next view definition excludes inactive customers: CREATE VIEW active_customer_vw (customer_id, first_name, last_name, email ) AS SELECT customer_id, first_name, last_name, concat(substr(email,1,2), '*****', substr(email, -4)) email FROM customer WHERE active = 1; If you provide this view to your marketing department, they will be able to avoid sending information to inactive customers, because the condition in the view’s where clause will always be included in their queries. NOTE Oracle Database users have another option for securing both rows and columns of a table: Virtual Private Database (VPD). VPD allows you to attach policies to your tables, after which the server will modify a user’s query as necessary to enforce the policies. For example, if you enact a policy that members of the sales and marketing departments can see only active customers, then the condition active = 1 will be added to all of their queries against the customer table. Data Aggregation Reporting applications generally require aggregated data, and views are a great way to make it appear as though data is being pre-aggregated and stored in the database. As an example, let’s say that an application generates a report each month showing the total sales for each film category, so that the managers can decide what new films to add to inventory. Rather than allowing the application developers to write queries against the base tables, you could provide them with the following view: CREATE VIEW sales_by_film_category AS SELECT c.name AS category, SUM(p.amount) AS total_sales FROM payment AS p INNER JOIN rental AS r ON p.rental_id = r.rental_id INNER JOIN inventory AS i ON r.inventory_id = i.inventory_id INNER JOIN film AS f ON i.film_id = f.film_id INNER JOIN film_category AS fc ON f.film_id = fc.film_id INNER JOIN category AS c ON fc.category_id = c.category_id GROUP BY c.name ORDER BY total_sales DESC; Using this approach gives you a great deal of flexibility as a database designer. If you decide at some point in the future that query performance would improve dramatically if the data were preaggregated in a table 1 rather than summed using a view, you could create a film_category_sales table, load it with aggregated data, and modify the sales_by_film_category view definition to retrieve data from this table. Afterward, all queries that use the sales_by_film_category view will retrieve data from the new film_category_sales table, meaning that users will see a performance improvement without needing to modify their queries. Hiding Complexity One of the most common reasons for deploying", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 166 + }, + { + "text": "the sales_by_film_category view definition to retrieve data from this table. Afterward, all queries that use the sales_by_film_category view will retrieve data from the new film_category_sales table, meaning that users will see a performance improvement without needing to modify their queries. Hiding Complexity One of the most common reasons for deploying views is to shield end users from complexity. For example, let’s say that a report is created each month showing information about all of the films, along with the film category, the number of actors appearing in the film, the total number of copies in inventory, and the number of rentals for each film. Rather than expecting the report designer to navigate six different tables to gather the necessary data, you could provide a view that looks as follows: CREATE VIEW film_stats AS SELECT f.film_id, f.title, f.description, f.rating, (SELECT c.name FROM category c INNER JOIN film_category fc ON c.category_id = fc.category_id WHERE fc.film_id = f.film_id) category_name, (SELECT count(*) FROM film_actor fa WHERE fa.film_id = f.film_id ) num_actors, (SELECT count(*) FROM inventory i WHERE i.film_id = f.film_id ) inventory_cnt, (SELECT count(*) FROM inventory i INNER JOIN rental r ON i.inventory_id = r.inventory_id WHERE i.film_id = f.film_id ) num_rentals FROM film f; This view definition is interesting because even though data from six different tables can be retrieved through the view, the from clause of the query only has one table (film). Data from the other five tables are generated using scalar subqueries. If someone uses this view but does not reference the category_name, num_actors, inventory_cnt, or num_rentals column, then none of the subqueries will be executed. This approach allows the view to be used for supplying descriptive information from the film table without unnecessarily joining five other tables. Joining Partitioned Data Some database designs break large tables into multiple pieces in order to improve performance. For example, if the payment table became large, the designers may decide to break it into two tables: payment_current, which holds the latest six months’ of data, and payment_historic, which holds all data up to six months ago. If a customer wants to see all the payments for a particular customer, you would need to query both tables. By creating a view that queries both tables and combines the results together, however, you can make it look like all payment data is stored in a single table. Here’s the view definition: CREATE VIEW payment_all (payment_id, customer_id, staff_id, rental_id, amount, payment_date, last_update ) AS SELECT payment_id, customer_id, staff_id, rental_id, amount, payment_date, last_update FROM payment_historic UNION ALL SELECT payment_id, customer_id, staff_id, rental_id, amount, payment_date, last_update FROM payment_current; Using a view in this case is a good idea because it allows the designers to change the structure of the underlying data without the need to force all database users to modify their queries. Updatable Views If you provide users with a set of views to use for data retrieval, what should you do if the users also need to modify the same data? It might seem a bit strange,", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 167 + }, + { + "text": "underlying data without the need to force all database users to modify their queries. Updatable Views If you provide users with a set of views to use for data retrieval, what should you do if the users also need to modify the same data? It might seem a bit strange, for example, to force the users to retrieve data using a view, but then allow them to directly modify the underlying table using update or insert statements. For this purpose, MySQL, Oracle Database, and SQL Server all allow you to modify data through a view, as long as you abide by certain restrictions. In the case of MySQL, a view is updatable if the following conditions are met: No aggregate functions are used (max(), min(), avg(), etc.). The view does not employ group by or having clauses. No subqueries exist in the select or from clause, and any subqueries in the where clause do not refer to tables in the from clause. The view does not utilize union, union all, or distinct. The from clause includes at least one table or updatable view. The from clause uses only inner joins if there is more than one table or view. To demonstrate the utility of updatable views, it might be best to start with a simple view definition and then to move to a more complex view. Updating Simple Views The view at the beginning of the chapter is about as simple as it gets, so let’s start there: CREATE VIEW customer_vw (customer_id, first_name, last_name, email ) AS SELECT customer_id, first_name, last_name, concat(substr(email,1,2), '*****', substr(email, -4)) email FROM customer; The customer_vw view queries a single table, and only one of the four columns is derived via an expression. This view definition doesn’t violate any of the restrictions listed earlier, so you can use it to modify data in the customer table. Let’s use the view to update Mary Smith’s last name to Smith-Allen: mysql> UPDATE customer_vw -> SET last_name = 'SMITH-ALLEN' -> WHERE customer_id = 1; Query OK, 1 row affected (0.11 sec) Rows matched: 1 Changed: 1 Warnings: 0 As you can see, the statement claims to have modified one row, but let’s check the underlying customer table just to be sure: mysql> SELECT first_name, last_name, email -> FROM customer -> WHERE customer_id = 1; +------------+-------------+-------------------------------+ | first_name | last_name | email | +------------+-------------+-------------------------------+ | MARY | SMITH-ALLEN | MARY.SMITH@sakilacustomer.org | +------------+-------------+-------------------------------+ 1 row in set (0.00 sec) While you can modify most of the columns in the view in this fashion, you will not be able to modify the email column, since it is derived from an expression: mysql> UPDATE customer_vw -> SET email = 'MARY.SMITH-ALLEN@sakilacustomer.org' -> WHERE customer_id = 1; ERROR 1348 (HY000): Column 'email' is not updatable In this case, it may not be a bad thing, since the main reason for creating the view was to obscure the email addresses. If you want to insert data using the customer_vw view, you are out of luck; views", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 168 + }, + { + "text": "= 1; ERROR 1348 (HY000): Column 'email' is not updatable In this case, it may not be a bad thing, since the main reason for creating the view was to obscure the email addresses. If you want to insert data using the customer_vw view, you are out of luck; views that contain derived columns cannot be used for inserting data, even if the derived columns are not included in the statement. For example, the next statement attempts to populate only the customer_id, first_name, and last_name columns using the customer_vw view: mysql> INSERT INTO customer_vw -> (customer_id, -> first_name, -> last_name) -> VALUES (99999,'ROBERT','SIMPSON'); ERROR 1471 (HY000): The target table customer_vw of the INSERT is not insertable-into Now that you have seen the limitations of simple views, the next section will demonstrate the use of a view that joins multiple tables. Updating Complex Views While single-table views are certainly common, many of the views that you come across will include multiple tables in the from clause of the underlying query. The next view, for example, joins the customer, address, city,and country tables so that all the data for customers can be easily queried: CREATE VIEW customer_details AS SELECT c.customer_id, c.store_id, c.first_name, c.last_name, c.address_id, c.active, c.create_date, a.address, ct.city, cn.country, a.postal_code FROM customer c INNER JOIN address a ON c.address_id = a.address_id INNER JOIN city ct ON a.city_id = ct.city_id INNER JOIN country cn ON ct.country_id = cn.country_id; You may use this view to update data in either the customer or the address table, as the following statements demonstrate: mysql> UPDATE customer_details -> SET last_name = 'SMITH-ALLEN', active = 0 -> WHERE customer_id = 1; Query OK, 1 row affected (0.10 sec) Rows matched: 1 Changed: 1 Warnings: 0 mysql> UPDATE customer_details -> SET address = '999 Mockingbird Lane' -> WHERE customer_id = 1; Query OK, 1 row affected (0.06 sec) Rows matched: 1 Changed: 1 Warnings: 0 The first statement modifies the customer.last_name and customer.active columns, whereas the second statement modifies the address.address column. You might be wondering what happens if you try to update columns from both tables in a single statement, so let’s find out: mysql> UPDATE customer_details -> SET last_name = 'SMITH-ALLEN', -> active = 0, -> address = '999 Mockingbird Lane' -> WHERE customer_id = 1; ERROR 1393 (HY000): Can not modify more than one base table through a join view 'sakila.customer_details' As you can see, you are allowed to modify both of the underlying tables separately, but not within a single statement. Next, let’s try to insert data into both tables for some new customers (customer_id = 9998 and 9999): mysql> INSERT INTO customer_details -> (customer_id, store_id, first_name, last_name, -> address_id, active, create_date) -> VALUES (9998, 1, 'BRIAN', 'SALAZAR', 5, 1, now()); Query OK, 1 row affected (0.23 sec) This statement, which only populates columns from the customer table, works fine. Let’s see what happens if we expand the column list to also include a column from the address table: mysql> INSERT INTO customer_details -> (customer_id, store_id,", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 169 + }, + { + "text": "'BRIAN', 'SALAZAR', 5, 1, now()); Query OK, 1 row affected (0.23 sec) This statement, which only populates columns from the customer table, works fine. Let’s see what happens if we expand the column list to also include a column from the address table: mysql> INSERT INTO customer_details -> (customer_id, store_id, first_name, last_name, -> address_id, active, create_date, address) -> VALUES (9999, 2, 'THOMAS', 'BISHOP', 7, 1, now(), -> '999 Mockingbird Lane'); ERROR 1393 (HY000): Can not modify more than one base table through a join view 'sakila.customer_details' This version, which includes columns spanning two different tables, raises an exception. In order to insert data through a complex view, you would need to know from where each column is sourced. Since many views are created to hide complexity from end users, this seems to defeat the purpose if the users need to have explicit knowledge of the view definition. NOTE Oracle Database and SQL Server also allow data to be inserted and updated through views, but, like MySQL, there are many restrictions. If you are willing to write some PL/SQL or Transact- SQL, however, you can use a feature called instead-of triggers, which allows you to essentially intercept insert, update, and delete statements against a view, and write custom code to incorporate the changes. Without this type of feature, there are usually too many restrictions to make updating through views a feasible strategy for nontrivial applications. Test Your Knowledge Test your understanding of views by working through the following exercises. When you’re done, compare your solutions with those in Appendix C. Exercise 14-1 Create a view definition that can be used by the following query to generate the given results: SELECT category_name, title, first_name, last_name FROM film_ctgry_actor WHERE last_name = 'FAWCETT'; +---------------------+---------------+------------+-----------+ | title | category_name | first_name | last_name | +---------------------+---------------+------------+-----------+ | ACE GOLDFINGER | Horror | BOB | FAWCETT | | ADAPTATION HOLES | Documentary | BOB | FAWCETT | | CHINATOWN GLADIATOR | New | BOB | FAWCETT | | CIRCUS YOUTH | Children | BOB | FAWCETT | | CONTROL ANTHEM | Comedy | BOB | FAWCETT | | DARES PLUTO | Animation | BOB | FAWCETT | | DARN FORRESTER | Action | BOB | FAWCETT | | DAZED PUNK | Games | BOB | FAWCETT | | DYNAMITE TARZAN | Classics | BOB | FAWCETT | | HATE HANDICAP | Comedy | BOB | FAWCETT | | HOMICIDE PEACH | Family | BOB | FAWCETT | | JACKET FRISCO | Drama | BOB | FAWCETT | | JUMANJI BLADE | New | BOB | FAWCETT | | LAWLESS VISION | Animation | BOB | FAWCETT | | LEATHERNECKS DWARFS | Travel | BOB | FAWCETT | | OSCAR GOLD | Animation | BOB | FAWCETT | | PELICAN COMFORTS | Documentary | BOB | FAWCETT | | PERSONAL LADYBUGS | Music | BOB | FAWCETT | | RAGING AIRPLANE | Sci-Fi | BOB | FAWCETT | | RUN PACIFIC | New | BOB | FAWCETT |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 170 + }, + { + "text": "| OSCAR GOLD | Animation | BOB | FAWCETT | | PELICAN COMFORTS | Documentary | BOB | FAWCETT | | PERSONAL LADYBUGS | Music | BOB | FAWCETT | | RAGING AIRPLANE | Sci-Fi | BOB | FAWCETT | | RUN PACIFIC | New | BOB | FAWCETT | | RUNNER MADIGAN | Music | BOB | FAWCETT | | SADDLE ANTITRUST | Comedy | BOB | FAWCETT | | SCORPION APOLLO | Drama | BOB | FAWCETT | | SHAWSHANK BUBBLE | Travel | BOB | FAWCETT | | TAXI KICK | Music | BOB | FAWCETT | | BERETS AGENT | Action | JULIA | FAWCETT | | BOILED DARES | Travel | JULIA | FAWCETT | | CHISUM BEHAVIOR | Family | JULIA | FAWCETT | | CLOSER BANG | Comedy | JULIA | FAWCETT | | DAY UNFAITHFUL | New | JULIA | FAWCETT | | HOPE TOOTSIE | Classics | JULIA | FAWCETT | | LUKE MUMMY | Animation | JULIA | FAWCETT | | MULAN MOON | Comedy | JULIA | FAWCETT | | OPUS ICE | Foreign | JULIA | FAWCETT | | POLLOCK DELIVERANCE | Foreign | JULIA | FAWCETT | | RIDGEMONT SUBMARINE | New | JULIA | FAWCETT | | SHANGHAI TYCOON | Travel | JULIA | FAWCETT | | SHAWSHANK BUBBLE | Travel | JULIA | FAWCETT | | THEORY MERMAID | Animation | JULIA | FAWCETT | | WAIT CIDER | Animation | JULIA | FAWCETT | +---------------------+---------------+------------+-----------+ 40 rows in set (0.00 sec) Exercise 14-2 The film rental company manager would like to have a report that includes the name of every country, along with the total payments for all customers who live in each country. Generate a view definition that queries the country table and uses a scalar subquery to calculate a value for a column named tot_payments 1 This view definition is included in the Sakila sample database, along with six others, several of which will be used in upcoming examples. Chapter 15. Metadata Along with storing all of the data that various users insert into a database, a database server also needs to store information about all of the database objects (tables, views, indexes, etc.) that were created to store this data. The database server stores this information, not surprisingly, in a database. This chapter discusses how and where this information, known as metadata, is stored, how you can access it, and how you can use it to build flexible systems. Data About Data Metadata is essentially data about data. Every time you create a database object, the database server needs to record various pieces of information. For example, if you were to create a table with multiple columns, a primary key constraint, three indexes, and a foreign key constraint, the database server would need to store all the following information: Table name Table storage information (tablespace, initial size, etc.) Storage engine Column names Column data types Default column values NOT NULL column", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 171 + }, + { + "text": "create a table with multiple columns, a primary key constraint, three indexes, and a foreign key constraint, the database server would need to store all the following information: Table name Table storage information (tablespace, initial size, etc.) Storage engine Column names Column data types Default column values NOT NULL column constraints Primary key columns Primary key name Name of primary key index Index names Index types (B-tree, bitmap) Indexed columns Index column sort order (ascending or descending) Index storage information Foreign key name Foreign key columns Associated table/columns for foreign keys This data is collectively known as the data dictionary or system catalog. The database server needs to store this data persistently, and it needs to be able to quickly retrieve this data in order to verify and execute SQL statements. Additionally, the database server must safeguard this data so that it can be modified only via an appropriate mechanism, such as the alter table statement. While standards exist for the exchange of metadata between different servers, every database server uses a different mechanism to publish metadata, such as: A set of views, such as Oracle Database’s user_tables and all_constraints views A set of system-stored procedures, such as SQL Server’s sp_tables procedure or Oracle Database’s dbms_metadata package A special database, such as MySQL’s information_schema database Along with SQL Server’s system-stored procedures, which are a vestige of its Sybase lineage, SQL Server also includes a special schema called information_schema that is provided automatically within each database. Both MySQL and SQL Server provide this interface to conform with the ANSI SQL:2003 standard. The remainder of this chapter discusses the information_schema objects that are available in MySQL and SQL Server. Information_Schema All of the objects available within the information_schema database (or schema, in the case of SQL Server) are views. Unlike the describe utility, which I used in several chapters of this book as a way to show the structure of various tables and views, the views within information_schema can be queried, and, thus, used programmatically (more on this later in the chapter). Here’s an example that demonstrates how to retrieve the names of all of the tables in the sakila database: mysql> SELECT table_name, table_type -> FROM information_schema.tables -> WHERE table_schema = 'sakila' -> ORDER BY 1; +----------------------------+------------+ | TABLE_NAME | TABLE_TYPE | +----------------------------+------------+ | actor | BASE TABLE | | actor_info | VIEW | | address | BASE TABLE | | category | BASE TABLE | | city | BASE TABLE | | country | BASE TABLE | | customer | BASE TABLE | | customer_list | VIEW | | film | BASE TABLE | | film_actor | BASE TABLE | | film_category | BASE TABLE | | film_list | VIEW | | film_text | BASE TABLE | | inventory | BASE TABLE | | language | BASE TABLE | | nicer_but_slower_film_list | VIEW | | payment | BASE TABLE | | rental | BASE TABLE | | sales_by_film_category | VIEW | | sales_by_store | VIEW | | staff | BASE", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 172 + }, + { + "text": "| | film_text | BASE TABLE | | inventory | BASE TABLE | | language | BASE TABLE | | nicer_but_slower_film_list | VIEW | | payment | BASE TABLE | | rental | BASE TABLE | | sales_by_film_category | VIEW | | sales_by_store | VIEW | | staff | BASE TABLE | | staff_list | VIEW | | store | BASE TABLE | +----------------------------+------------+ 23 rows in set (0.00 sec) As you can see, the information_schema.tables view includes both tables and views; if you want to exclude the views, simply add another condition to the where clause: mysql> SELECT table_name, table_type -> FROM information_schema.tables -> WHERE table_schema = 'sakila' -> AND table_type = 'BASE TABLE' -> ORDER BY 1; +---------------+------------+ | TABLE_NAME | TABLE_TYPE | +---------------+------------+ | actor | BASE TABLE | | address | BASE TABLE | | category | BASE TABLE | | city | BASE TABLE | | country | BASE TABLE | | customer | BASE TABLE | | film | BASE TABLE | | film_actor | BASE TABLE | | film_category | BASE TABLE | | film_text | BASE TABLE | | inventory | BASE TABLE | | language | BASE TABLE | | payment | BASE TABLE | | rental | BASE TABLE | | staff | BASE TABLE | | store | BASE TABLE | +---------------+------------+ 16 rows in set (0.00 sec) If you are only interested in information about views, you can query information_schema.views. Along with the view names, you can retrieve additional information, such as a flag that shows whether a view is updatable: mysql> SELECT table_name, is_updatable -> FROM information_schema.views -> WHERE table_schema = 'sakila' -> ORDER BY 1; +----------------------------+--------------+ | TABLE_NAME | IS_UPDATABLE | +----------------------------+--------------+ | actor_info | NO | | customer_list | YES | | film_list | NO | | nicer_but_slower_film_list | NO | | sales_by_film_category | NO | | sales_by_store | NO | | staff_list | YES | +----------------------------+--------------+ 7 rows in set (0.00 sec) Column information for both tables and views is available via the columns view. The following query shows column information for the film table: mysql> SELECT column_name, data_type, -> character_maximum_length char_max_len, -> numeric_precision num_prcsn, numeric_scale num_scale -> FROM information_schema.columns -> WHERE table_schema = 'sakila' AND table_name = 'film' -> ORDER BY ordinal_position; +----------------------+-----------+--------------+-----------+- ----------+ | COLUMN_NAME | DATA_TYPE | char_max_len | num_prcsn | num_scale | +----------------------+-----------+--------------+-----------+- ----------+ | film_id | smallint | NULL | 5 | 0 | | title | varchar | 255 | NULL | NULL | | description | text | 65535 | NULL | NULL | | release_year | year | NULL | NULL | NULL | | language_id | tinyint | NULL | 3 | 0 | | original_language_id | tinyint | NULL | 3 | 0 | | rental_duration | tinyint | NULL | 3 | 0 | | rental_rate | decimal | NULL | 4 | 2 | | length | smallint | NULL | 5 | 0 | | replacement_cost | decimal | NULL |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 173 + }, + { + "text": "original_language_id | tinyint | NULL | 3 | 0 | | rental_duration | tinyint | NULL | 3 | 0 | | rental_rate | decimal | NULL | 4 | 2 | | length | smallint | NULL | 5 | 0 | | replacement_cost | decimal | NULL | 5 | 2 | | rating | enum | 5 | NULL | NULL | | special_features | set | 54 | NULL | NULL | | last_update | timestamp | NULL | NULL | NULL | +----------------------+-----------+--------------+-----------+- ----------+ 13 rows in set (0.00 sec) The ordinal_position column is included merely as a means to retrieve the columns in the order in which they were added to the table. You can retrieve information about a table’s indexes via the information_schema.statistics view as demonstrated by the following query, which retrieves information for the indexes built on the rental table: mysql> SELECT index_name, non_unique, seq_in_index, column_name -> FROM information_schema.statistics -> WHERE table_schema = 'sakila' AND table_name = 'rental' -> ORDER BY 1, 3; +---------------------+------------+--------------+------------- -+ | INDEX_NAME | NON_UNIQUE | SEQ_IN_INDEX | COLUMN_NAME | +---------------------+------------+--------------+------------- -+ | idx_fk_customer_id | 1 | 1 | customer_id | | idx_fk_inventory_id | 1 | 1 | inventory_id | | idx_fk_staff_id | 1 | 1 | staff_id | | PRIMARY | 0 | 1 | rental_id | | rental_date | 0 | 1 | rental_date | | rental_date | 0 | 2 | inventory_id | | rental_date | 0 | 3 | customer_id | +---------------------+------------+--------------+------------- -+ 7 rows in set (0.02 sec) The rental table has a total of five indexes, one of which has three columns (rental_date) and one of which is a unique index (PRIMARY) used for the primary key constraint. You can retrieve the different types of constraints (foreign key, primary key, unique) that have been created via the information_schema.table_constraints view. Here’s a query that retrieves all of the constraints in the sakila schema: mysql> SELECT constraint_name, table_name, constraint_type -> FROM information_schema.table_constraints -> WHERE table_schema = 'sakila' -> ORDER BY 3,1; +---------------------------+---------------+-----------------+ | constraint_name | table_name | constraint_type | +---------------------------+---------------+-----------------+ | fk_address_city | address | FOREIGN KEY | | fk_city_country | city | FOREIGN KEY | | fk_customer_address | customer | FOREIGN KEY | | fk_customer_store | customer | FOREIGN KEY | | fk_film_actor_actor | film_actor | FOREIGN KEY | | fk_film_actor_film | film_actor | FOREIGN KEY | | fk_film_category_category | film_category | FOREIGN KEY | | fk_film_category_film | film_category | FOREIGN KEY | | fk_film_language | film | FOREIGN KEY | | fk_film_language_original | film | FOREIGN KEY | | fk_inventory_film | inventory | FOREIGN KEY | | fk_inventory_store | inventory | FOREIGN KEY | | fk_payment_customer | payment | FOREIGN KEY | | fk_payment_rental | payment | FOREIGN KEY | | fk_payment_staff | payment | FOREIGN KEY | | fk_rental_customer | rental | FOREIGN KEY | | fk_rental_inventory | rental | FOREIGN KEY | | fk_rental_staff | rental | FOREIGN KEY | | fk_staff_address | staff | FOREIGN KEY | | fk_staff_store", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 174 + }, + { + "text": "| fk_payment_rental | payment | FOREIGN KEY | | fk_payment_staff | payment | FOREIGN KEY | | fk_rental_customer | rental | FOREIGN KEY | | fk_rental_inventory | rental | FOREIGN KEY | | fk_rental_staff | rental | FOREIGN KEY | | fk_staff_address | staff | FOREIGN KEY | | fk_staff_store | staff | FOREIGN KEY | | fk_store_address | store | FOREIGN KEY | | fk_store_staff | store | FOREIGN KEY | | PRIMARY | film | PRIMARY KEY | | PRIMARY | film_actor | PRIMARY KEY | | PRIMARY | staff | PRIMARY KEY | | PRIMARY | film_category | PRIMARY KEY | | PRIMARY | store | PRIMARY KEY | | PRIMARY | actor | PRIMARY KEY | | PRIMARY | film_text | PRIMARY KEY | | PRIMARY | address | PRIMARY KEY | | PRIMARY | inventory | PRIMARY KEY | | PRIMARY | customer | PRIMARY KEY | | PRIMARY | category | PRIMARY KEY | | PRIMARY | language | PRIMARY KEY | | PRIMARY | city | PRIMARY KEY | | PRIMARY | payment | PRIMARY KEY | | PRIMARY | country | PRIMARY KEY | | PRIMARY | rental | PRIMARY KEY | | idx_email | customer | UNIQUE | | idx_unique_manager | store | UNIQUE | | rental_date | rental | UNIQUE | +---------------------------+---------------+-----------------+ 41 rows in set (0.02 sec) Table 15-1 shows many of the information_schema views that are available in MySQL version 8.0. Table 15-1. Information_schema views View name Provides information about… Schemata Databases Tables Tables and views Columns Columns of tables and views Statistics Indexes User_Privileges Who has privileges on which schema objects Schema_Privileges Who has privileges on which databases Table_Privileges Who has privileges on which tables Column_Privileges Who has privileges on which columns of which tables Character_Sets What character sets are available Collations What collations are available for which character sets Collation_Character_Set_Appli cability Which character sets are available for which collation Table_Constraints The unique, foreign key, and primary key constraints Key_Column_Usage The constraints associated with each key column Routines Stored routines (procedures and functions) Views Views Triggers Table triggers Plugins Server plug-ins Engines Available storage engines Partitions Table partitions Events Scheduled events ProcessList Running processes Referential_Constraints Foreign keys Parameters Stored procedure and function parameters Profiling User profiling information While some of these views, such as engines, events, and plugins, are specific to MySQL, many of these views are available in SQL Server as well. If you are using Oracle Database, please consult the online Oracle Database Reference Guide for information about the user_, all_, and dba_ views. Working with Metadata As I mentioned earlier, having the ability to retrieve information about your schema objects via SQL queries opens up some interesting possibilities. This section shows several ways in which you can make use of metadata in your applications. Schema Generation Scripts While some project teams include a full-time database designer who oversees the design and implementation of the database, many projects take the “design-by-committee” approach, allowing multiple people", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 175 + }, + { + "text": "up some interesting possibilities. This section shows several ways in which you can make use of metadata in your applications. Schema Generation Scripts While some project teams include a full-time database designer who oversees the design and implementation of the database, many projects take the “design-by-committee” approach, allowing multiple people to create database objects. After several weeks or months of development, you may need to generate a script that will create the various tables, indexes, views, and so on that the team has deployed. Although a variety of tools and utilities will generate these types of scripts for you, you can also query the information_schema views and generate the script yourself. As an example, let’s build a script that will create the sakila.category table. Here’s the command used to build the table, which I extracted from the script used to build the example database: CREATE TABLE category ( category_id TINYINT UNSIGNED NOT NULL AUTO_INCREMENT, name VARCHAR(25) NOT NULL, last_update TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (category_id) )ENGINE=InnoDB DEFAULT CHARSET=utf8; Although it would certainly be easier to generate the script with the use of a procedural language (e.g., Transact-SQL or Java), since this is a book about SQL I’m going to write a single query that will generate the create table statement. The first step is to query the information_schema.columns table to retrieve information about the columns in the table: mysql> SELECT 'CREATE TABLE category (' create_table_statement -> UNION ALL -> SELECT cols.txt -> FROM -> (SELECT concat(' ',column_name, ' ', column_type, -> CASE -> WHEN is_nullable = 'NO' THEN ' not null' -> ELSE '' -> END, -> CASE -> WHEN extra IS NOT NULL AND extra LIKE 'DEFAULT_GENERATED%' -> THEN concat(' DEFAULT ',column_default,substr(extra,18)) -> WHEN extra IS NOT NULL THEN concat(' ', extra) -> ELSE '' -> END, -> ',') txt -> FROM information_schema.columns -> WHERE table_schema = 'sakila' AND table_name = 'category' -> ORDER BY ordinal_position -> ) cols -> UNION ALL -> SELECT ')'; +--------------------------------------------------------------- --------+ | create_table_statement | +--------------------------------------------------------------- --------+ | CREATE TABLE category ( | | category_id tinyint(3) unsigned not null auto_increment, | | name varchar(25) not null , | | last_update timestamp not null DEFAULT CURRENT_TIMESTAMP | | on update CURRENT_TIMESTAMP, | | ) | +--------------------------------------------------------------- --------+ 5 rows in set (0.00 sec) Well, that got us pretty close; we just need to add queries against the table_constraints and key_column_usage views to retrieve information about the primary key constraint: mysql> SELECT 'CREATE TABLE category (' create_table_statement -> UNION ALL -> SELECT cols.txt -> FROM -> (SELECT concat(' ',column_name, ' ', column_type, -> CASE -> WHEN is_nullable = 'NO' THEN ' not null' -> ELSE '' -> END, -> CASE -> WHEN extra IS NOT NULL AND extra LIKE 'DEFAULT_GENERATED%' -> THEN concat(' DEFAULT ',column_default,substr(extra,18)) -> WHEN extra IS NOT NULL THEN concat(' ', extra) -> ELSE '' -> END, -> ',') txt -> FROM information_schema.columns -> WHERE table_schema = 'sakila' AND table_name = 'category' -> ORDER BY ordinal_position -> )", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 176 + }, + { + "text": "WHEN extra IS NOT NULL AND extra LIKE 'DEFAULT_GENERATED%' -> THEN concat(' DEFAULT ',column_default,substr(extra,18)) -> WHEN extra IS NOT NULL THEN concat(' ', extra) -> ELSE '' -> END, -> ',') txt -> FROM information_schema.columns -> WHERE table_schema = 'sakila' AND table_name = 'category' -> ORDER BY ordinal_position -> ) cols -> UNION ALL -> SELECT concat(' constraint primary key (') -> FROM information_schema.table_constraints -> WHERE table_schema = 'sakila' AND table_name = 'category' -> AND constraint_type = 'PRIMARY KEY' -> UNION ALL -> SELECT cols.txt -> FROM -> (SELECT concat(CASE WHEN ordinal_position > 1 THEN ' ,' -> ELSE ' ' END, column_name) txt -> FROM information_schema.key_column_usage -> WHERE table_schema = 'sakila' AND table_name = 'category' -> AND constraint_name = 'PRIMARY' -> ORDER BY ordinal_position -> ) cols -> UNION ALL -> SELECT ' )' -> UNION ALL -> SELECT ')'; +--------------------------------------------------------------- --------+ | create_table_statement | +--------------------------------------------------------------- --------+ | CREATE TABLE category ( | | category_id tinyint(3) unsigned not null auto_increment, | | name varchar(25) not null , | | last_update timestamp not null DEFAULT CURRENT_TIMESTAMP | | on update CURRENT_TIMESTAMP, | | constraint primary key ( | | category_id | | ) | | ) | +--------------------------------------------------------------- --------+ 8 rows in set (0.02 sec) To see whether the statement is properly formed, I’ll paste the query output into the mysql tool (I’ve changed the table name to category2 so that it won’t step on our existing table): mysql> CREATE TABLE category2 ( -> category_id tinyint(3) unsigned not null auto_increment, -> name varchar(25) not null , -> last_update timestamp not null DEFAULT CURRENT_TIMESTAMP -> on update CURRENT_TIMESTAMP, -> constraint primary key ( -> category_id -> ) -> ); Query OK, 0 rows affected (0.61 sec) The statement executed without errors, and there is now a category2 table in the sakila database. In order for the query to generate a well- formed create table statement for any table, more work is required (such as handling indexes and foreign key constraints), but I’ll leave that as an exercise. Deployment Verification Many organizations allow for database maintenance windows, wherein existing database objects may be administered (such as adding/dropping partitions) and new schema objects and code can be deployed. After the deployment scripts have been run, it’s a good idea to run a verification script to ensure that the new schema objects are in place with the appropriate columns, indexes, primary keys, and so forth. Here’s a query that returns the number of columns, number of indexes, and number of primary key constraints (0 or 1) for each table in the sakila schema: mysql> SELECT tbl.table_name, -> (SELECT count(*) FROM information_schema.columns clm -> WHERE clm.table_schema = tbl.table_schema -> AND clm.table_name = tbl.table_name) num_columns, -> (SELECT count(*) FROM information_schema.statistics sta -> WHERE sta.table_schema = tbl.table_schema -> AND sta.table_name = tbl.table_name) num_indexes, -> (SELECT count(*) FROM information_schema.table_constraints tc -> WHERE tc.table_schema = tbl.table_schema -> AND tc.table_name = tbl.table_name -> AND tc.constraint_type = 'PRIMARY KEY') num_primary_keys -> FROM information_schema.tables tbl -> WHERE tbl.table_schema = 'sakila' AND tbl.table_type", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 177 + }, + { + "text": "(SELECT count(*) FROM information_schema.statistics sta -> WHERE sta.table_schema = tbl.table_schema -> AND sta.table_name = tbl.table_name) num_indexes, -> (SELECT count(*) FROM information_schema.table_constraints tc -> WHERE tc.table_schema = tbl.table_schema -> AND tc.table_name = tbl.table_name -> AND tc.constraint_type = 'PRIMARY KEY') num_primary_keys -> FROM information_schema.tables tbl -> WHERE tbl.table_schema = 'sakila' AND tbl.table_type = 'BASE TABLE' -> ORDER BY 1; +---------------+-------------+-------------+------------------+ | TABLE_NAME | num_columns | num_indexes | num_primary_keys | +---------------+-------------+-------------+------------------+ | actor | 4 | 2 | 1 | | address | 9 | 3 | 1 | | category | 3 | 1 | 1 | | city | 4 | 2 | 1 | | country | 3 | 1 | 1 | | customer | 9 | 7 | 1 | | film | 13 | 4 | 1 | | film_actor | 3 | 3 | 1 | | film_category | 3 | 3 | 1 | | film_text | 3 | 3 | 1 | | inventory | 4 | 4 | 1 | | language | 3 | 1 | 1 | | payment | 7 | 4 | 1 | | rental | 7 | 7 | 1 | | staff | 11 | 3 | 1 | | store | 4 | 3 | 1 | +---------------+-------------+-------------+------------------+ 16 rows in set (0.01 sec) You could execute this statement before and after the deployment and then verify any differences between the two sets of results before declaring the deployment a success. Dynamic SQL Generation Some languages, such as Oracle’s PL/SQL and Microsoft’s Transact-SQL, are supersets of the SQL language, meaning that they include SQL statements in their grammar along with the usual procedural constructs, such as “if-then-else” and “while.” Other languages, such as Java, include the ability to interface with a relational database, but do not include SQL statements in the grammar, meaning that all SQL statements must be contained within strings. Therefore, most relational database servers, including SQL Server, Oracle Database, and MySQL, allow SQL statements to be submitted to the server as strings. Submitting strings to a database engine rather than utilizing its SQL interface is generally known as dynamic SQL execution. Oracle’s PL/SQL language, for example, includes an execute immediate command, which you can use to submit a string for execution, while SQL Server includes a system stored procedure called sp_executesql for executing SQL statements dynamically. MySQL provides the statements prepare, execute, and deallocate to allow for dynamic SQL execution. Here’s a simple example: mysql> SET @qry = 'SELECT customer_id, first_name, last_name FROM customer'; Query OK, 0 rows affected (0.00 sec) mysql> PREPARE dynsql1 FROM @qry; Query OK, 0 rows affected (0.00 sec) Statement prepared mysql> EXECUTE dynsql1; +-------------+-------------+--------------+ | customer_id | first_name | last_name | +-------------+-------------+--------------+ | 505 | RAFAEL | ABNEY | | 504 | NATHANIEL | ADAM | | 36 | KATHLEEN | ADAMS | | 96 | DIANA | ALEXANDER | ... | 31 | BRENDA | WRIGHT | | 318 | BRIAN | WYMAN | | 402", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 178 + }, + { + "text": "first_name | last_name | +-------------+-------------+--------------+ | 505 | RAFAEL | ABNEY | | 504 | NATHANIEL | ADAM | | 36 | KATHLEEN | ADAMS | | 96 | DIANA | ALEXANDER | ... | 31 | BRENDA | WRIGHT | | 318 | BRIAN | WYMAN | | 402 | LUIS | YANEZ | | 413 | MARVIN | YEE | | 28 | CYNTHIA | YOUNG | +-------------+-------------+--------------+ 599 rows in set (0.02 sec) mysql> DEALLOCATE PREPARE dynsql1; Query OK, 0 rows affected (0.00 sec) The set statement simply assigns a string to the qry variable, which is then submitted to the database engine (for parsing, security checking, and optimization) using the prepare statement. After executing the statement by calling execute, the statement must be closed using deallocate prepare, which frees any database resources (e.g., cursors) that have been utilized during execution. The next example shows how you could execute a query that includes placeholders so that conditions can be specified at runtime: mysql> SET @qry = 'SELECT customer_id, first_name, last_name FROM customer WHERE customer_id = ?'; Query OK, 0 rows affected (0.00 sec) mysql> PREPARE dynsql2 FROM @qry; Query OK, 0 rows affected (0.00 sec) Statement prepared mysql> SET @custid = 9; Query OK, 0 rows affected (0.00 sec) mysql> EXECUTE dynsql2 USING @custid; +-------------+------------+-----------+ | customer_id | first_name | last_name | +-------------+------------+-----------+ | 9 | MARGARET | MOORE | +-------------+------------+-----------+ 1 row in set (0.00 sec) mysql> SET @custid = 145; Query OK, 0 rows affected (0.00 sec) mysql> EXECUTE dynsql2 USING @custid; +-------------+------------+-----------+ | customer_id | first_name | last_name | +-------------+------------+-----------+ | 145 | LUCILLE | HOLMES | +-------------+------------+-----------+ 1 row in set (0.00 sec) mysql> DEALLOCATE PREPARE dynsql2; Query OK, 0 rows affected (0.00 sec) In this sequence, the query contains a placeholder (the ? at the end of the statement) so that the customer ID value can be submitted at runtime. The statement is prepared once and then executed twice, once for customer ID 9, and again for customer ID 145, after which the statement is closed. What, you may wonder, does this have to do with metadata? Well, if you are going to use dynamic SQL to query a table, why not build the query string using metadata rather than hardcoding the table definition? The following example generates the same dynamic SQL string as the previous example, but it retrieves the column names from the information_schema.columns view: mysql> SELECT concat('SELECT ', -> concat_ws(',', cols.col1, cols.col2, cols.col3, cols.col4, -> cols.col5, cols.col6, cols.col7, cols.col8, cols.col9), -> ' FROM customer WHERE customer_id = ?') -> INTO @qry -> FROM -> (SELECT -> max(CASE WHEN ordinal_position = 1 THEN column_name -> ELSE NULL END) col1, -> max(CASE WHEN ordinal_position = 2 THEN column_name -> ELSE NULL END) col2, -> max(CASE WHEN ordinal_position = 3 THEN column_name -> ELSE NULL END) col3, -> max(CASE WHEN ordinal_position = 4 THEN column_name -> ELSE NULL END) col4, -> max(CASE WHEN ordinal_position = 5 THEN column_name -> ELSE NULL END)", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 179 + }, + { + "text": "max(CASE WHEN ordinal_position = 2 THEN column_name -> ELSE NULL END) col2, -> max(CASE WHEN ordinal_position = 3 THEN column_name -> ELSE NULL END) col3, -> max(CASE WHEN ordinal_position = 4 THEN column_name -> ELSE NULL END) col4, -> max(CASE WHEN ordinal_position = 5 THEN column_name -> ELSE NULL END) col5, -> max(CASE WHEN ordinal_position = 6 THEN column_name -> ELSE NULL END) col6, -> max(CASE WHEN ordinal_position = 7 THEN column_name -> ELSE NULL END) col7, -> max(CASE WHEN ordinal_position = 8 THEN column_name -> ELSE NULL END) col8, -> max(CASE WHEN ordinal_position = 9 THEN column_name -> ELSE NULL END) col9 -> FROM information_schema.columns -> WHERE table_schema = 'sakila' AND table_name = 'customer' -> GROUP BY table_name -> ) cols; Query OK, 1 row affected (0.00 sec) mysql> SELECT @qry; mysql> SELECT @qry; +--------------------------------------------------------------- -----+ | @qry | +--------------------------------------------------------------- -----+ | SELECT customer_id,store_id,first_name,last_name,email, address_id,active,create_date,last_update FROM customer WHERE customer_id = ? | +--------------------------------------------------------------- -----+ 1 row in set (0.00 sec) mysql> PREPARE dynsql3 FROM @qry; Query OK, 0 rows affected (0.00 sec) Statement prepared mysql> SET @custid = 45; Query OK, 0 rows affected (0.00 sec) mysql> EXECUTE dynsql3 USING @custid; +-------------+----------+------------+-----------+------------- ----------------------+------------+--------+------------------- --+---------------------+ | customer_id | store_id | first_name | last_name | email | address_id | active | create_date | last_update | +-------------+----------+------------+-----------+------------- ----------------------+------------+--------+------------------- --+---------------------+ | 45 | 1 | JANET | PHILLIPS | JANET.PHILLIPS@sakilacustomer.org | 49 | 1 | 2006- 02-14 22:04:36 | 2006-02-15 04:57:20 | +-------------+----------+------------+-----------+------------- ----------------------+------------+--------+------------------- --+---------------------+ 1 row in set (0.00 sec) mysql> DEALLOCATE PREPARE dynsql3; Query OK, 0 rows affected (0.00 sec) The query pivots the first nine columns in the customer table, builds a query string using the concat and concat_ws functions, and assigns the string to the qry variable. The query string is then executed as before. NOTE Generally, it would be better to generate the query using a procedural language that includes looping constructs, such as Java, PL/SQL, Transact-SQL, or MySQL’s Stored Procedure Language. However, I wanted to demonstrate a pure SQL example, so I had to limit the number of columns retrieved to some reasonable number, which in this example is nine. Test Your Knowledge The following exercises are designed to test your understanding of metadata. When you’re finished, please see Appendix C for the solutions. Exercise 15-1 Write a query that lists all of the indexes in the sakila schema. Include the table names. Exercise 15-2 Write a query that generates output that can be used to create all of the indexes on the sakila.customer table. Output should be of the form: \"ALTER TABLE ADD INDEX ()\" Chapter 16. Analytic Functions Data volumes have been growing at a staggering pace, and organizations are having difficulty storing all of it, not to mention trying to make sense of it. While data analysis has traditionally occurred outside of the database server, using specialized tools or languages such as Excel, R, and Python, the SQL language includes a robust set of functions useful for analytical processing. Common functionality such", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 180 + }, + { + "text": "all of it, not to mention trying to make sense of it. While data analysis has traditionally occurred outside of the database server, using specialized tools or languages such as Excel, R, and Python, the SQL language includes a robust set of functions useful for analytical processing. Common functionality such as generating rankings or calculating sub-totals can be easily accomplished within the database server without the need to export the data and load it into another tool. Analytic Function Concepts After the database server has completed all of the steps necessary to evaluate a query, including joining, filtering, grouping, and sorting, the result set is complete and ready to be returned to the caller. Imagine if you could pause the query execution at this point and take a walk through the result set while it is still held in memory; what types of analysis might you want to do? If your result set contains sales data, perhaps you might want to generate rankings for salespeople or regions, or calculate percentage differences between one time period to another. If you are generating results for a financial report, perhaps you would like to calculate subtotals for each report section, and a grand total for the final section. Using analytic functions, you can do all of these things and more. Before diving into the details, the following subsections describe the mechanisms used by several of the most commonly used analytic functions. Data Windows Let’s say you have written a query which generates monthly sales totals across an entire year. You may want to find the maximum monthly sales across the full year, which would involve finding the maximum value across the entire result set. However, you may also want to find the maximum monthly sales for each quarter, which would require the result set to be split into four pieces. To accommodate this type of analysis, analytic functions include the ability to group rows into windows, which effectively partition the data for use by the analytic function without changing the overall result set. Windows are defined using the over clause combined with the partition by sub clause, as demonstrated by the following query: mysql> SELECT quarter(payment_date) quarter, -> monthname(payment_date) month_nm, -> sum(amount) monthly_sales, -> max(sum(amount)) -> over () max_mnth_sales, -> max(sum(amount)) -> over (partition by quarter(payment_date)) max_qrtr_sales -> FROM payment -> WHERE year(payment_date) = 2005 -> GROUP BY quarter(payment_date), monthname(payment_date); +---------+----------+---------------+----------------+--------- -------+ | quarter | month_nm | monthly_sales | max_mnth_sales | max_qrtr_sales | +---------+----------+---------------+----------------+--------- -------+ | 2 | May | 4824.43 | 28373.89 | 9631.88 | | 2 | June | 9631.88 | 28373.89 | 9631.88 | | 3 | July | 28373.89 | 28373.89 | 28373.89 | | 3 | August | 24072.13 | 28373.89 | 28373.89 | +---------+----------+---------------+----------------+--------- -------+ 4 rows in set (0.00 sec) This query calculates the monthly sales for each month (monthly_sales), and then includes two analytic functions to calculate the maximum monthly sales (max_mnth_sales) and the maximum sales within each quarter (max_qrtr_sales). Both analytic functions include an", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 181 + }, + { + "text": "24072.13 | 28373.89 | 28373.89 | +---------+----------+---------------+----------------+--------- -------+ 4 rows in set (0.00 sec) This query calculates the monthly sales for each month (monthly_sales), and then includes two analytic functions to calculate the maximum monthly sales (max_mnth_sales) and the maximum sales within each quarter (max_qrtr_sales). Both analytic functions include an over clause, but the first one is empty, indicating that the window should include the entire result set, whereas the second one specifies that the window should only include rows within the same quarter. Data windows may contain anywhere from a single row to all of the rows in the result set, and different analytic functions can define different data windows. Localized Sorting Along with partitioning your result set into data windows for your analytic functions, you may also specify a sort order. For example, if you want to define a ranking number for each month, where the value 1 is given to the month having the highest sales, you will need to specify which column (or columns) to use for the ranking: mysql> SELECT quarter(payment_date) quarter, -> monthname(payment_date) month_nm, -> sum(amount) monthly_sales, -> rank() over (order by sum(amount) desc) sales_rank -> FROM payment -> WHERE year(payment_date) = 2005 -> GROUP BY quarter(payment_date), monthname(payment_date) -> ORDER BY 1, month(payment_date); +---------+----------+---------------+------------+ | quarter | month_nm | monthly_sales | sales_rank | +---------+----------+---------------+------------+ | 2 | May | 4824.43 | 4 | | 2 | June | 9631.88 | 3 | | 3 | July | 28373.89 | 1 | | 3 | August | 24072.13 | 2 | +---------+----------+---------------+------------+ 4 rows in set (0.00 sec) This query includes a call to the rank function, which will be covered in the next section, and specifies that the sum of the amount column be used to generate the rankings, with the values sorted in descending order. Thus, the month having the highest sales (July, in this case) will be given a ranking of 1. MULTIPLE ORDER BY CLAUSES The previous example contains two order by clauses; one at the end of the query to determine how the result set should be sorted, and another within the rank function to determine how the rankings should be allocated. While it is unfortunate that the same clause is used for different purposes, keep in mind that even if you are using analytic functions with one or more order by clauses, you will still need an order by clause at the end of your query if you want the result set to be sorted in a particular way. In some cases, you will want to use both the partition by and order by sub clauses in the same analytic function call. For example, the previous example can be modified to provide a different set of rankings per quarter, rather than a single ranking across the entire result set: mysql> SELECT quarter(payment_date) quarter, -> monthname(payment_date) month_nm, -> sum(amount) monthly_sales, -> rank() over (partition by quarter(payment_date) -> order by sum(amount) desc) qtr_sales_rank -> FROM payment -> WHERE year(payment_date) = 2005", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 182 + }, + { + "text": "modified to provide a different set of rankings per quarter, rather than a single ranking across the entire result set: mysql> SELECT quarter(payment_date) quarter, -> monthname(payment_date) month_nm, -> sum(amount) monthly_sales, -> rank() over (partition by quarter(payment_date) -> order by sum(amount) desc) qtr_sales_rank -> FROM payment -> WHERE year(payment_date) = 2005 -> GROUP BY quarter(payment_date), monthname(payment_date) -> ORDER BY 1, month(payment_date); +---------+----------+---------------+----------------+ | quarter | month_nm | monthly_sales | qtr_sales_rank | +---------+----------+---------------+----------------+ | 2 | May | 4824.43 | 2 | | 2 | June | 9631.88 | 1 | | 3 | July | 28373.89 | 1 | | 3 | August | 24072.13 | 2 | +---------+----------+---------------+----------------+ 4 rows in set (0.00 sec) While these examples were designed to illustrate the use of the over clause, the following sections will describe in detail the various analytic functions. Ranking People love to rank things. If you visit your favorite news/sports / travel sites, you’ll see headlines similar to the following: Top 10 Vacation Values Best Mutual Fund Returns Pre-Season College Football Rankings Top 100 Songs of All Time Companies also like to generate rankings, but for more practical purposes. Knowing which products are the best/worst sellers, or which geographic regions generate the least/most revenue help organizations to make strategic decisions. Ranking Functions There are multiple ranking functions available in the SQL standard, with each one taking a different approach to how ties are handled: Row_Num ber Returns a unique number for each row, with rankings arbitrarily assigned in case of a tie. Rank Returns the same ranking in case of a tie, with gaps in the rankings. Dense_Ran k Returns the same ranking in case of a tie, with no gaps in the rankings. Let’s look at an example to help illustrate the differences. Let’s say that the marketing department wants to identify the top 10 customers so they can be offered a free film rental. The following query determines the number of film rentals for each customer and sorts the results in descending order: mysql> SELECT customer_id, count(*) num_rentals -> FROM rental -> GROUP BY customer_id -> ORDER BY 2 desc; +-------------+-------------+ | customer_id | num_rentals | +-------------+-------------+ | 148 | 46 | | 526 | 45 | | 236 | 42 | | 144 | 42 | | 75 | 41 | | 469 | 40 | | 197 | 40 | | 137 | 39 | | 468 | 39 | | 178 | 39 | | 459 | 38 | | 410 | 38 | | 5 | 38 | | 295 | 38 | | 257 | 37 | | 366 | 37 | | 176 | 37 | | 198 | 37 | | 267 | 36 | | 439 | 36 | | 354 | 36 | | 348 | 36 | | 380 | 36 | | 29 | 36 | | 371 | 35 | | 403 | 35 | | 21 | 35 | ... | 136 | 15 |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 183 + }, + { + "text": "267 | 36 | | 439 | 36 | | 354 | 36 | | 348 | 36 | | 380 | 36 | | 29 | 36 | | 371 | 35 | | 403 | 35 | | 21 | 35 | ... | 136 | 15 | | 248 | 15 | | 110 | 14 | | 281 | 14 | | 61 | 14 | | 318 | 12 | +-------------+-------------+ 599 rows in set (0.16 sec) Looking at the results, the third and fourth customers in the result set both rented 42 films; should they both receive the same ranking of 3? And if so, should the customer with 41 rentals be given the ranking 4, or should we skip one and assign ranking 5? To see how each function handles ties when assigning rankings, the next query adds three more columns, each one employing a different ranking function: mysql> SELECT customer_id, count(*) num_rentals, -> row_number() over (order by count(*) desc) row_number_rnk, -> rank() over (order by count(*) desc) rank_rnk, -> dense_rank() over (order by count(*) desc) dense_rank_rnk -> FROM rental -> GROUP BY customer_id -> ORDER BY 2 desc; +-------------+-------------+----------------+----------+------- ---------+ | customer_id | num_rentals | row_number_rnk | rank_rnk | dense_rank_rnk | +-------------+-------------+----------------+----------+------- ---------+ | 148 | 46 | 1 | 1 | 1 | | 526 | 45 | 2 | 2 | 2 | | 144 | 42 | 3 | 3 | 3 | | 236 | 42 | 4 | 3 | 3 | | 75 | 41 | 5 | 5 | 4 | | 197 | 40 | 6 | 6 | 5 | | 469 | 40 | 7 | 6 | 5 | | 468 | 39 | 10 | 8 | 6 | | 137 | 39 | 8 | 8 | 6 | | 178 | 39 | 9 | 8 | 6 | | 5 | 38 | 11 | 11 | 7 | | 295 | 38 | 12 | 11 | 7 | | 410 | 38 | 13 | 11 | 7 | | 459 | 38 | 14 | 11 | 7 | | 198 | 37 | 16 | 15 | 8 | | 257 | 37 | 17 | 15 | 8 | | 366 | 37 | 18 | 15 | 8 | | 176 | 37 | 15 | 15 | 8 | | 348 | 36 | 21 | 19 | 9 | | 354 | 36 | 22 | 19 | 9 | | 380 | 36 | 23 | 19 | 9 | | 439 | 36 | 24 | 19 | 9 | | 29 | 36 | 19 | 19 | 9 | | 267 | 36 | 20 | 19 | 9 | | 50 | 35 | 26 | 25 | 10 | | 506 | 35 | 37 | 25 | 10 | | 368 |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 184 + }, + { + "text": "| 9 | | 29 | 36 | 19 | 19 | 9 | | 267 | 36 | 20 | 19 | 9 | | 50 | 35 | 26 | 25 | 10 | | 506 | 35 | 37 | 25 | 10 | | 368 | 35 | 32 | 25 | 10 | | 91 | 35 | 27 | 25 | 10 | | 371 | 35 | 33 | 25 | 10 | | 196 | 35 | 28 | 25 | 10 | | 373 | 35 | 34 | 25 | 10 | | 204 | 35 | 29 | 25 | 10 | | 381 | 35 | 35 | 25 | 10 | | 273 | 35 | 30 | 25 | 10 | | 21 | 35 | 25 | 25 | 10 | | 403 | 35 | 36 | 25 | 10 | | 274 | 35 | 31 | 25 | 10 | | 66 | 34 | 42 | 38 | 11 | ... | 136 | 15 | 594 | 594 | 30 | | 248 | 15 | 595 | 594 | 30 | | 110 | 14 | 597 | 596 | 31 | | 281 | 14 | 598 | 596 | 31 | | 61 | 14 | 596 | 596 | 31 | | 318 | 12 | 599 | 599 | 32 | +-------------+-------------+----------------+----------+------- ---------+ 599 rows in set (0.01 sec) The third column uses the row_number function to assign a unique ranking to each row, without regard to ties. Each of the 599 rows are assigned a number from 1 to 599, with the ranking value arbitrarily assigned for customers having the same number of film rentals. The next two columns, however, assign the same ranking in case of a tie, but the difference lies in whether not a gap is left in the ranking values after a tie. Looking at row 5 of the result set, you can see that the rank function skips the value 4 and assigns the value 5, whereas the dense_rank function assigns the value 4. To get back to the original request, how would you identify the top 10 customers? There are 3 possible solutions: Use the row_number function to identify customers ranked from 1 to 10, which results in exactly 10 customers in this example, but in other cases might exclude customers having the same number of rentals as the 10th ranked customer. Use the rank function to identify customers ranked 10 or less, which also results in exactly 10 customers. Use the dense_rank function to identify customers ranked 10 or less, which yields a list of 37 customers. If there are no ties in your result set, then any of these functions will suffice, but for many situations the rank function may be the best option. Generating Multiple Rankings The example in the previous section generates a single", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 185 + }, + { + "text": "10 or less, which yields a list of 37 customers. If there are no ties in your result set, then any of these functions will suffice, but for many situations the rank function may be the best option. Generating Multiple Rankings The example in the previous section generates a single ranking across the entire set of customers, but what if you want to generate multiple sets of rankings within the same result set? To extend the prior example, let’s say the marketing department decides to offer free film rentals to the top 5 customers every month. To generate the data, the rental_month column can be added to the previous query: mysql> SELECT customer_id, -> monthname(rental_date) rental_month, -> count(*) num_rentals -> FROM rental -> GROUP BY customer_id, monthname(rental_date) -> ORDER BY 2, 3 desc; +-------------+--------------+-------------+ | customer_id | rental_month | num_rentals | +-------------+--------------+-------------+ | 119 | August | 18 | | 15 | August | 18 | | 569 | August | 18 | | 148 | August | 18 | | 141 | August | 17 | | 21 | August | 17 | | 266 | August | 17 | | 418 | August | 17 | | 410 | August | 17 | | 342 | August | 17 | | 274 | August | 16 | ... | 281 | August | 2 | | 318 | August | 1 | | 75 | February | 3 | | 155 | February | 2 | | 175 | February | 2 | | 516 | February | 2 | | 361 | February | 2 | | 269 | February | 2 | | 208 | February | 2 | | 53 | February | 2 | ... | 22 | February | 1 | | 472 | February | 1 | | 148 | July | 22 | | 102 | July | 21 | | 236 | July | 20 | | 75 | July | 20 | | 91 | July | 19 | | 30 | July | 19 | | 64 | July | 19 | | 137 | July | 19 | ... | 339 | May | 1 | | 485 | May | 1 | | 116 | May | 1 | | 497 | May | 1 | | 180 | May | 1 | +-------------+--------------+-------------+ 2466 rows in set (0.02 sec) In order to create a new set of rankings for each month, you will need to add something to the rank function in order to describe how to divide the result set into different data windows (months, in this case). This is done using the partition by clause, which is added to the over clause: mysql> SELECT customer_id, -> monthname(rental_date) rental_month, -> count(*) num_rentals, -> rank() over (partition by monthname(rental_date) -> order by count(*) desc) rank_rnk -> FROM rental -> GROUP BY customer_id, monthname(rental_date) -> ORDER BY 2, 3 desc; +-------------+--------------+-------------+----------+ | customer_id | rental_month |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 186 + }, + { + "text": "by clause, which is added to the over clause: mysql> SELECT customer_id, -> monthname(rental_date) rental_month, -> count(*) num_rentals, -> rank() over (partition by monthname(rental_date) -> order by count(*) desc) rank_rnk -> FROM rental -> GROUP BY customer_id, monthname(rental_date) -> ORDER BY 2, 3 desc; +-------------+--------------+-------------+----------+ | customer_id | rental_month | num_rentals | rank_rnk | +-------------+--------------+-------------+----------+ | 569 | August | 18 | 1 | | 119 | August | 18 | 1 | | 148 | August | 18 | 1 | | 15 | August | 18 | 1 | | 141 | August | 17 | 5 | | 410 | August | 17 | 5 | | 418 | August | 17 | 5 | | 21 | August | 17 | 5 | | 266 | August | 17 | 5 | | 342 | August | 17 | 5 | | 144 | August | 16 | 11 | | 274 | August | 16 | 11 | ... | 164 | August | 2 | 596 | | 318 | August | 1 | 599 | | 75 | February | 3 | 1 | | 457 | February | 2 | 2 | | 53 | February | 2 | 2 | | 354 | February | 2 | 2 | | 352 | February | 1 | 24 | | 373 | February | 1 | 24 | | 148 | July | 22 | 1 | | 102 | July | 21 | 2 | | 236 | July | 20 | 3 | | 75 | July | 20 | 3 | | 91 | July | 19 | 5 | | 354 | July | 19 | 5 | | 30 | July | 19 | 5 | | 64 | July | 19 | 5 | | 137 | July | 19 | 5 | | 526 | July | 19 | 5 | | 366 | July | 19 | 5 | | 595 | July | 19 | 5 | | 469 | July | 18 | 13 | ... | 457 | May | 1 | 347 | | 356 | May | 1 | 347 | | 481 | May | 1 | 347 | | 10 | May | 1 | 347 | +-------------+--------------+-------------+----------+ 2466 rows in set (0.03 sec) Looking at the results, you can see that the rankings are reset to 1 for each month. In order to generate the desired results for the marketing department (top 5 customers from each month), you can simply wrap the previous query in a subquery, and add a filter condition to exclude any rows with a ranking higher than 5: SELECT customer_id, rental_month, num_rentals, rank_rnk ranking FROM (SELECT customer_id, monthname(rental_date) rental_month, count(*) num_rentals, rank() over (partition by monthname(rental_date) order by count(*) desc) rank_rnk FROM rental GROUP BY customer_id, monthname(rental_date) ) cust_rankings WHERE rank_rnk <= 5 ORDER BY rental_month, num_rentals desc, rank_rnk; Since analytic functions", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 187 + }, + { + "text": "with a ranking higher than 5: SELECT customer_id, rental_month, num_rentals, rank_rnk ranking FROM (SELECT customer_id, monthname(rental_date) rental_month, count(*) num_rentals, rank() over (partition by monthname(rental_date) order by count(*) desc) rank_rnk FROM rental GROUP BY customer_id, monthname(rental_date) ) cust_rankings WHERE rank_rnk <= 5 ORDER BY rental_month, num_rentals desc, rank_rnk; Since analytic functions can only be used in the SELECT clause, you will often need to nest queries if you need to do any filtering or grouping based on the results from the analytic function. Reporting Functions Along with generating rankings, another common use for analytic functions is to find outliers (e.g. min or max values) or to generate sums or averages across an entire data set. For these types of uses, you will be using aggregate functions (min , max, avg, sum, count), but instead of using them with a group by clause, you will pair them with an over clause. Here’s an example that generates monthly and grand totals for all payments of $10 or higher: mysql> SELECT monthname(payment_date) payment_month, -> amount, -> sum(amount) -> over (partition by monthname(payment_date)) monthly_total, -> sum(amount) over () grand_total -> FROM payment -> WHERE amount >= 10 -> ORDER BY 1; +---------------+--------+---------------+-------------+ | payment_month | amount | monthly_total | grand_total | +---------------+--------+---------------+-------------+ | August | 10.99 | 521.53 | 1262.86 | | August | 11.99 | 521.53 | 1262.86 | | August | 10.99 | 521.53 | 1262.86 | | August | 10.99 | 521.53 | 1262.86 | ... | August | 10.99 | 521.53 | 1262.86 | | August | 10.99 | 521.53 | 1262.86 | | August | 10.99 | 521.53 | 1262.86 | | July | 10.99 | 519.53 | 1262.86 | | July | 10.99 | 519.53 | 1262.86 | | July | 10.99 | 519.53 | 1262.86 | | July | 10.99 | 519.53 | 1262.86 | ... | July | 10.99 | 519.53 | 1262.86 | | July | 10.99 | 519.53 | 1262.86 | | July | 10.99 | 519.53 | 1262.86 | | June | 10.99 | 165.85 | 1262.86 | | June | 10.99 | 165.85 | 1262.86 | | June | 10.99 | 165.85 | 1262.86 | | June | 10.99 | 165.85 | 1262.86 | | June | 10.99 | 165.85 | 1262.86 | | June | 10.99 | 165.85 | 1262.86 | | June | 10.99 | 165.85 | 1262.86 | | June | 10.99 | 165.85 | 1262.86 | | June | 11.99 | 165.85 | 1262.86 | | June | 10.99 | 165.85 | 1262.86 | | June | 10.99 | 165.85 | 1262.86 | | June | 10.99 | 165.85 | 1262.86 | | June | 10.99 | 165.85 | 1262.86 | | June | 10.99 | 165.85 | 1262.86 | | June | 10.99 | 165.85 | 1262.86 | | May | 10.99 | 55.95 | 1262.86 | | May | 10.99 | 55.95 | 1262.86 | | May | 10.99 | 55.95 | 1262.86 | | May |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 188 + }, + { + "text": "1262.86 | | June | 10.99 | 165.85 | 1262.86 | | June | 10.99 | 165.85 | 1262.86 | | May | 10.99 | 55.95 | 1262.86 | | May | 10.99 | 55.95 | 1262.86 | | May | 10.99 | 55.95 | 1262.86 | | May | 10.99 | 55.95 | 1262.86 | | May | 11.99 | 55.95 | 1262.86 | +---------------+--------+---------------+-------------+ 114 rows in set (0.01 sec) The grand_total column contains the same value ($1,262.86) for every row because the over clause is empty, which specifies that the summation be done over the entire result set. The monthly_total column, however, contains a different value for each month, since there is a partition by clause specifying that the result set be split into multiple data windows (one for each month). While it may seem of little value to include a column such as grand_total having the same value for every row, these types of columns can also be used for calculations, as shown by the next query: mysql> SELECT monthname(payment_date) payment_month, -> sum(amount) month_total, -> round(sum(amount) / sum(sum(amount)) over () -> * 100, 2) pct_of_total -> FROM payment -> GROUP BY monthname(payment_date); +---------------+-------------+--------------+ | payment_month | month_total | pct_of_total | +---------------+-------------+--------------+ | May | 4824.43 | 7.16 | | June | 9631.88 | 14.29 | | July | 28373.89 | 42.09 | | August | 24072.13 | 35.71 | | February | 514.18 | 0.76 | +---------------+-------------+--------------+ 5 rows in set (0.04 sec) This query calculates the total payments for each month by summing the amount column, and then calculates the percentage of the total payments for each month by summing the monthly sums to use as the denominator in the calculation. Reporting functions may also be used for comparisons, such as the next query, which uses a case expression to determine whether a monthly total is the max, min, or somewhere in the middle: mysql> SELECT monthname(payment_date) payment_month, -> sum(amount) month_total, -> CASE sum(amount) -> WHEN max(sum(amount)) over () THEN 'Highest' -> WHEN min(sum(amount)) over () THEN 'Lowest' -> ELSE 'Middle' -> END descriptor -> FROM payment -> GROUP BY monthname(payment_date); +---------------+-------------+------------+ | payment_month | month_total | descriptor | +---------------+-------------+------------+ | May | 4824.43 | Middle | | June | 9631.88 | Middle | | July | 28373.89 | Highest | | August | 24072.13 | Middle | | February | 514.18 | Lowest | +---------------+-------------+------------+ 5 rows in set (0.04 sec) The descriptor column acts as a quasi-ranking function, in that it helps identify the top/bottom values across a set of rows. Window Frames As described earlier in the chapter, data windows for analytic functions are defined using the partition by clause, which allows you to group rows by common values. But what if you need even finer control over which rows to include in a data window? For example, perhaps you want to generate a running total starting from the beginning of the year and up to the current row. For these types", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 189 + }, + { + "text": "you to group rows by common values. But what if you need even finer control over which rows to include in a data window? For example, perhaps you want to generate a running total starting from the beginning of the year and up to the current row. For these types of calculations, you can include a “frame” sub clause to define exactly which rows to include in the data window. Here’s a query that sums payments for each week, and includes a reporting function to calculate the rolling sum: mysql> SELECT yearweek(payment_date) payment_week, -> sum(amount) week_total, -> sum(sum(amount)) -> over (order by yearweek(payment_date) -> rows unbounded preceding) rolling_sum -> FROM payment -> GROUP BY yearweek(payment_date) -> ORDER BY 1; +--------------+------------+-------------+ | payment_week | week_total | rolling_sum | +--------------+------------+-------------+ | 200521 | 2847.18 | 2847.18 | | 200522 | 1977.25 | 4824.43 | | 200524 | 5605.42 | 10429.85 | | 200525 | 4026.46 | 14456.31 | | 200527 | 8490.83 | 22947.14 | | 200528 | 5983.63 | 28930.77 | | 200530 | 11031.22 | 39961.99 | | 200531 | 8412.07 | 48374.06 | | 200533 | 10619.11 | 58993.17 | | 200534 | 7909.16 | 66902.33 | | 200607 | 514.18 | 67416.51 | +--------------+------------+-------------+ 11 rows in set (0.04 sec) The rolling_sum column expression includes the rows unbounded preceding sub clause to define a data window from the beginning of the result set up to and including the current row. The data window consists of a single row for the first row in the result set, two rows for the second row, etc. The value for the last row is the summation of the entire result set. Along with rolling sums, you can calculate rolling averages. Here’s a query that calculates a 3-week rolling average of total payments: mysql> SELECT yearweek(payment_date) payment_week, -> sum(amount) week_total, -> avg(sum(amount)) -> over (order by yearweek(payment_date) -> rows between 1 preceding and 1 following) rolling_3wk_avg -> FROM payment -> GROUP BY yearweek(payment_date) -> ORDER BY 1; +--------------+------------+-----------------+ | payment_week | week_total | rolling_3wk_avg | +--------------+------------+-----------------+ | 200521 | 2847.18 | 2412.215000 | | 200522 | 1977.25 | 3476.616667 | | 200524 | 5605.42 | 3869.710000 | | 200525 | 4026.46 | 6040.903333 | | 200527 | 8490.83 | 6166.973333 | | 200528 | 5983.63 | 8501.893333 | | 200530 | 11031.22 | 8475.640000 | | 200531 | 8412.07 | 10020.800000 | | 200533 | 10619.11 | 8980.113333 | | 200534 | 7909.16 | 6347.483333 | | 200607 | 514.18 | 4211.670000 | +--------------+------------+-----------------+ 11 rows in set (0.03 sec) The rolling_3wk_avg column defines a data window consisting of the current row, the prior row, and the next row. The data window will therefore consist of 3 rows, except for the first and last rows, which will have a data window consisting of just 2 rows (since there is no prior row for the first row, and no next row for the last row). Specifying a number of rows for your data window", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 190 + }, + { + "text": "therefore consist of 3 rows, except for the first and last rows, which will have a data window consisting of just 2 rows (since there is no prior row for the first row, and no next row for the last row). Specifying a number of rows for your data window works fine in many cases, but if there are gaps in your data you might want to try a different approach. In the previous result set, for example, there is data for weeks 200521, 200522, and 200524, but no date for week 200523. If you want to specify a date interval rather than a number of rows, you can specify a range for your data window, as shown in the next query: mysql> SELECT date(payment_date), sum(amount), -> avg(sum(amount)) over (order by date(payment_date) -> range between interval 3 day preceding -> and interval 3 day following) 7_day_avg -> FROM payment -> WHERE payment_date BETWEEN '2005-07-01' AND '2005-09-01' -> GROUP BY date(payment_date) -> ORDER BY 1; +--------------------+-------------+-------------+ | date(payment_date) | sum(amount) | 7_day_avg | +--------------------+-------------+-------------+ | 2005-07-05 | 128.73 | 1603.740000 | | 2005-07-06 | 2131.96 | 1698.166000 | | 2005-07-07 | 1943.39 | 1738.338333 | | 2005-07-08 | 2210.88 | 1766.917143 | | 2005-07-09 | 2075.87 | 2049.390000 | | 2005-07-10 | 1939.20 | 2035.628333 | | 2005-07-11 | 1938.39 | 2054.076000 | | 2005-07-12 | 2106.04 | 2014.875000 | | 2005-07-26 | 160.67 | 2046.642500 | | 2005-07-27 | 2726.51 | 2206.244000 | | 2005-07-28 | 2577.80 | 2316.571667 | | 2005-07-29 | 2721.59 | 2388.102857 | | 2005-07-30 | 2844.65 | 2754.660000 | | 2005-07-31 | 2868.21 | 2759.351667 | | 2005-08-01 | 2817.29 | 2795.662000 | | 2005-08-02 | 2726.57 | 2814.180000 | | 2005-08-16 | 111.77 | 1973.837500 | | 2005-08-17 | 2457.07 | 2123.822000 | | 2005-08-18 | 2710.79 | 2238.086667 | | 2005-08-19 | 2615.72 | 2286.465714 | | 2005-08-20 | 2723.76 | 2630.928571 | | 2005-08-21 | 2809.41 | 2659.905000 | | 2005-08-22 | 2576.74 | 2649.728000 | | 2005-08-23 | 2523.01 | 2658.230000 | +--------------------+-------------+-------------+ 24 rows in set (0.03 sec) The 7_day_avg column specifies a range of +/- 3 days and will include only those rows whose payment_date values fall within that range. For the 2005-08-16 calculation, for example, only the values for 08-16, 08-17, 08-19, and 08-20 are included, since there are no rows for the 3 prior dates (08-13 through 08-15). Lag and Lead Along with computing sums and averages over a data window, another common reporting task involves comparing values from one row to another. For example, if you are generating monthly sales totals, you may be asked to create a column showing the percentage difference from the prior month, which will require a way to retrieve the monthly sales total from the previous row. This can be accomplished using the lag function, which will retrieve a column value from a prior row in the result set, or the lead function, which will retrieve a column value from a", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 191 + }, + { + "text": "month, which will require a way to retrieve the monthly sales total from the previous row. This can be accomplished using the lag function, which will retrieve a column value from a prior row in the result set, or the lead function, which will retrieve a column value from a following row. Here’s an example using both functions: mysql> SELECT yearweek(payment_date) payment_week, -> sum(amount) week_total, -> lag(sum(amount), 1) -> over (order by yearweek(payment_date)) prev_wk_tot, -> lead(sum(amount), 1) -> over (order by yearweek(payment_date)) next_wk_tot -> FROM payment -> GROUP BY yearweek(payment_date) -> ORDER BY 1; +--------------+------------+-------------+-------------+ | payment_week | week_total | prev_wk_tot | next_wk_tot | +--------------+------------+-------------+-------------+ | 200521 | 2847.18 | NULL | 1977.25 | | 200522 | 1977.25 | 2847.18 | 5605.42 | | 200524 | 5605.42 | 1977.25 | 4026.46 | | 200525 | 4026.46 | 5605.42 | 8490.83 | | 200527 | 8490.83 | 4026.46 | 5983.63 | | 200528 | 5983.63 | 8490.83 | 11031.22 | | 200530 | 11031.22 | 5983.63 | 8412.07 | | 200531 | 8412.07 | 11031.22 | 10619.11 | | 200533 | 10619.11 | 8412.07 | 7909.16 | | 200534 | 7909.16 | 10619.11 | 514.18 | | 200607 | 514.18 | 7909.16 | NULL | +--------------+------------+-------------+-------------+ 11 rows in set (0.03 sec) Looking at the results, , the weekly total of 8,490.43 for week 200527 also appears in the next_wk_tot column for week 200525, as well as in the prev_wk_tot column for week 200528. Since there is no row prior to 200521 in the result set, the value generated by the lag function is NULL for the first row; likewise, the value generated by the lead function is NULL for the last row in the result set. Both lag and lead allow for an optional 2nd parameter (which defaults to 1) to describe the number of rows prior/following from which to retrieve the column value. Here’s how you could use the lag function to generate the percentage difference from the prior week: mysql> SELECT yearweek(payment_date) payment_week, -> sum(amount) week_total, -> round((sum(amount) - lag(sum(amount), 1) -> over (order by yearweek(payment_date))) -> / lag(sum(amount), 1) -> over (order by yearweek(payment_date)) -> * 100, 1) pct_diff -> FROM payment -> GROUP BY yearweek(payment_date) -> ORDER BY 1; +--------------+------------+----------+ | payment_week | week_total | pct_diff | +--------------+------------+----------+ | 200521 | 2847.18 | NULL | | 200522 | 1977.25 | -30.6 | | 200524 | 5605.42 | 183.5 | | 200525 | 4026.46 | -28.2 | | 200527 | 8490.83 | 110.9 | | 200528 | 5983.63 | -29.5 | | 200530 | 11031.22 | 84.4 | | 200531 | 8412.07 | -23.7 | | 200533 | 10619.11 | 26.2 | | 200534 | 7909.16 | -25.5 | | 200607 | 514.18 | -93.5 | +--------------+------------+----------+ 11 rows in set (0.07 sec) Comparing values from different rows in the same result set is a common practice in reporting systems, so you will likely find many uses for the lag and lead functions. Test Your Knowledge The", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 192 + }, + { + "text": "-25.5 | | 200607 | 514.18 | -93.5 | +--------------+------------+----------+ 11 rows in set (0.07 sec) Comparing values from different rows in the same result set is a common practice in reporting systems, so you will likely find many uses for the lag and lead functions. Test Your Knowledge The following exercises are designed to test your understanding of analytic functions. When you’re finished, please see Appendix C. For all exercises in this section, use the following data set from the Sales_Fact table: Sales_Fact +---------+----------+-----------+ | year_no | month_no | tot_sales | +---------+----------+-----------+ | 2019 | 1 | 19228 | | 2019 | 2 | 18554 | | 2019 | 3 | 17325 | | 2019 | 4 | 13221 | | 2019 | 5 | 9964 | | 2019 | 6 | 12658 | | 2019 | 7 | 14233 | | 2019 | 8 | 17342 | | 2019 | 9 | 16853 | | 2019 | 10 | 17121 | | 2019 | 11 | 19095 | | 2019 | 12 | 21436 | | 2020 | 1 | 20347 | | 2020 | 2 | 17434 | | 2020 | 3 | 16225 | | 2020 | 4 | 13853 | | 2020 | 5 | 14589 | | 2020 | 6 | 13248 | | 2020 | 7 | 8728 | | 2020 | 8 | 9378 | | 2020 | 9 | 11467 | | 2020 | 10 | 13842 | | 2020 | 11 | 15742 | | 2020 | 12 | 18636 | +---------+----------+-----------+ 24 rows in set (0.00 sec) Exercise 16-1 Write a query that retrieves every row from Sales_Fact, and add a column to generate a ranking based on the tot_sales column values. The highest value should receive a ranking of 1, and the lowest a ranking of 24. Exercise 16-2 Modify the query from exercise 16-1 to generate two sets of rankings from 1 to 12, one for 2019 data and one for 2020. Exercise 16-3 Write a query that retrieves all 2020 data, and include a column which will contain the tot_sales value from the previous month. Chapter 17. Working with Large Databases In the early days of relational databases, hard drive capacity was measured in megabytes, and databases were generally easy to administer simply because they couldn’t get very large. Today, however, hard drive capacity has ballooned to 15TB, a modern disk array can store over 4PB of data, and storage in the cloud is essentially limitless. At some point along this journey, individual database tables started to become unwieldy, with row counts into the billions. Some databases have become too big to fit on even the largest database appliances, causing designers to find ways to spread data across many different databases, or to look to alternate technologies for data storage. This chapter looks at some of the strategies which have evolved for handling very large databases. Partitioning When exactly does a database table become “too big”?", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 193 + }, + { + "text": "largest database appliances, causing designers to find ways to spread data across many different databases, or to look to alternate technologies for data storage. This chapter looks at some of the strategies which have evolved for handling very large databases. Partitioning When exactly does a database table become “too big”? If you ask this question to 10 different data architects/administrators/developers, you will likely get 10 different answers. Most people, however, would agree that the following tasks become more difficult and/or time consuming as a table grows past a few million rows: Query execution requiring full table scans Index creation/rebuild Data archival/deletion Generation of table/index statistics Table relocation (e.g. move to a different tablespace) Database backups These tasks can start as routine when a database is small, then become time consuming as more and more data accumulates, and then become problematic/impossible due to limited administrative time windows. The best way to prevent administrative issues from occurring in the future is to break large tables into pieces, or partitions, when the table is first created (although tables can be partitioned later, it is easier to do so initially). Administrative tasks can be performed on individual partitions, often in parallel, and some tasks can skip one or more partitions entirely. Partitioning Concepts Table partitioning was introduced in the late 1990’s by Oracle, but since then every major database server has added the ability to partition tables and indexes. When a table is partitioned, two or more table partitions are created, each having the exact same definition, but with non-overlapping subsets of data. For example, a table containing sales data could be partitioned by month using the column containing the sale date, or it could be partitioned by geographic region using the state/province code. Once a table has been partitioned, the table itself becomes a virtual concept; the partitions hold the data, and any indexes are built on the data in the partitions. However, the database users can still interact with the table without knowing that the table had been partitioned. This is similar in concept to a view, in that the users interact with schema objects which are interfaces rather than actual tables. While every partition must have the same schema definition (columns, column types, etc.), there are several administrative features which can differ for each partition: Partitions may be stored on different tablespaces, which can be on different physical storage tiers Partitions can be compressed using different compression schemes Local indexes (more on this shortly) can be dropped for some partitions Table statistics can be frozen on some partitions, while being periodically refreshed on others Individual partitions can be pinned into memory, or stored in the database’s flash storage tier Thus, table partitioning allows for flexibility with data storage and administration, while still presenting the simplicity of a single table to your user community. Table Partitioning The partitioning scheme available in most relational databases is horizontal partitioning, which assigns entire rows to exactly one partition. Tables may also be partitioned vertically, which", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 194 + }, + { + "text": "partitioning allows for flexibility with data storage and administration, while still presenting the simplicity of a single table to your user community. Table Partitioning The partitioning scheme available in most relational databases is horizontal partitioning, which assigns entire rows to exactly one partition. Tables may also be partitioned vertically, which involves assigning sets of columns to different partitions, but this must be done manually. When partitioning a table horizontally, you must choose a partition key, which is the column whose values are used to assign a row to a particular partition. In most cases, a table’s partition key consists of a single column, and a partitioning function is applied to this column to determine in which partition each row should reside. Index Partitioning If your partitioned table has indexes, you will get to choose whether a particular index should stay intact, known as a global index, or be broken into pieces such that each partition has its own index, which is called a local index. Global indexes span all partitions of the table and are useful for queries which do not specify a value for the partition key. For example, let’s say your table is partitioned on the sale_date column, and a user executes the following query: SELECT sum(amount) FROM sales WHERE geo_region_cd = 'US' Since this query does not include a filter condition on the sale_date column, the server will need to search every partition in order to find the total US sales. If a global index is built on the geo_region_cd column, however, then the server could use this index to quickly find all of the rows containing US sales. Partitioning Methods While each database server has their own unique partitioning features, the next three sections describe the common partitioning methods available across most servers. Range Partitioning Range partitioning was the first partitioning method to be implemented, and it is still one of the most-widely used. While range partitioning can be used for several different column types, the most common usage is to break up tables by date ranges. For example, a table named sales could be partitioned using the sale_date column such that data for each week is stored in a different partition: mysql> CREATE TABLE sales -> (sale_id INT NOT NULL, -> cust_id INT NOT NULL, -> store_id INT NOT NULL, -> sale_date DATE NOT NULL, -> amount DECIMAL(9,2) -> ) -> PARTITION BY RANGE (yearweek(sale_date)) -> (PARTITION s1 VALUES LESS THAN (202002), -> PARTITION s2 VALUES LESS THAN (202003), -> PARTITION s3 VALUES LESS THAN (202004), -> PARTITION s4 VALUES LESS THAN (202005), -> PARTITION s5 VALUES LESS THAN (202006), -> PARTITION s999 VALUES LESS THAN (MAXVALUE) -> ); Query OK, 0 rows affected (1.78 sec) This statement creates six different partitions; one for each of the first five weeks of 2020, and a sixth partition named s999 to hold any rows beyond week 5 of year 2020. For this table, the yearweek(sale_date) expression is used as the partitioning function, and the sale_date column serves", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 195 + }, + { + "text": "sec) This statement creates six different partitions; one for each of the first five weeks of 2020, and a sixth partition named s999 to hold any rows beyond week 5 of year 2020. For this table, the yearweek(sale_date) expression is used as the partitioning function, and the sale_date column serves as the partitioning key. To see the metadata about your partitioned tables, you can use the partitions table in the information_schema database: mysql> SELECT partition_name, partition_method, partition_expression -> FROM information_schema.partitions -> WHERE table_name = 'sales' -> ORDER BY partition_ordinal_position; +----------------+------------------+-------------------------+ | PARTITION_NAME | PARTITION_METHOD | PARTITION_EXPRESSION | +----------------+------------------+-------------------------+ | s1 | RANGE | yearweek(`sale_date`,0) | | s2 | RANGE | yearweek(`sale_date`,0) | | s3 | RANGE | yearweek(`sale_date`,0) | | s4 | RANGE | yearweek(`sale_date`,0) | | s5 | RANGE | yearweek(`sale_date`,0) | | s999 | RANGE | yearweek(`sale_date`,0) | +----------------+------------------+-------------------------+ 6 rows in set (0.00 sec) One of the administrative tasks which will need to be performed on the sales table involves generating new partitions to hold future data (to keep data from being added to the MAXVALUE partition). Different databases handle this in different ways, but in MySQL you could use the reorganize partition clause of the alter table command to split the s999 partition into three pieces: ALTER TABLE sales REORGANIZE PARTITION s999 INTO (PARTITION s6 VALUES LESS THAN (202007), PARTITION s7 VALUES LESS THAN (202008), PARTITION s999 VALUES LESS THAN (MAXVALUE) ); If you execute the previous metadata query again, you will now see 8 partitions: mysql> SELECT partition_name, partition_method, partition_ex pression -> FROM information_schema.partitions -> WHERE table_name = 'sales' -> ORDER BY partition_ordinal_position; +----------------+------------------+----------------------- --+ | PARTITION_NAME | PARTITION_METHOD | PARTITION_EXPRESSION | +----------------+------------------+----------------------- --+ | s1 | RANGE | yearweek(`sale_date`,0 ) | | s2 | RANGE | yearweek(`sale_date`,0 ) | | s3 | RANGE | yearweek(`sale_date`,0 ) | | s4 | RANGE | yearweek(`sale_date`,0 ) | | s5 | RANGE | yearweek(`sale_date`,0 ) | | s6 | RANGE | yearweek(`sale_date`,0 ) | | s7 | RANGE | yearweek(`sale_date`,0 ) | | s999 | RANGE | yearweek(`sale_date`,0 ) | +----------------+------------------+----------------------- --+ 8 rows in set (0.00 sec) Next, let’s add a couple of rows to the table: mysql> INSERT INTO sales -> VALUES -> (1, 1, 1, '2020-01-18', 2765.15), -> (2, 3, 4, '2020-02-07', 5322.08); Query OK, 2 rows affected (0.18 sec) Records: 2 Duplicates: 0 Warnings: 0 The table now has 2 rows, but into which partitions where they inserted? To find out, let’s use the partition sub clause of the from clause to count the number of rows in each partition: mysql> SELECT concat('# of rows in S1 = ', count(*)) partition_rowcount -> FROM sales PARTITION (s1) UNION ALL -> SELECT concat('# of rows in S2 = ', count(*)) partition_rowcount -> FROM sales PARTITION (s2) UNION ALL -> SELECT concat('# of rows in S3 = ', count(*)) partition_rowcount -> FROM sales PARTITION (s3) UNION ALL -> SELECT concat('# of rows in S4 = ', count(*)) partition_rowcount -> FROM sales PARTITION (s4) UNION ALL ->", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 196 + }, + { + "text": "in S2 = ', count(*)) partition_rowcount -> FROM sales PARTITION (s2) UNION ALL -> SELECT concat('# of rows in S3 = ', count(*)) partition_rowcount -> FROM sales PARTITION (s3) UNION ALL -> SELECT concat('# of rows in S4 = ', count(*)) partition_rowcount -> FROM sales PARTITION (s4) UNION ALL -> SELECT concat('# of rows in S5 = ', count(*)) partition_rowcount -> FROM sales PARTITION (s5) UNION ALL -> SELECT concat('# of rows in S6 = ', count(*)) partition_rowcount -> FROM sales PARTITION (s6) UNION ALL -> SELECT concat('# of rows in S7 = ', count(*)) partition_rowcount -> FROM sales PARTITION (s7) UNION ALL -> SELECT concat('# of rows in S999 = ', count(*)) partition_rowcount -> FROM sales PARTITION (s999); +-----------------------+ | partition_rowcount | +-----------------------+ | # of rows in S1 = 0 | | # of rows in S2 = 1 | | # of rows in S3 = 0 | | # of rows in S4 = 0 | | # of rows in S5 = 1 | | # of rows in S6 = 0 | | # of rows in S7 = 0 | | # of rows in S999 = 0 | +-----------------------+ 8 rows in set (0.00 sec) The results show that one row was inserted into partition S2, and the other row was inserted into the S5 partition. The ability to query a specific partition involves having knowledge of the partitioning scheme, so it is unlikely that your user community will be executing these types of queries, but it is commonly used for administrative activities. List Partitioning If the column chosen as the partitioning key contains state codes (e.g. CA, TX, VA, etc.), currencies (e.g. USD, EUR, JPY, etc.), or some other enumerated set of values, you may want to utilize list partitioning, which allows you to specify which values will be assigned to each partition. For example, let’s say that the sales table includes the column geo_region_cd, which contains the following values: +---------------+--------------------------+ | geo_region_cd | description | +---------------+--------------------------+ | US_NE | United States North East | | US_SE | United States South East | | US_MW | United States Mid West | | US_NW | United States North West | | US_SW | United States South West | | CAN | Canada | | MEX | Mexico | | EUR_E | Eastern Europe | | EUR_W | Western Europe | | CHN | China | | JPN | Japan | | IND | India | | KOR | Korea | +---------------+--------------------------+ 13 rows in set (0.00 sec) You could group these values into geographic regions, and create a partition for each one, as in: mysql> CREATE TABLE sales -> (sale_id INT NOT NULL, -> cust_id INT NOT NULL, -> store_id INT NOT NULL, -> sale_date DATE NOT NULL, -> geo_region_cd VARCHAR(6) NOT NULL, -> amount DECIMAL(9,2) -> ) -> PARTITION BY LIST COLUMNS (geo_region_cd) -> (PARTITION NORTHAMERICA VALUES IN ('US_NE','US_SE','US_MW', -> 'US_NW','US_SW','CAN','MEX'), -> PARTITION EUROPE VALUES IN ('EUR_E','EUR_W'), -> PARTITION ASIA VALUES", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 197 + }, + { + "text": "NULL, -> cust_id INT NOT NULL, -> store_id INT NOT NULL, -> sale_date DATE NOT NULL, -> geo_region_cd VARCHAR(6) NOT NULL, -> amount DECIMAL(9,2) -> ) -> PARTITION BY LIST COLUMNS (geo_region_cd) -> (PARTITION NORTHAMERICA VALUES IN ('US_NE','US_SE','US_MW', -> 'US_NW','US_SW','CAN','MEX'), -> PARTITION EUROPE VALUES IN ('EUR_E','EUR_W'), -> PARTITION ASIA VALUES IN ('CHN','JPN','IND') -> ); Query OK, 0 rows affected (1.13 sec) The table has three partitions, where each partition includes a set of two or more geo_region_cd values. Next, let’s add a few rows to the table: mysql> INSERT INTO sales -> VALUES -> (1, 1, 1, '2020-01-18', 'US_NE', 2765.15), -> (2, 3, 4, '2020-02-07', 'CAN', 5322.08), -> (3, 6, 27, '2020-03-11', 'KOR', 4267.12); ERROR 1526 (HY000): Table has no partition for value from column_list It looks like there was a problem, and the error message indicates that one of the geographic region codes was not assigned to a partition. Looking at the create table statement, I see that I forgot to add Korea to the ASIA partition. This can be fixed using alter table statement: mysql> ALTER TABLE sales REORGANIZE PARTITION ASIA INTO -> (PARTITION ASIA VALUES IN ('CHN','JPN','IND', 'KOR')); Query OK, 0 rows affected (1.28 sec) Records: 0 Duplicates: 0 Warnings: 0 That seemed to do the trick, but let’s check the metadata just to be sure: mysql> SELECT partition_name, partition_expression, -> partition_description -> FROM information_schema.partitions -> WHERE table_name = 'sales' -> ORDER BY partition_ordinal_position; +----------------+----------------------+----------------------- ----------+ | PARTITION_NAME | PARTITION_EXPRESSION | PARTITION_DESCRIPTION | +----------------+----------------------+----------------------- ----------+ | NORTHAMERICA | `geo_region_cd` | 'US_NE','US_SE','US_MW','US_NW',| | | | 'US_SW','CAN','MEX' | | EUROPE | `geo_region_cd` | 'EUR_E','EUR_W' | | ASIA | `geo_region_cd` | 'CHN','JPN','IND','KOR' | +----------------+----------------------+----------------------- ----------+ 3 rows in set (0.00 sec) Korea has indeed been added to the ASIA partition, and the data insertion will now proceed without any issues: mysql> INSERT INTO sales -> VALUES -> (1, 1, 1, '2020-01-18', 'US_NE', 2765.15), -> (2, 3, 4, '2020-02-07', 'CAN', 5322.08), -> (3, 6, 27, '2020-03-11', 'KOR', 4267.12); Query OK, 3 rows affected (0.26 sec) Records: 3 Duplicates: 0 Warnings: 0 While range partitioning allows for a “maxvalue” partition to catch any rows which don’t map to any other partition, it’s important to keep in mind that list partitioning doesn’t provide for a spillover partition. Thus, any time you need to add another column value (e.g. the company starts selling products in Australia), you will need to modify the partitioning definition before rows with the new value can be added to the table. Hash Partitioning If your partition key column doesn’t lend itself to range or list partitioning, there is a third option which endeavors to distribute rows evenly across a set of partitions. The server does this by applying a hashing function to the column value, and this type of partitioning is (not surprisingly) called hash partitioning. Unlike list partitioning, where the column chosen as the partitioning key should only contain a small number of values, hash partitioning works best when the partitioning key column contains a large number of", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 198 + }, + { + "text": "hashing function to the column value, and this type of partitioning is (not surprisingly) called hash partitioning. Unlike list partitioning, where the column chosen as the partitioning key should only contain a small number of values, hash partitioning works best when the partitioning key column contains a large number of distinct values. Here’s another version of the sales table, but with 4 hash partitions generated by hashing the values in the cust_id column: mysql> CREATE TABLE sales -> (sale_id INT NOT NULL, -> cust_id INT NOT NULL, -> store_id INT NOT NULL, -> sale_date DATE NOT NULL, -> amount DECIMAL(9,2) -> ) -> PARTITION BY HASH (cust_id) -> PARTITIONS 4 -> (PARTITION H1, -> PARTITION H2, -> PARTITION H3, -> PARTITION H4 -> ); Query OK, 0 rows affected (1.50 sec) When rows are added to the sales table, they will be evenly distributed across the four partitions, which I named H1, H2, H3, and H4. In order to see how good a job it does, let’s add 16 rows, each with a different value for the cust_id column: mysql> INSERT INTO sales -> VALUES -> (1, 1, 1, '2020-01-18', 1.1), (2, 3, 4, '2020-02-07', 1.2), -> (3, 17, 5, '2020-01-19', 1.3), (4, 23, 2, '2020-02-08', 1.4), -> (5, 56, 1, '2020-01-20', 1.6), (6, 77, 5, '2020-02-09', 1.7), -> (7, 122, 4, '2020-01-21', 1.8), (8, 153, 1, '2020-02- 10', 1.9), -> (9, 179, 5, '2020-01-22', 2.0), (10, 244, 2, '2020-02- 11', 2.1), -> (11, 263, 1, '2020-01-23', 2.2), (12, 312, 4, '2020-02- 12', 2.3), -> (13, 346, 2, '2020-01-24', 2.4), (14, 389, 3, '2020-02- 13', 2.5), -> (15, 472, 1, '2020-01-25', 2.6), (16, 502, 1, '2020-02- 14', 2.7); Query OK, 16 rows affected (0.19 sec) Records: 16 Duplicates: 0 Warnings: 0 If the hashing function does a good job of distributing the rows evenly, we should ideally see 4 rows in each of the 4 partitions: mysql> SELECT concat('# of rows in H1 = ', count(*)) partition_rowcount -> FROM sales PARTITION (h1) UNION ALL -> SELECT concat('# of rows in H2 = ', count(*)) partition_rowcount -> FROM sales PARTITION (h2) UNION ALL -> SELECT concat('# of rows in H3 = ', count(*)) partition_rowcount -> FROM sales PARTITION (h3) UNION ALL -> SELECT concat('# of rows in H4 = ', count(*)) partition_rowcount -> FROM sales PARTITION (h4); +---------------------+ | partition_rowcount | +---------------------+ | # of rows in H1 = 4 | | # of rows in H2 = 5 | | # of rows in H3 = 3 | | # of rows in H4 = 4 | +---------------------+ 4 rows in set (0.00 sec) Given that only 16 rows were inserted, this is a pretty good distribution, and as the number of rows increases, each partition should contain close to 25% of the rows as long as there are a reasonably large number of distinct values for the cust_id column. Composite Partitioning If you need finer-grained control of how data is allocated to your partitions, you can employ composite partitioning,", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 199 + }, + { + "text": "number of rows increases, each partition should contain close to 25% of the rows as long as there are a reasonably large number of distinct values for the cust_id column. Composite Partitioning If you need finer-grained control of how data is allocated to your partitions, you can employ composite partitioning, which allows you to use two different types of partitioning for the same table. With composite partitioning, the first partitioning method defines the partitions, and the second partitioning method defines the subpartitions. Here’s an example, again using the sales table, but utilizing both range and hash partitioning: mysql> CREATE TABLE sales -> (sale_id INT NOT NULL, -> cust_id INT NOT NULL, -> store_id INT NOT NULL, -> sale_date DATE NOT NULL, -> amount DECIMAL(9,2) -> ) -> PARTITION BY RANGE (yearweek(sale_date)) -> SUBPARTITION BY HASH (cust_id) -> (PARTITION s1 VALUES LESS THAN (202002) -> (SUBPARTITION s1_h1, -> SUBPARTITION s1_h2, -> SUBPARTITION s1_h3, -> SUBPARTITION s1_h4), -> PARTITION s2 VALUES LESS THAN (202003) -> (SUBPARTITION s2_h1, -> SUBPARTITION s2_h2, -> SUBPARTITION s2_h3, -> SUBPARTITION s2_h4), -> PARTITION s3 VALUES LESS THAN (202004) -> (SUBPARTITION s3_h1, -> SUBPARTITION s3_h2, -> SUBPARTITION s3_h3, -> SUBPARTITION s3_h4), -> PARTITION s4 VALUES LESS THAN (202005) -> (SUBPARTITION s4_h1, -> SUBPARTITION s4_h2, -> SUBPARTITION s4_h3, -> SUBPARTITION s4_h4), -> PARTITION s5 VALUES LESS THAN (202006) -> (SUBPARTITION s5_h1, -> SUBPARTITION s5_h2, -> SUBPARTITION s5_h3, -> SUBPARTITION s5_h4), -> PARTITION s999 VALUES LESS THAN (MAXVALUE) -> (SUBPARTITION s999_h1, -> SUBPARTITION s999_h2, -> SUBPARTITION s999_h3, -> SUBPARTITION s999_h4) -> ); Query OK, 0 rows affected (9.72 sec) There are 6 partitions, each having 4 subpartitions, for a total of 24 subpartitions. Next, let’s re-insert the 16 rows from the earlier example for hash partitioning: mysql> INSERT INTO sales -> VALUES -> (1, 1, 1, '2020-01-18', 1.1), (2, 3, 4, '2020-02-07', 1.2), -> (3, 17, 5, '2020-01-19', 1.3), (4, 23, 2, '2020-02-08', 1.4), -> (5, 56, 1, '2020-01-20', 1.6), (6, 77, 5, '2020-02-09', 1.7), -> (7, 122, 4, '2020-01-21', 1.8), (8, 153, 1, '2020-02- 10', 1.9), -> (9, 179, 5, '2020-01-22', 2.0), (10, 244, 2, '2020-02- 11', 2.1), -> (11, 263, 1, '2020-01-23', 2.2), (12, 312, 4, '2020-02- 12', 2.3), -> (13, 346, 2, '2020-01-24', 2.4), (14, 389, 3, '2020-02- 13', 2.5), -> (15, 472, 1, '2020-01-25', 2.6), (16, 502, 1, '2020-02- 14', 2.7); Query OK, 16 rows affected (0.22 sec) Records: 16 Duplicates: 0 Warnings: 0 When you query the sales table, you can retrieve data from one of the partitions, in which case you retrieve data from the 4 subpartitions associated with the partition: mysql> SELECT * -> FROM sales PARTITION (s3); +---------+---------+----------+------------+--------+ | sale_id | cust_id | store_id | sale_date | amount | +---------+---------+----------+------------+--------+ | 5 | 56 | 1 | 2020-01-20 | 1.60 | | 15 | 472 | 1 | 2020-01-25 | 2.60 | | 3 | 17 | 5 | 2020-01-19 | 1.30 | | 7 | 122 | 4 | 2020-01-21 | 1.80 | | 13 | 346 | 2 | 2020-01-24", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 200 + }, + { + "text": "| 56 | 1 | 2020-01-20 | 1.60 | | 15 | 472 | 1 | 2020-01-25 | 2.60 | | 3 | 17 | 5 | 2020-01-19 | 1.30 | | 7 | 122 | 4 | 2020-01-21 | 1.80 | | 13 | 346 | 2 | 2020-01-24 | 2.40 | | 9 | 179 | 5 | 2020-01-22 | 2.00 | | 11 | 263 | 1 | 2020-01-23 | 2.20 | +---------+---------+----------+------------+--------+ 7 rows in set (0.00 sec) Since the table is subpartitioned, you may also retrieve data from a single subpartition: mysql> SELECT * -> FROM sales PARTITION (s3_h3); +---------+---------+----------+------------+--------+ | sale_id | cust_id | store_id | sale_date | amount | +---------+---------+----------+------------+--------+ | 7 | 122 | 4 | 2020-01-21 | 1.80 | | 13 | 346 | 2 | 2020-01-24 | 2.40 | +---------+---------+----------+------------+--------+ 2 rows in set (0.00 sec) This query retrieves data only from the s3_h3 subpartition of the s3 partition. Partitioning Benefits One major advantage to partitioning is that you may only need to interact with as few as one partition, rather than the entire table. For example, if your table is range-partitioned on the sales_date column, and you execute a query which includes a filter condition such as WHERE sales_date BETWEEN '2019-12-01' AND '2020-01-15' , the server will check the table’s metadata to determine which partitions actually need to be included. This concept is called partition pruning, and it is one of the biggest advantages of table partitioning. Similarly, if you execute a query which includes a join to a partitioned table, and the query includes a condition on the partitioning column, the server can exclude any partitions which do not contain data pertinent to the query. This is known as partition-wise joins, and it is similar to partition pruning in that only those partitions which contain data needed by the query will be included. From an administrative standpoint, one of the main benefits to partitioning is the ability to quickly delete data which is no longer needed. For example, financial data may be required to be kept online for seven years; if a table has been partitioned based on transaction dates, any partitions holding data greater than seven years old can be dropped. Another administrative advantage to partitioned tables is the ability to perform updates on multiple partitions simultaneously, which can greatly reduce the time needed to touch every row in a table. Sharding Let’s say you have been hired as the Data Architect for a new social media company. You are told to expect approximately 1 billion users, each of whom will generate 3.7 messages per day on average, and that the data must be available indefinitely. After performing a few calculations, you determine that you would exhaust the biggest available relational database platform in less than a year. One possibility would be to partition not just individual tables, but the entire database. Known as sharding, this approach partitions the data across multiple databases (called shards), so it", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 201 + }, + { + "text": "performing a few calculations, you determine that you would exhaust the biggest available relational database platform in less than a year. One possibility would be to partition not just individual tables, but the entire database. Known as sharding, this approach partitions the data across multiple databases (called shards), so it is similar to table partitioning, but on a larger scale and with far more complexity. If you were to employ this strategy for the social media company, you might decide to implement 100 separate databases, each one hosting the data for approximately 10 million users. Sharding is a complex topic, and since this is an introductory book I will refrain from going into details, but here are a few of the issues which would need to be addressed: You will need to choose a sharding key, which is the value used to determine to which database to connect. While large tables will be divided into pieces, with individual rows assigned to a single shard, smaller reference tables may need to be replicated to all shards, and a strategy needs to be defined for how reference data can be modified and changes propagated to all shards. If individual shards become too large (e.g. the social media company now has 2 billion users), you will need a plan for adding more shards and redistributing data across the shards. When you need to make schema changes, you will need to have a strategy for deploying the changes across all of the shards so that all schemas stay in synch. If application logic needs to access data stored in two or more shards, you need to have a strategy for how to query across multiple databases, and also how to implement transactions across multiple databases. If this seems complicated, that’s because it is, and by the late 2000’s many companies began looking for new approaches. The next section looks at other strategies for handling very large data sets, but completely outside the realm of relational databases. Big Data After spending some time weighing the pros and cons of sharding, let’s say that you (the Data Architect of the social media company) decide to investigate other approaches. Rather than attempting to forge your own path, you might benefit from reviewing the work done by other companies which deal with massive amounts of data: companies like Amazon, Google, Facebook, and Twitter. Together, the set of technologies pioneered by these companies (and others) have been branded as Big Data, which has become an industry buzzword but has several possible definitions. However, one way to define the boundaries of Big Data is with the “3 V’s”: 1. Volume, which in this context generally means billions or trillions of data points 2. Velocity, which is a measure of how quickly data arrives 3. Variety, meaning that data is not always structured (as in rows and columns in a relational database) but can also be unstructured (e.g. emails, videos, photos, audio files, etc.) So, one way to characterize Big Data", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 202 + }, + { + "text": "data points 2. Velocity, which is a measure of how quickly data arrives 3. Variety, meaning that data is not always structured (as in rows and columns in a relational database) but can also be unstructured (e.g. emails, videos, photos, audio files, etc.) So, one way to characterize Big Data is any system designed to handle a huge amount of data of various formats arriving at a rapid pace. The following sections offer a quick description of some of the Big Data technologies which have evolved over the past 15 years or so. Hadoop Hadoop is best described as an ecosystem, or a set of technologies and tools which work together. Some of the major components include: Hadoop Distributed File System (HDFS), which, like the name implies, enables file management across a large number of servers. MapReduce, which is a technology used to process large amounts of structured and unstructured data by breaking a task into many small pieces which can be run in parallel across many servers. YARN, which is a resource manager and job scheduler for HDFS Together, these technologies allow for the storage and processing of files across hundreds or even thousands of servers acting as a single logical system. While Hadoop is widely used, querying the data using MapReduce generally requires a programmer, which has led to the development of several SQL interfaces, including Hive, Impala, and Drill. NoSQL and Document Databases In a relational database, data must generally conform to a pre-defined schema consisting of tables made up of columns holding numbers, strings, dates, etc. What happens, however, if the structure of the data isn’t known beforehand, or if the structure is known but changes frequently? The answer for many companies is to combine both the data and schema definition into documents using a format such as XML (Extensible Markup Language) or JSON (JavaScript Object Notation), and then store the documents in a database. By doing so, various types of data can be stored in the same database without the need to make schema modifications, which makes storage easier but puts the burden on query and analytic tools to make sense of the data stored in the documents. Document databases are a subset of what are called NoSQL databases, which typically store data using a simple key-value mechanism. For example, using a document database such as MongoDB, you could utilize the User ID as the key to store a JSON document containing all of the customer’s data, and other users can read the schema stored within the document to make sense of the data stored within. Cloud Computing Prior to the advent of Big Data, most companies had to build their own data centers to house the database, web, and application servers used across the enterprise. With the advent of cloud computing, you can choose to essentially outsource your data center to platforms such as Amazon Web Services (AWS), Microsoft Azure, or Google Cloud. One of the biggest benefits to hosting your services in the", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 203 + }, + { + "text": "house the database, web, and application servers used across the enterprise. With the advent of cloud computing, you can choose to essentially outsource your data center to platforms such as Amazon Web Services (AWS), Microsoft Azure, or Google Cloud. One of the biggest benefits to hosting your services in the cloud is instant scalability, which allows you to quickly dial up or down the amount of computing power needed to run your services. Startups love these platforms because they can start writing code without spending any money upfront for servers, storage, networks, or software licenses. As far as databases are concerned, a quick look at AWS’s database and analytics offerings yields the following options: Relational databases (MySQL, Aurora, PostgreSQL, MariaDB, Oracle, and SQL Server) In-memory database (ElastiCache) Data Warehousing database (Redshift) NoSQL database (DynamoDB) Document database (DocumentDB) Graph database (Neptune) Time Series database (TimeStream) Hadoop (EMR) Data Lakes (Lake Formation) While relational databases had dominated the landscape up until the mid 2000’s, it’s pretty easy to see that companies are now mixing and matching various platforms, and that relational databases will become less and less prevalent over time. Future of SQL So if relational databases are becoming less popular, what does that mean for the future of SQL? While many people think of SQL as the query language used for relational databases, the SQL language has taken on a life of its own in recent years, mostly because it is the used by millions of people (both programmers and non-programmers) and has been embedded into thousands of applications (such as just about every reporting and analytic engine). There have also been many attempts to graft the SQL language onto non-relational databases over the years, such as Hive for Hadoop. One tool that I find particularly interesting is Apache Drill, which is a SQL engine which can query many kinds of data, stored either locally or on just about any distributed file system. In a later chapter, I will demonstrate how to use Apache Drill to query the Sakila sample data set, both in MySQL and in MongoDB. Chapter 18. SQL and Big Data While most of the content in this book covers the various features of the SQL language when using a relational database such as MySQL, the data landscape has changed quite a bit over the past decade, and SQL is changing to meet the needs of today’s rapidly evolving environments. Many organizations which had used relational databases exclusively just a few years ago are now also housing data in Hadoop clusters, data lakes, and NoSQL databases. At the same time, companies are struggling to find ways to gain insights from the ever-growing volumes of data, and the fact that this data is now spread across multiple data stores, perhaps both onsite and in the cloud, makes this a daunting task. Because SQL is used by millions of people and has been integrated into thousands of applications, it makes sense to leverage SQL to harness this data and make", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 204 + }, + { + "text": "that this data is now spread across multiple data stores, perhaps both onsite and in the cloud, makes this a daunting task. Because SQL is used by millions of people and has been integrated into thousands of applications, it makes sense to leverage SQL to harness this data and make it actionable. Before this can happen, however, the SQL language needs to evolve in order to work with semi-structured and unstructured data, and a new breed of tools has emerged to meet this challenge. This chapter will use one of these tools to demonstrate how data in different formats and stored on different servers can be brought together. Apache Drill There have been numerous tools and interfaces developed to allow SQL access to data stored in Hadoop, NoSQL, Spark, and cloud-based distributed file systems. Examples include Hive, which was one of the first attempts to allow users to query data stored in Hadoop, and Spark SQL, which is a library used to query data stored in various formats from within Spark. One relative newcomer is the open-source Apache Drill, which first hit the scene in 2015 and has some compelling features: facilitates queries across multiple data formats, including delimited data, JSON, Parquet, and log files connects to relational databases, Hadoop, NoSQL, HBase, Kafka allows creation of custom plug-ins to connect to most any other data store requires no up-front schema definitions supports the SQL:2003 standard works with popular BI tools like Tableau Using Drill, you can connect to any number of data sources and begin querying, without the need to first set up a metadata repository. While it is beyond the scope of this book to discuss the installation and configuration options for Apache Drill, if you are interested in learning more I highly recommend “Learning Apache Drill” by Charles Givre and Paul Rogers (O’Reilly). Drill and MySQL Let’s start by running some Drill queries against the Sakila sample database used for the examples in this book. After loading the JDBC driver for MySQL and configuring Drill to connect to my local MySQL database, I should be able to run most of the example queries from earlier chapters. The first step is to choose a database: apache drill (information_schema)> use mysql.sakila; +------+------------------------------------------+ | ok | summary | +------+------------------------------------------+ | true | Default schema changed to [mysql.sakila] | +------+------------------------------------------+ 1 row selected (0.062 seconds) After choosing the database, Drill includes a simple command to show all of the tables available in the chosen schema: apache drill (mysql.sakila)> show tables; +--------------+----------------------------+ | TABLE_SCHEMA | TABLE_NAME | +--------------+----------------------------+ | mysql.sakila | actor | | mysql.sakila | address | | mysql.sakila | category | | mysql.sakila | city | | mysql.sakila | country | | mysql.sakila | customer | | mysql.sakila | film | | mysql.sakila | film_actor | | mysql.sakila | film_category | | mysql.sakila | film_text | | mysql.sakila | inventory | | mysql.sakila | language | | mysql.sakila | payment | | mysql.sakila | rental | | mysql.sakila | sales |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 205 + }, + { + "text": "| mysql.sakila | customer | | mysql.sakila | film | | mysql.sakila | film_actor | | mysql.sakila | film_category | | mysql.sakila | film_text | | mysql.sakila | inventory | | mysql.sakila | language | | mysql.sakila | payment | | mysql.sakila | rental | | mysql.sakila | sales | | mysql.sakila | staff | | mysql.sakila | store | | mysql.sakila | actor_info | | mysql.sakila | customer_list | | mysql.sakila | film_list | | mysql.sakila | nicer_but_slower_film_list | | mysql.sakila | sales_by_film_category | | mysql.sakila | sales_by_store | | mysql.sakila | staff_list | +--------------+----------------------------+ 24 rows selected (0.147 seconds) Everything looks good, so it’s time to run some queries. Here’s a simple 2- table join from the Joins chapter: apache drill (mysql.sakila)> SELECT a.address_id, a.address, ct.city . . . . . . . . . . . . . )> FROM address a . . . . . . . . . . . . . )> INNER JOIN city ct . . . . . . . . . . . . . )> ON a.city_id = ct.city_id . . . . . . . . . . . . . )> WHERE a.district = 'California'; +------------+------------------------+----------------+ | address_id | address | city | +------------+------------------------+----------------+ | 6 | 1121 Loja Avenue | San Bernardino | | 18 | 770 Bydgoszcz Avenue | Citrus Heights | | 55 | 1135 Izumisano Parkway | Fontana | | 116 | 793 Cam Ranh Avenue | Lancaster | | 186 | 533 al-Ayn Boulevard | Compton | | 218 | 226 Brest Manor | Sunnyvale | | 274 | 920 Kumbakonam Loop | Salinas | | 425 | 1866 al-Qatif Avenue | El Monte | | 599 | 1895 Zhezqazghan Drive | Garden Grove | +------------+------------------------+----------------+ 9 rows selected (3.523 seconds) The next query comes from the Grouping and Aggregates chapter and includes both a Group By and a Having clause: apache drill (mysql.sakila)> SELECT fa.actor_id, f.rating, . . . . . . . . . . . . . )> count(*) num_films . . . . . . . . . . . . . )> FROM film_actor fa . . . . . . . . . . . . . )> INNER JOIN film f . . . . . . . . . . . . . )> ON fa.film_id = f.film_id . . . . . . . . . . . . . )> WHERE f.rating IN ('G','PG') . . . . . . . . . . . . . )> GROUP BY fa.actor_id, f.rating . . . . . . . . . . . . . )> HAVING count(*) > 9; +----------+--------+-----------+ | actor_id | rating | num_films | +----------+--------+-----------+ | 137 | PG | 10 | | 37 | PG | 12 | | 180 | PG | 12 | | 7 | G | 10 | | 83 | G | 14 | | 129 | G | 12 |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 206 + }, + { + "text": "| actor_id | rating | num_films | +----------+--------+-----------+ | 137 | PG | 10 | | 37 | PG | 12 | | 180 | PG | 12 | | 7 | G | 10 | | 83 | G | 14 | | 129 | G | 12 | | 111 | PG | 15 | | 44 | PG | 12 | | 26 | PG | 11 | | 92 | PG | 12 | | 17 | G | 12 | | 158 | PG | 10 | | 147 | PG | 10 | | 14 | G | 10 | | 102 | PG | 11 | | 133 | PG | 10 | +----------+--------+-----------+ 16 rows selected (0.277 seconds) Finally, here’s a query from the Analytic Functions chapter which includes 3 different ranking functions: apache drill (mysql.sakila)> SELECT customer_id, count(*) num_rentals, . . . . . . . . . . . . . )> row_number() . . . . . . . . . . . . . )> over (order by count(*) desc) . . . . . . . . . . . . . )> row_number_rnk, . . . . . . . . . . . . . )> rank() . . . . . . . . . . . . . )> over (order by count(*) desc) rank_rnk, . . . . . . . . . . . . . )> dense_rank() . . . . . . . . . . . . . )> over (order by count(*) desc) . . . . . . . . . . . . . )> dense_rank_rnk . . . . . . . . . . . . . )> FROM rental . . . . . . . . . . . . . )> GROUP BY customer_id . . . . . . . . . . . . . )> ORDER BY 2 desc; +-------------+-------------+----------------+----------+------- ---------+ | customer_id | num_rentals | row_number_rnk | rank_rnk | dense_rank_rnk | +-------------+-------------+----------------+----------+------- ---------+ | 148 | 46 | 1 | 1 | 1 | | 526 | 45 | 2 | 2 | 2 | | 144 | 42 | 3 | 3 | 3 | | 236 | 42 | 4 | 3 | 3 | | 75 | 41 | 5 | 5 | 4 | | 197 | 40 | 6 | 6 | 5 | … | 248 | 15 | 595 | 594 | 30 | | 61 | 14 | 596 | 596 | 31 | | 110 | 14 | 597 | 596 | 31 | | 281 | 14 | 598 | 596 | 31 | | 318 | 12 | 599 | 599 | 32 | +-------------+-------------+----------------+----------+------- ---------+ 599 rows selected (1.827 seconds) It looks like Drill does a pretty good job querying MySQL, but you will need to keep in mind that Drill works with many", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 207 + }, + { + "text": "14 | 598 | 596 | 31 | | 318 | 12 | 599 | 599 | 32 | +-------------+-------------+----------------+----------+------- ---------+ 599 rows selected (1.827 seconds) It looks like Drill does a pretty good job querying MySQL, but you will need to keep in mind that Drill works with many relational databases, not just MySQL, so some features of the language may differ (e.g. data conversion functions). For more information, you can read about Drill’s SQL implementation at http://drill.apache.org/docs/sql-reference/. Drill and MongoDB After using Drill to query the sample Sakila data in MySQL, I felt that the next logical step would be to convert the Sakila data to another commonly-used format, store it in a non-relational database, and use Drill to query the data. I decided to convert the data to JSON (JavaScript Object Notation) and store it in MongoDB, which is one of the more popular NoSQL platforms for document storage. Fortunately, I discovered that fellow author Guy Harrison had the same idea a couple of years ago, and he was kind enough to share his files with me. Drill includes a plug-in for MongoDB, so it was relatively easy to load Guy’s JSON files into Mongo and begin writing queries. Before diving into the queries, let’s take a look at the structure of the JSON files, since it isn’t in normalized form. The first of the two JSON files is films.json: {\"_id\":1, \"Actors\":[ 1 {\"First name\":\"PENELOPE\",\"Last name\":\"GUINESS\",\"actorId\":1}, {\"First name\":\"CHRISTIAN\",\"Last name\":\"GABLE\",\"actorId\":10}, {\"First name\":\"LUCILLE\",\"Last name\":\"TRACY\",\"actorId\":20}, {\"First name\":\"SANDRA\",\"Last name\":\"PECK\",\"actorId\":30}, {\"First name\":\"JOHNNY\",\"Last name\":\"CAGE\",\"actorId\":40}, {\"First name\":\"MENA\",\"Last name\":\"TEMPLE\",\"actorId\":53}, {\"First name\":\"WARREN\",\"Last name\":\"NOLTE\",\"actorId\":108}, {\"First name\":\"OPRAH\",\"Last name\":\"KILMER\",\"actorId\":162}, {\"First name\":\"ROCK\",\"Last name\":\"DUKAKIS\",\"actorId\":188}, {\"First name\":\"MARY\",\"Last name\":\"KEITEL\",\"actorId\":198}], \"Category\":\"Documentary\", \"Description\":\"A Epic Drama of a Feminist And a Mad Scientist who must Battle a Teacher in The Canadian Rockies\", \"Length\":\"86\", \"Rating\":\"PG\", \"Rental Duration\":\"6\", \"Replacement Cost\":\"20.99\", \"Special Features\":\"Deleted Scenes,Behind the Scenes\", \"Title\":\"ACADEMY DINOSAUR\"}, {\"_id\":2, \"Actors\":[ {\"First name\":\"BOB\",\"Last name\":\"FAWCETT\",\"actorId\":19}, {\"First name\":\"MINNIE\",\"Last name\":\"ZELLWEGER\",\"actorId\":85}, {\"First name\":\"SEAN\",\"Last name\":\"GUINESS\",\"actorId\":90}, {\"First name\":\"CHRIS\",\"Last name\":\"DEPP\",\"actorId\":160}], \"Category\":\"Horror\", \"Description\":\"A Astounding Epistle of a Database Administrator And a Explorer who must Find a Car in Ancient China\", \"Length\":\"48\", \"Rating\":\"G\", \"Rental Duration\":\"3\", \"Replacement Cost\":\"12.99\", \"Special Features\":\"Trailers,Deleted Scenes\", \"Title\":\"ACE GOLDFINGER\"}, ... {\"_id\":999, \"Actors\":[ {\"First name\":\"CARMEN\",\"Last name\":\"HUNT\",\"actorId\":52}, {\"First name\":\"MARY\",\"Last name\":\"TANDY\",\"actorId\":66}, {\"First name\":\"PENELOPE\",\"Last name\":\"CRONYN\",\"actorId\":104}, {\"First name\":\"WHOOPI\",\"Last name\":\"HURT\",\"actorId\":140}, {\"First name\":\"JADA\",\"Last name\":\"RYDER\",\"actorId\":142}], \"Category\":\"Children\", \"Description\":\"A Fateful Reflection of a Waitress And a Boat who must Discover a Sumo Wrestler in Ancient China\", \"Length\":\"101\", \"Rating\":\"R\", \"Rental Duration\":\"5\", \"Replacement Cost\":\"28.99\", \"Special Features\":\"Trailers,Deleted Scenes\", \"Title\":\"ZOOLANDER FICTION\"} {\"_id\":1000, \"Actors\":[ {\"First name\":\"IAN\",\"Last name\":\"TANDY\",\"actorId\":155}, {\"First name\":\"NICK\",\"Last name\":\"DEGENERES\",\"actorId\":166}, {\"First name\":\"LISA\",\"Last name\":\"MONROE\",\"actorId\":178}], \"Category\":\"Comedy\", \"Description\":\"A Intrepid Panorama of a Mad Scientist And a Boy who must Redeem a Boy in A Monastery\", \"Length\":\"50\", \"Rating\":\"NC-17\", \"Rental Duration\":\"3\", \"Replacement Cost\":\"18.99\", \"Special Features\": \"Trailers,Commentaries,Behind the Scenes\", \"Title\":\"ZORRO ARK\"} There are 1,000 documents in this collection, and each document contains a number of scalar attributes (Title, Rating, _id) but also includes an array called Actors, which contains 1 to N elements consisting of the actorId, First Name, and Last Name attributes for every actor appearing in the film. Therefore, this file contains all of the data found in the Actor, Film, and Film_Actor tables within", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 208 + }, + { + "text": "(Title, Rating, _id) but also includes an array called Actors, which contains 1 to N elements consisting of the actorId, First Name, and Last Name attributes for every actor appearing in the film. Therefore, this file contains all of the data found in the Actor, Film, and Film_Actor tables within the MySQL Sakila database. The second file is customer.json, which combines data from the Customer, Address, City, Country, Rental, and Payment tables from the MySQL Sakila database: {\"_id\":1, \"Address\":\"1913 Hanoi Way\", \"City\":\"Sasebo\", \"Country\":\"Japan\", \"District\":\"Nagasaki\", \"First Name\":\"MARY\", \"Last Name\":\"SMITH\", \"Phone\":\"28303384290\", \"Rentals\":[ {\"rentalId\":1185, \"filmId\":611, \"staffId\":2, \"Film Title\":\"MUSKETEERS WAIT\", \"Payments\":[ {\"Payment Id\":3,\"Amount\":5.99,\"Payment Date\":\"2005-06-15 00:54:12\"}], \"Rental Date\":\"2005-06-15 00:54:12.0\", \"Return Date\":\"2005-06-23 02:42:12.0\"}, {\"rentalId\":1476, \"filmId\":308, \"staffId\":1, \"Film Title\":\"FERRIS MOTHER\", \"Payments\":[ {\"Payment Id\":5,\"Amount\":9.99,\"Payment Date\":\"2005-06-15 21:08:46\"}], \"Rental Date\":\"2005-06-15 21:08:46.0\", \"Return Date\":\"2005-06-25 02:26:46.0\"}, ... {\"rentalId\":14825, \"filmId\":317, \"staffId\":2, \"Film Title\":\"FIREBALL PHILADELPHIA\", \"Payments\":[ {\"Payment Id\":30,\"Amount\":1.99,\"Payment Date\":\"2005-08-22 01:27:57\"}], \"Rental Date\":\"2005-08-22 01:27:57.0\", \"Return Date\":\"2005-08-27 07:01:57.0\"} ] } This file contains 599 entries (only 1 is shown above), which is loaded into Mongo as 599 documents in the customers collection. Each document contains the information about a single customer, along with all of the rentals and associated payments made by that customer. Furthermore, the documents contain nested arrays, since each rental in the Rentals array also contains an array of Payments. After the JSON files have been loaded, the Mongo database contains two collections (films and customers), and the data in these collections spans 9 different tables from the MySQL Sakila database. This is a fairly typical scenario, since application programmers typically work with collections, and generally prefer not to deconstruct their data for storage into normalized relational tables. The challenge from an SQL perspective is to determine how to flatten this data so that it behaves as if it were stored in multiple tables. To illustrate, let’s construct the following query against the films collection: find all actors who have appeared in 10 or more films rated either G or PG. Here’s what the raw data looks like: apache drill (mongo.sakila)> SELECT Rating, Actors . . . . . . . . . . . . . )> FROM films . . . . . . . . . . . . . )> WHERE Rating IN ('G','PG'); +--------+------------------------------------------------------ ----------+ | Rating | Actors | +--------+------------------------------------------------------ ----------+ | PG |[{\"First name\":\"PENELOPE\",\"Last name\":\"GUINESS\",\"actorId\":\"1\"}, {\"First name\":\"FRANCES\",\"Last name\":\"DAY- LEWIS\",\"actorId\":\"48\"}, {\"First name\":\"ANNE\",\"Last name\":\"CRONYN\",\"actorId\":\"49\"}, {\"First name\":\"RAY\",\"Last name\":\"JOHANSSON\",\"actorId\":\"64\"}, {\"First name\":\"PENELOPE\",\"Last name\":\"CRONYN\",\"actorId\":\"104\"}, {\"First name\":\"HARRISON\",\"Last name\":\"BALE\",\"actorId\":\"115\"}, {\"First name\":\"JEFF\",\"Last name\":\"SILVERSTONE\",\"actorId\":\"180\"}, {\"First name\":\"ROCK\",\"Last name\":\"DUKAKIS\",\"actorId\":\"188\"}] | | PG |[{\"First name\":\"UMA\",\"Last name\":\"WOOD\",\"actorId\":\"13\"}, {\"First name\":\"HELEN\",\"Last name\":\"VOIGHT\",\"actorId\":\"17\"}, {\"First name\":\"CAMERON\",\"Last name\":\"STREEP\",\"actorId\":\"24\"}, {\"First name\":\"CARMEN\",\"Last name\":\"HUNT\",\"actorId\":\"52\"}, {\"First name\":\"JANE\",\"Last name\":\"JACKMAN\",\"actorId\":\"131\"}, {\"First name\":\"BELA\",\"Last name\":\"WALKEN\",\"actorId\":\"196\"}] | ... | G |[{\"First name\":\"ED\",\"Last name\":\"CHASE\",\"actorId\":\"3\"}, {\"First name\":\"JULIA\",\"Last name\":\"MCQUEEN\",\"actorId\":\"27\"}, {\"First name\":\"JAMES\",\"Last name\":\"PITT\",\"actorId\":\"84\"}, {\"First name\":\"CHRISTOPHER\",\"Last name\":\"WEST\",\"actorId\":\"163\"}, {\"First name\":\"MENA\",\"Last name\":\"HOPPER\",\"actorId\":\"170\"}] | +--------+------------------------------------------------------ ----------+ 372 rows selected (0.432 seconds) The Actors field is an array of one or more Actor documents. In order to interact with this data as if it were a table, the flatten command can be used to turn the array into a nested table containing 3 fields: apache drill (mongo.sakila)> SELECT f.Rating, flatten(Actors) actor_list . .", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 209 + }, + { + "text": "Actors field is an array of one or more Actor documents. In order to interact with this data as if it were a table, the flatten command can be used to turn the array into a nested table containing 3 fields: apache drill (mongo.sakila)> SELECT f.Rating, flatten(Actors) actor_list . . . . . . . . . . . . . )> FROM films f . . . . . . . . . . . . . )> WHERE f.Rating IN ('G','PG'); +--------+------------------------------------------------------ ----------+ | Rating | actor_list | +--------+------------------------------------------------------ ----------+ | PG | {\"First name\":\"PENELOPE\",\"Last name\":\"GUINESS\",\"actorId\":\"1\"} | | PG | {\"First name\":\"FRANCES\",\"Last name\":\"DAY- LEWIS\",\"actorId\":\"48\"}| | PG | {\"First name\":\"ANNE\",\"Last name\":\"CRONYN\",\"actorId\":\"49\"} | | PG | {\"First name\":\"RAY\",\"Last name\":\"JOHANSSON\",\"actorId\":\"64\"} | | PG | {\"First name\":\"PENELOPE\",\"Last name\":\"CRONYN\",\"actorId\":\"104\"} | | PG | {\"First name\":\"HARRISON\",\"Last name\":\"BALE\",\"actorId\":\"115\"} | | PG | {\"First name\":\"JEFF\",\"Last name\":\"SILVERSTONE\",\"actorId\":\"180\"}| | PG | {\"First name\":\"ROCK\",\"Last name\":\"DUKAKIS\",\"actorId\":\"188\"} | | PG | {\"First name\":\"UMA\",\"Last name\":\"WOOD\",\"actorId\":\"13\"} | | PG | {\"First name\":\"HELEN\",\"Last name\":\"VOIGHT\",\"actorId\":\"17\"} | | PG | {\"First name\":\"CAMERON\",\"Last name\":\"STREEP\",\"actorId\":\"24\"} | | PG | {\"First name\":\"CARMEN\",\"Last name\":\"HUNT\",\"actorId\":\"52\"} | | PG | {\"First name\":\"JANE\",\"Last name\":\"JACKMAN\",\"actorId\":\"131\"} | | PG | {\"First name\":\"BELA\",\"Last name\":\"WALKEN\",\"actorId\":\"196\"} | ... | G | {\"First name\":\"ED\",\"Last name\":\"CHASE\",\"actorId\":\"3\"} | | G | {\"First name\":\"JULIA\",\"Last name\":\"MCQUEEN\",\"actorId\":\"27\"} | | G | {\"First name\":\"JAMES\",\"Last name\":\"PITT\",\"actorId\":\"84\"} | | G | {\"First name\":\"CHRISTOPHER\",\"Last name\":\"WEST\",\"actorId\":\"163\"}| | G | {\"First name\":\"MENA\",\"Last name\":\"HOPPER\",\"actorId\":\"170\"} | +--------+------------------------------------------------------ ----------+ 2,119 rows selected (0.718 seconds) | This query returns 2,119 rows, rather than the 372 rows returned by the previous query, which indicates that there are an average of 5.7 actors appearing in each G or PG film. This query can then be wrapped in a subquery and used to group the data by rating and actor, as in : apache drill (mongo.sakila)> SELECT g_pg_films.Rating, . . . . . . . . . . . . . )> g_pg_films.actor_list.`First name` first_name, . . . . . . . . . . . . . )> g_pg_films.actor_list.`Last name` last_name, . . . . . . . . . . . . . )> count(*) num_films . . . . . . . . . . . . . )> FROM . . . . . . . . . . . . . )> (SELECT f.Rating, flatten(Actors) actor_list . . . . . . . . . . . . . )> FROM films f . . . . . . . . . . . . . )> WHERE f.Rating IN ('G','PG') . . . . . . . . . . . . . )> ) g_pg_films . . . . . . . . . . . . . )> GROUP BY g_pg_films.Rating, . . . . . . . . . . . . . )> g_pg_films.actor_list.`First name`, . . . . . . . . . . . . . )> g_pg_films.actor_list.`Last name` . . . . . . . . . . . . . )> HAVING count(*) > 9; +--------+------------+-------------+-----------+ | Rating | first_name | last_name |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 210 + }, + { + "text": ". . . . . )> g_pg_films.actor_list.`First name`, . . . . . . . . . . . . . )> g_pg_films.actor_list.`Last name` . . . . . . . . . . . . . )> HAVING count(*) > 9; +--------+------------+-------------+-----------+ | Rating | first_name | last_name | num_films | +--------+------------+-------------+-----------+ | PG | JEFF | SILVERSTONE | 12 | | G | GRACE | MOSTEL | 10 | | PG | WALTER | TORN | 11 | | PG | SUSAN | DAVIS | 10 | | PG | CAMERON | ZELLWEGER | 15 | | PG | RIP | CRAWFORD | 11 | | PG | RICHARD | PENN | 10 | | G | SUSAN | DAVIS | 13 | | PG | VAL | BOLGER | 12 | | PG | KIRSTEN | AKROYD | 12 | | G | VIVIEN | BERGEN | 10 | | G | BEN | WILLIS | 14 | | G | HELEN | VOIGHT | 12 | | PG | VIVIEN | BASINGER | 10 | | PG | NICK | STALLONE | 12 | | G | DARYL | CRAWFORD | 12 | | PG | MORGAN | WILLIAMS | 10 | | PG | FAY | WINSLET | 10 | +--------+------------+-------------+-----------+ 18 rows selected (0.466 seconds) The inner query uses the flatten command to create one row for every actor who has appeared in a G or PG movie, and the outer query simply performs a grouping on this data set. Next, let’s try to write a query against the customers collection in Mongo. This should prove a bit more challenging, since each document contains an array of film rentals, each of which contains an array of payments. To make it a little more interesting, let’s also join to the films collection in order to see how Drill handles joins. The query should return all customers who have spent more then $80 to rent films rated either G or PG. Here’s what it looks like: apache drill (mongo.sakila)> SELECT first_name, last_name, . . . . . . . . . . . . . )> sum(cast(cust_payments.payment_data.Amount . . . . . . . . . . . . . )> as decimal(4,2))) tot_payments . . . . . . . . . . . . . )> FROM . . . . . . . . . . . . . )> (SELECT cust_data.first_name, . . . . . . . . . . . . . )> cust_data.last_name, . . . . . . . . . . . . . )> f.Rating, . . . . . . . . . . . . . )> flatten(cust_data.rental_data.Payments) . . . . . . . . . . . . . )> payment_data . . . . . . . . . . . . . )> FROM films f . . . . . . . . . . . . . )>", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 211 + }, + { + "text": ". . )> flatten(cust_data.rental_data.Payments) . . . . . . . . . . . . . )> payment_data . . . . . . . . . . . . . )> FROM films f . . . . . . . . . . . . . )> INNER JOIN . . . . . . . . . . . . . )> (SELECT c.`First Name` first_name, . . . . . . . . . . . . . )> c.`Last Name` last_name, . . . . . . . . . . . . . )> flatten(c.Rentals) rental_data . . . . . . . . . . . . . )> FROM customers c . . . . . . . . . . . . . )> ) cust_data . . . . . . . . . . . . . )> ON f._id = cust_data.rental_data.filmID . . . . . . . . . . . . . )> WHERE f.Rating IN ('G','PG') . . . . . . . . . . . . . )> ) cust_payments . . . . . . . . . . . . . )> GROUP BY first_name, last_name . . . . . . . . . . . . . )> HAVING . . . . . . . . . . . . . )> sum(cast(cust_payments.payment_data.Amount . . . . . . . . . . . . . )> as decimal(4,2))) > 80; +------------+-----------+--------------+ | first_name | last_name | tot_payments | +------------+-----------+--------------+ | ELEANOR | HUNT | 85.80 | | GORDON | ALLARD | 85.86 | | CLARA | SHAW | 86.83 | | JACQUELINE | LONG | 86.82 | | KARL | SEAL | 89.83 | | PRISCILLA | LOWE | 95.80 | | MONICA | HICKS | 85.82 | | LOUIS | LEONE | 95.82 | | JUNE | CARROLL | 88.83 | | ALICE | STEWART | 81.82 | +------------+-----------+--------------+ 10 rows selected (1.658 seconds) The innermost query, which I named cust_data, flattens the Rental array so that the cust_payments query can join to the films collection and also flatten the Payment array. The outermost query groups the data by customer name and applies a having clause to filter out customers who spent $80 or less on films rated G or PG. Drill with Multiple Data Sources So far, I have used Drill to join multiple tables stored in the same database, but what if the data is stored in different databases? For example, let’s say the customer/rental/payment data is stored in MongoDB, but the catalog of film/actor data is stored in MySQL. As long as Drill is configured to connect to both databases, you just need to describe where to find the data. Here’s the query from the previous section, but instead of joining to the films collection stored in MongoDB the join specifies the film table stored in MySQL: apache drill (mongo.sakila)> SELECT first_name,", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 212 + }, + { + "text": "Drill is configured to connect to both databases, you just need to describe where to find the data. Here’s the query from the previous section, but instead of joining to the films collection stored in MongoDB the join specifies the film table stored in MySQL: apache drill (mongo.sakila)> SELECT first_name, last_name, . . . . . . . . . . . . . )> sum(cast(cust_payments.paymen t_data.Amount . . . . . . . . . . . . . )> as decimal(4,2))) tot_p ayments . . . . . . . . . . . . . )> FROM . . . . . . . . . . . . . )> (SELECT cust_data.first_name, . . . . . . . . . . . . . )> cust_data.last_name, . . . . . . . . . . . . . )> f.Rating, . . . . . . . . . . . . . )> flatten(cust_data.rental_da ta.Payments) . . . . . . . . . . . . . )> payment_data . . . . . . . . . . . . . )> FROM mysql.sakila.film f . . . . . . . . . . . . . )> INNER JOIN . . . . . . . . . . . . . )> (SELECT c.`First Name` first _name, . . . . . . . . . . . . . )> c.`Last Name` last_name, . . . . . . . . . . . . . )> flatten(c.Rentals) rental _data . . . . . . . . . . . . . )> FROM mongo.sakila.customers c . . . . . . . . . . . . . )> ) cust_data . . . . . . . . . . . . . )> ON f.film_id = . . . . . . . . . . . . . )> cast(cust_data.rental_dat a.filmID as integer) . . . . . . . . . . . . . )> WHERE f.rating IN ('G','PG') . . . . . . . . . . . . . )> ) cust_payments . . . . . . . . . . . . . )> GROUP BY first_name, last_name . . . . . . . . . . . . . )> HAVING . . . . . . . . . . . . . )> sum(cast(cust_payments.paymen t_data.Amount . . . . . . . . . . . . . )> as decimal(4,2))) > 80; +------------+-----------+--------------+ | first_name | last_name | tot_payments | +------------+-----------+--------------+ | LOUIS | LEONE | 95.82 | | JACQUELINE | LONG | 86.82 | | CLARA | SHAW | 86.83 | | ELEANOR | HUNT | 85.80 | | JUNE | CARROLL | 88.83 | | PRISCILLA | LOWE | 95.80 | | ALICE | STEWART | 81.82 | | MONICA | HICKS | 85.82 | | GORDON |", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 213 + }, + { + "text": "| LONG | 86.82 | | CLARA | SHAW | 86.83 | | ELEANOR | HUNT | 85.80 | | JUNE | CARROLL | 88.83 | | PRISCILLA | LOWE | 95.80 | | ALICE | STEWART | 81.82 | | MONICA | HICKS | 85.82 | | GORDON | ALLARD | 85.86 | | KARL | SEAL | 89.83 | +------------+-----------+--------------+ 10 rows selected (1.874 seconds) Since I’m using multiple databases in the same query, I specified the full path to each table/collection to make it clear as to where the data is being sourced. This is where Drill really shines, since I can combine data from multiple sources in the same query without having to transform and load the data from one source to another. Someday, we may look back on relational databases with a feeling of nostalgia, similar to how we remember things like floppy discs, company pensions, and civility. SQL will likely live on, however, and it will be tools like Apache Drill that help keep SQL relevant for years to come. 1 For those readers who would benefit from a high-level overview of the new and emerging database trends, I recommend Guy Harrison’s “Next Generation Databases”, published in 2015 by Apress. About the Author Alan Beaulieu has been designing, building, and implementing custom database applications for more than 25 years. He is the author of Learning SQL, Second Edition, and coauthored Mastering Oracle SQL, Second Edition (both O’Reilly), and has written an online course on SQL for the University of California. He currently runs his own consulting company that specializes in database design and development in the fields of financial services and telecommunications. Alan has a BS in operations research from the Cornell University School of Engineering. He lives in Massachusetts with his wife and two daughters.", + "source": "learning-sql-generate-manipulate-and-retrieve-data.pdf", + "chunk_id": 214 + } +] \ No newline at end of file