Appendix for Unwarranted Warrants? An Empirical Analysis of Judicial Review in Search and Seizure

* * *

APPENDIX I: EXAMPLES OF WARRANT AFFIDAVITS

Figure I.A: Standard DUI Warrant Example

Figure I.B: “Form” Affidavit for General Warrant

Figure I.C: “Form” Affidavit for DUI Blood Draw

Figure I.D: Example Affidavit and Warrant Lacking Specificity

Figure I.E: Minute Warrant Example

APPENDIX II: ADDITIONAL FIGURES AND TABLES

Table II.A(1): Warrant Approval Time (Minutes) by Mixed Crime Category

Crime Category		Obs.	Min.	First Decile	First Quart.		Med.	Adj. Mean	Third Quart.	Max.
Violent		1,595	0:08	1:08	2:00	3:35		8:51	6:56	721:31
Property		1,055	0:10	1:16	2:03	3:35		8:42	6:12	1,058:03
Drug		1,245	0:08	1:05	1:55	3:12		6:53	5:31	662:40
Vice/Morals		1,589	0:10	1:11	2:08	3:51		10:07	7:34	1,150:30
Motor Vehicle		6,375	0:01	0:47	1:14	1:59		2:49	3:11	133:20
Other Crimes*		4,613	0:07	1:07	1:58	3:26		7:25	6:16	1,109:13
All Approvals		29,810	0:01	0:57	1:37	2:50		6:47	5:13	2,581:37
Notes: Minutes are reported as Minutes:Seconds. “Adj. Mean” reflects the average warrant decision time in minutes once outliers were removed. “Mixed Crime” categories are determined based on whether a given warrant is coded as having that category (that is, double counting occurs). For example, if a warrant is coded under both “Property” and “Drug” crimes, the same warrant would appear in both rows of the table. * “Other Crimes” are those warrants that were coded to crime types other than the five main categories included above.

Table II.A(2): Warrant Approval Length (Words) by Mixed Crime Category

Crime Category	Obs.	First Quart.	Med.	Third Quart.	Adj. Mean*
Violent		1,595	935	1,255	1,768	1,719.07
Property		1,055	886	1,198	1,633	1,373.62
Drug		1,245	862	1,161	1,567	1,295.63
Vice/Morals		1,589	968	1,392	2,299	2,340.72
Motor Vehicle		6,376	566	663	805	712.23
Other Crimes		4,613	935	1,335	1,870	1,521.54
All Approvals		25,601	564	979	1,489	1,264.74
Notes: Minutes are reported as Minutes:Seconds. “Adj. Mean” reflects the average warrant decision time in minutes once outliers were removed.

Figure II.A: District-Level,Overall Review Times

Table II.B: Regressions on Review Time by Submission Sequence

Table II.B presents the results of our preferred regression model using subsets of data that include only first submissions (column 1) and resubmissions (column 2). We also include the full-data model from previous tables (column 3, a reprint of column 6b in Table 9 in the main body of this Article) for comparison.

As expected, the model using only first submissions produces nearly identical results to the full-data regression. However, although the regression that only analyzes resubmissions includes fewer than 2,000 observations, we see a statistically significant and substantively sizable relationship between race/ethnicity and review time. On average, white judges spend nearly seven minutes more on review than their nonwhite counterparts. However, the fact that this relationship only exists when analyzing the resubmissions dataset complicates the implications that can be drawn. It is not clear why such a massive correlation would only manifest on rereview, and this data is likely influenced by selection bias and small-sample variability, as the only submissions that are included are those that have previously been denied. As a result, we do not believe those results justify any solid conclusions or implications.

Table II.C: Regressions on Outcome (Approval or Not) by Submission Sequence

In Table II.C we present the results of our preferred model on the likelihood of approval using datasets broken down by first submissions (column 1), resubmissions (column 2), and the full dataset (column 3, which mirrors column 4 in Table 10). Given that the vast majority of submissions were approved in the first round, it is not surprising that our results for only first submissions closely match those that come from an analysis on all submissions. Having a JD and previously working as a prosecutor are still correlated with a lower likelihood of approval, and the gender and race/ethnicity relationships match those in the full dataset.

We also present the results of our preferred model using only resubmissions, although we do so with the caution that the resulting sample size (just over 1,500 observations) is so small that standard errors are likely to be large enough to mask any significant relationships. Indeed, the indicator for whether the reviewing judge has a law degree was omitted from this model due to multicollinearity. Except for a small but statistically significant correlation between tenure and approval rate — something that was not present in the tests using the full dataset — we do not see any notable relationships.

APPENDIX III: SUPPLEMENTAL EXPLANATIONS OF DATA AND METHODS

A.  Detailed Description of Dataset Sourcing and Cleaning

1.  PDF Images and Machine-Algorithm Text Analysis. — Using the Utah Administrative Office of the Courts (AOC) database, Xchange,¹ we obtained PDF images of every unsealed digital affidavit, warrant, and return submitted and approved (denied and retracted warrants are not available, as explained in section II.A of this Article) through the e-Warrants system from December 1, 2013, to January 29, 2020.² Xchange is ostensibly designed for easy access by parties and attorneys in criminal (and civil) cases, but it is also provided to academic researchers on an as-needed basis. Because the Xchange system explicitly prohibits electronic scraping tools (that is, code or programs that automatically download files en masse), we manually downloaded each of the relevant warrant PDFs one by one.

At an individual level, procuring substantive information from the PDF files is fairly simple. Some of the data — submitting officer name; law enforcement agency; court location; and the dates of the affidavit, warrant, and return — are already available as bulk metadata via the AOC, and the data contain all the information necessary to match each entry with the corresponding PDF image. Other key datapoints such as reviewing judge name and document length can be determined simply by reviewing the last page of the affidavit. Even the more substantively complex aspects that we hoped to identify such as the legal “type” of the warrant and the length of individual sections within the warrants can be determined in short order by reading over the affidavit language.

However, our dataset included nearly 33,500 such affidavits,³ so an individual, manual coding of them is impracticably labor- and cost-intensive. Therefore, we turned to natural language processing (NLP) methods, a computational approach, to help with the analysis of the warrants. The warrants were converted from their original image format as PDF documents to raw text using a combination of two methods. First, the files were converted to text using the PDFminer library in Python 3.⁴ However, the textual information from the judge’s seal could not be accurately converted using this method. Thus, a second method, PyTesseract, a Python wrapper for Google’s proprietary OCR Engine,⁵ was also employed to both check the accuracy of the first conversion method and to convert the text from the seal. The PDF images contained the full text of the affidavit for the search warrant, the warrant itself, and the return for the search warrant, as well as some metadata for the corresponding documents.

Relevant data was then extracted from the documents using regular expressions (or regex), an algorithmic method for advanced pattern matching in textual data. Using regex, the file was divided into its three documents (the affidavit, the warrant, and the return), and metadata about the warrant number, jurisdiction, dates, references to criminal statutes, officers submitting or returning the warrant (including their names, agencies, and ranks), and judge were generated for each file. These regular expressions were developed over several iterations and evaluated for their precision and recall to ensure that they were maximally accurate. In our final test set of 50 warrants, the accuracy of these regular expressions yielded 100% accuracy, although in the larger dataset of over 33,000 warrants, small numbers of errors were observed.

From the affidavits, two additional blocks of text were extracted using a large language model (BERT⁶): (1) an explanation of the facts to provide grounds for the search warrant, and (2) a description of the submitting officer’s experience that qualifies them to make the relevant judgments about the grounds in the search warrant. Word counts were then generated using Python for each affidavit, warrant, and return, as well as the descriptions of facts and officer experience. While the text identification process for the facts and experience are much more complex and, consequently, less accurate, they provide us with an accurate enough measure to use in some of our alternative analyses presented in the main body of this Article.

2.  Electronic Warrants Administrative Data. — The second of our key datasets comes from the largely untapped data from the Utah Department of Public Safety (DPS).⁷ Critical to our study, and the element that makes our dataset so novel, is the metadata recorded by the electronic warrant system. These include timestamps at the second level for when (1) the officer submitted the affidavit, (2) the magistrate viewed the affidavit, (3) the magistrate made a decision on the warrant application (retraction, approval, or denial), and (4) the officer served the warrant. The data contains a warrant identification number, which allows us to easily connect it with the PDF image data, but does not have geographic, judge, or other information. The DPS retains data start-ing from April 6, 2016, although only starting on March 28, 2017, does the data include a timestamp for when the judge viewed the affidavit.

We obtained this dataset through a Government Records and Access Management Act⁸ (GRAMA, Utah’s equivalent to FOIA) Request, which was initially denied but granted on administrative appeal. The DPS data for this study ends on January 29, 2020 (the date our data request was fulfilled).

3.  Imputed Reviewing-Judge Data. — As we alluded to earlier, our dataset only includes PDF images for approved affidavits. In other words, the small percentage of affidavits that are submitted for review and subsequently denied by the reviewing judge or withdrawn by the submitting officer will have administrative metadata (like timestamps) but no PDF images that allow us to create data derived from the text of the affidavit itself. While, in section I.D of this Article, we discuss the ways in which this prevents us from conducting certain analyses, we are able to impute the reviewing judge for a denied or retracted affidavit by reconstructing the county-level shifts for each day covered in our broader dataset. This allows us to conduct some judge-level analyses on those denied and retracted affidavits, such as exploring the likelihood of approval in Part IV of this Article.

We created this shift data using a combination of schedules provided to us by county administration and the reviewing-judge patterns that existed in the PDF image data we were able to scrape. Starting with the PDF-derived data, research assistants ordered the dataset by county and then by submission date and tracked the approving-judge name by hand, noting any clear changes in the judge name and when those changes occurred. Once the entirety of a given county’s data was coded, we attempted to identify patterns in the variations that translated to identifiable week- or month-level “shifts.”⁹ Overlaying those patterns back onto the existing data, we then calculated the judge who approved the majority of the warrant affidavits during a given shift and assigned that judge as the “on-call” judge.¹⁰ This process provided us with clear shift data in all but two of Utah’s judicial districts (the Second and Fifth Judicial Districts).¹¹

With the shift data, we then needed to determine what the parameters of our shift imputations would be. On the one hand, the most conservative assumption would be to assign all warrants reviewed during a given shift to the judge assigned to that shift. On the other hand, we have confirmed through interviews with law enforcement officers and judges (and some instances in the data) that the actual process is more complicated and that affidavits are not always sent to the judge who is on call during a given shift.

For example, in one of the warrants in our dataset, an officer submitted a warrant affidavit during shift one, was denied by the judge (anonymous in our data), resubmitted during the next shift, was denied again, and resubmitted during a third shift, where the warrant was denied for the last time. We might be justified in assuming that the reviewing judge for each submission was the on-call judge at the time of each submission, but it is also possible that the officer resubmitted the warrant both times to the judge who had originally reviewed the warrant given her familiarity with the contents.

While we have some intuitions regarding the most likely path a warrant took in a given scenario (informed by some anecdotes shared by officers we interviewed), the various permutations in the data are sufficiently complex that any attempt at systematically imputing each of them made the analysis unwieldy. As a result, our dataset adopts the more conservative approach — where the on-call judge is assumed to be the judge who reviewed an affidavit submitted then denied or retracted during that shift. Any approved warrants during that period retained the reviewing judge that we scraped from the PDF images for the purposes of our analysis.

4.  Hand-Coded Data. — While most of the data we use in this study is available in bulk metadata form or is electronically scraped from the PDF images, some of the data required manual efforts.

Although our analysis is completely anonymous on the judge level, we do draw some correlations between the amount of time reviewing and likelihood of approval for a warrant affidavit and the characteristics of the reviewing judge. Some of this data comes from Utah’s Judicial Performance Evaluation Commission (JPEC), a state-sponsored committee that conducts regular performance evaluations of each judge or magistrate in Utah, primarily as a mechanism for providing information to Utah’s voters during the judges’ retention elections.¹² Its dataset not only provides detailed results from these evaluations (some of which we use in our study) but also compiles clean, up-to-date registries of the judiciary, including judges’ names, tenures, and courts.¹³

We add to this base dataset with a rich set of hand-coded biographical variables on each judge, including their gender, race/ethnicity, legal education, professional experience (whether they were formerly a prosecutor or a criminal defense attorney), and the time and manner of appointment. This information was gathered by research assistants using a mix of the judge biographies included on court websites and online search engines (like Google).

In addition to the previously described administrative data, our qualitative analysis also relies on two in-depth hand-coded sources of data. First, we make use of a hand-coded random sample of 231 affidavits, warrants, and returns from the Utah AOC database. Second, we similarly review 40 warrants that were approved in a minute or less. We discuss our use of these samples in detail in Part V of this Article.

5.  Dataset Merging and Additional Cleaning. — Due primarily to the fact that the recording of first viewing timestamps in the DPS data only began in March 2017, the primary dataset used in this study limits all the datasets (AOC PDF data, hand-coded judge data, shift data) from March 2017 to near the end of January 2020.¹⁴ Additionally, as we describe in section III.A of this Article, we exclude outlier review times for portions of our analysis.

B.  Jaccard Similarity

While computational approaches for measuring similarity have been widely used in other fields, it has only recently been applied to analyzing legal texts.¹⁵ An extended outline of the technicalities of Jaccard similarity is beyond the scope of this Article,¹⁶ but we feel it warrants a brief explanation. Like most measures for linguistic similarity, Jaccard similarity (also called the Jaccard index) involves the direct comparison of two distinct datasets and computes a ratio of dataset observations that are the same relative to the overall number of observations in those data.¹⁷ In the case of this study, we compute word-level Jaccard coefficients for each of the approved affidavits in our dataset against every other affidavit. While including all affidavits (denials and retractions) in this analysis would be ideal, we are once again limited by the fact that only the PDF images of the approved affidavits are retained in the warrant database.

Using a simplistic but germane example of search warrants, imagine that the entirety of a warrant affidavit, Affidavit A, stated only that “the suspect was seen pocketing a bag with unidentified contents.” A second affidavit, Affidavit B, stating that “the suspect was seen pocketing a bag with white contents,” would have a Jaccard coefficient of 0.9 because there are eighteen similar observations between the two affidavits (nine words in each affidavit) and twenty observations in total. Similarly, comparing Affidavit A to a third affidavit, Affidavit C, that states that “the suspect was seen pocketing a bag with white powder” would produce a Jaccard coefficient of 0.8 because only sixteen of the twenty observations are similar. Conversely, the Jaccard coefficient for the comparison of Affidavits B and C would be 0.9. Computing the Jaccard similarity scores for full-length affidavits is more computationally complex than these single-sentence examples, but the resulting coefficient is still straightforward in interpretation.

Using Jaccard similarity scores naturally comes with some limitations, although we feel that they are mitigated somewhat by the general goal behind this alternative analysis. First, the Jaccard scores do not tell us anything about why there is a difference between the two affidavits, only the size of the difference. A Jaccard coefficient for two affidavits might be 0.5 because 50% of one affidavit is completely different than another affidavit, or it might be that the second affidavit contains all of the text of the first but also includes substantially more language that was not included in the first affidavit at all. Second, a Jaccard coefficient is content neutral, which is to say that the word-to-word similarity does not account for the relative substantive importance of some words over others (like facts vs. nonfacts). Using the example from above, the comparison between Affidavits A and B would yield the same coefficient as a comparison between Affidavit A and a fourth affidavit that states: “A suspect was seen pocketing a bag with unidentified contents.” Even though the difference between the first and second affidavit (that is, a bag with unidentified contents vs. one with white contents) is clearly more substantively important than the mere change in articles between the first and fourth (“the suspect” vs. “a suspect”), the Jaccard similarity score would be the same. However, because we are only broadly concerned with overall similarity for the purposes of excluding boilerplate-type observations from our dataset, neither of these limitations prove to be overly restrictive.