IRS RESPONSE TO RECOMMENDATION: The approach used by the IRS is guided by the 2014 Economic Report of the President, Chapter 7: Evaluation as a Tool for Improving Federal Programs, which outlines methods for conducting rigorous impact evaluations:

“A strong impact evaluation needs a strategy for constructing more valid comparisons—specifically, for identifying ‘treatment’ and ‘control’ groups for which differences in outcomes can reasonably be attributed to the program or intervention rather than to some other factor. Impact evaluations conducted using rigorous, high-quality methods provide the greatest confidence that observed changes in outcomes targeted by the program are indeed attributable to the program or intervention.”

The report also outlines the types of approaches available for creating a valid comparison group. “Although the classic impact evaluation design entails random assignment of recipients into treatment and control groups as part of the experiment, the goal of constructing valid comparisons sometimes can be achieved by taking advantage of natural variation that produces as-if randomness, an approach referred to as a quasi-experiment. Quasi-experiments can be much less expensive than traditional large-scale random assignment experiments.”  The IRS employs both randomized control trials and analysis of quasi-experimental settings for behavioral insights testing with the goal of enhancing both taxpayer service and enforcement. Current and recent efforts include reporting, filing, and payment compliance nudges, outreach promoting use of IRS (e.g., electronic payment) and partner (e.g., VITA) services, quasi-experimental analysis of factors promoting voluntary compliance (1099-K, 1099-B basis reporting, FATCA and OVDP, as well as influences of prior enforcement efforts), nudges to encourage take-up of tax benefits (e.g. EITC and AOTC), and a variety of behavioral nudges to promote issue resolution and future compliance (examples in Collection include employment tax early intervention pilots, two notice redesign pilots, a pre-emptive notice pilot, and a lien pilot).

The IRS has dedicated resources towards behavioral interventions, including the creation of the Behavioral Insights team to promote the dissemination and application of behavioral insights across the IRS. The Behavioral Insights Team and associated Community of Practice have developed resources and procedures to facilitate the sharing of knowledge related to behavioral insights, providing a foundation for future application and extension of this work. The resources include best practices and examples of successful applications both inside and outside of the IRS, summarized in a Behavioral Insights Toolkit. Collaboration with behavioral researchers in academia and in other parts of the government help the IRS continue to bring the best available behavioral research to promote effective tax administration.

CORRECTIVE ACTION: N/A

TAS RESPONSE: The IRS has started to use RCTs and quasi-experimental settings to test BIs. It has also established a BIs team and developed a BIs Toolkit. These steps should help the IRS move toward more routine use of BIs to improve tax administration. However, the IRS response does not suggest that it has revised procedural guidance or provided instructions to staff (e.g., instructions governing campaigns) requiring those charged with addressing compliance problems to consider alternative treatments that incorporate BIs or to measure the effect of any treatments using RCTs or quasi-experiments. The IRS should issue such guidance, as recommended.

ADOPTED, PARTIALLY ADOPTED or NOT ADOPTED: Partially Adopted

OPEN or CLOSED: Closed

DUE DATE FOR ACTION (if left open): N/A

IRS RESPONSE TO RECOMMENDATION: ​​The IRS is using several methods for disseminating results of RCTs and quasi-experimental studies internally and externally. Internally, the Research Planning and Prioritization Executive Steering Committee, the Research Directors Coordination Council, The Behavioral Research Community of Practice all provide forums where results from RCTs and quasi-experimental studies are often reported. The Behavioral Insights Team is also working with HCO on developing a Behavioral Insights knowledge base for consolidating and disseminating baseline and new insights on from RCTs, quasi-experimental studies, and related behavioral research. Externally, the IRS hosts an annual research conference, participates in many other tax and research conferences, and promotes transparency of these evidence-based findings through academic partnerships such as those managed by the Joint Statistical Research Program. The associated presentations and papers are made public either through Tax Stats pages on irs.gov or through the websites and journals of the organizations where the results are presented, promoting transparency and external review.

CORRECTIVE ACTION: N/A

TAS RESPONSE: The IRS has made significant strides in cataloging baseline BI insights as well as the BI research that it is undertaking. The National Taxpayer Advocate applauds the IRS for working with the Human Capital Office (HCO) on a knowledge base for consolidating and disseminating baseline and new BIs. However, the IRS should improve its BI repository or find another way to preserve its research results so that they are readily available to IRS employees in different functions, even if the results did not appear to reveal new insights. Information about what does not work is nearly as important as information about what does.

Similarly, the IRS has continued past practices, which disclose some of its research to external stakeholders, such as by allowing IRS researchers to draft and publish journal articles. Without doing a literature review and, in some cases, paying for access, however, it is difficult to learn what various IRS researchers have submitted for publication or stated at conferences. Moreover, some IRS researchers may not send their work for publication and if they do, it may not be accepted. While the transparency of the IRS research conference is a step in the right direction, the IRS should consider requiring that abstracts, presentations, and papers and other deliverables be available for free (redacted, if necessary) on an IRS website, if they were written by IRS employees or funded by the IRS and delivered to a client or target audience. Such a policy would help both internal and external stakeholders find the information they need to evaluate and potentially extend the IRS’s prior work, without checking or requesting that someone else check an internal repository.

ADOPTED, PARTIALLY ADOPTED or NOT ADOPTED: Partially Adopted

OPEN or CLOSED: Closed

DUE DATE FOR ACTION (if left open): N/A

IRS RESPONSE TO RECOMMENDATION: ​Ongoing research and analysis seeks to better disentangle the various factors and agency actions contributing to compliance.  A 2014 OECD Forum on Tax Administration report on measuring tax compliance outcomes provides a useful overview of the associated measurement and attribution issues:

“For outcome measures to be fit for purpose, they must be based on reasonable evidence to ensure the measurement is reliable. A related but separate issue is attribution reliability. An outcome measure can be fit for purpose without being attributed. Direct attribution cannot be expected of a measure if the cause and effect in reality is not direct. This is particularly so at the strategic level, where outcome measure may be used as indicators of the health of the overall tax administration system. For this purpose an outcome measure does not need to be attributed to the specific actions of the revenue body. For example, overall filing on time can be measured reliably but is not directly attributable to the revenue body’s actions. In contrast, at operational level a fit for purpose effectiveness measure needs to have reliable attribution to enable revenue bodies to identify which interventions work and which are not working as intended.”

As discussed in the narrative response, the IRS commonly reports compliance and revenue outcomes as part of results presented in the academic research community and the Taxpayer Assistance Blueprint. The IRS also provides annual statistics on alternative treatments such as AUR and ASFR in the IRS Data Book. The IRS conducts ongoing review of the most appropriate content for the Data Book and will evaluate options to address this issue more comprehensively going forward. It is worth noting that current enforcement statistics include the results of both soft notice campaigns as well as more traditional enforcement methods. Development of new statistics involving estimation methods are subject to OMB guidelines.

CORRECTIVE ACTION: N/A

TAS RESPONSE: The IRS response references the Taxpayer Assistance Blueprint (TAB) and data on the AUR and ASFR programs in the IRS Data Book as examples of transparency concerning the performance of its programs. The information contained in these publications is incomplete. For example, the TAB references various Taxpayer Usage Surveys and Taxpayer Experience Surveys, but a full analysis of the survey results does not appear to be available to the public. By contrast, when TAS conducts a survey it publishes both the results and the survey instrument. For the ASFR and AUR programs, the IRS Data Book only reports very basic data such as the number of closures and assessments. As noted above, these statistics provide stakeholders with no ability to evaluate the success or failure of these programs in achieving the desired outcome. For example, the IRS does not report the number of erroneous assessments, or amount of abatements, the ultimate outcomes its ASFR or AUR closures, or the taxpayer’s future compliance.

In addition, the IRS response seems to suggest that it cannot develop a measure of voluntary compliance revenue resulting solely from the IRS’s efforts, rather than from other causes. However, that is exactly what the IRS’s BI team is doing when it uses RCTs or quasi-experiments to isolate the effect of the IRS’s BI treatments from other causes. It is unclear why the IRS could not extend this methodology, even if it starts by using it just to estimate the service revenue from campaigns, improvements resulting from the application of BIs, or similar initiatives.

The IRS is apparently concerned that it could not develop measures of service revenue that meet the OMB guideline, which requires that “influential scientific or statistical information” must be “capable of being substantially reproduced.” This is not an unrealistically high standard, however. It only requires that “independent reanalysis of the original or supporting data using the same methods would generate similar analytical results, subject to an acceptable degree of imprecision.” The IRS response does not explain why a reasonable methodology that addresses all of the concerns expressed by internal and external stakeholders could not meet this standard.

ADOPTED, PARTIALLY ADOPTED or NOT ADOPTED: Partially Adopted

OPEN or CLOSED: Closed

DUE DATE FOR ACTION (if left open): N/A

IRS RESPONSE TO RECOMMENDATION: ​Discussed above in 1-3. In addition, as recognized in footnote 70 of the TAS Report, there is a GAO requirement for current enforcement ROI using the current enforcement revenue reporting methods. We concur that the current enforcement revenue reporting includes no indirect effect estimates on subsequent voluntary reporting (positive or negative).

CORRECTIVE ACTION: N/A

TAS RESPONSE: The IRS response does not address the National Taxpayer Advocate’s recommendation to correct or discontinue the IRS’s misleading definition of “enforcement” revenue. The response also misconstrues GAO’s recommendation that it “review disparities in the ratios of direct [enforcement] revenue yield to costs,” as a “requirement” to retain a misleading definition of “enforcement” revenue If GAO could require the IRS to take action, it would not need to make recommendations.

Moreover, GAO has not recommended that the IRS retain its misleading definition of “enforcement” revenue. Indeed, it suggested the definition of “enforcement” yield should change when it also recommended the IRS “explore the potential of estimating the marginal influence of enforcement activity on voluntary compliance.” The IRS’s response to GAO also said there were problems with its use of direct “enforcement” revenue (presumably, as currently defined) to allocate resources when it stated: “The IRS is committed to the optimal allocation of our resources that is why… we account for factors other than just direct return on investment when allocating resources across programs or categories of work…” Furthermore, segregating and reporting service revenues is consistent with both GAO’s recommendations and the IRS’s response to GAO.

ADOPTED, PARTIALLY ADOPTED or NOT ADOPTED: Not Adopted

OPEN or CLOSED: Closed

DUE DATE FOR ACTION (if left open): N/A

IRS RESPONSE TO RECOMMENDATION: ​​​As stated above, the IRS is evaluating performance measures in support of its vision for the future.  Frameworks, such as that produced by the Organization for Economic Cooperation and Development, as well as practices used by tax authorities outside the U.S. provide useful models.   The IRS’s official statistics are carefully compiled in compliance with policies and directives issued by the Statistical Policy Branch of the Office of Budget and Management’s Office of Information and Regulatory Affairs. Guidelines governing information disseminated by Federal agencies are intended to maximizing quality, objectivity, utility and integrity, emphasizing reproducibility and peer review of methods used to produce statistics.   These principles must guide any new IRS performance measures.

CORRECTIVE ACTION: N/A

TAS RESPONSE: The IRS response states that it is evaluating performance measures in support of its vision for the future, as recommended. Because the IRS response cites an OECD report, which supports the National Taxpayer Advocate’s recommendation (as discussed above), the IRS seems to be saying that it agrees with the National Taxpayer Advocate’s concerns about the IRS’s current output-oriented metrics. Thus, its hesitancy to adopt the National Taxpayer Advocate’s recommendation in full appears to stem from concerns about the “reproducibility and peer review of methods used to produce statistics.”

In other words, the IRS seems concerned that it may not be able to measure or estimate the effect of its activities on voluntary compliance in a way that is objective and reproducible. While this is a valid concern, if the IRS explicitly acknowledges that it needs to focus more on the total effect of its activities on voluntary compliance and less on the outputs of various enforcement functions, as recommended, it can work toward that goal, which is not as difficult as the IRS response suggests.

As discussed above, some measures of voluntary compliance are relatively easy to quantify or estimate, such as future filing and payment compliance. Even some types of reporting noncompliance are easy to detect (e.g., math errors and mismatches). While it may be difficult to say with absolute certainty what caused a taxpayer’s compliance or noncompliance following some interaction with the IRS, as the IRS acknowledges, the current direct “enforcement” revenue statistics are not computed with certainty either. Moreover, the IRS has begun to use RCT and field experiments, which can provide reasonable estimates of the effect of its activities on future compliance. If properly designed, these estimates can be generalized. The IRS should have the confidence to report these results on a regular basis and move toward more holistic metrics, as recommended.

ADOPTED, PARTIALLY ADOPTED or NOT ADOPTED: Not Adopted

OPEN or CLOSED: Closed

DUE DATE FOR ACTION (if left open): N/A