unitizeR - The Interactive Environment

unitize vs review

unitizer offers three functions to access the interactive review environment: unitize, unitize_dir, and review. unitize is used when you either want to generate a unitizer from a test file, or when you want to compare the re-evaluation of a test file to an existing unitizer. untize_dir does what unitize does, except for a whole directory at a time.

review is a helper function used when you want to review the contents of an existing unitizer. This is useful if you grow uncertain about tests that you previously approved and want to ensure they actually do what you want them to. You can review and potentially remove items from a unitizer with review.

Both these functions use the same interactive environment, though rules therein are slightly different. For example, in review, all the tests are considered passed since there is nothing to compare them to, and the interactive environment will step you through all the passed tests. unitize will normally omit passed tests from the review process.

We will focus on unitize for the rest of this vignette since most of the commentary about it applies equally to unitize_dir and review.

Example Set-up

To examine the interactive environment more thoroughly we will go back to the demo (you can run it with demo(unitizer)). This is the unitizer prompt right after our first failed test when our unitizer.fastlm implementation was returning the wrong values:

> get_slope(res)
unitizer test fails on value mismatch:
*value* mismatch: Mean relative difference: 6943055624
@@ .ref @@
-    [1] 101
@@ .new @@
+    [1] 701248618125

Selecting A Test to Review

unitize will present to you all the tests that require review, but if you wish to review a specific test you can use the P (for Previous) and B (for Browse) commands. These commands can come in handy if you realize that you incorrectly accepted or rejected an earlier test, but do not wish to quit unitizer completely and lose all the other properly reviewed tests. P just steps you back to the previous test. B gives you the option to go back to any previously reviewed test.

P is trivially straightforward, so we will not discuss it further. We will type B at the prompt of our second failed test to examine what it does:

unitizer> B
    *1. library(unitizer.fastlm)   .  .  .  .  .  .  .  .  .  .  .  .        -:-
    *2. dat <- data.frame(x = 1:100, y = (1:100)^2)  .  .  .  .  .  .        -:-
    *3. res <- fastlm(dat$x, dat$y)   .  .  .  .  .  .  .  .  .  .  .        -:-
     4. res   .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .   Failed:N
     5. get_slope(res) .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .   Failed:-
     6. get_rsq(res)   .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .   Passed:-
     7. fastlm(1:100, 1:10)  .  .  .  .  .  .  .  .  .  .  .  .  .  .   Passed:-

What test do you wish to review (input a test number, [U]nreviewed)?
unitizer>

The [B]rowse option produces a list of all the tests in the order in which they appear in the test file. You can type the number of a test to review it, or U to go to the first test that hasn’t been reviewed (more on this in a minute). We will examine the line for test #5 in more detail:

     5. get_slope(res) .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .   Failed:-
     ^   ^                                                                ^    ^
     |   |                                                                |    |
     |   +--  Deparsed test expression                    Test status ----+    |
     |                                                                         |
     +- Test ID                                                    User Input -+

The value and order of the test IDs shouldn’t mean anything to you other than being the number to type in if you wish to review that test. Tests that have a * to the left of the test id are expessions that are not reviewed or checked by unitizer (we call these ignored tests).

The test status (see tests outcomes) indicates the outcome of comparison of the reference test in the unitizer store to the newly evaluated ones. The first four tests are ignored tests, so they do not have a status. The User Input column marks which tests have been reviewed and what the user decision was. In this case we had reviewed test #2 and decided not to keep it (hence the “N”). Typically neither ignored tests nor passed tests require user input so they will typically have a “-” as the user input, as will tests that would be reviewed, but haven’t been yet.

Typing U at the review prompt will take you to the first unreviewed test. Since ignored tests and passed tests are not typically reviewed, U will take you to the first unreviewed test that is neither passed nor ignored.

If we type 4 at the prompt, we get:

You are re-reviewing a test; previous selection was: "N"

# Our fast computations do not produce the same results as our
# original tests so they fail.  If you need more detail than the
# provided diff you may use `.new`/`.NEW` or `.ref`/`.REF`.
#
# You should reject these tests by typing 'N' at the prompt since
# they are incorrect.

> res
unitizer test fails on value mismatch:
*value* mismatch: mean relative difference: 19854602162
@@ .ref @@
-     intercept        slope          rsq
-  -1717.000000   101.000000     0.938679
   attr(,"class")
   [1] "fastlm"
@@ .new @@
+      intercept         slope           rsq
+  -3.541306e+13  7.012486e+11  9.386790e-01
   attr(,"class")
   [1] "fastlm"
unitizer>

unitizer tells us we are re-reviewing this test and that previously we had chosen not to keep the new version. At this point we could re-examine the test, and potentially change our previous selection. unitizer also re-displays any comments that were in the source file either ahead of the test or on the same line as the test. We used this feature to document the demo.

You can jump ahead to any test from the review menu, even tests that are typically not reviewed (i.e. ignored or passed, though if you go to those you will be brought back to the review menu once you complete the review because those tests are not part of the normal review flow).

If you skip ahead some tests and then get to the end of the review cycle unitizer will warn you about unreviewed tests.

Finalizing unitizer

Let’s accept the 5th test, which brings us to this prompt:

unitizer> Y

= Finalize Unitizer ============================================================

You will IRREVERSIBLY modify 'tests/unitizer/fastlm1.unitizer' by:
- Replacing 1 out of 2 failed tests

Update unitizer ([Y]es, [N]o, [P]rev, [B]rowse, [R]erun)?
unitizer>

In this case we were reviewing a unitizer with two failed tests, one of which we chose to update with the newer value. unitizer will summarize for you all the changes that it is about to make to the unitizer store. If you type Y at the prompt, the existing unitizer store will be overwritten with the new version you just reviewed.

If you are unsure about the changes you just approved for the unitizer, you can re-review them with R or B. You can also quit without saving your changes by typing N, but once you do so you will no longer be able to recover your changes.

Quitting unitizer

At any point you may quit unitizer by typing Q at the unitizer prompt. If you have already reviewed tests, you will be given the opportunity to save what you have done so far as you would when finalizing the unitizer. Note that if you chose to quit unitizer may exit without giving you the opportunity to review the tests. This will happen if:

• You did not make any changes to the unitizer (e.g. if you chose N at failed tests, you are keeping the reference value, so the unitizer is not actually changing).
• And test evaluation took less than getOption("unitizer.prompt.b4.quit.time") seconds (currently 10 seconds).

If you end up in the R debugger from a unitizer (e.g. via debug or recover), quitting the debugger with “Q” will force-exit you from the session without a chance to save any changes.

Differences in review Mode

review works exactly like unitize, except that passed tests are automatically queued up for review, and that the only test statuses you should see are “Passed” or “-”, the latter for ignored tests.

As Compared To The Standard R Prompt

The unitizer prompt is designed to emulate the standard R prompt. For the most part you can type any expression that you would type at the R prompt and get the same result as you would there. This means you can examine the objects created by your test script, run R computations, etc.

There are, however, some subtle differences created by the structure of the evaluation environments unitizer uses:

• Even though you can see objects produced by tests, you can not actually remove them with rm.
• Any objects you create at the interactive prompt are only available for the test you are currently reviewing, so do not expect them to still be there at subsequent prompts.
• All expressions are evaluated with options(warn=1) or greater.
• Other subtle issues discussed at length in the Reproducible Tests Vignette.
• There are special unitizer objects .new, .NEW, .ref, and .REF that let you review the results of tests (we will discuss these next).
• ls and traceback/.traceback are masked by special unitizer versions (we will also discuss this next); you can use base::ls/base::traceback if you need the originals.
• You will have access to any objects created through the pre argument to unitize, though they will not show up in a call to ls.

.new and .ref

As we saw in the demo there are special objects available at the prompt: .new (except for removed/deleted tests), and for all but new tests, .ref. These objects contain the values produced by the newly evaluated test (.new) and by the test when it was previously run and accepted (.ref). .new might seem a bit superfluous since the user can always re-evaluate the test expression at the unitizer prompt to review the value, but if that evaluation is slow you can save a little time by using .new. .ref is the only option you have to see what the test used to produce back when it was first accepted into the unitizer store.

.new and .ref contain the values produced by the tests, but sometimes it is useful to access other aspects of the test evaluation. To do so you can use .NEW and .REF:

• .NEW prints general information about the test.
• .NEW$value returns the test value; equivalent to typing .new at the prompt.
• .NEW$conditions returns the list of conditions produced by the test.
• .NEW$messsage returns the stderr captured during test evaluation.
• .NEW$output returns the screen output captured during test evaluation (note often this will be similar to what you get from .new or .NEW$value since typing those expressions at the prompt leads to the value being printed).
• .NEW$call returns the test expression.
• .NEW$aborted returns whether the test expression invoked an “abort” restart (e.g. called stop at some point).

You can substitute .REF for .NEW in any of the above, provided that .REF is defined (i.e. that will not work when you are reviewing new tests since there is no corresponding reference test for those by definition).

If both .NEW and .REF are defined, then .DIFF will be defined too. .DIFF has the same structure as .NEW and it contains the result of evaluating diffobj::diffObj between each component of .NEW and .REF. .diff is shorthand for .DIFF$value. If there are state differences (e.g. search path) you will be able to view those with .DIFF$state.

ls

Using ls at the unitizer prompt calls an unitizer version of the function (you can call the original with base::ls()). This is what happens when we type ls() at the first failed test in the unitizer we’ve been reviewing in this vignette:

$`objects in new test env:`
[1] "res" "x"   "y"

$`objects in ref test env:`
[1] "res" "x"   "y"

$`unitizer objects:`
[1] ".new" ".NEW" ".ref" ".REF"

Use `ref(.)` to access objects in ref test env
`.new` / `.ref` for test value, `.NEW` / `.REF` for details.
unitizer>

This special version of ls highlights that our environment is more complex than that at the typical R prompt. This is necessary to allow us to review both the newly evaluated objects as well as the objects from the reference unitizer store to compare them for differences. For instance, in this example, we can see that there are both new and reference copies of the res, x, and y objects. The reference copies are from the previous time we ran unitizer. ls also notes what unitizer special objects are available.

When you type at the prompt the name of one of the objects ls lists, you will see the newly evaluated version of that variable. If you wish to see the reference value, then use the ref function:

unitizer> res
    intercept         slope           rsq
-3.541306e+13  7.012486e+11  9.386790e-01
attr(,"class")
[1] "fastlm"
unitizer> ref(res)
   intercept        slope          rsq
-1717.000000   101.000000     0.938679
attr(,"class")
[1] "fastlm"

Note that at times when you use ls at the unitizer promopt you may see something along the lines of:

$`objects in ref test env:`
[1] "res" "x*"   "y'"

where object names have symbols such as * or ' appended to them. This happens because unitizer does not store the entire environment structure of the reference tests. Here is a description of the possible situations you can run into:

• * Object existed during reference test evaluation, but is no longer available
• ' Object existed during reference test evaluation, and still does, but it has a different value than it did during reference test evaluation
• ** Object exists now, but did not exist during reference test evaluation

For more discussion see ?"healEnvs,unitizerItems,unitizer-method" and the discussion of Patchwork Reference Environments.

Objects assigned right before a test are part of that test’s environment so will always be available.