--- title: "unitizeR - The Interactive Environment" author: "Brodie Gaslam" output: rmarkdown::html_vignette: toc: true css: styles.css vignette: > %\VignetteIndexEntry{3 - Interactive Environment} %\VignetteEngine{knitr::rmarkdown} %\usepackage[utf8]{inputenc} --- ## Overview ### `unitize` vs `review` `unitizer` offers three functions to access the interactive review environment: `unitize`, `unitize_dir`, and `review`. `unitize` is used when you either want to generate a `unitizer` from a test file, or when you want to compare the re-evaluation of a test file to an existing `unitizer`. `untize_dir` does what `unitize` does, except for a whole directory at a time. `review` is a helper function used when you want to review the contents of an existing `unitizer`. This is useful if you grow uncertain about tests that you previously approved and want to ensure they actually do what you want them to. You can review and potentially remove items from a `unitizer` with `review`. Both these functions use the same interactive environment, though rules therein are slightly different. For example, in `review`, all the tests are considered passed since there is nothing to compare them to, and the interactive environment will step you through all the passed tests. `unitize` will normally omit passed tests from the review process. We will focus on `unitize` for the rest of this vignette since most of the commentary about it applies equally to `unitize_dir` and `review`. ### Example Set-up To examine the interactive environment more thoroughly we will go back to the demo (you can run it with `demo(unitizer)`). This is the `unitizer` prompt right after our first failed test when our `unitizer.fastlm` implementation was returning the wrong values: ``` > get_slope(res) unitizer test fails on value mismatch: *value* mismatch: Mean relative difference: 6943055624 @@ .ref @@ - [1] 101 @@ .new @@ + [1] 701248618125 ``` ## `unitizer` Commands Much like the `browser()` prompt, the `unitizer` prompt accepts several special expressions that allow you to control `unitizer` behavior. What the expressions are and what they do depends on context. We will review them in the context of the failed test described above. Look at what the `unitizer` prompt stated before we started reviewing our failed tests: ``` - Failed ----------------------------------------------------------------------- The 2 tests in this section failed because the new evaluations do not match the reference values from the store. Overwrite with new results ([Y]es, [N]o, [P]rev, [B]rowse, [R]erun, [Q]uit, [H]elp)? ``` This clearly lays out all the special commands available to us: * `Y` will accept the new value as the correct reference value to use for a test. * `N` will keep the previous reference value as the reference value for future tests. * `P` takes us back to the previously reviewed test (see "Test Navigation" next). * `B` allows us to navigate to any previously reviewed test (see "Test Navigation" next). * `R` toggles re-run mode; when you complete review or exit, `unitizer` will re-run the tests, which is useful if you made changes to your source code and re-installed your package from the `unitizer` prompt. * `Q` quits `unitizer` (see "Quitting `unitizer`"). * `H` provides contextual help. If you type any of those letters into the `unitizer` prompt you will cause `unitizer` to respond as described above instead of evaluating the expression as it would be at the normal R console prompt. If you have a variable assigned to one of those letters and you wish to access it, you can do so with any of `get`, `(`, `print`, etc. For example, suppose we stored something in `Y`, then to access it all these commands would work: * `(Y)` * `get("Y")` * `print(Y)` `unitizer` checks for an exact match of a user expression to the special command letters, so something like `(Y)` does not match `Y` which allows you to reach the value stored in `Y`. If at any time you forget what `unitizer` options are available to you you can just hit the "ENTER" key and `unitizer` will re-print the options to screen. You can accept all unreviewed tests in a sub-section, section, or unitizer with `YY`, `YYY`, and `YYYY` respectively. You can also reject them with `NN`, `NNN`, and `NNNN`. Please note that accepting multiple tests without reviewing them is **a really bad idea**, and you should only resort to these shortcuts when you are absolutely certain of what you are doing. The most common use case for these shortcuts is to drop multiple removed tests from a `unitizer`. ## Test Navigation ### Selecting A Test to Review `unitize` will present to you all the tests that require review, but if you wish to review a specific test you can use the `P` (for Previous) and `B` (for Browse) commands. These commands can come in handy if you realize that you incorrectly accepted or rejected an earlier test, but do not wish to quit `unitizer` completely and lose all the other properly reviewed tests. `P` just steps you back to the previous test. `B` gives you the option to go back to any previously reviewed test. `P` is trivially straightforward, so we will not discuss it further. We will type `B` at the prompt of our second failed test to examine what it does: ``` unitizer> B *1. library(unitizer.fastlm) . . . . . . . . . . . . -:- *2. dat <- data.frame(x = 1:100, y = (1:100)^2) . . . . . . -:- *3. res <- fastlm(dat$x, dat$y) . . . . . . . . . . . -:- 4. res . . . . . . . . . . . . . . . . . . . Failed:N 5. get_slope(res) . . . . . . . . . . . . . . . . Failed:- 6. get_rsq(res) . . . . . . . . . . . . . . . . Passed:- 7. fastlm(1:100, 1:10) . . . . . . . . . . . . . . Passed:- What test do you wish to review (input a test number, [U]nreviewed)? unitizer> ``` The `[B]rowse` option produces a list of all the tests in the order in which they appear in the test file. You can type the number of a test to review it, or U to go to the first test that hasn't been reviewed (more on this in a minute). We will examine the line for test #5 in more detail: ``` 5. get_slope(res) . . . . . . . . . . . . . . . . Failed:- ^ ^ ^ ^ | | | | | +-- Deparsed test expression Test status ----+ | | | +- Test ID User Input -+ ``` The value and order of the test IDs shouldn't mean anything to you other than being the number to type in if you wish to review that test. Tests that have a `*` to the left of the test id are expessions that are not reviewed or checked by `unitizer` (we call these [ignored tests](u2_tests.html#what-constitutes-a-test)). The test status (see [tests outcomes](u2_tests.html#test-outcomes)) indicates the outcome of comparison of the reference test in the `unitizer` store to the newly evaluated ones. The first four tests are ignored tests, so they do not have a status. The User Input column marks which tests have been reviewed and what the user decision was. In this case we had reviewed test #2 and decided not to keep it (hence the "N"). Typically neither ignored tests nor passed tests require user input so they will typically have a "-" as the user input, as will tests that would be reviewed, but haven't been yet. Typing `U` at the review prompt will take you to the first unreviewed test. Since ignored tests and passed tests are not typically reviewed, `U` will take you to the first unreviewed test that is neither passed nor ignored. If we type 4 at the prompt, we get: ``` You are re-reviewing a test; previous selection was: "N" # Our fast computations do not produce the same results as our # original tests so they fail. If you need more detail than the # provided diff you may use `.new`/`.NEW` or `.ref`/`.REF`. # # You should reject these tests by typing 'N' at the prompt since # they are incorrect. > res unitizer test fails on value mismatch: *value* mismatch: mean relative difference: 19854602162 @@ .ref @@ - intercept slope rsq - -1717.000000 101.000000 0.938679 attr(,"class") [1] "fastlm" @@ .new @@ + intercept slope rsq + -3.541306e+13 7.012486e+11 9.386790e-01 attr(,"class") [1] "fastlm" unitizer> ``` `unitizer` tells us we are re-reviewing this test and that previously we had chosen not to keep the new version. At this point we could re-examine the test, and potentially change our previous selection. `unitizer` also re-displays any comments that were in the source file either ahead of the test or on the same line as the test. We used this feature to document the demo. You can jump ahead to any test from the review menu, even tests that are typically not reviewed (i.e. ignored or passed, though if you go to those you will be brought back to the review menu once you complete the review because those tests are not part of the normal review flow). If you skip ahead some tests and then get to the end of the review cycle `unitizer` will warn you about unreviewed tests. ### Finalizing `unitizer` Let's accept the 5th test, which brings us to this prompt: ``` unitizer> Y = Finalize Unitizer ============================================================ You will IRREVERSIBLY modify 'tests/unitizer/fastlm1.unitizer' by: - Replacing 1 out of 2 failed tests Update unitizer ([Y]es, [N]o, [P]rev, [B]rowse, [R]erun)? unitizer> ``` In this case we were reviewing a `unitizer` with two failed tests, one of which we chose to update with the newer value. `unitizer` will summarize for you all the changes that it is about to make to the `unitizer` store. If you type `Y` at the prompt, the existing `unitizer` store will be overwritten with the new version you just reviewed. If you are unsure about the changes you just approved for the `unitizer`, you can re-review them with `R` or `B`. You can also quit without saving your changes by typing `N`, but once you do so you will no longer be able to recover your changes. ### Quitting `unitizer` At any point you may quit `unitizer` by typing `Q` at the `unitizer` prompt. If you have already reviewed tests, you will be given the opportunity to save what you have done so far as you would when finalizing the `unitizer`. Note that if you chose to quit `unitizer` may exit without giving you the opportunity to review the tests. This will happen if: * You did not make any changes to the `unitizer` (e.g. if you chose `N` at failed tests, you are keeping the reference value, so the `unitizer` is not actually changing). * And test evaluation took less than `getOption("unitizer.prompt.b4.quit.time")` seconds (currently 10 seconds). If you end up in the R debugger from a `unitizer` (e.g. via `debug` or `recover`), quitting the debugger with "Q" will force-exit you from the session without a chance to save any changes. ### Differences in `review` Mode `review` works exactly like `unitize`, except that passed tests are automatically queued up for review, and that the only test statuses you should see are "Passed" or "-", the latter for ignored tests. ## Evaluating Expressions at the `unitizer` Prompt ### As Compared To The Standard R Prompt The `unitizer` prompt is designed to emulate the standard R prompt. For the most part you can type any expression that you would type at the R prompt and get the same result as you would there. This means you can examine the objects created by your test script, run R computations, etc. There are, however, some subtle differences created by the structure of the evaluation environments `unitizer` uses: * Even though you can see objects produced by tests, you can not actually remove them with `rm`. * Any objects you create at the interactive prompt are only available for the test you are currently reviewing, so do not expect them to still be there at subsequent prompts. * All expressions are evaluated with `options(warn=1)` or greater. * Other subtle issues discussed at length in the [Reproducible Tests Vignette](u4_reproducible-tests.html#test-environments). * There are special `unitizer` objects `.new`, `.NEW`, `.ref`, and `.REF` that let you review the results of tests (we will discuss these next). * `ls` and `traceback`/`.traceback` are masked by special `unitizer` versions (we will also discuss this next); you can use `base::ls`/`base::traceback` if you need the originals. * You will have access to any objects created through the `pre` argument to `unitize`, though they will not show up in a call to `ls`. ### `.new` and `.ref` As we saw in the demo there are special objects available at the prompt: `.new` (except for removed/deleted tests), and for all but new tests, `.ref`. These objects contain the values produced by the newly evaluated test (`.new`) and by the test when it was previously run and accepted (`.ref`). `.new` might seem a bit superfluous since the user can always re-evaluate the test expression at the `unitizer` prompt to review the value, but if that evaluation is slow you can save a little time by using `.new`. `.ref` is the only option you have to see what the test used to produce back when it was first accepted into the `unitizer` store. `.new` and `.ref` contain the values produced by the tests, but sometimes it is useful to access other aspects of the test evaluation. To do so you can use `.NEW` and `.REF`: * `.NEW` prints general information about the test. * `.NEW$value` returns the test value; equivalent to typing `.new` at the prompt. * `.NEW$conditions` returns the list of conditions produced by the test. * `.NEW$messsage` returns the stderr captured during test evaluation. * `.NEW$output` returns the screen output captured during test evaluation (note often this will be similar to what you get from `.new` or `.NEW$value` since typing those expressions at the prompt leads to the value being printed). * `.NEW$call` returns the test expression. * `.NEW$aborted` returns whether the test expression invoked an "abort" restart (e.g. called `stop` at some point). You can substitute `.REF` for `.NEW` in any of the above, provided that `.REF` is defined (i.e. that will not work when you are reviewing new tests since there is no corresponding reference test for those by definition). If both `.NEW` and `.REF` are defined, then `.DIFF` will be defined too. `.DIFF` has the same structure as `.NEW` and it contains the result of evaluating `diffobj::diffObj` between each component of `.NEW` and `.REF`. `.diff` is shorthand for `.DIFF$value`. If there are state differences (e.g. search path) you will be able to view those with `.DIFF$state`. ### `ls` Using `ls` at the `unitizer` prompt calls an `unitizer` version of the function (you can call the original with `base::ls()`). This is what happens when we type `ls()` at the first failed test in the `unitizer` we've been reviewing in this vignette: ``` $`objects in new test env:` [1] "res" "x" "y" $`objects in ref test env:` [1] "res" "x" "y" $`unitizer objects:` [1] ".new" ".NEW" ".ref" ".REF" Use `ref(.)` to access objects in ref test env `.new` / `.ref` for test value, `.NEW` / `.REF` for details. unitizer> ``` This special version of `ls` highlights that our environment is more complex than that at the typical R prompt. This is necessary to allow us to review both the newly evaluated objects as well as the objects from the reference `unitizer` store to compare them for differences. For instance, in this example, we can see that there are both new and reference copies of the `res`, `x`, and `y` objects. The reference copies are from the previous time we ran `unitizer`. `ls` also notes what `unitizer` special objects are available. When you type at the prompt the name of one of the objects `ls` lists, you will see the newly evaluated version of that variable. If you wish to see the reference value, then use the `ref` function: ``` unitizer> res intercept slope rsq -3.541306e+13 7.012486e+11 9.386790e-01 attr(,"class") [1] "fastlm" unitizer> ref(res) intercept slope rsq -1717.000000 101.000000 0.938679 attr(,"class") [1] "fastlm" ``` Note that at times when you use `ls` at the `unitizer` promopt you may see something along the lines of: ``` $`objects in ref test env:` [1] "res" "x*" "y'" ``` where object names have symbols such as `*` or `'` appended to them. This happens because `unitizer` does not store the entire environment structure of the reference tests. Here is a description of the possible situations you can run into: * `*` Object existed during reference test evaluation, but is no longer available * `'` Object existed during reference test evaluation, and still does, but it has a different value than it did during reference test evaluation * `**` Object exists now, but did not exist during reference test evaluation For more discussion see `?"healEnvs,unitizerItems,unitizer-method"` and the discussion of [Patchwork Reference Environments](u4_reproducible-tests.html#patchwork-reference-environments). Objects assigned right before a test are part of that test's environment so will always be available. ## `traceback` / `.traceback` Errors that occur during test evaluation are handled, so they do not register in the normal R traceback mechanism. `unitizer` stores the traces from the test evaluation and makes them available via internal versions of `traceback`/`.traceback` that mask the base ones at the interactive `unitizer` prompt. They behave similarly but not identically to the `base` counterparts. In particular, parameter `x` must be NULL. You can access the `base` versions with e.g. `base::traceback`, but those will not display any tracebacks generated by `unitizer`-evaluated code. ## `unitize_dir` `unitize_dir` adds a layer of navigation. Here is what you see after running it on the demo package directory test directory: ``` > (.unitizer.fastlm <- copy_fastlm_to_tmpdir()) # package directory > unitize_dir(.unitizer.fastlm) Inferred test directory location: private/var/folders/56/qcx6p6f94695mh7yw- q9m6z_80000gq/T/RtmpJO7kjd/file43ac57df6164/unitizer.fastlm/tests/unitizer Summary of files in common directory 'tests/unitizer': Pass Fail New *1. fastlm1.R - - 4 *2. fastlm2.R - - 1 *3. unitizer.fastlm.R - - 3 ..................................... - - 8 Legend: * `unitizer` requires review Type number of unitizer to review, 'A' to review all that require review unitizer> ``` Each listing corresponds to a test file. If you were to type `1` at the prompt then you would see the equivalent of the `unitize` process in the demo, since "fastlm1.R" is the file we `unitize` in the demo. The `*` ahead of each file indicates that the file has tests that require review. In this case, all the files have new tests. After we type `1` and go through the `unitize` process for "fastlm1.R" we are returned to the `unitize_dir` prompt: ``` unitizer updated Summary of files in common directory 'tests/unitizer': Pass Fail New $1. fastlm1.R ? ? ? *2. fastlm2.R - - 1 *3. unitizer.fastlm.R - - 3 ..................................... ? ? ? Legend: * `unitizer` requires review $ `unitizer` has been updated and needs to be re-evaluted to recompute summary Type number of unitizer to review, 'A' to review all that require review, 'R' to re-run all updated unitizer> ``` Because we updated "fastlm.R", the statistics `unitize_dir` collected when it first ran all the tests are out of date, which is why they show up as question marks. The `$` also indicates that "fastlm1.R" stats are out of date. There is nothing wrong with this, and you do not need to do anything about it, but if you want you can re-run any unitizers that need to be updated by typing "R" at the prompt. This is what happens if we do so: ``` unitizer> R Summary of files in common directory 'tests/unitizer': Pass Fail New 1. fastlm1.R 4 - - *2. fastlm2.R - - 1 *3. unitizer.fastlm.R - - 3 ..................................... 4 - 4 * `unitizer` requires review Type number of unitizer to review, 'A' to review all that require review unitizer> ``` You can now see that we added all the tests, and upon re-running, they all passed since the source code for `unitizer.fastlm` has not changed. Notice how there is no `*` ahead of the first test anymore. Another option for reviewing tests is to type "A" at the prompt, which would cause `unitize_dir` to put you through each test file that requires review in sequence.