This site is from a past semester! The current version will be here when the new semester starts.

tP: Practical Exam Dry RuntP: Grading


tP: Practical Exam

tP: Practical Exam Dry Run (PE-D)

PE-D Overview

What: The latest release of the v1.3 period is subjected to a round of peer acceptance/system testing, also called the Practical Exam (PE) Dry Run as this round of testing will be similar to the graded Practical Exam that will be done at v1.4.

When, where: uses a 40 minute slot at the start of week 11 lecture slot. It will be conducted via Zoom. If you can't find another suitable location to join the Zoom meeting, you may do so from the lecture venue.

Grading: The PE dry run affects your grade in the following ways.

  • You can earn up to 2 bonus marks for your performance in the PE-D.
    Graded based on the usefulness and the quality of your bug reports, as evaluated by the receiving team.
  • PE dry run is a way for you to practice for the actual PE.
  • Taking part in the PE dry run will earn you participation points.
  • There is no penalty for bugs reported in your product. Every bug you find is a win-win for you and the team whose product you are testing.

Why:

  • To train you to do manual testing, bug reporting, bug bug fixing, communicating with users/testers/developers, evaluating products etc.
  • To help you improve your product before the final submission.

PE-D Preparation

  • Ensure that you have accepted the invitation to join the GitHub org used by the module. Go to https://github.com/nus-cs2103-AY2223S2 to accept the invitation.

  • Ensure you have access to a computer that is able to run module projects e.g. has the right Java version.

  • Ensure you can run CATcher on your computer. You should have done this when you smoke-tested CATcher earlier in the semester.

  • Have a good screen grab tool with annotation features so that you can quickly take a screenshot of a bug, annotate it, and post in the issue tracker.
    You can use Ctrl+V to paste a picture from the clipboard into a text box in a bug report.

  • [Optional] Have a good screen recording tool if you plan to use screen recording clips as part of your bug reports. Ensure that your screen recording tool can create small files as CATcher doesn't allow files bigger than 10Mb.
    As the CATcher support for uploading screen recordings is new and limited, use it only if strictly necessary -- use screenshots for other cases.

  • Download the product to be tested.

    PE Dry Run (at v1.3)
    • After you have been notified which team to test (likely to be in the morning of PE-D day), download the latest JAR file from the team's releases page.
    PE (at v1.4)
    • After you have been notified of the download location, download the zip file that bears your name. The password required to unzip it will be given to you at the start of the PE.

    Testing tips

    Use easy-to-remember patterns in test data. For example, if you use 12345678 as a phone number while testing and it appears as 2345678 somewhere else in the UI, you can easily spot that the first digit has gone missing. But if you used a random number instead, detecting that bug won't be as easy. Similarly, if you use Alice Bee, Benny Lee, Charles Pereira as test data (note how the names start with letters A, B, C), it will be easy to detect if one goes missing, or they appear in the incorrect order.

    Go wide before you go deep. Do a light testing of all features first. That will give you a better idea of which features are likely to be more buggy. Spending equal time for all features or testing in the order the features appear in the UG is not always the best approach.

    PE-D During the session

    Use the CATcher Web app for reporting bugs. More instructions will be given during the PE-D e.g., which session to use.

    Use MS Teams (not Zoom) to contact prof if you need help during the session. Use Zoom chat only if you don't get a response via MS Teams.

    How many bugs to report?

    Report as many bugs as you can find during the given time. Take longer if you need (unlike the PE, PE-D is not timed strictly). If you can't find many bugs at this stage when the product is largely untested, you are unlikely to be able to find enough bugs in the better-tested final submission later. In that case, all the more reasons to spend more time and find more bugs now.

    Evaluation rubric:

    • Meets expectations: 3-5 good bug reports
    • Exceeds expectations: more than 5 good bug reports

    A good bug report,

    [a] has a descriptive title,
    [b] has enough details,
    [c] severity/type labels chosen are not too far off,
    [d] is written in a non-confrontational tone, and
    [e] points out a potentially problematic behavior (or a good way to improve the product)

    As you can't be sure which of your bug reports will be considered as good (criterion [e] in particular) by the receiving team, we recommend that you aim to submit at least 8-10 bug reports in total. The median bug reports count in the last round was 9.

    PE and PE-D are manual testing sessions. Using test automation tools or scripting is not allowed.

    Test the product and report bugs as described below, when the prof informs you to begin testing.

    Testing instructions for PE and PE-D
    a) Launching the JAR file
    • Get the jar file to be tested:
      PE Dry Run (at v1.3)
      • Download the latest JAR file from the team's releases page, if you haven't done this already.
      PE (at v1.4)
      • Download the zip file from the given location (to be given to you at least a few hours before the PE), if you haven't done that already.
      • The file is zipped using a two-part password.
        • We will email you the second part in advance, via email (it's unique to each student). Keep it safe, and have it ready at the start of the PE.
        • At the start of the PE, we'll give you the first part of the password (common to the whole class). Use combined password to unzip the file, which should give you another zip file with the name suffix _inner.zip.
        • Unzip that second zip file normally (no password required). That will give you a folder containing the JAR file to test and other PDF files needed for the PE. Warning: do not run the JAR file while it is still inside the zip file.
          Ignore the padding_file found among the extracted files. Its only purpose is to mask the true size of the JAR file so that someone cannot guess which team they will be testing based on the zip file size.
        • You can try above steps using the this sample zip file if you wish (first part of the password: password1-, second part: password2 i.e., you should use password1-password2 to unzip it).

      • Put the JAR file in an empty folder in which the app is allowed to create files (i.e., do not use a write-protected folder).
        In rare cases, the team could have submitted a ZIP file instead of a JAR file. In that case, unzip that file into the target folder.
      • Open a command window. Run the java -version command to ensure you are using Java 11.
        Do this again even if you did this before, as your OS might have auto-updated the default Java version to a newer version.
      • Check the UG to see if there are extra things you need to do before launching the JAR file e.g., download another file from somewhere
        You may visit the team's releases page on GitHub if they have provided some extra files you need to download.
      • Launch the jar file using the java -jar command rather than double-clicking (reason: to ensure the jar file is using the same java version that you verified above). Use double-clicking as a last resort.
        We strongly recommend surrounding the jar filename with double quotes, in case special characters in the filename causes the java -jar command to break.
        e.g., java -jar "[CS2103-F18-1][Task Pro].jar"
        Windows users: use the DOS prompt or the PowerShell (not the WSL terminal) to run the JAR file.
        Linux users: If the JAR fails with an error labelled Gdk-CRITICAL (happens in Wayland display servers), try running it using GDK_BACKEND=x11 java -jar jar_file_name.jar command instead.

      If the product doesn't work at all: If the product fails catastrophically e.g., cannot even launch, or even the basic commands crash the app, do the following:

      1. Check the UG of the team, to see if there are extra things you need to do before launching the JAR.
        Confirm that you are using Java 11 and using the java -jar command to run the JAR, as explained in points above.
      2. Contact our head TA via MS Teams (name: Kim Hyeongcheol, NUSNET: dcskh) and give him
        (a) a screenshot of the error message, and
        (b) your GitHub username.
      3. Wait for him to give you a fallback team to test.
        Contact the prof (via MS Teams) if you didn't get a response from Kim within 5 minutes.
      4. Delete bug reports you submitted for the previous team (if any), using CATcher.
      5. You should not go back to testing the previous team after you've been given a fallback team to test.
      b) What to test
        PE Dry Run (at v1.3)
        • Test the product based on the User Guide available from their GitHub website https://{team-id}.github.io/tp/UserGuide.html.
        • Do system testing first i.e., does the product work as specified by the documentation?. If there is time left, you can do acceptance testing as well i.e., does the product solve the problem it claims to solve?.
        PE (at v1.4)

        • Test based on the Developer Guide (Appendix named Instructions for Manual Testing) and the User Guide. The testing instructions in the Developer Guide can provide you some guidance but if you follow those instructions strictly, you are unlikely to find many bugs. You can deviate from the instructions to probe areas that are more likely to have bugs.

        • The DG appendix named Planned Enhancements (if it exists) gives some enhancements the team is planning for the near future. The feature flaws these enhancements address are 'known' -- reporting them will not earn you any credit.
          However, you can report type.FeatureFlaws bugs if you think these enhancements themselves are flawed/inadequate.
          You can also report type.DocumentationBug bugs if any of the enhancements in this list combines more than one enhancement.

        • As before, do both system testing and acceptance testing but give priority to system testing as those bugs can earn you more credit.

        • Be careful when copying commands from the UG (PDF version) to the software as some PDF viewers can affect the pasted text. If that happens, you might want to open the UG in a different PDF viewer.

        c) What bugs to report?
          PE Dry Run (at v1.3)
          • You may report functionality bugs, UG bugs, and feature flaws.

          • You can also post suggestions on how to improve the product.
            Be diplomatic when reporting bugs or suggesting improvements. For example, instead of criticising the current behavior, simply suggest alternatives to consider.
          PE (at v1.4)
          • Report functionality bugs:

          • Do not post suggestions but if the product is missing a critical functionality that makes the product less useful to the intended user, it can be reported as a bug of type Type.FeatureFlaw. The dev team is allowed to reject bug reports framed as mere suggestions or/and lacking in a convincing justification as to why the omission of that functionality is problematic.

          • You may also report documentation bugs (UG bugs in particular) but keep in mind that there is another time (i.e., part II) set aside for reporting documentation bugs too.

          d) How to report bugs
          • Post bugs as you find them (i.e., do not wait to post all bugs at the end) because bug reports created/modified after the allocated time will not count.
            Using CATcher
            • Launch CATcher, and login to the correct profile (when CATcher asks, consent to creating a new repo):
              • PE Dry Run: CS2103/T PE Dry run
              • PE: CS2103/T PE
            • Post bugs using CATcher.
            Not using CATcher

            Issues created for PE-D and PE need to be in a precise format for our grading scripts to work. Incorrectly-formatted responses will have to discarded. Therefore, you are not allowed to use the GitHub interface for PE-D and PE activities, unless you have obtained our permission first.

            • Post bug reports in the following repo you created earlier:
              • PE Dry Run: ped
              • PE: pe
            • The whole description of the bug should be in the issue description i.e., do not add comments to the issue.
            e) Bug report format
            • Each bug should be a separate issue i.e., do not report multiple problems in the same bug report.
              If there are multiple bugs in the same report, the dev team will select only one of the bugs in the report and discard the others.
            • When reporting similar bugs, it is safer to report them as separate bugs because there is no penalty for reporting duplicates while putting multiple bugs in the same report can reduce your bug count (see the previous point). But as submitting multiple bug reports take extra time, if you are quite sure they will be considered as duplicates by the dev team later, you can report them together, to save time.
            • Write good quality bug reports; poor quality or incorrect bug reports will not earn credit.
            • Use a descriptive title.
            • Give a good description of the bug with steps to reproduce, expected, actual, and screenshots. If the receiving team cannot reproduce the bug, you will not be able to get credit for it.
            • Assign exactly one severity.* label to the bug report. Bug reports without a severity label are considered severity.Low (lower severity bugs earn lower credit)

            Bug Severity labels:

            • severity.VeryLow : A flaw that is purely cosmetic and does not affect usage e.g., a typo/spacing/layout/color/font issues in the docs or the UI that doesn't affect usage. Only cosmetic problems should have this label.
            • severity.Low : A flaw that is unlikely to affect normal operations of the product. Appears only in very rare situations and causes a minor inconvenience only.
            • severity.Medium : A flaw that causes occasional inconvenience to some users but they can continue to use the product.
            • severity.High : A flaw that affects most users and causes major problems for users. i.e., only problems that make the product almost unusable for most users should have this label.

            When applying for documentation bugs, replace user with reader.

            • Assign exactly one type.* label to the issue.

            Type labels:

            • type.FunctionalityBug: A functionality does not work as specified/expected.
            • type.FeatureFlaw: Some functionality missing from a feature delivered in v1.4 in a way that the feature becomes less useful to the intended target user for normal usage. i.e., the feature is not 'complete'. In other words, an acceptance-testing bug that falls within the scope of v1.4 features.
              These issues are counted against the product design aspect of the project. Therefore, other design problems (e.g., low testability, mismatches to the target user/problem, project constraint violations etc.) can be put in this category as well.
              Features that work as specified by the UG but should have been designed to work differently (from the end-user's point of view) fall in this category too.
            • type.DocumentationBug: A flaw in the documentation e.g., a missing step, a wrong instruction, typos
            • If you need to include < or > symbols in your bug report, use HTML equivalents for 'less than' and 'greater than' instead (i.e., &lt; and &gt; e.g., x &lt; y instead of x < y).
              Reason: CATcher and GitHub strips out content wrapped in < and >, for security reasons.

            PE-D After the session

            • The relevant bug reports will be transferred to your issue tracker within a day after the session is over. Once you have received the bug reports for your product, you can decide whether you will act on reported issues before the final submission v1.4. For some issues, the correct decision could be to reject or postpone to a version beyond v1.4.
              Reminder: There is no penalty for any of the bugs you received in the PE-D.

            Dealing with "What the h___ the tester was thinking?" type bug reports

            Some bug reports will make you angry because they seem baseless, wrong, rude etc. It's still possible to get value from such bug reports though:

            • After you got over the initial indignation, dig deeper to see if there's even the slightest possibility that there is a bug. For example, consider this scenario:
              1. The tester claims a certain command doesn't work.
              2. All your team members tried the exact same command and it works as advertised. What the h___ the tester is trying to pull here?
              3. In reality, the error is actually caused by a duplicate entry in the database resulting from a previous command; the tester didn't mention that command in the bug report (because s/he didn't realize the two are connected).
            • The reported bug might be non-existent but the tester's screw up can indicate other areas to improve. For example, the tester reports a missing feature that is clearly mentioned as 'not implemented' in the UG, but perhaps the UG can be improved to make that fact harder to miss?
            • What exactly about the bug report that makes you angry? Remind yourself not to do the same offence when you report bugs yourself in future.

            Use the pain of dealing with this kind of bug reports as an opportunity to develop the following mindset:

            (a) The product is guilty until proven innocent: If the bug report has even a hint of something amiss with the product, it's your (not the tester's) responsibility to try and prove if it is really a problem or not. Why? because finding a bug is a win for you -- as you can then fix it and thereby avoid the embarrassment of releasing a buggy product.

            (b) A crappy bug report is better than no bug report: If the bug actually exists, it is better to have some indication about it than none at all. In a real project, a tester that fails to find bugs can cause more harm to your career than a tester who finds bugs but doesn't report them well.

            (c) I used to get angry at bug reports, but not any more: If you work hard, take pride in the quality of your work, it's no wonder that you get angry when others find faults with your work incorrectly. Aim to move past that phase where you take bug reports personally. The sooner you can tackle any sh*tty bug report calmly and objectively, the sooner you'll rise to the 'professional' software engineer level.

            • If you have received stray bug reports (i.e., bug reports that don't seem to be about your project), do let us know ASAP (email the prof).
            • You can navigate to the original bug report (via the back-link provided in the bug report given to you) and post in that issue thread to communicate with the tester who reported the bug e.g. to ask for more info, etc. However, the tester is not obliged to respond. Note that simply replying to the bug report in your own repo will not notify the tester.
              • Do not argue with the tester to try to convince that person that your way is correct/better. If at all, you can gently explain the rationale for the current behavior but do not waste time getting involved in long arguments. If you think the suggestion/bug is unreasonable, just thank the tester for their view and discontinue to discussion.
            • Aim to do a systematic triaging of issues received. Some suggestions:
              • Close duplicate issues.
              • Use labels (create new labels if necessary) to,
                • differentiate bugs from the rest (e.g., feature suggestions/flaws).
                • indicate priority of the bugs that need fixing.
              • Assign each bug to the person who should fix it.
            • You may ignore type/severity.* labels given by the tester. They will not affect you or the tester either way -- they were there just for the testers to practice. You may apply your own type/severity labels if you wish.
              In particular, beware of simply following the type.* given by the tester; it is your job to decide the correct type of the issue. e.g., What the tester labeled as a bug might actually be a feature flaw.
            • If a bug report is simply a feature suggestion, you can take note of it and close it (to reduce clutter in the issue tracker, and to make it easy for the teaching team to track your progress on dealing with PE-D issues). Similarly, you can close PE-D issues not relevant to v1.3.

            Note that listing bugs as 'known bugs' in the UG or specifying unreasonable constraints in the UG to make bugs 'out of scope' will not exempt those bugs from the final grading. That is, PE testers can still earn credit for reporting those bugs and you will still be penalized for them.
            However, a product is allowed to have 'known limitations' (e.g., a daily expense tracking application meant for students is unable to handle expenses larger than $999) as long as they don't degrade the product's use within the intended scope. They will not be penalized.

            Even bugs inherited from AB3 need to be fixed. As mentioned in a previous week, even bugs you inherited from AB3 need fixing (because "we inherited it from the previous dev team" is not a valid excuse to leave a bug unfixed). If you are unsure if something is such a bug that need fixing, please post in the forum.

            Identify bugs you missed in the PE-D: Visit the issue tracker of the team you tested and see bugs reported by others who tested the same product. Identify bugs you missed (if any). That knowledge might help you find similar bugs in your own product as well as find more bugs during the PE.


            tP: Practical Exam Dry RuntP: Grading