The Software Testing, Verification and Validation course is structured as a semester course and has been planned for 6 ECTS (1.5 hours per week of lectures and 1.5 hours per week of recitation classes). This document describes its organization, the program contents, bibliography, schedule, assessment components, and the teaching methodology for the 2022/2023 instance.
[Syllabus] [Schedule] [Assessment] [Lectures] [Recitation classes] [Assignments] [Miscellaneous]
Teaching language: Portuguese but suitable for English-speaking students.
Pre-requirements (aka prior knowledge): Knowledge in programming is required. Code examples are (typically) in Java, although easily transferrable to other programming languages.
Goals: The main objective of this course is to introduce students to the issues related to software quality, the terminology used in the Verification and Validation (VV) area, and to explore/practice different Verification and Validation techniques needed to build high-quality software systems. At the end of the semester, students are expected to be able to effectively design and to perform a Verification and Validation plan on a software project. More specifically, students are expected to be able to:
Programme: The programme for the 2022/2023 instance of the Software Testing, Verification and Validation course is organized as the following:
Note that this programme is subject to change and could be updated as the semester progresses, especially to help focus on requested topics or to support learning.
Teaching methods and learning activities: The continuous enrollment of students in this course is promoted through the study and discussion of the course topics distributed every week. To improve the regular and effective development of autonomous learning processes of each student, in-class and/or away assignments are given on a weekly basis. During the course, the students are expected to work in a pre-defined open-source project where they are required to apply the concepts presented during the course, in particular, different Verification and Validation strategies and tools.
Lecture classes are used to expose the concepts and fundamental aspects of Verification and Validation. Whenever possible, concepts are either formally exposed along with presentation and discussion of examples from real-life applications or introduced in the context of students’ assignments. Lecture classes are scheduled to work as a synchronous presential model, i.e., all lectures take place Wednesday 11:30am – 1:00pm at room B006.
Recitation classes are used to further help students understand the topics exposed in each lecture and to practice one or more Verification and Validation strategies and/or tools. Recitation classes can also be used to clarify and/or discuss any question related to students’ assignments. Recitation classes are scheduled to work as a synchronous presential model, i.e., all classes will take place Friday 11:30am – 1:00pm at room B107 (for M.EIC students) and Friday 9:30am – 11:00am at room B109 (for MESW students).
Bibliography: under development
Faculty:
Communication:
Course’s staff to Students: All announcements are made through moodle.
Students to Course’s staff: Students should feel free to
contact the course’s staff by e-mail if they have any question related
to the course or use moodle’s forum,
M.EIC’s link / moodle’s forum,
MESW’s link instead if they think their question may be interesting
for others. Course’s staff is committed to answer any question as
quickly as possible and are happy to set up ad hoc in person or Zoom
meetings whenever possible. Note that any contact by e-mail
must be through the official university’s e-mail
address (i.e., @fe.up.pt or @g.uporto.pt). Any
e-mail received from a non @fe.up.pt or
@g.uporto.pt address would ignored by the course’s staff
and therefore not answered.
Effort required to complete this course: The amount of work required to successfully complete this course corresponds to 6 ECTS (European Credit Transfer System), i.e., 6 ECTS x 27 hours each = 162 hours of work from day one to the end of the semester. Of these 162 hours, 42 hours (25.9%) corresponds to the time spent in classes. The remaining 120 hours must be devoted to autonomous work, that is 8.6 hours/week, on average, considering a 14 weeks semester.
Students should feel free to give the course’s staff feedback on how much time the course is taking for them.
The grading of each student is be based on the following components:
The final grade is calculated according to the formula: Grade = 50% x ((A1 + A2 + A3 + A4 + A5 + A6 + A7 + A8 + A9) / 9) + 50% x Exam (E)
To successfully complete this course, students must achieve:
Note: students with a special status (e.g., part-time, athletic, etc) which are allowed to not attend classes, must complete all components successfully as any other regular student.
To put in practice the theoretical fundamentals of Verification and Validation strategies presented in each lecture class and informally exemplified in each recitation class, students (in groups of two) must apply different VV strategies on a pre-assigned open-source project.
That is, every week (starting on the 3rd week), and until further notice, each group of two students must apply the VV strategy discussed in the previous week on a pre-assigned open-source project.
Here is a description of the procedure.
On week N, Wednesday 1:00pm – 3:00pm, the course’s staff introduces students to a Verification or Validation strategy X. On the same week N, Friday 11:30am – 1:00pm (for M.EIC students) and Friday 9:30am – 11:00am (for MESW students), the course’s staff present some tools that may assist developers / testers at performing X.
On week N, Friday 1:01pm, each group may start performing X on their pre-assigned open-source project.
On week N+1 (consult the assignments section for more info), Friday 11:59:00pm, each group must upload to moodle a copy of the tests developed for their project. It is only required that one element of each group submit the developed solution.
Find the list of groups and more details of the assigned open-source projects in here.
Late work policy: Any homework submitted after the official deadline may receive feedback but no score. Exceptions to this policy could be made only in extraordinary circumstances, e.g., in case of a family or medical emergency. Exceptions would be subject of consideration by the course’s staff if requested at least 3 days before each assignment’s deadline.
A final written 120 minutes exam covering all topics mentioned during the course will take place January 19, 2023 at 2:30pm (room B338) and February 2, 2023 at 2:30pm (room B329).
As a future IT professional, we expect from students an irreproachable attitude, in both ethical and moral terms, as regulated by the University policy on academic integrity ((PT) Código Ético de Conduta Académica da U.Porto). Students may contact directly the U.Porto’s Ethics Committee for more information on this.
Some of the assignments are designed to be done in groups of two students. We expect that group members collaborate with one another, but that groups work independently from one another, not exchanging results with other groups. Within groups, we expect that students are honest about their contribution to the group’s work. This implies not taking credit for others’ work and not covering for group members that have not contributed to the group.
The rest of this academic integrity content is taken from the policy used in CMU’s course 17-313 and 15-214, which is in here reused almost directly except a few minor modifications.
“Students may not copy any part of a solution to a problem that was written by another student, or was developed together with another student, or was copied from another unauthorized source such as the Internet. Students may not look at another student’s solution, even if they have completed their own, nor may they knowingly give their solution to another student or leave their solution where another student can see it.
Here are some examples of behavior that are inappropriate:
- Copying or retyping, or referring to, files or parts of files (such as source code, written text, or unit tests) from another person or source (whether in final or draft form, regardless of the permissions set on the associated files) while producing your own. This is true even if your version includes minor modifications such as style or variable name changes or minor logic modifications.
- Getting help that you do not fully understand, and from someone whom you do not acknowledge on your solution.
- Writing, using, or submitting a program that attempts to alter or erase grading information or otherwise compromise security of course resources.
- Lying to course staff.
- Giving copies of work to others, or allowing someone else to copy or refer to your code or written assignment to produce their own, either in draft or final form. This includes making your work publicly available in a way that other students (current or future) can access your solutions, even if others’ access is accidental or incidental to your goals.
- Coaching others step-by-step without them understanding your help.
If any of your work contains any statement that was not written by you, you must put it in quotes and cite the source. If you are paraphrasing an idea you read elsewhere, you must acknowledge the source. Using existing material without proper citation is plagiarism, a form of cheating. If there is any question about whether the material is permitted, you must get permission in advance. Note that we will be using automated systems to detect software plagiarism.
It is not considered cheating to clarify vague points in the assignments, lectures, lecture notes; to give help or receive help in using the computer systems, compilers, debuggers, profilers, or other facilities; or to discuss ideas at a very high level, without referring to or producing code.
Any violation of this policy is considered a fraud. The minimum penalty for students, or groups of students, caught in a fraud such as plagiarism (either as plagiarists and plagiarized) will have a zero grade in the correspondent assignment, quiz, or any other formal evaluation. Such incidents will also be reported through University channels, with possible additional disciplinary actions (see the above-linked University policy on academic integrity).
Some assignments are specifically noted as group projects. For those, interpret ‘you’ in the preceding paragraphs as ‘you and your partner(s)’.”
Student should feel free to reach out the course’s staff for any question or clarification about this policy.
Lecture classes are scheduled to work as a synchronous presential model, i.e., all lectures take place Wednesday 11:30am – 1:00pm at room B006. The following schedule describes the planing status for the course and the covered concepts. Note that it is subject to change and could be updated as the semester progresses, especially to help focus on requested topics or support learning.
| Week | Date | Topics | Slides link |
|---|---|---|---|
| #1 | September 14, 2022 | — | — |
| #2 | September 21, 2022 | Introduction to Software Verification and Software Validation, Static testing | |
| #3 | September 28, 2022 | Black-box Testing I – Equivalence Class Partitioning / Category Partition | |
| #4 | October 05, 2022 | Bank holiday | — |
| #5 | October 12, 2022 | Black-box Testing II – Boundary Value Analysis | |
| #6 | October 19, 2022 | Black-box Testing III – Model-based Testing | |
| #7 | October 26, 2022 | White-box Testing I – Structural Testing (Line and Decision coverage) | |
| #8 | November 02, 2022 | FEUP’s week | — |
| #9 | November 09, 2022 | White-box Testing II – Structural/Logical Testing (Condition coverage, Path coverage, Modified Condition/Decision Coverage (MC/DC)) | |
| #10 | November 16, 2022 | White-box Testing III – Dataflow Testing | |
| #11 | November 23, 2022 | White-box Testing IV – Mutation Testing | |
| #12 | November 30, 2022 | Integration Testing, System Testing, Acceptance Testing, and Regression Testing | |
| #13 | December 07, 2022 | Advanced software testing: e.g., Search-based Software Testing | |
| #14 | December 14, 2022 | Exam Exercises |
Recitation classes are scheduled to work as a synchronous presential model, i.e., all classes will take place Friday 11:30am – 1:00pm at room B107 (for M.EIC students) and Friday 9:30am – 11:00am at room B109 (for MESW students).
| Week | Date | Topics | Artifacts link |
|---|---|---|---|
| #1 | September 16, 2022 | — | — |
| #2 | September 23, 2022 | Static testing | slides, exercises |
| #3 | September 30, 2022 | Black-box Testing I – Equivalence Class Partitioning / Category Partition | exercises |
| #4 | October 07, 2022 | Development of the project | — |
| #5 | October 14, 2022 | Black-box Testing II – Boundary Value Analysis | exercises |
| #6 | October 21, 2022 | Black-box Testing III – Model-based Testing | exercises |
| #7 | October 28, 2022 | White-box Testing I – Structural Testing (Line and Decision coverage) | exercises |
| #8 | November 04, 2022 | FEUP’s week | — |
| #9 | November 11, 2022 | White-box Testing II – Structural/Logical Testing (Condition coverage, Path coverage, Modified Condition/Decision Coverage (MC/DC)) | exercises |
| #10 | November 18, 2022 | White-box Testing III – Dataflow Testing | exercises |
| #11 | November 25, 2022 | White-box Testing IV – Mutation Testing | exercises |
| #12 | December 02, 2022 | Project | — |
| #13 | December 09, 2022 | Project | — |
| #14 | December 16, 2022 | Exam Exercises |
| Week | Topic of Evaluation | Deadline | Submission link |
|---|---|---|---|
| #1 | — | — | — |
| #2 | — | — | — |
| #3 | Static Testing | September 30, 2022 11:59:00pm | M.EIC’s moodle / MESW’s moodle |
| #4 | Equivalence Class Partitioning / Category Partition | October 07, 2022 11:59:00pm | M.EIC’s moodle / MESW’s moodle |
| #5 | — | — | — |
| #6 | Boundary Value Analysis | October 21, 2022 11:59:00pm | M.EIC’s moodle / MESW’s moodle |
| #7 | — | — | — |
| #8 | Model-based Testing | M.EIC’s moodle / MESW’s moodle | |
| #9 | Line and Decision Coverage | November 11, 2022 11:59:00pm | M.EIC’s moodle / MESW’s moodle |
| #10 | |||
| #11 | |||
| #12 | Data-flow Testing | December 2, 2022 11:59:00pm | M.EIC’s moodle / MESW’s moodle |
| #13 | Mutation Testing | December 9, 2022 11:59:00pm | M.EIC’s moodle / MESW’s moodle |
| #14 | — | — | — |
The main goal of assignment #1 is to perform static testing in an open-source project. In general, we expect you to select two static testing tools, integrate the selected tools in your project’s build system, perform static testing, and report your findings. In detail:
pom.xml file). Useful resources:
In the end, you should write an informal report (either a PDF or a markdown file) (either in Portuguese or English) where you must discuss and explain, in your own words, the following:
What should you submit/deliver? Place your informal
report (either a PDF or a markdown file) in your project directory and
zip the entire directory. The zip file must be named as
groupXX_projectYY.zip, where XX is your group
identifier (e.g., 03) and YY is your project
identifier (e.g., 02). Note: reports in any other format
than PDF or MD, submissions named incorrectly, i.e., with an invalid
identifier, with an incomplete identifier, and/or with an incorrect
extension (e.g., anything else than zip) will have a 10% penalization in
their final score.
The main goal of assignment #2 is to perform black-box testing in an open-source project. In general, we expect you to perform ‘Category-Partition’ in your pre-assigned project, write unit test cases using the JUnit framework, and report your findings. In detail:
jpass.util for the JPass projectcom.todotxt.todotxttouch.util for the jdotxt projectde.dominik_geyer.jtimesched.project for the jTimeSchedsrc/test/java.Note: It is suggested you look at the project’s documentation, rather than looking at the project’s source code while performing ‘Category-Partition’. The documentation of each project is available in here.
In the end, you should write an informal report (either a PDF or a markdown file) (either in Portuguese or English) where you must discuss and explain, in your own words, the following:
What should you submit/deliver? Place your informal
report (either a PDF or a markdown file) in your project directory and
zip the entire directory. The zip file must be named as
groupXX_projectYY.zip, where XX is your group
identifier (e.g., 03) and YY is your project
identifier (e.g., 02). Note: reports in any other format
than PDF or MD, submissions named incorrectly, i.e., with an invalid
identifier, with an incomplete identifier, and/or with an incorrect
extension (e.g., anything else than zip) will have a 10% penalization in
their final score.
The main goal of assignment #3 is to perform black-box testing in an open-source project. In general, we expect you to perform ‘Boundary Value Analysis’ in your pre-assigned project, write unit test cases using the JUnit framework, and report your findings. In detail:
src/test/java.In the end, you should write an informal report (either a PDF or a markdown file) (either in Portuguese or English) where you must discuss and explain, in your own words, the following:
What should you submit/deliver? Place your informal
report (either a PDF or a markdown file) in your project directory and
zip the entire directory. The zip file must be named as
groupXX_projectYY.zip, where XX is your group
identifier (e.g., 03) and YY is your project
identifier (e.g., 02). Note: reports in any other format
than PDF or MD, submissions named incorrectly, i.e., with an invalid
identifier, with an incomplete identifier, and/or with an incorrect
extension (e.g., anything else than zip) will have a 10% penalization in
their final score.
The main goal of assignment #4 is to perform black-box testing in an open-source project. In general, we expect you to perform ‘Model-based Software Testing’ in your pre-assigned project, write system test cases using the QF-Test tool (download link available in here), and report your findings. In detail:
In the end, you should write an informal report (either a PDF or a markdown file) (either in Portuguese or English) where you must discuss and explain, in your own words, the following:
What should you submit/deliver? Place your informal
report (either a PDF or a markdown file) in your project directory and
zip the entire directory. The zip file must be named as
groupXX_projectYY.zip, where XX is your group
identifier (e.g., 03) and YY is your project
identifier (e.g., 02). Note: reports in any other format
than PDF or MD, submissions named incorrectly, i.e., with an invalid
identifier, with an incomplete identifier, and/or with an incorrect
extension (e.g., anything else than zip) will have a 10% penalization in
their final score.
The main goal of assignment #5 is to perform white-box testing in an open-source project. In general, we expect you to perform ‘Structural Testing’ testing in your pre-assigned project, write unit test cases using the JUnit framework, collect code coverage using the JaCoCo Code Coverage Library for Java, and report your findings. In detail:
mvn test.true and
false outcome of an if statement but not
necessarily all trues and falses of all
conditions in an if statement.In the end, you should write an informal report (either a PDF or a markdown file) (either in Portuguese or English) where you must discuss and explain, in your own words, the following:
@ParameterizedTest
annotation), assert expected exceptions (with the
@Test(expected = ...) annotation), type of asserts
(assertTrue, assertNull, …), etc.What should you submit/deliver? Place your informal
report (either a PDF or a markdown file) in your project directory and
zip the entire directory. The zip file must be named as
groupXX_projectYY.zip, where XX is your group
identifier (e.g., 03) and YY is your project
identifier (e.g., 02). Note: reports in any other format
than PDF or MD, submissions named incorrectly, i.e., with an invalid
identifier, with an incomplete identifier, and/or with an incorrect
extension (e.g., anything else than zip) will have a 10% penalization in
their final score.
The main goal of assignment #8 is to perform white-box testing in an open-source project. In general, we expect you to perform ‘Dataflow Testing’ in your pre-assigned project, write unit test cases using the JUnit framework, and report your findings. In detail:
src/test/java.In the end, you should write an informal report (either a PDF or a markdown file) (either in Portuguese or English) where you must discuss and explain, in your own words, the following:
What should you submit/deliver? Place your informal
report (either a PDF or a markdown file) in your project directory and
zip the entire directory. The zip file must be named as
groupXX_projectYY.zip, where XX is your group
identifier (e.g., 03) and YY is your project
identifier (e.g., 02). Note: reports in any other format
than PDF or MD, submissions named incorrectly, i.e., with an invalid
identifier, with an incomplete identifier, and/or with an incorrect
extension (e.g., anything else than zip) will have a 10% penalization in
their final score.
The main goal of assignment #9 is to perform white-box testing in an open-source project. In general, we expect you to perform ‘Mutation’ testing in your pre-assigned project, write unit test cases using the JUnit framework, collect mutation score coverage using the PIT library for Java, and report your findings. In detail:
In the end, you should write an informal report (either a PDF or a markdown file) (either in Portuguese or English) where you must discuss and explain, in your own words, the following:
What should you submit/deliver? Place your informal
report (either a PDF or a markdown file) in your project directory and
zip the entire directory. The zip file must be named as
groupXX_projectYY.zip, where XX is your group
identifier (e.g., 03) and YY is your project
identifier (e.g., 02). Note: reports in any other format
than PDF or MD, submissions named incorrectly, i.e., with an invalid
identifier, with an incomplete identifier, and/or with an incorrect
extension (e.g., anything else than zip) will have a 10% penalization in
their final score.
Each of the following projects require:
apache-maven-3.8.6-bin.zipapache-maven-3.8.6-bin.zip<extracted directory>/bin. On Linux/MacOS, run
export PATH="<extracted directory>/bin:$PATH". (You
might have to run the export everytime you restart the
computer. For a more permanent solution, please consider adding that
command to your bash profile.)| Project id | Project name | Project source code | Project documentation |
|---|---|---|---|
| 1 | JPass | link | link |
| 2 | jTimeSched | link | link |
| 3 | jdotxt | link | link |
To bundle the project’s source code and all its dependencies, you
must run mvn package. This command should create the
target/jdotxt-0.4.9-SNAPSHOT-jar-with-dependencies.jar
file for the jdotxt
project.target/jtimesched-1.2-SNAPSHOT-jar-with-dependencies.jar
file for the jTimeSched
project.target/jpass-0.1.21-SNAPSHOT.jar file for the JPass project.which could then be executed as
java -jar target/jdotxt-0.4.9-SNAPSHOT-jar-with-dependencies.jarjava -jar target/jtimesched-1.2-SNAPSHOT-jar-with-dependencies.jarjava -jar target/jpass-0.1.21-SNAPSHOT.jarHere is the list of groups and the assigned open-source projects.
| Degree | Group id | Student id | Project id |
|---|---|---|---|
| M.EIC | 01 | 201806613 | 01 |
| M.EIC | 01 | 201806554 | 01 |
| M.EIC | 02 | 201806551 | 03 |
| M.EIC | 02 | 201806538 | 03 |
| M.EIC | 03 | 201806593 | 02 |
| M.EIC | 03 | 201806560 | 02 |
| M.EIC | 04 | 201806529 | 02 |
| M.EIC | 04 | 201806317 | 02 |
| M.EIC | 05 | 201806528 | 01 |
| M.EIC | 05 | 201806185 | 01 |
| M.EIC | 06 | 201703884 | 03 |
| M.EIC | 06 | 201806658 | 03 |
| M.EIC | 07 | 201806230 | 02 |
| M.EIC | 07 | 201800175 | 02 |
| M.EIC | 08 | 201808546 | 03 |
| M.EIC | 08 | 201806878 | 03 |
| M.EIC | 09 | 201800149 | 03 |
| M.EIC | 09 | 201800170 | 03 |
| M.EIC | 10 | 201806490 | 02 |
| M.EIC | 10 | 201806505 | 02 |
| M.EIC | 11 | 202102686 | 01 |
| M.EIC | 11 | 201704618 | 01 |
| M.EIC | 12 | 201806716 | 01 |
| M.EIC | 12 | 201805386 | 01 |
| MESW | 01 | 202103399 | 02 |
| MESW | 01 | 202100688 | 02 |
| MESW | 02 | 202202458 | 03 |
| MESW | 02 | 202200592 | 03 |
| MESW | 03 | 202200590 | 02 |
| MESW | 03 | 202202459 | 02 |
| MESW | 04 | 202202457 | 02 |
| MESW | 04 | 202204413 | 02 |
| MESW | 05 | 201202379 | 03 |
| MESW | 05 | 202202456 | 03 |
| MESW | 06 | 202204410 | 02 |
| MESW | 06 | 202202875 | 02 |
| MESW | 07 | 202204407 | 01 |
| MESW | 07 | 201902431 | 01 |
| MESW | 08 | 202204408 | 01 |
| MESW | 08 | 201208144 | 01 |
| MESW | — | 202204409 | — |
| MESW | 10 | 202204411 | 01 |
| MESW | 10 | ????????? | 01 |
| MESW | 11 | 202103397 | 03 |
| MESW | 11 | 202202453 | 03 |
| MESW | 11 | 201906505 | 03 |