Asking review for PR "Refactor/enrich the tests of the analysis part of modules"

[haeter525]

Title: Refactor/enrich the tests of the analysis part of modules

Description

Please refer to here. I want to write tests to increase the test coverage for Quark. This is the first PR.

In this PR, I focus on these files.

1. quark.py
2. apkinfo.py
3. pyeval.py
4. registerobject.py
5. tableobject.py

Why are these files first?

1. A risk assessment shows that those files have the highest priorities to write tests.
2. Those files compose the Dalvik bytecode loader and the well-known five-stage analysis in Quark.
3. The upcoming replacement of Androguard mainly makes changes to those files.

How are you going to do it?

In summary, I follow the guideline and the strategies mentioned in the issue.

For the existing tests

Divide them into individual methods according to their test scenarios.

The goal is to make scenarios distinct for spotting the potential tests.

For the potential tests

1. To analyze the test coverage of each method input, I will apply the Equivalence Class Partitioning strategy.
2. To provide test data for those uncovered test scenarios, I will work with the Boundary Value Analysis strategy.
3. To deliver test code with enough quality, I will follow the coding guideline mentioned in the above issue.

Code Changes

• Add tests to cover the different input conditions.
• Modify the name of the existing tests.

Note
I found a strangle behavior in _invoke() at pyeval.py. No matter the invoked function gives a return value or not, it changes the value of the return register ret_stack.

Files	# Tests added for normal inputs	# Tests added for error inputs	# Tests modified
test_pyeval.py	9	5	5
test_quark.py	3	8	5
test_tableobject.py	3	3	4
test_registerobject.py	3	3	-

Related Discussion

1. Enrich Quark tests gsoc2021-ShengFengLu#1
2. Asking review for PR "Refactor/enrich the tests of the analysis part of modules" #173

---

Hi @pulorsok

The above is an issue I'm going to submit. Can you kindly do a review on this? I will appreciate that!

[pulorsok]

Hi, @haeter525 nicely work!

Here are my advice and questions.

---

• Separate each test into individual methods.
• Add lost tests for each module.

I suggest you add more details to explain these two things.

to enrich the tests for the following files.

1. pyeval.py
2. registerobject.py
3. tableobject.py

Why "only" these three files you want to enrich the tests?

1. A risk assessment on all files shows that those files have the highest priorities to writing unit tests.

Can you explain how you do the risk assessment and how do you evaluate priority? I've read the issue but still don't understand it clearly.

• For the existing tests, I will apply the guideline to separate them into individual methods.

Is it necessary to apply all existing tests?

[haeter525]

Hi @pulorsok , thank you again for your help!

I suggest you add more details to explain these two things.

I agree with you! I have added more details into the "How are you going to do it?" section. Please have a look.

Why "only" these three files you want to enrich the tests?

I have divided the analysis core of Quark into two partitions.

1. The Dalvik Bytecode Loader (pyeval.py, registerobject.py, and tableobject.py)
2. The other four-stage analysis procedure (quark.py, apkinfo.py, analysis.py, etc.)

The issue contains only the first partition. I will open another one for the next part.

Please read here.

Can you explain how you do the risk assessment and how do you evaluate priority? I've read the issue but still don't understand it clearly.

The risk assessment procedure can refer to the page. It can evaluate the risk priorities for files in Quark.

Risk Assessment

Overall, the risk of a file is composed of the probability and the severity. There are two steps.

1. Measure the probability and severity levels for codes in a file.
   • Probability Level - Frequent, Probable, Occasional, etc.
   • Severity Level - Catastrophic, Critical, Marginal, etc.
2. Evaluate the risk level by using the risk assessment matrix given by the page.

To be clear, here is the definition for each term.

• Uncertain events are crashes, freezings, or any unexpected behaviors in a file.
• Risk is a measure of the overall result of an uncertain event in a program. (Serious, High, Medium, and Low)
• Probability is the chance of an uncertain event occurring.
• Severity is the impact when an uncertain event occurs.

The steps in detail.

1. Measure the probability according to two dimensions.
   1. How complex is the code in the file?
   2. How many times does the code in the file expect to run?
2. Measure the severity according to three aspects.
   1. How does the file react to an uncertain event? (Crashes, freezing, or unaffected)
   2. How is the outcome of the file affected by an uncertain event? (Correct outcome, wrong outcome, or nothing)
   3. How to monitor the event that happened in the file? (debugging-needed, in the screen, or the log files)
3. Evaluate the risk stage according to the given matrix.

Is it necessary to apply all existing tests?

Yes, the goal is to make the test scenarios distinct for spotting the potential ones.

[pulorsok]

Hi @haeter525,
Please correct me if I am wrong.
Here is how you evaluated the risk of files:

1. Measure and classified the probability (e.g., Frequent, Probable, Occasional) and severity (e.g., Catastrophic, Critical, Marginal) levels for each code in files.
2. According to the probability and severity levels using the risk assessment matrix to evaluate the risk stages.

If that is correct, how you defined each probability and severity level.

1. How much code complexity and execute frequency are considered the probability level "Frequent"?
2. Which cases are considered the severity level "Critical"?

[haeter525]

Yes, that's how it works!

Here are the level details for the probability. According to the execution frequency, may apply at most one rank bias on a file (e.g., Frequent and Probable below.).

Probability Level	File Relates To
Frequent	the major parts of a project that run one or more times
Probable	the other major parts of a project
Occasional	the data providers for the above, or the minor parts of a project
Remote	the nested data structures or result presentation
Improbable	the simple data structures or the command lines
Eliminate	the constants and logo

And here are the details for the severity.

Severity Level	Impacts on Quark
Catastrophic	crashes, or freezing
Critical	incorrect analysis result
Marginal	convenience features such as updating, APIs loss
Negligible	incorrect report generation

The severity level "Critical" now includes analysis.py and tools.py.

[pulorsok]

Nice, I think you're good to go.

[haeter525]

Hi @pulorsok ,

I have reworded statements to the top comment. Can you kindly have a look again? Sorry for taking your time.

Here are the modifications.

1. The issue title
2. The file list
3. The heading of the section "Why are these files?".
4. The second element in the above section

The reason is to be in line with a conclusion with the mentor.

I will write tests in the following sequence, submit one PR for each item correspondingly.

1. The analysis core of Quark (quark/Object/*.py)
2. The side components (quark/*.py)
3. The overall tests for each stage

Here is the link. Thank you for your kindly help.

[pulorsok]

Correct me if I'm wrong.
You will refactor/enrich these files first because you think they have high priority, and you will refactor/enrich other files later?

When do you plan to refactor/enrich other files?