unnecessary java file extracted when compile AOSP with codeql

[thor509]

Hi
I'm trying to run CodeQL on AOSP with this command:
codeql database create new-database --working-dir=./ --source-root=./ --language=java --command='make services'
many unnecessary java files were extracted by codeql in output/soong/.intermediates, some javac-extractor logs are as following:

    [2021-09-14 15:13:27] [javac-extractor-21811] Processing file /media/test/AOSP/output/soong/.intermediates/frameworks/base/services/core/statslog-art-java-gen/gen/com/android/internal/art/ArtStatsLog.java
    [2021-09-14 15:13:28] [javac-extractor-21811] Writing trap file for: com.android.internal.art.ArtStatsLog 0.0-0 null
    [2021-09-14 15:13:28] [javac-extractor-21811] Processing file /media/test/AOSP/output/soong/.intermediates/frameworks/base/services/core/services.core.unboosted/android_common/gen/logtags/frameworks/base/services/core/java/com/android/server/EventLogTags.java
    [2021-09-14 15:13:29] [javac-extractor-21811] Writing trap file for: com.android.server.EventLogTags 0.0-0 null
    [2021-09-14 15:13:29] [javac-extractor-21811] Processing file /media/test/AOSP/output/soong/.intermediates/frameworks/base/services/core/services.core.unboosted/android_common/gen/logtags/frameworks/base/services/core/java/com/android/server/am/EventLogTags.java
    [2021-09-14 15:13:29] [javac-extractor-21811] Writing trap file for: com.android.server.am.EventLogTags 0.0-0 null
    [2021-09-14 15:13:29] [javac-extractor-21811] Processing file /media/test/AOSP/output/soong/.intermediates/frameworks/base/services/core/services.core.unboosted/android_common/gen/logtags/frameworks/base/services/core/java/com/android/server/wm/EventLogTags.java
    [2021-09-14 15:13:29] [javac-extractor-21811] Writing trap file for: com.android.server.wm.EventLogTags 0.0-0 null
    [2021-09-14 15:13:29] [javac-extractor-21811] Processing file /media/test/AOSP/output/soong/.intermediates/frameworks/base/services/core/services.core.unboosted/android_common/gen/logtags/frameworks/base/services/core/java/com/android/server/policy/EventLogTags.java
    [2021-09-14 15:13:29] [javac-extractor-21811] Writing trap file for: com.android.server.policy.EventLogTags 0.0-0 null
    [2021-09-14 15:13:29] [javac-extractor-21811] Processing file /media/test/AOSP/output/soong/.intermediates/frameworks/base/services/core/services.core.unboosted/android_common/javac/srcjars/frameworks/base/services/core/java/android/app/usage/UsageStatsManagerInternal.java
    [2021-09-14 15:13:30] [javac-extractor-21811] Writing trap file for: android.app.usage.UsageStatsManagerInternal 0.0-0 null
    [2021-09-14 15:13:30] [javac-extractor-21811] Writing trap file for: android.app.usage.UsageStatsManagerInternal.AppUsageLimitData 0.0-0 null
    [2021-09-14 15:13:30] [javac-extractor-21811] Processing file /media/test/AOSP/output/soong/.intermediates/frameworks/base/services/core/services.core.unboosted/android_common/javac/srcjars/frameworks/base/services/core/java/android/content/pm/PackageManagerInternal.java

My question is how can I exclude these unnecessary java files while compile AOSP?

[hvitved]

Hi.

I don't believe any of the log messages above are error messages, but merely an indication that the relevant (generated) .java files got imported. Are the generated files causing any problems for the analysis? Normally, including also generated files in the database is encouraged, as it can improve e.g. security queries where there is data-flow through generated code.

[thor509]

Hi, thanks for the reply.
In Android Open Source Project, this will extract many extra java classes with same name and package, but with different implementations. Such as following query:

from Class c
where c.getName().matches("PendingIntent") 
select c, c.getPackage()

The result is:

The source file are in different locations:

The last class within /AOSP/aosp/frameworks/base/core/java/android/app/PendingIntent.java is what I want to analyze.
The other four are like this:

I think the extra java classes extracted cause problems for the analysis. One problem is that codeql event can't get the right methods of PendingIntent class, and the AST parse is failed.

[adityasharad]

@thor509 I believe your assessment is correct. aosp/frameworks/base/core/java/android/app/PendingIntent.java is the actual source class, and the others are stubs produced during the build process. The CodeQL Java extractor currently assumes that each of these files are the same, and only extracts one of them. We have previously observed the same problem on Intent.java.

Although we are aware of the problem, we haven't yet solved it on the CodeQL side. Would it be possible to change your build command so that it doesn't generate the stub files? Then CodeQL will only see the 'correct' PendingIntent.java.

[thor509]

@adityasharad Currently I have not found a way to solve this problem by changing build command. Is there any way to specify
source file location to be extracted? The --source-root arg did't take effect.

[atorralba]

Hey @thor509, we recently introduced a new extractor option in the CodeQL CLI that you might find useful for this use case. By using --extractor-option exclude=<glob>, you can exclude directories from extraction and maintain them in the build. That should be helpful to exclude the stubs for e.g. PendingIntent and Intent that AOSP uses, and ensure that the real implementations end up in the database.

Example:

--extractor-option exclude='**/.intermediates/**/*'