Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the reader for the fci L1C Africa files #2778

Open
wants to merge 13 commits into
base: main
Choose a base branch
from

Conversation

ClementLaplace
Copy link

@ClementLaplace ClementLaplace commented Apr 9, 2024

I have added the possibility to read the data for AFRICA files related to the FCI instrument for the L1C level.

  • Tests added
  • Fully documented
  • Add your name to AUTHORS.md if not there already

@ameraner ameraner added enhancement code enhancements, features, improvements component:readers labels Apr 9, 2024
Copy link
Member

@ameraner ameraner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Clement for this nice work! I have left some comments and hints in the code.

Please also expand in the Filehandler docstring the fact that we support also the African data now, with link to the AfricaPUG.

satpy/etc/readers/fci_l1c_nc.yaml Outdated Show resolved Hide resolved
satpy/etc/readers/fci_l1c_nc.yaml Outdated Show resolved Hide resolved
satpy/etc/readers/fci_l1c_nc.yaml Outdated Show resolved Hide resolved
satpy/readers/fci_l1c_nc.py Outdated Show resolved Hide resolved
satpy/readers/fci_l1c_nc.py Outdated Show resolved Hide resolved
satpy/tests/reader_tests/test_fci_l1c_nc.py Outdated Show resolved Hide resolved
satpy/tests/reader_tests/test_fci_l1c_nc.py Outdated Show resolved Hide resolved
satpy/tests/reader_tests/test_fci_l1c_nc.py Outdated Show resolved Hide resolved
satpy/tests/reader_tests/test_fci_l1c_nc.py Outdated Show resolved Hide resolved
satpy/tests/reader_tests/test_fci_l1c_nc.py Outdated Show resolved Hide resolved
Copy link

codecov bot commented Apr 17, 2024

Codecov Report

Attention: Patch coverage is 97.36842% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 95.95%. Comparing base (f5a8532) to head (54fbb11).
Report is 124 commits behind head on main.

Files Patch % Lines
satpy/readers/fci_l1c_nc.py 63.63% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2778      +/-   ##
==========================================
+ Coverage   95.94%   95.95%   +0.01%     
==========================================
  Files         379      379              
  Lines       53673    53958     +285     
==========================================
+ Hits        51494    51774     +280     
- Misses       2179     2184       +5     
Flag Coverage Δ
behaviourtests 4.08% <0.00%> (-0.01%) ⬇️
unittests 96.04% <97.36%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@coveralls
Copy link

coveralls commented Apr 17, 2024

Pull Request Test Coverage Report for Build 9205754552

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 148 of 152 (97.37%) changed or added relevant lines in 3 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+96.0%) to 96.03%

Changes Missing Coverage Covered Lines Changed/Added Lines %
satpy/readers/fci_l1c_nc.py 7 11 63.64%
Totals Coverage Status
Change from base Build 8732515933: 96.0%
Covered Lines: 51646
Relevant Lines: 53781

💛 - Coveralls

@ClementLaplace ClementLaplace marked this pull request as ready for review April 23, 2024 11:59
@ameraner ameraner requested a review from sjoro April 23, 2024 15:36
@sjoro
Copy link
Collaborator

sjoro commented Apr 24, 2024

a general comment about the fci_l1c_nc.yaml-file. this file repeats key required_netcdf_variables 18 times under different file-types and it seems to be the same set for each file-type. has anyone tried using "anchors and aliases" in these files? that is, create anchor &required_netcdf_variables only once somewhere at the top of the file and use an alias *required_netcdf_variables where needed... should look something like:

&required_netcdf_variables:
  - attr/platform
  - data/{channel_name}/measured/start_position_row
  - data/{channel_name}/measured/end_position_row
  - data/{channel_name}/measured/radiance_to_bt_conversion_coefficient_wavenumber
  - data/{channel_name}/measured/radiance_to_bt_conversion_coefficient_a
  - data/{channel_name}/measured/radiance_to_bt_conversion_coefficient_b
  - data/{channel_name}/measured/radiance_to_bt_conversion_constant_c1
  - data/{channel_name}/measured/radiance_to_bt_conversion_constant_c2
  - data/{channel_name}/measured/radiance_unit_conversion_coefficient
  - data/{channel_name}/measured/channel_effective_solar_irradiance
  - data/{channel_name}/measured/effective_radiance
  - data/{channel_name}/measured/x
  - data/{channel_name}/measured/y
  - data/{channel_name}/measured/pixel_quality
  - data/{channel_name}/measured/index_map
  - data/mtg_geos_projection
  - data/swath_direction
  - data/swath_number
  - index
  - state/celestial/earth_sun_distance
  - state/celestial/subsolar_latitude
  - state/celestial/subsolar_longitude
  - state/celestial/sun_satellite_distance
  - state/platform/platform_altitude
  - state/platform/subsatellite_latitude
  - state/platform/subsatellite_longitude
  - time

fci_l1c_af_vis_06:
  file_reader: !!python/name:satpy.readers.fci_l1c_nc.FCIL1cNCFileHandler
  file_patterns:
    [
      "{pflag}_{location_indicator},{data_designator},MTI{spacecraft_id:1d}-{data_source}-1C-RRAD-3KM-{coverage}-VIS06-{component1}-x-{component3}-{purpose}-{format}_{oflag}_{originator}_{processing_time:%Y%m%d%H%M%S}_{facility_or_tool}_{environment}_{start_time:%Y%m%d%H%M%S}_{end_time:%Y%m%d%H%M%S}_{processing_mode}_{special_compression}_{disposition_mode}_{repeat_cycle_in_day:>04d}_{erraneous_count_in_repeat_cycle:>04d}.nc",
      "{pflag}_{location_indicator},{data_designator},MTI{spacecraft_id:1d}-{data_source}-1C-RRAD-1KM-{coverage}-VIS06-{component1}-x-{component3}-{purpose}-{format}_{oflag}_{originator}_{processing_time:%Y%m%d%H%M%S}_{facility_or_tool}_{environment}_{start_time:%Y%m%d%H%M%S}_{end_time:%Y%m%d%H%M%S}_{processing_mode}_{special_compression}_{disposition_mode}_{repeat_cycle_in_day:>04d}_{erraneous_count_in_repeat_cycle:>04d}.nc",
    ]
  expected_segments: 1
  required_netcdf_variables:
    *required_netcdf_variables
  variable_name_replacements:
    channel_name:
      - vis_06

if this works it would reduce the number of lines in the file by 400+ lines (already 2500 lines long!!) and would be easier to read and maintain.
here's a short tutorial i found:
https://batalin.dev/posts/yaml-anchors-and-aliases/

@ClementLaplace can you make a quick test and see if it works? if yes, please modify the PR accordingly.

Copy link
Collaborator

@sjoro sjoro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice work @ClementLaplace ! a few comments/questions for improved(?) readability and maintainability.

satpy/tests/reader_tests/test_fci_l1c_nc.py Outdated Show resolved Hide resolved
satpy/tests/reader_tests/test_fci_l1c_nc.py Outdated Show resolved Hide resolved
@sjoro sjoro self-requested a review April 26, 2024 06:42
Copy link
Collaborator

@sjoro sjoro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

last nitpicks and editorial comments. thanks clement for taking care of this feature!

satpy/etc/readers/fci_l1c_nc.yaml Outdated Show resolved Hide resolved
satpy/etc/readers/fci_l1c_nc.yaml Outdated Show resolved Hide resolved
satpy/tests/reader_tests/test_fci_l1c_nc.py Outdated Show resolved Hide resolved
satpy/tests/reader_tests/test_fci_l1c_nc.py Outdated Show resolved Hide resolved
satpy/tests/reader_tests/test_fci_l1c_nc.py Outdated Show resolved Hide resolved
…_nc.yaml file, erase _ that is in front for some constant
@sjoro sjoro self-requested a review May 8, 2024 08:53
Copy link
Collaborator

@sjoro sjoro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ameraner ameraner added this to the v0.49.0 milestone May 22, 2024
Copy link
Member

@mraspaud mraspaud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work, especially with refactoring the tests! They are still quite intricate, but it's getting better :) some quick fixes mentioned inline, but after that it's good to merge IMO

satpy/etc/readers/fci_l1c_nc.yaml Outdated Show resolved Hide resolved
satpy/etc/readers/fci_l1c_nc.yaml Show resolved Hide resolved
satpy/etc/readers/fci_l1c_nc.yaml Outdated Show resolved Hide resolved
satpy/readers/fci_l1c_nc.py Outdated Show resolved Hide resolved
satpy/tests/reader_tests/test_fci_l1c_nc.py Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component:readers enhancement code enhancements, features, improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants