Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Structure Change for Deaths #12

Open
nimzoking opened this issue Aug 14, 2020 · 9 comments
Open

Structure Change for Deaths #12

nimzoking opened this issue Aug 14, 2020 · 9 comments
Assignees
@xenatisch
Labels
help wanted Extra attention is needed

Comments

@nimzoking
Copy link

nimzoking commented Aug 14, 2020 •
edited

Has the structure changed for api calls for information relating to deaths? I keep getting 404 error codes using this structure:

Request failed .... 404 - Not Found
Response .......... {"response":"Invalid parameter 'newDeathsByDeathDate' in the requested JSON structure. Did you mean 'newDeaths28DaysByPublishDate'?","status_code":404,"status":"Not Found"}

Tried checking the documentation and there is no mention of this field.

Changing the field to newDeaths28DaysByPublishDate, stopped the python error, but no data was returned.

Have I missed a change announcement somewhere?

Thank you.
@xenatisch
Copy link
Contributor

Release of death figures as ...DeathsByDeathDate has been discontinued.

Re the newDeaths28DaysByPublishDate, could you please share your code here?

@xenatisch xenatisch self-assigned this Aug 14, 2020
@xenatisch xenatisch added the help wanted Extra attention is needed label Aug 14, 2020
@grandpamartin
Copy link

I can answer nimzoking’s query having spent some time looking at this particular issue.

The documentation defining the valid list of metrics for the API’s structure was apparently updated on 13th of August 2020 at 17:30pm. In order to find the list, one has to click on the link ‘See a list of valid metrics for structure’ in the Developer’s guide on the Coronavirus dashboard (https://coronavirus.data.gov.uk/developers-guide).

The API documentation on https://publichealthengland.github.io/coronavirus-dashboard-api-python-sdk/pages/examples/general_use.html#structure points to the same Developer’s guide for the metrics.

It is not very helpful that the examples of how to format the Structure on both websites include two metrics are no longer valid; ie "newDeathsByDeathDate" and "cumDeathsByDeathDate".

@nimzoking
Copy link
Author

Thank you, I did missed the URL to expand, and was simply concentrating on the examples.

That said, I still get no data returned.

Example Return:

{'data': [{'Date': '2020-08-17', 'areaName': 'Manchester', 'newCasesByPublishDate': 21, 'cumCasesByPublishDate': 3809, 'newDeaths28DaysByPublishDate': None, 'cumDeaths28DaysByPublishDate': None}], 'lastUpdate': '2020-08-17T14:15:27.000000Z', 'length': 1, 'totalPages': 1}

Is None the expected result for the cumulative cases? I am getting a result from cumulative cases, but nothing for cumulative Deaths. Can I safely assume, no deaths in previous 28 days?

Structure I have set up:

cases_and_deaths = {
"Date": "date",
"areaName": "areaName",
"newCasesByPublishDate": "newCasesByPublishDate",
"cumCasesByPublishDate": "cumCasesByPublishDate",
"newDeaths28DaysByPublishDate": "newDeaths28DaysByPublishDate",
"cumDeaths28DaysByPublishDate": "cumDeaths28DaysByPublishDate"
}

@grandpamartin
Copy link

grandpamartin commented Aug 17, 2020 via email

@xenatisch xenatisch reopened this Aug 17, 2020
@xenatisch
Copy link
Contributor

xenatisch commented Aug 17, 2020 •
edited

@nimzoking Yes, it is the expected answer. Cases by PublishDate are only available for nations, not UTLAs or LTLAs.

Try newCasesBySpecimenDate or cumCasesBySpecimenDate instead. You should get the full data then.

We are hoping to unify the cases data and make them available by SpecimenDate soon.

Hope this helps.

@grandpamartin
Copy link

grandpamartin commented Aug 17, 2020 via email

@xenatisch
Copy link
Contributor

@grandpamartin Oh you mean the 28-day deaths. Yes, it was only the latest at LA-level today. We are waiting for the data to come through... It'll be pushed to the database and released as soon as we receive it. Our ETL, database, and API infrastructures are set up and ready to go...

kathsherratt added a commit to epiforecasts/covidregionaldata that referenced this issue Aug 17, 2020
@kathsherratt
Working UK API
4f1aea3 
UK API now works properly and is integrated to return data as with other countries in the package.

- Own internal function to interact with API, so no  dependence on the PHE R package `ukcovid19`
- Data are returned clean (with standardised variable names, as with other countries) but alongside all the raw variables
- The old UK code is still here but is commented out. Suggest we either
  - delete,
  - move to a back-up folder in case the new code breaks (!)
  - pull out any useful bits and store separately - as it has some useful tables of lower level region authority codes

Outstanding issues (no action required yet):
- Deaths data at a regional level is returned as NA. This is as it comes from the gov.uk dashboard, but this is expected to change and data added soon. See [issue here](UKHSA-Internal/coronavirus-dashboard-api-python-sdk#12)
- Similarly there is a suggestion that England cases may "soon" be available by PublishDate, which would be better as that is consistent with other UK nations
@grandpamartin
Copy link

I have just checked the Dashboard and found that Daily deaths within 28 days of positive test by date reported are now shown for Nations and Regions. The North West region has the most at 6 new deaths. By a little trial and error using the API I found that Kirklees has 1 newDeaths28DaysByPublishDate. The cumDeaths28DaysByPublishDate column is empty. When I looked at Manchester, 0 is shown for the new deaths and the cumulative column is also empty. Seems to mean that the API is working down to utla level at least.
Incidentally, cases are being shown for Nations and below by Specimen Date (as a time series) and for the latest date only by Publish Date as well. It seems that the UK case data is only by Publish Date since Northern Ireland does not submit by Specimen Dates (I think).

@xenatisch
Copy link
Contributor

@grandpamartin cumDeaths28DaysByPublishDate is indeed empty. We're working on it. We added a bunch of new metrics today again... do have a look.

The API is working regardless of level, it's just that we don't have the data at all levels yet.

We don't have all the metrics for every region. This is primarily because different DAs/LAs have been producing different metrics. We are working to standardise them and make at least one metric available for everything... but it takes some time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xenatisch xenatisch

Labels
help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@xenatisch @nimzoking @grandpamartin