Swchab conversion error

[adrianbn]

Tried with multiple accounts and I always get the following output:

docker run --rm -v $PWD/exports/:/var/tmp/e2g-input -v .:/var/tmp/e2g-output --env GHOSTFOLIO_ACCOUNT_ID=<some-acct-id>  --env DEBUG_LOGGING=true dickwolff/export-to-ghostfolio

> [email protected] watch
> tsx ./src/watcher.ts

[i] Watching /var/tmp/e2g-input..
[i] Found Personal-Positions-2024-05-19.csv!
[i] Determined the file type to be of kind 'ibkr'.
[i] Restored 0 ISIN-symbol pairs and 0 symbols from cache..
[i] Processing file using IBKR converter
[e] An error ocurred while processing.
[e] Error details: Error: An error ocurred while parsing!
[e] Moving file to output..
[i] Stop container as usePolling is set to false..

Then when trying to import to Ghostfolio I get this error:

activities.0.currency is not valid
{
  "Positions for account Personal ...436 as of 04:48 PM ET, 2024/05/19": null,
  "": null
}

[dickwolff]

Hi there! Looks like a header line is being processed as a transaction. Could you maybe provide a (partial) export so I can reproduce this and make a fix? Thanks!

[adrianbn]

Here are the first few lines from one of the accounts:

"Positions for account Personal ...436 as of 04:48 PM ET, 2024/05/19","","","","","","","","","","","","","","","",""
"","","","","","","","","","","","","","","","",""
"Symbol","Description","Quantity","Price","Price Change %","Price Change $","Market Value","Day Change %","Day Change $","Cost Basis","Gain/Loss %","Gain/Loss $","Ratings","Reinvest Dividends?","Capital Gains?","% Of Account","Security Type"
"BNDX","VANGUARD TOTAL INTERNATIONAL BND ETF","330","$48.73","-0.27%","-$0.13","$1608.9","-0.27%","-$42.9","$1892.17","-15.03%","-$2844.27","--","No","--","4.06%","ETFs & Closed End Funds"

Then there are more entries like the BNDX one, and the account total at the end that looks like this:

"Account Total","--","--","--","--","--","$3961.6","0.2%","$80.17","$3413.54","13.55%","$462.64","--","--","--","--","--"

Exact dollar values and percentages are made up.

[dickwolff]

Thank you! I will take a look at it this week and get back to you!

[dickwolff]

@adrianbn The headers you posted don’t seem to match those of the sample export. The sample export is taken from the account history if I recall correctly (see the image in the readme). It seems like your export was taken from another screen and looks like it’s some kind of account position export (so the current status, not the transactions). Can you confirm this to me?

[adrianbn]

Hi @dickwolff,

Yes. I tried to import the positions because the history also was failing. This is what happens when I try to import the history:

> [email protected] watch
> tsx ./src/watcher.ts

[i] Watching /var/tmp/e2g-input..
[i] Found Designated_Bene_Joint_XXXYYY_Transactions_20240520-165958.csv!
[i] Determined the file type to be of kind 'schwab'.
[i] Restored 0 ISIN-symbol pairs and 0 symbols from cache..
[i] Processing file using Schwab converter
[i] Read CSV file. Start processing..

[e] An error ocurred while trying to retrieve symbol TDA TO DW&O TRANSFER (line 38)!

[e] An error ocurred while processing.
[e] Error details: BadRequestError: Missing required query parameter=q
[e] Moving file to output..
[i] Stop container as usePolling is set to false..
npm notice
npm notice New minor version of npm available! 10.5.2 -> 10.8.0
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v10.8.0>
npm notice Run `npm install -g [email protected]` to update!
npm notice

Then when I try to import the output file to Ghostfolio

activities.0.currency is not valid
{
  "Date": "05/16/2024 as of 05/15/2024",
  "Action": "Bank Interest",
  "Symbol": null,
  "Description": "BANK INT …XXX-YYYYYY TD BANK USA NA",
  "Quantity": null,
  "Price": null,
  "Fees & Comm": null,
  "Amount": "$3.11"
}

I think the line reporting isn't accurate (it may be skipping the headers and counting from 0?). I think this is the offending line

"11/07/2023","Internal Transfer","","TDA TO DW&O TRANSFER","","","","$7.06"

I tried with another of my Schwab accounts as well and the history export produces the following error:

> [email protected] watch
> tsx ./src/watcher.ts

[i] Watching /var/tmp/e2g-input..
[i] Found Individual_XXXYYY_Transactions_20240520-170620.csv!
[i] Determined the file type to be of kind 'schwab'.
[i] Restored 0 ISIN-symbol pairs and 0 symbols from cache..
[i] Processing file using Schwab converter
[i] Read CSV file. Start processing..

[e] An error ocurred while trying to retrieve symbol Tfr BANK OF AMERICA, N, XXXX YYYYYY ZZZ (line 30)!

[e] An error ocurred while processing.
[e] Error details: BadRequestError: Missing required query parameter=q
[e] Moving file to output..
[i] Stop container as usePolling is set to false..
npm notice
npm notice New minor version of npm available! 10.5.2 -> 10.8.0
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v10.8.0>
npm notice Run `npm install -g [email protected]` to update!
npm notice

I think the offending line is this

"04/08/2024","MoneyLink Transfer","","Tfr BANK OF AMERICA, N, XXXX YYYYYY ZZZ","","","","-$30.00"

[dickwolff]

Gotcha, thanks for the detailed analysis. One more thing to check: which version of the tool are you running?

[adrianbn]

Pulling from latest, `-v` reports 10.5.2.

…

On Mon, May 20, 2024 at 10:55 AM Dick Wolff ***@***.***> wrote: Gotcha, thanks for the detailed analysis. One more thing to check: which version of the tool are you running? — Reply to this email directly, view it on GitHub <#75 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA6NPRMIM5TOHGIPEDHMDR3ZDI2IRAVCNFSM6AAAAABH6VAM7OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRQHEZDQMRQGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[dickwolff]

That would be npm, but I meant the version of Export to Ghostfolio. The container version from Docker hub

[adrianbn]

sha256:06e91d68af18c6c853f4fa94593aaa0cf194cc8325d8b41331bdb89b6e741be9, 0.13.1 according to the labels; this is latest from last week or so.

[dickwolff]

Thanks! I'll probably have some time tonight to look into the issue and will get back to you when I have something for you to test 😅

[dickwolff]

I think the issue does relate to the "internal transfer" line, at least based on your information. Thanks again for the detailed infromation. I have added "transfer" to the ignore list, and that seems to work now.

Can you pull the 0.13.2-beta image and let me know if this resolved the issue? After your confirmation I will merge the change.

[adrianbn]

We're making progress but I am still having issues. One of my accounts has this problem:

> [email protected] watch
> tsx ./src/watcher.ts

[i] Watching /var/tmp/e2g-input..
[i] Found Designated_Bene_Joint_XXX436_Transactions_20240520-165958.csv!
[i] Determined the file type to be of kind 'schwab'.
[i] Starting Export to Ghostfolio v0.13.2
[i] Restored 0 ISIN-symbol pairs and 0 symbols from cache..
[i] Processing file using Schwab converter
[i] Read CSV file. Start processing..

[e] An error ocurred while trying to retrieve symbol TDA TRAN - NON-TAXABLE DIVIDENDS (CMF) (line 40)!

[e] An error ocurred while processing.
[e] Error details: BadRequestError: Missing required query parameter=q
[e] Moving file to output..
[i] Stop container as usePolling is set to false..

Offending line is this

"11/07/2023","Non-Qualified Div","","TDA TRAN - NON-TAXABLE DIVIDENDS (CMF)","","","","$72.06"

This likely needs to be treated as a cash deposit I am guessing.

The other account shows no errors on the parsing, but it doesn't upload to ghostfolio. Pretty much every line errors out with:

activities.0.each value in type must be one of the following values: BUY, DIVIDEND, FEE, INTEREST, ITEM, LIABILITY, SELL
activities.1.each value in type must be one of the following values: BUY, DIVIDEND, FEE, INTEREST, ITEM, LIABILITY, SELL
activities.2.each value in type must be one of the following values: BUY, DIVIDEND, FEE, INTEREST, ITEM, LIABILITY, SELL
{
  "accountId": "a71e7a8c-dc55-4160-a461-4b6d9b752476",
  "comment": "",
  "fee": 0,
  "quantity": 29,
  "unitPrice": 0,
  "currency": "USD",
  "dataSource": "YAHOO",
  "date": "2024-05-17T00:00:00+00:00",
  "symbol": "ZZZZ"
}

[dickwolff]

The first error is a weird one. There are non-qualified dividends in the sample export as well, which I have got from another user when creating the Schwab converter. Those do have a symbol, like 09/29/2023,Non-Qualified Div,ARCC,ARES CAPITAL CORP,,,,$177.70 However your example does not seem to have a symbol. So that instance probably needs to be treated like a cash deposit, yes.

The last errors are due to the fact that the records don't have a type (buy, sell, etc) so that's probably a line that doesn't get converted correctly. What kind of record type do the lines have? So for 2023","Non-Qualified Div","",", the "Non-Qualified Div" part.

[dickwolff]

Sorry for the late check-in, I have been busy lately.

I added the non-qualified div as a cash position this morning. Could you try and run the tool again with the latest version, 0.13.2-beta? Thanks!

[dickwolff]

Merged the changes, closing issue due to inactivity. Please open a new issue should there be something else.