Compare commits

..

11 Commits

Author SHA1 Message Date
49a975c43c Implement encrypted transaction caching for GoCardless adapter
- Reduces GoCardless API calls by up to 99% through intelligent caching of transaction data
- Secure AES-GCM encryption with PBKDF2 key derivation (200k iterations) for at-rest storage
- Automatic range merging and transaction deduplication to minimize storage and API usage
- Cache-first approach with automatic fetching of uncovered date ranges
- Comprehensive test suite with 30 unit tests covering all cache operations and edge cases
- Thread-safe implementation with in-memory caching and encrypted disk persistence
2025-11-21 22:44:41 +01:00
9442d71e84 Add input validation for transaction amounts and currencies
- Validate amounts are non-zero and within reasonable bounds (≤1B)
- Validate currency codes are 3 uppercase ASCII letters
- Apply validation to main and foreign amounts/currencies
- Add comprehensive tests for validation logic
- Maintain graceful error handling for invalid data
2025-11-21 20:16:19 +01:00
d185ca36fd Add JJ version control requirement to AGENTS.md 2025-11-21 20:12:07 +01:00
74d362b412 Handle expired agreements and rewrite README
- Implement robust End User Agreement expiry detection and handling
- Add graceful error recovery for failed accounts
- Rewrite README.md to focus on user benefits
- Add documentation guidelines to AGENTS.md
2025-11-21 19:30:54 +01:00
a4fcea1afe Mask details in debug traces. 2025-11-21 17:40:39 +01:00
cf5e6eee08 Implemented debug logging to debug_logs/ 2025-11-21 17:16:26 +01:00
f7e96bcf35 Add specs for debug logging. 2025-11-21 16:31:38 +01:00
6f293730ee Differentiate between human and LLM debugging 2025-11-21 16:30:55 +01:00
fcd59b7fc5 Implement logic 2025-11-21 14:32:23 +01:00
4b5dc6f59a Remove old stuff 2025-11-19 20:40:47 +00:00
6903ba6924 Add high-level planning 2025-11-19 20:40:14 +00:00
47 changed files with 855 additions and 4912 deletions

View File

@@ -136,8 +136,6 @@ mod tests {
- Write code in appropriate modules following the hexagonal architecture - Write code in appropriate modules following the hexagonal architecture
- Keep core business logic separate from external integrations - Keep core business logic separate from external integrations
- Use workspace dependencies consistently - Use workspace dependencies consistently
- When working from a spec, update the spec with the current status as soon as you finish something
- **MANDATORY**: After making ANY code change, complete the Post-Change Verification Checklist (see Code Quality section below)
### 2. Testing ### 2. Testing
- Write tests alongside code in `#[cfg(test)]` modules - Write tests alongside code in `#[cfg(test)]` modules
@@ -150,28 +148,14 @@ mod tests {
- Use `cargo fmt` for formatting - Use `cargo fmt` for formatting
- Use `cargo clippy` for linting - Use `cargo clippy` for linting
- Ensure documentation for public APIs - Ensure documentation for public APIs
- **MANDATORY: ALWAYS format and lint after making ANY change, and fix ALL linting errors and warnings**
- When a change is end-user visible, update the README.md. Use the README.md documentation guidelines
- Always clean up unused code. No todo's or unused code is allowed after a change. Remove unused variables, functions, imports, etc. Do NOT hide unused code with underscores - delete it!
#### Post-Change Verification Checklist (MANDATORY)
After making ANY code change, you MUST run these commands and fix any issues:
1. **Format code**: `cargo fmt --all`
2. **Run linter**: `cargo clippy --all-targets --all-features -- -D warnings`
3. **Run tests**: `cargo test --workspace`
4. **Build project**: `cargo build --workspace`
5. **Clean up unused code**: Remove any unused variables, functions, imports, etc.
**FAILURE TO COMPLETE THIS CHECKLIST WILL RESULT IN CODE REJECTION**
### 4. Commit Standards ### 4. Commit Standards
- *Always* ensure the workspace compiles: `cargo build --workspace`
- Commit both code and tests together - Commit both code and tests together
- Write clear, descriptive commit messages, focusing on user benefits over technical details. Use prose over bullet points - Write clear, descriptive commit messages
- Ensure the workspace compiles: `cargo build --workspace`
### Version Control ### Version Control
- **Use JJ (Jujutsu)** as the primary tool for all source control operations due to its concurrency and conflict-free design. Use a specialized agent if available - **Use JJ (Jujutsu)** as the primary tool for all source control operations due to its concurrency and conflict-free design
- **Git fallback**: Only for complex operations unsupported by JJ (e.g., interactive rebasing) - **Git fallback**: Only for complex operations unsupported by JJ (e.g., interactive rebasing)
## Project Structure Guidelines ## Project Structure Guidelines
@@ -186,20 +170,6 @@ After making ANY code change, you MUST run these commands and fix any issues:
- **firefly/**: Firefly III API integration - **firefly/**: Firefly III API integration
- Each adapter implements the appropriate port trait - Each adapter implements the appropriate port trait
### Commands Module (`banks2ff/src/commands/`)
- **sync.rs**: Sync command handler
- **accounts/**: Account management commands
- **mod.rs**: Account command dispatch
- **link.rs**: Account linking logic and LinkCommands dispatch
- **list.rs**: Account listing handler
- **status.rs**: Account status handler
- **transactions/**: Transaction management commands
- **mod.rs**: Transaction command dispatch
- **list.rs**: Transaction listing handler
- **cache.rs**: Cache status handler
- **clear.rs**: Cache clearing handler
- **list.rs**: Source/destination listing handler
### Client Libraries ### Client Libraries
- **gocardless-client/**: Standalone GoCardless API wrapper - **gocardless-client/**: Standalone GoCardless API wrapper
- **firefly-client/**: Standalone Firefly III API wrapper - **firefly-client/**: Standalone Firefly III API wrapper
@@ -237,4 +207,4 @@ After making ANY code change, you MUST run these commands and fix any issues:
### Technical Documentation ### Technical Documentation
- **docs/architecture.md**: Detailed technical specifications, implementation details, and developer-focused content - **docs/architecture.md**: Detailed technical specifications, implementation details, and developer-focused content
- **specs/**: Implementation planning, API specifications, and historical context - **specs/**: Implementation planning, API specifications, and historical context
- **Code Comments**: Use sparingly for implementation details. *Do* explain complex logic - **Code Comments**: Use for implementation details and complex logic explanations

231
Cargo.lock generated
View File

@@ -198,12 +198,9 @@ dependencies = [
"bytes", "bytes",
"chrono", "chrono",
"clap", "clap",
"comfy-table",
"dialoguer",
"dotenvy", "dotenvy",
"firefly-client", "firefly-client",
"gocardless-client", "gocardless-client",
"hkdf",
"http", "http",
"hyper", "hyper",
"mockall", "mockall",
@@ -216,7 +213,6 @@ dependencies = [
"serde_json", "serde_json",
"sha2", "sha2",
"task-local-extensions", "task-local-extensions",
"temp-env",
"thiserror", "thiserror",
"tokio", "tokio",
"tracing", "tracing",
@@ -417,17 +413,6 @@ version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b05b61dc5112cbb17e4b6cd61790d9845d13888356391624cbe7e41efeac1e75" checksum = "b05b61dc5112cbb17e4b6cd61790d9845d13888356391624cbe7e41efeac1e75"
[[package]]
name = "comfy-table"
version = "7.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b03b7db8e0b4b2fdad6c551e634134e99ec000e5c8c3b6856c65e8bbaded7a3b"
dependencies = [
"crossterm",
"unicode-segmentation",
"unicode-width",
]
[[package]] [[package]]
name = "concurrent-queue" name = "concurrent-queue"
version = "2.5.0" version = "2.5.0"
@@ -437,19 +422,6 @@ dependencies = [
"crossbeam-utils", "crossbeam-utils",
] ]
[[package]]
name = "console"
version = "0.16.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b430743a6eb14e9764d4260d4c0d8123087d504eeb9c48f2b2a5e810dd369df4"
dependencies = [
"encode_unicode",
"libc",
"once_cell",
"unicode-width",
"windows-sys 0.61.2",
]
[[package]] [[package]]
name = "core-foundation" name = "core-foundation"
version = "0.9.4" version = "0.9.4"
@@ -481,29 +453,6 @@ version = "0.8.21"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d0a5c400df2834b80a4c3327b3aad3a4c4cd4de0629063962b03235697506a28" checksum = "d0a5c400df2834b80a4c3327b3aad3a4c4cd4de0629063962b03235697506a28"
[[package]]
name = "crossterm"
version = "0.29.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d8b9f2e4c67f833b660cdb0a3523065869fb35570177239812ed4c905aeff87b"
dependencies = [
"bitflags 2.10.0",
"crossterm_winapi",
"document-features",
"parking_lot",
"rustix",
"winapi",
]
[[package]]
name = "crossterm_winapi"
version = "0.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "acdd7c62a3665c7f6830a51635d9ac9b23ed385797f70a83bb8bafe9c572ab2b"
dependencies = [
"winapi",
]
[[package]] [[package]]
name = "crypto-common" name = "crypto-common"
version = "0.1.7" version = "0.1.7"
@@ -543,18 +492,6 @@ version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "092966b41edc516079bdf31ec78a2e0588d1d0c08f78b91d8307215928642b2b" checksum = "092966b41edc516079bdf31ec78a2e0588d1d0c08f78b91d8307215928642b2b"
[[package]]
name = "dialoguer"
version = "0.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "25f104b501bf2364e78d0d3974cbc774f738f5865306ed128e1e0d7499c0ad96"
dependencies = [
"console",
"shell-words",
"tempfile",
"zeroize",
]
[[package]] [[package]]
name = "difflib" name = "difflib"
version = "0.4.0" version = "0.4.0"
@@ -583,15 +520,6 @@ dependencies = [
"syn 2.0.110", "syn 2.0.110",
] ]
[[package]]
name = "document-features"
version = "0.2.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d4b8a88685455ed29a21542a33abd9cb6510b6b129abadabdcef0f4c55bc8f61"
dependencies = [
"litrs",
]
[[package]] [[package]]
name = "dotenvy" name = "dotenvy"
version = "0.15.7" version = "0.15.7"
@@ -610,12 +538,6 @@ version = "1.15.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "48c757948c5ede0e46177b7add2e67155f70e33c07fea8284df6576da70b3719" checksum = "48c757948c5ede0e46177b7add2e67155f70e33c07fea8284df6576da70b3719"
[[package]]
name = "encode_unicode"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "34aa73646ffb006b8f5147f3dc182bd4bcb190227ce861fc4a4844bf8e3cb2c0"
[[package]] [[package]]
name = "encoding_rs" name = "encoding_rs"
version = "0.8.35" version = "0.8.35"
@@ -631,16 +553,6 @@ version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f"
[[package]]
name = "errno"
version = "0.3.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
dependencies = [
"libc",
"windows-sys 0.61.2",
]
[[package]] [[package]]
name = "event-listener" name = "event-listener"
version = "2.5.3" version = "2.5.3"
@@ -656,12 +568,6 @@ dependencies = [
"instant", "instant",
] ]
[[package]]
name = "fastrand"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "37909eebbb50d72f9059c3b6d82c0463f2ff062c9e95845c43a6c9c0355411be"
[[package]] [[package]]
name = "find-msvc-tools" name = "find-msvc-tools"
version = "0.1.5" version = "0.1.5"
@@ -776,7 +682,7 @@ version = "1.13.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "49a9d51ce47660b1e808d3c990b4709f2f415d928835a17dfd16991515c46bce" checksum = "49a9d51ce47660b1e808d3c990b4709f2f415d928835a17dfd16991515c46bce"
dependencies = [ dependencies = [
"fastrand 1.9.0", "fastrand",
"futures-core", "futures-core",
"futures-io", "futures-io",
"memchr", "memchr",
@@ -864,18 +770,6 @@ dependencies = [
"wasi 0.11.1+wasi-snapshot-preview1", "wasi 0.11.1+wasi-snapshot-preview1",
] ]
[[package]]
name = "getrandom"
version = "0.3.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd"
dependencies = [
"cfg-if",
"libc",
"r-efi",
"wasip2",
]
[[package]] [[package]]
name = "ghash" name = "ghash"
version = "0.5.1" version = "0.5.1"
@@ -949,15 +843,6 @@ version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fc0fef456e4baa96da950455cd02c081ca953b141298e41db3fc7e36b1da849c" checksum = "fc0fef456e4baa96da950455cd02c081ca953b141298e41db3fc7e36b1da849c"
[[package]]
name = "hkdf"
version = "0.12.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7b5f8eb2ad728638ea2c7d47a21db23b7b58a72ed6a38256b8a1849f15fbbdf7"
dependencies = [
"hmac",
]
[[package]] [[package]]
name = "hmac" name = "hmac"
version = "0.12.1" version = "0.12.1"
@@ -1269,24 +1154,12 @@ version = "0.2.177"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2874a2af47a2325c2001a6e6fad9b16a53b802102b528163885171cf92b15976" checksum = "2874a2af47a2325c2001a6e6fad9b16a53b802102b528163885171cf92b15976"
[[package]]
name = "linux-raw-sys"
version = "0.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "df1d3c3b53da64cf5760482273a98e575c651a67eec7f77df96b5b642de8f039"
[[package]] [[package]]
name = "litemap" name = "litemap"
version = "0.8.1" version = "0.8.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6373607a59f0be73a39b6fe456b8192fcc3585f602af20751600e974dd455e77" checksum = "6373607a59f0be73a39b6fe456b8192fcc3585f602af20751600e974dd455e77"
[[package]]
name = "litrs"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "11d3d7f243d5c5a8b9bb5d6dd2b1602c0cb0b9db1621bafc7ed66e35ff9fe092"
[[package]] [[package]]
name = "lock_api" name = "lock_api"
version = "0.4.14" version = "0.4.14"
@@ -1587,12 +1460,6 @@ dependencies = [
"proc-macro2", "proc-macro2",
] ]
[[package]]
name = "r-efi"
version = "5.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f"
[[package]] [[package]]
name = "radium" name = "radium"
version = "0.7.0" version = "0.7.0"
@@ -1839,19 +1706,6 @@ dependencies = [
"serde_json", "serde_json",
] ]
[[package]]
name = "rustix"
version = "1.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cd15f8a2c5551a84d56efdc1cd049089e409ac19a3072d5037a17fd70719ff3e"
dependencies = [
"bitflags 2.10.0",
"errno",
"libc",
"linux-raw-sys",
"windows-sys 0.61.2",
]
[[package]] [[package]]
name = "rustls" name = "rustls"
version = "0.21.12" version = "0.21.12"
@@ -2003,12 +1857,6 @@ dependencies = [
"lazy_static", "lazy_static",
] ]
[[package]]
name = "shell-words"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "24188a676b6ae68c3b2cb3a01be17fbf7240ce009799bb56d5b1409051e78fde"
[[package]] [[package]]
name = "shlex" name = "shlex"
version = "1.3.0" version = "1.3.0"
@@ -2155,28 +2003,6 @@ dependencies = [
"pin-utils", "pin-utils",
] ]
[[package]]
name = "temp-env"
version = "0.3.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "96374855068f47402c3121c6eed88d29cb1de8f3ab27090e273e420bdabcf050"
dependencies = [
"parking_lot",
]
[[package]]
name = "tempfile"
version = "3.23.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2d31c77bdf42a745371d260a26ca7163f1e0924b64afa0b688e61b5a9fa02f16"
dependencies = [
"fastrand 2.3.0",
"getrandom 0.3.4",
"once_cell",
"rustix",
"windows-sys 0.61.2",
]
[[package]] [[package]]
name = "termtree" name = "termtree"
version = "0.5.1" version = "0.5.1"
@@ -2433,18 +2259,6 @@ version = "1.0.22"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9312f7c4f6ff9069b165498234ce8be658059c6728633667c526e27dc2cf1df5" checksum = "9312f7c4f6ff9069b165498234ce8be658059c6728633667c526e27dc2cf1df5"
[[package]]
name = "unicode-segmentation"
version = "1.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f6ccf251212114b54433ec949fd6a7841275f9ada20dddd2f29e9ceea4501493"
[[package]]
name = "unicode-width"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254"
[[package]] [[package]]
name = "universal-hash" name = "universal-hash"
version = "0.5.1" version = "0.5.1"
@@ -2534,15 +2348,6 @@ version = "0.11.1+wasi-snapshot-preview1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b"
[[package]]
name = "wasip2"
version = "1.0.1+wasi-0.2.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0562428422c63773dad2c345a1882263bbf4d65cf3f42e90921f787ef5ad58e7"
dependencies = [
"wit-bindgen",
]
[[package]] [[package]]
name = "wasm-bindgen" name = "wasm-bindgen"
version = "0.2.105" version = "0.2.105"
@@ -2617,28 +2422,6 @@ version = "0.25.4"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5f20c57d8d7db6d3b86154206ae5d8fba62dd39573114de97c2cb0578251f8e1" checksum = "5f20c57d8d7db6d3b86154206ae5d8fba62dd39573114de97c2cb0578251f8e1"
[[package]]
name = "winapi"
version = "0.3.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419"
dependencies = [
"winapi-i686-pc-windows-gnu",
"winapi-x86_64-pc-windows-gnu",
]
[[package]]
name = "winapi-i686-pc-windows-gnu"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6"
[[package]]
name = "winapi-x86_64-pc-windows-gnu"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
[[package]] [[package]]
name = "windows-core" name = "windows-core"
version = "0.62.2" version = "0.62.2"
@@ -2961,12 +2744,6 @@ dependencies = [
"tokio", "tokio",
] ]
[[package]]
name = "wit-bindgen"
version = "0.46.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f17a85883d4e6d00e8a97c586de764dabcc06133f7f1d55dce5cdc070ad7fe59"
[[package]] [[package]]
name = "writeable" name = "writeable"
version = "0.6.2" version = "0.6.2"
@@ -3046,12 +2823,6 @@ dependencies = [
"synstructure", "synstructure",
] ]
[[package]]
name = "zeroize"
version = "1.8.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b97154e67e32c85465826e8bcc1c59429aaaf107c1e4a9e53c8d8ccd5eff88d0"
[[package]] [[package]]
name = "zerotrie" name = "zerotrie"
version = "0.2.3" version = "0.2.3"

View File

@@ -24,7 +24,7 @@ rust_decimal = { version = "1.33", features = ["serde-float"] }
async-trait = "0.1" async-trait = "0.1"
dotenvy = "0.15" dotenvy = "0.15"
clap = { version = "4.4", features = ["derive", "env"] } clap = { version = "4.4", features = ["derive", "env"] }
reqwest = { version = "0.11", default-features = false, features = ["json", "rustls-tls"] } reqwest = { version = "0.11", default-features = false, features = ["json", "multipart", "rustls-tls"] }
url = "2.5" url = "2.5"
wiremock = "0.5" wiremock = "0.5"
tokio-test = "0.4" tokio-test = "0.4"
@@ -32,14 +32,3 @@ mockall = "0.11"
reqwest-middleware = "0.2" reqwest-middleware = "0.2"
hyper = { version = "0.14", features = ["full"] } hyper = { version = "0.14", features = ["full"] }
bytes = "1.0" bytes = "1.0"
comfy-table = "7.1"
http = "0.2"
task-local-extensions = "0.1"
aes-gcm = "0.10"
pbkdf2 = "0.12"
hkdf = "0.12"
rand = "0.8"
sha2 = "0.10"
temp-env = "0.3"
dialoguer = "0.12"
walkdir = "2.4"

View File

@@ -1,6 +1,6 @@
# Banks2FF # Banks2FF
A robust command-line tool to synchronize bank transactions between various sources and destinations. Currently supports GoCardless (formerly Nordigen) to Firefly III, with extensible architecture for additional sources and destinations. A robust command-line tool to synchronize bank transactions from GoCardless (formerly Nordigen) to Firefly III.
## ✨ Key Benefits ## ✨ Key Benefits
@@ -11,7 +11,6 @@ A robust command-line tool to synchronize bank transactions between various sour
- **Reliable Operation**: Continues working even when some accounts need attention - **Reliable Operation**: Continues working even when some accounts need attention
- **Safe Preview Mode**: Test changes before applying them to your finances - **Safe Preview Mode**: Test changes before applying them to your finances
- **Rate Limit Aware**: Works within API limits to ensure consistent access - **Rate Limit Aware**: Works within API limits to ensure consistent access
- **Smart Account Linking**: Automatically match bank accounts to Firefly III accounts, with interactive and intelligent manual linking options
## 🚀 Quick Start ## 🚀 Quick Start
@@ -32,90 +31,24 @@ A robust command-line tool to synchronize bank transactions between various sour
### Usage ### Usage
```bash ```bash
# Sync all accounts (automatic date range) # Sync all accounts (automatic date range)
cargo run -p banks2ff -- sync gocardless firefly cargo run -p banks2ff
# Preview changes without saving # Preview changes without saving
cargo run -p banks2ff -- --dry-run sync gocardless firefly cargo run -p banks2ff -- --dry-run
# Sync specific date range # Sync specific date range
cargo run -p banks2ff -- sync gocardless firefly --start 2023-01-01 --end 2023-01-31 cargo run -p banks2ff -- --start 2023-01-01 --end 2023-01-31
# List available sources and destinations
cargo run -p banks2ff -- sources
cargo run -p banks2ff -- destinations
# Inspect accounts
cargo run -p banks2ff -- accounts list
cargo run -p banks2ff -- accounts list gocardless # Only GoCardless accounts
cargo run -p banks2ff -- accounts list firefly # Only Firefly III accounts
cargo run -p banks2ff -- accounts status
# Manage account links
cargo run -p banks2ff -- accounts link list
cargo run -p banks2ff -- accounts link create # Interactive mode - guided account selection
cargo run -p banks2ff -- accounts link create "Account Name" # Smart mode - auto-detect source/destination
cargo run -p banks2ff -- accounts link create <source> <dest> # Direct mode - for scripts
# Inspect transactions and cache
cargo run -p banks2ff -- transactions list <account_id>
cargo run -p banks2ff -- transactions cache-status
``` ```
## 🖥️ CLI Structure
Banks2FF uses a structured command-line interface with the following commands:
- `sync <SOURCE> <DESTINATION>` - Synchronize transactions between source and destination
- `sources` - List all available source types
- `destinations` - List all available destination types
- `accounts list [gocardless|firefly]` - List all discovered accounts (optionally filter by adapter type)
- `accounts status` - Show sync status for all accounts
- `accounts link` - Manage account links between sources and destinations (with interactive and smart modes)
- `transactions list <account_id>` - Show transaction information for a specific account
- `transactions cache-status` - Display cache status and statistics
- `transactions clear-cache` - Clear transaction cache (implementation pending)
Use `cargo run -p banks2ff -- --help` for detailed command information.
## 📋 What It Does ## 📋 What It Does
Banks2FF automatically: Banks2FF automatically:
1. Connects to your bank accounts via GoCardless 1. Connects to your bank accounts via GoCardless
2. Discovers accounts and provides intelligent linking between GoCardless and Firefly III 2. Finds matching accounts in your Firefly III instance
3. Downloads new transactions since your last sync 3. Downloads new transactions since your last sync
4. Adds them to Firefly III (avoiding duplicates) 4. Adds them to Firefly III (avoiding duplicates)
5. Handles errors gracefully - keeps working even if some accounts have issues 5. Handles errors gracefully - keeps working even if some accounts have issues
The account linking system automatically matches accounts by IBAN, but also provides interactive tools for manual linking when needed.
## 🔗 Smart Account Linking
Banks2FF provides multiple ways to link your bank accounts to Firefly III accounts:
### Interactive Mode
```bash
cargo run -p banks2ff -- accounts link create
```
Shows you unlinked bank accounts and guides you through selecting destination accounts with human-readable names.
### Smart Resolution
```bash
cargo run -p banks2ff -- accounts link create "Main Checking"
```
Automatically finds accounts by name, IBAN, or ID and shows appropriate linking options.
### Direct Linking (for Scripts)
```bash
cargo run -p banks2ff -- accounts link create <source_id> <destination_id>
```
Perfect for automation - uses exact account IDs for reliable scripting.
### Key Features
- **Auto-Linking**: Automatically matches accounts with identical IBANs during sync
- **Manual Override**: Create custom links when auto-matching isn't sufficient
- **Constraint Enforcement**: One bank account can only link to one Firefly account (prevents duplicates)
- **Human-Friendly**: Uses account names and masked IBANs for easy identification
## 🔐 Secure Transaction Caching ## 🔐 Secure Transaction Caching
Banks2FF automatically caches your transaction data to make future syncs much faster: Banks2FF automatically caches your transaction data to make future syncs much faster:
@@ -129,10 +62,7 @@ The cache requires `BANKS2FF_CACHE_KEY` to be set in your `.env` file for secure
## 🔧 Troubleshooting ## 🔧 Troubleshooting
- **Unknown source/destination?** Use `sources` and `destinations` commands to see what's available - **Account not syncing?** Check that the IBAN matches between GoCardless and Firefly III
- **Account not syncing?** Check that the IBAN matches between GoCardless and Firefly III, or use `accounts link create` for interactive linking
- **Can't find account by name?** Use `accounts list` to see all available accounts with their IDs and names
- **Link creation failed?** Each bank account can only link to one Firefly account - check existing links with `accounts link list`
- **Missing transactions?** The tool syncs from the last transaction date forward - **Missing transactions?** The tool syncs from the last transaction date forward
- **Rate limited?** The tool automatically handles API limits and retries appropriately - **Rate limited?** The tool automatically handles API limits and retries appropriately

View File

@@ -29,20 +29,14 @@ gocardless-client = { path = "../gocardless-client" }
reqwest-middleware = { workspace = true } reqwest-middleware = { workspace = true }
hyper = { workspace = true } hyper = { workspace = true }
bytes = { workspace = true } bytes = { workspace = true }
http = { workspace = true } http = "0.2"
task-local-extensions = { workspace = true } task-local-extensions = "0.1"
# Encryption dependencies # Encryption dependencies
aes-gcm = { workspace = true } aes-gcm = "0.10"
pbkdf2 = { workspace = true } pbkdf2 = "0.12"
hkdf = { workspace = true } rand = "0.8"
rand = { workspace = true } sha2 = "0.10"
sha2 = { workspace = true }
# CLI formatting dependencies
comfy-table = { workspace = true }
dialoguer = { workspace = true }
[dev-dependencies] [dev-dependencies]
mockall = { workspace = true } mockall = { workspace = true }
temp-env = { workspace = true }

View File

@@ -1,42 +1,84 @@
use crate::core::config::Config;
use crate::core::models::{Account, AccountSummary, BankTransaction};
use crate::core::ports::{TransactionDestination, TransactionMatch};
use anyhow::Result;
use async_trait::async_trait; use async_trait::async_trait;
use chrono::NaiveDate; use anyhow::Result;
use tracing::instrument;
use crate::core::ports::{TransactionDestination, TransactionMatch};
use crate::core::models::BankTransaction;
use firefly_client::client::FireflyClient; use firefly_client::client::FireflyClient;
use firefly_client::models::{ use firefly_client::models::{TransactionStore, TransactionSplitStore, TransactionUpdate, TransactionSplitUpdate};
TransactionSplitStore, TransactionSplitUpdate, TransactionStore, TransactionUpdate,
};
use rust_decimal::Decimal;
use std::str::FromStr;
use std::sync::Arc; use std::sync::Arc;
use tokio::sync::Mutex; use tokio::sync::Mutex;
use tracing::instrument; use rust_decimal::Decimal;
use std::str::FromStr;
use chrono::NaiveDate;
pub struct FireflyAdapter { pub struct FireflyAdapter {
client: Arc<Mutex<FireflyClient>>, client: Arc<Mutex<FireflyClient>>,
config: Config,
} }
impl FireflyAdapter { impl FireflyAdapter {
pub fn new(client: FireflyClient, config: Config) -> Self { pub fn new(client: FireflyClient) -> Self {
Self { Self {
client: Arc::new(Mutex::new(client)), client: Arc::new(Mutex::new(client)),
config,
} }
} }
} }
#[async_trait] #[async_trait]
impl TransactionDestination for FireflyAdapter { impl TransactionDestination for FireflyAdapter {
#[instrument(skip(self))]
async fn resolve_account_id(&self, iban: &str) -> Result<Option<String>> {
let client = self.client.lock().await;
let accounts = client.search_accounts(iban).await?;
// Look for exact match on IBAN, ensuring account is active
for acc in accounts.data {
// Filter for active accounts only (default is usually active, but let's check if attribute exists)
// Note: The Firefly API spec v6.4.4 Account object has 'active' attribute as boolean.
let is_active = acc.attributes.active.unwrap_or(true);
if !is_active {
continue;
}
if let Some(acc_iban) = acc.attributes.iban {
if acc_iban.replace(" ", "") == iban.replace(" ", "") {
return Ok(Some(acc.id));
}
}
}
Ok(None)
}
#[instrument(skip(self))]
async fn get_active_account_ibans(&self) -> Result<Vec<String>> {
let client = self.client.lock().await;
// Get all asset accounts. Note: Pagination might be needed if user has > 50 accounts.
// For typical users, 50 is enough. If needed we can loop pages.
// The client `get_accounts` method hardcodes limit=default. We should probably expose a list_all method or loop here.
// For now, let's assume page 1 covers it or use search.
let accounts = client.get_accounts("").await?; // Argument ignored in current impl
let mut ibans = Vec::new();
for acc in accounts.data {
let is_active = acc.attributes.active.unwrap_or(true);
if is_active {
if let Some(iban) = acc.attributes.iban {
if !iban.is_empty() {
ibans.push(iban);
}
}
}
}
Ok(ibans)
}
#[instrument(skip(self))] #[instrument(skip(self))]
async fn get_last_transaction_date(&self, account_id: &str) -> Result<Option<NaiveDate>> { async fn get_last_transaction_date(&self, account_id: &str) -> Result<Option<NaiveDate>> {
let client = self.client.lock().await; let client = self.client.lock().await;
// Fetch latest 1 transaction // Fetch latest 1 transaction
let tx_list = client let tx_list = client.list_account_transactions(account_id, None, None).await?;
.list_account_transactions(account_id, None, None)
.await?;
if let Some(first) = tx_list.data.first() { if let Some(first) = tx_list.data.first() {
if let Some(split) = first.attributes.transactions.first() { if let Some(split) = first.attributes.transactions.first() {
@@ -51,24 +93,18 @@ impl TransactionDestination for FireflyAdapter {
} }
#[instrument(skip(self))] #[instrument(skip(self))]
async fn find_transaction( async fn find_transaction(&self, account_id: &str, tx: &BankTransaction) -> Result<Option<TransactionMatch>> {
&self,
account_id: &str,
tx: &BankTransaction,
) -> Result<Option<TransactionMatch>> {
let client = self.client.lock().await; let client = self.client.lock().await;
// Search window: +/- 3 days // Search window: +/- 3 days
let start_date = tx.date - chrono::Duration::days(3); let start_date = tx.date - chrono::Duration::days(3);
let end_date = tx.date + chrono::Duration::days(3); let end_date = tx.date + chrono::Duration::days(3);
let tx_list = client let tx_list = client.list_account_transactions(
.list_account_transactions(
account_id, account_id,
Some(&start_date.format("%Y-%m-%d").to_string()), Some(&start_date.format("%Y-%m-%d").to_string()),
Some(&end_date.format("%Y-%m-%d").to_string()), Some(&end_date.format("%Y-%m-%d").to_string())
) ).await?;
.await?;
// Filter logic // Filter logic
for existing_tx in tx_list.data { for existing_tx in tx_list.data {
@@ -119,26 +155,10 @@ impl TransactionDestination for FireflyAdapter {
date: tx.date.format("%Y-%m-%d").to_string(), date: tx.date.format("%Y-%m-%d").to_string(),
amount: tx.amount.abs().to_string(), amount: tx.amount.abs().to_string(),
description: tx.description.clone(), description: tx.description.clone(),
source_id: if !is_credit { source_id: if !is_credit { Some(account_id.to_string()) } else { None },
Some(account_id.to_string()) source_name: if is_credit { tx.counterparty_name.clone().or(Some("Unknown Sender".to_string())) } else { None },
} else { destination_id: if is_credit { Some(account_id.to_string()) } else { None },
None destination_name: if !is_credit { tx.counterparty_name.clone().or(Some("Unknown Recipient".to_string())) } else { None },
},
source_name: if is_credit {
tx.counterparty_name.clone()
} else {
None
},
destination_id: if is_credit {
Some(account_id.to_string())
} else {
None
},
destination_name: if !is_credit {
tx.counterparty_name.clone()
} else {
None
},
currency_code: Some(tx.currency.clone()), currency_code: Some(tx.currency.clone()),
foreign_amount: tx.foreign_amount.map(|d| d.abs().to_string()), foreign_amount: tx.foreign_amount.map(|d| d.abs().to_string()),
foreign_currency_code: tx.foreign_currency.clone(), foreign_currency_code: tx.foreign_currency.clone(),
@@ -163,109 +183,6 @@ impl TransactionDestination for FireflyAdapter {
external_id: Some(external_id.to_string()), external_id: Some(external_id.to_string()),
}], }],
}; };
client client.update_transaction(id, update).await.map_err(|e| e.into())
.update_transaction(id, update)
.await
.map_err(|e| e.into())
}
#[instrument(skip(self))]
async fn discover_accounts(&self) -> Result<Vec<Account>> {
let client = self.client.lock().await;
let accounts = client.get_accounts("").await?;
let mut result = Vec::new();
// Cache the accounts
let encryption = crate::core::encryption::Encryption::new(self.config.cache.key.clone());
let mut cache =
crate::core::cache::AccountCache::load(self.config.cache.directory.clone(), encryption);
for acc in accounts.data {
let is_active = acc.attributes.active.unwrap_or(true);
if is_active {
// Cache the full account details
let ff_account = crate::core::cache::FireflyAccount {
id: acc.id.clone(),
name: acc.attributes.name.clone(),
account_type: acc.attributes.account_type.clone(),
iban: acc.attributes.iban.clone(),
active: acc.attributes.active,
order: acc.attributes.order,
created_at: acc.attributes.created_at.clone(),
account_role: acc.attributes.account_role.clone(),
object_group_id: acc.attributes.object_group_id.clone(),
object_group_title: acc.attributes.object_group_title.clone(),
object_group_order: acc.attributes.object_group_order,
currency_id: acc.attributes.currency_id.clone(),
currency_name: acc.attributes.currency_name.clone(),
currency_code: acc.attributes.currency_code.clone(),
currency_symbol: acc.attributes.currency_symbol.clone(),
currency_decimal_places: acc.attributes.currency_decimal_places,
primary_currency_id: acc.attributes.primary_currency_id.clone(),
primary_currency_name: acc.attributes.primary_currency_name.clone(),
primary_currency_code: acc.attributes.primary_currency_code.clone(),
primary_currency_symbol: acc.attributes.primary_currency_symbol.clone(),
primary_currency_decimal_places: acc.attributes.primary_currency_decimal_places,
opening_balance: acc.attributes.opening_balance.clone(),
pc_opening_balance: acc.attributes.pc_opening_balance.clone(),
debt_amount: acc.attributes.debt_amount.clone(),
pc_debt_amount: acc.attributes.pc_debt_amount.clone(),
notes: acc.attributes.notes.clone(),
monthly_payment_date: acc.attributes.monthly_payment_date.clone(),
credit_card_type: acc.attributes.credit_card_type.clone(),
account_number: acc.attributes.account_number.clone(),
bic: acc.attributes.bic.clone(),
opening_balance_date: acc.attributes.opening_balance_date.clone(),
liability_type: acc.attributes.liability_type.clone(),
liability_direction: acc.attributes.liability_direction.clone(),
interest: acc.attributes.interest.clone(),
interest_period: acc.attributes.interest_period.clone(),
include_net_worth: acc.attributes.include_net_worth,
longitude: acc.attributes.longitude,
latitude: acc.attributes.latitude,
zoom_level: acc.attributes.zoom_level,
last_activity: acc.attributes.last_activity.clone(),
};
cache.insert(crate::core::cache::CachedAccount::Firefly(Box::new(
ff_account,
)));
cache.save();
result.push(Account {
id: acc.id,
name: Some(acc.attributes.name),
iban: acc.attributes.iban,
currency: "EUR".to_string(),
});
}
}
Ok(result)
}
async fn list_accounts(&self) -> Result<Vec<AccountSummary>> {
let encryption = crate::core::encryption::Encryption::new(self.config.cache.key.clone());
let cache =
crate::core::cache::AccountCache::load(self.config.cache.directory.clone(), encryption);
let mut summaries = Vec::new();
// Use cached account data for display
for (account_id, cached_account) in &cache.accounts {
if let crate::core::cache::CachedAccount::Firefly(ff_account) = cached_account {
let summary = AccountSummary {
id: account_id.clone(),
name: Some(ff_account.name.clone()),
iban: ff_account.iban.clone().unwrap_or_else(|| "".to_string()),
currency: ff_account
.currency_code
.clone()
.unwrap_or_else(|| "EUR".to_string()),
};
summaries.push(summary);
}
}
Ok(summaries)
} }
} }

View File

@@ -0,0 +1,59 @@
use std::collections::HashMap;
use std::fs;
use std::path::Path;
use serde::{Deserialize, Serialize};
use tracing::warn;
use crate::adapters::gocardless::encryption::Encryption;
#[derive(Debug, Serialize, Deserialize, Default)]
pub struct AccountCache {
/// Map of Account ID -> IBAN
pub accounts: HashMap<String, String>,
}
impl AccountCache {
fn get_path() -> String {
let cache_dir = std::env::var("BANKS2FF_CACHE_DIR").unwrap_or_else(|_| "data/cache".to_string());
format!("{}/accounts.enc", cache_dir)
}
pub fn load() -> Self {
let path = Self::get_path();
if Path::new(&path).exists() {
match fs::read(&path) {
Ok(encrypted_data) => match Encryption::decrypt(&encrypted_data) {
Ok(json_data) => match serde_json::from_slice(&json_data) {
Ok(cache) => return cache,
Err(e) => warn!("Failed to parse cache file: {}", e),
},
Err(e) => warn!("Failed to decrypt cache file: {}", e),
},
Err(e) => warn!("Failed to read cache file: {}", e),
}
}
Self::default()
}
pub fn save(&self) {
let path = Self::get_path();
match serde_json::to_vec(self) {
Ok(json_data) => match Encryption::encrypt(&json_data) {
Ok(encrypted_data) => {
if let Err(e) = fs::write(&path, encrypted_data) {
warn!("Failed to write cache file: {}", e);
}
},
Err(e) => warn!("Failed to encrypt cache: {}", e),
},
Err(e) => warn!("Failed to serialize cache: {}", e),
}
}
pub fn get_iban(&self, account_id: &str) -> Option<String> {
self.accounts.get(account_id).cloned()
}
pub fn insert(&mut self, account_id: String, iban: String) {
self.accounts.insert(account_id, iban);
}
}

View File

@@ -1,42 +1,30 @@
use crate::adapters::gocardless::mapper::map_transaction;
use crate::adapters::gocardless::transaction_cache::AccountTransactionCache;
use crate::core::cache::{AccountCache, CachedAccount, GoCardlessAccount};
use crate::core::config::Config;
use crate::core::encryption::Encryption;
use crate::core::models::{
Account, AccountStatus, AccountSummary, BankTransaction, CacheInfo, TransactionInfo,
};
use crate::core::ports::TransactionSource;
use anyhow::Result;
use async_trait::async_trait; use async_trait::async_trait;
use chrono::NaiveDate; use chrono::NaiveDate;
use anyhow::Result;
use tracing::{info, instrument, warn};
use crate::core::ports::TransactionSource;
use crate::core::models::{Account, BankTransaction};
use crate::adapters::gocardless::mapper::map_transaction;
use crate::adapters::gocardless::cache::AccountCache;
use crate::adapters::gocardless::transaction_cache::AccountTransactionCache;
use gocardless_client::client::GoCardlessClient; use gocardless_client::client::GoCardlessClient;
use tracing::{debug, info, instrument, warn};
use std::collections::HashMap;
use std::sync::Arc; use std::sync::Arc;
use std::collections::HashMap;
use tokio::sync::Mutex; use tokio::sync::Mutex;
pub struct GoCardlessAdapter { pub struct GoCardlessAdapter {
client: Arc<Mutex<GoCardlessClient>>, client: Arc<Mutex<GoCardlessClient>>,
cache: Arc<Mutex<AccountCache>>, cache: Arc<Mutex<AccountCache>>,
transaction_caches: Arc<Mutex<HashMap<String, AccountTransactionCache>>>, transaction_caches: Arc<Mutex<HashMap<String, AccountTransactionCache>>>,
config: Config,
encryption: Encryption,
} }
impl GoCardlessAdapter { impl GoCardlessAdapter {
pub fn new(client: GoCardlessClient, config: Config) -> Self { pub fn new(client: GoCardlessClient) -> Self {
let encryption = Encryption::new(config.cache.key.clone());
Self { Self {
client: Arc::new(Mutex::new(client)), client: Arc::new(Mutex::new(client)),
cache: Arc::new(Mutex::new(AccountCache::load( cache: Arc::new(Mutex::new(AccountCache::load())),
config.cache.directory.clone(),
encryption.clone(),
))),
transaction_caches: Arc::new(Mutex::new(HashMap::new())), transaction_caches: Arc::new(Mutex::new(HashMap::new())),
config,
encryption,
} }
} }
} }
@@ -74,20 +62,14 @@ impl TransactionSource for GoCardlessAdapter {
if let Some(agreement_id) = &req.agreement { if let Some(agreement_id) = &req.agreement {
match client.is_agreement_expired(agreement_id).await { match client.is_agreement_expired(agreement_id).await {
Ok(true) => { Ok(true) => {
debug!( warn!("Skipping requisition {} - agreement {} has expired", req.id, agreement_id);
"Skipping requisition {} - agreement {} has expired",
req.id, agreement_id
);
continue; continue;
} }
Ok(false) => { Ok(false) => {
// Agreement is valid, proceed // Agreement is valid, proceed
} }
Err(e) => { Err(e) => {
warn!( warn!("Failed to check agreement {} expiry: {}. Skipping requisition.", agreement_id, e);
"Failed to check agreement {} expiry: {}. Skipping requisition.",
agreement_id, e
);
continue; continue;
} }
} }
@@ -95,42 +77,18 @@ impl TransactionSource for GoCardlessAdapter {
if let Some(req_accounts) = req.accounts { if let Some(req_accounts) = req.accounts {
for acc_id in req_accounts { for acc_id in req_accounts {
// Always fetch fresh account data during sync // 1. Check Cache
let mut iban_opt = cache.get_iban(&acc_id);
// 2. Fetch if missing
if iban_opt.is_none() {
match client.get_account(&acc_id).await { match client.get_account(&acc_id).await {
Ok(basic_account) => { Ok(details) => {
// Also try to fetch account details let new_iban = details.iban.unwrap_or_default();
let details_result = client.get_account_details(&acc_id).await; cache.insert(acc_id.clone(), new_iban.clone());
let gc_account = GoCardlessAccount {
id: basic_account.id.clone(),
iban: basic_account.iban,
owner_name: basic_account.owner_name,
status: basic_account.status,
institution_id: basic_account.institution_id,
created: basic_account.created,
last_accessed: basic_account.last_accessed,
// Include details if available
name: details_result
.as_ref()
.ok()
.and_then(|d| d.account.name.clone()),
display_name: details_result
.as_ref()
.ok()
.and_then(|d| d.account.display_name.clone()),
product: details_result
.as_ref()
.ok()
.and_then(|d| d.account.product.clone()),
cash_account_type: details_result
.as_ref()
.ok()
.and_then(|d| d.account.cash_account_type.clone()),
};
cache.insert(CachedAccount::GoCardless(Box::new(gc_account)));
cache.save(); cache.save();
} iban_opt = Some(new_iban);
},
Err(e) => { Err(e) => {
// If rate limit hit here, we might want to skip this account and continue? // If rate limit hit here, we might want to skip this account and continue?
// But get_account is critical to identify the account. // But get_account is critical to identify the account.
@@ -139,12 +97,9 @@ impl TransactionSource for GoCardlessAdapter {
continue; continue;
} }
} }
}
let iban = cache let iban = iban_opt.unwrap_or_default();
.get_account_data(&acc_id)
.and_then(|acc| acc.iban())
.unwrap_or("")
.to_string();
let mut keep = true; let mut keep = true;
if let Some(ref wanted) = wanted_set { if let Some(ref wanted) = wanted_set {
@@ -156,54 +111,39 @@ impl TransactionSource for GoCardlessAdapter {
} }
if keep { if keep {
// Try to get account name from cache if available
let name = cache.get_account(&acc_id).and_then(|acc| match acc {
CachedAccount::GoCardless(gc_acc) => gc_acc.name.clone(),
_ => None,
});
accounts.push(Account { accounts.push(Account {
id: acc_id, id: acc_id,
name, iban,
iban: Some(iban),
currency: "EUR".to_string(), currency: "EUR".to_string(),
}); });
} }
// Optimization: Stop if we found all wanted accounts // Optimization: Stop if we found all wanted accounts
if wanted_set.is_some() && found_count >= target_count && target_count > 0 { if let Some(_) = wanted_set {
info!( if found_count >= target_count && target_count > 0 {
"Found all {} wanted accounts. Stopping search.", info!("Found all {} wanted accounts. Stopping search.", target_count);
target_count
);
return Ok(accounts); return Ok(accounts);
} }
} }
} }
} }
}
info!("Found {} matching accounts in GoCardless", accounts.len()); info!("Found {} matching accounts in GoCardless", accounts.len());
Ok(accounts) Ok(accounts)
} }
#[instrument(skip(self))] #[instrument(skip(self))]
async fn get_transactions( async fn get_transactions(&self, account_id: &str, start: NaiveDate, end: NaiveDate) -> Result<Vec<BankTransaction>> {
&self,
account_id: &str,
start: NaiveDate,
end: NaiveDate,
) -> Result<Vec<BankTransaction>> {
let mut client = self.client.lock().await; let mut client = self.client.lock().await;
client.obtain_access_token().await?; client.obtain_access_token().await?;
// Load or get transaction cache // Load or get transaction cache
let mut caches = self.transaction_caches.lock().await; let mut caches = self.transaction_caches.lock().await;
let cache = caches.entry(account_id.to_string()).or_insert_with(|| { let cache = caches.entry(account_id.to_string()).or_insert_with(|| {
let encryption = self.encryption.clone(); AccountTransactionCache::load(account_id).unwrap_or_else(|_| AccountTransactionCache {
let cache_dir = self.config.cache.directory.clone(); account_id: account_id.to_string(),
AccountTransactionCache::load(account_id, cache_dir.clone(), encryption.clone()) ranges: Vec::new(),
.unwrap_or_else(|_| {
AccountTransactionCache::new(account_id.to_string(), cache_dir, encryption)
}) })
}); });
@@ -215,43 +155,27 @@ impl TransactionSource for GoCardlessAdapter {
// Fetch missing ranges // Fetch missing ranges
for (range_start, range_end) in uncovered_ranges { for (range_start, range_end) in uncovered_ranges {
let response_result = client let response_result = client.get_transactions(
.get_transactions(
account_id, account_id,
Some(&range_start.to_string()), Some(&range_start.to_string()),
Some(&range_end.to_string()), Some(&range_end.to_string())
) ).await;
.await;
match response_result { match response_result {
Ok(response) => { Ok(response) => {
let raw_txs = response.transactions.booked.clone(); let raw_txs = response.transactions.booked.clone();
raw_transactions.extend(raw_txs.clone()); raw_transactions.extend(raw_txs.clone());
cache.store_transactions(range_start, range_end, raw_txs); cache.store_transactions(range_start, range_end, raw_txs);
info!( info!("Fetched {} transactions for account {} in range {}-{}", response.transactions.booked.len(), account_id, range_start, range_end);
"Fetched {} transactions for account {} in range {}-{}", },
response.transactions.booked.len(),
account_id,
range_start,
range_end
);
}
Err(e) => { Err(e) => {
let err_str = e.to_string(); let err_str = e.to_string();
if err_str.contains("429") { if err_str.contains("429") {
warn!( warn!("Rate limit reached for account {} in range {}-{}. Skipping.", account_id, range_start, range_end);
"Rate limit reached for account {} in range {}-{}. Skipping.",
account_id, range_start, range_end
);
continue; continue;
} }
if err_str.contains("401") if err_str.contains("401") && (err_str.contains("expired") || err_str.contains("EUA")) {
&& (err_str.contains("expired") || err_str.contains("EUA")) warn!("EUA expired for account {} in range {}-{}. Skipping.", account_id, range_start, range_end);
{
debug!(
"EUA expired for account {} in range {}-{}. Skipping.",
account_id, range_start, range_end
);
continue; continue;
} }
return Err(e.into()); return Err(e.into());
@@ -271,389 +195,7 @@ impl TransactionSource for GoCardlessAdapter {
} }
} }
info!( info!("Total {} transactions for account {} in range {}-{}", transactions.len(), account_id, start, end);
"Total {} transactions for account {} in range {}-{}",
transactions.len(),
account_id,
start,
end
);
Ok(transactions) Ok(transactions)
} }
#[instrument(skip(self))]
async fn list_accounts(&self) -> Result<Vec<AccountSummary>> {
let cache = self.cache.lock().await;
let mut summaries = Vec::new();
// Use cached account data for display - only GoCardless accounts
for (account_id, cached_account) in &cache.accounts {
if let crate::core::cache::CachedAccount::GoCardless(_) = cached_account {
if let Some(account_data) = cache.get_account_data(account_id) {
let summary = AccountSummary {
id: account_id.clone(),
name: account_data.display_name(),
iban: account_data.iban().unwrap_or("").to_string(),
currency: "EUR".to_string(), // GoCardless primarily uses EUR
};
summaries.push(summary);
}
}
}
Ok(summaries)
}
#[instrument(skip(self))]
async fn get_account_status(&self) -> Result<Vec<AccountStatus>> {
let account_cache = self.cache.lock().await;
let mut statuses = Vec::new();
// Iterate through cached GoCardless accounts
for (account_id, cached_account) in &account_cache.accounts {
if let crate::core::cache::CachedAccount::GoCardless(_) = cached_account {
// Try to load the transaction cache for this account
let transaction_cache = AccountTransactionCache::load(
account_id,
self.config.cache.directory.clone(),
self.encryption.clone(),
);
let iban = account_cache
.get_account_data(account_id)
.and_then(|acc| acc.iban())
.unwrap_or("Unknown")
.to_string();
match transaction_cache {
Ok(cache) => {
let transaction_count =
cache.ranges.iter().map(|r| r.transactions.len()).sum();
let last_sync_date = cache.ranges.iter().map(|r| r.end_date).max();
statuses.push(AccountStatus {
account_id: account_id.clone(),
iban,
last_sync_date,
transaction_count,
status: if transaction_count > 0 {
"synced"
} else {
"pending"
}
.to_string(),
});
}
Err(_) => {
// No transaction cache found for this account
statuses.push(AccountStatus {
account_id: account_id.clone(),
iban,
last_sync_date: None,
transaction_count: 0,
status: "pending".to_string(),
});
}
}
}
}
Ok(statuses)
}
#[instrument(skip(self))]
async fn get_transaction_info(&self, account_id: &str) -> Result<TransactionInfo> {
let caches = self.transaction_caches.lock().await;
if let Some(cache) = caches.get(account_id) {
let total_count = cache.ranges.iter().map(|r| r.transactions.len()).sum();
let date_range = if cache.ranges.is_empty() {
None
} else {
let min_date = cache.ranges.iter().map(|r| r.start_date).min();
let max_date = cache.ranges.iter().map(|r| r.end_date).max();
min_date.and_then(|min| max_date.map(|max| (min, max)))
};
let last_updated = cache.ranges.iter().map(|r| r.end_date).max();
Ok(TransactionInfo {
account_id: account_id.to_string(),
total_count,
date_range,
last_updated,
})
} else {
Ok(TransactionInfo {
account_id: account_id.to_string(),
total_count: 0,
date_range: None,
last_updated: None,
})
}
}
#[instrument(skip(self))]
async fn get_cache_info(&self) -> Result<Vec<CacheInfo>> {
let mut infos = Vec::new();
// Account cache
let account_cache = self.cache.lock().await;
infos.push(CacheInfo {
account_id: None,
cache_type: "account".to_string(),
entry_count: account_cache.accounts.len(),
total_size_bytes: 0, // Not tracking size
last_updated: None, // Not tracking
});
// Transaction caches
let transaction_caches = self.transaction_caches.lock().await;
for (account_id, cache) in transaction_caches.iter() {
infos.push(CacheInfo {
account_id: Some(account_id.clone()),
cache_type: "transaction".to_string(),
entry_count: cache.ranges.len(),
total_size_bytes: 0, // Not tracking
last_updated: cache.ranges.iter().map(|r| r.end_date).max(),
});
}
Ok(infos)
}
#[instrument(skip(self))]
async fn discover_accounts(&self) -> Result<Vec<Account>> {
self.get_accounts(None).await
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::core::cache::{CachedAccount, GoCardlessAccount};
use crate::core::config::Config;
use gocardless_client::models::Transaction;
fn create_test_config() -> Config {
create_test_config_with_suffix("")
}
fn create_test_config_with_suffix(suffix: &str) -> Config {
Config {
gocardless: crate::core::config::GoCardlessConfig {
url: "https://test.com".to_string(),
secret_id: "test".to_string(),
secret_key: "test".to_string(),
},
firefly: crate::core::config::FireflyConfig {
url: "https://test.com".to_string(),
api_key: "test".to_string(),
},
cache: crate::core::config::CacheConfig {
directory: format!("tmp/test-cache-status{}", suffix),
key: "test-key-for-status".to_string(),
},
logging: crate::core::config::LoggingConfig {
level: "info".to_string(),
},
}
}
fn create_test_gc_account(id: &str, iban: &str) -> GoCardlessAccount {
GoCardlessAccount {
id: id.to_string(),
iban: Some(iban.to_string()),
owner_name: Some("Test Owner".to_string()),
status: Some("READY".to_string()),
institution_id: Some("TEST_BANK".to_string()),
created: Some("2024-01-01T00:00:00Z".to_string()),
last_accessed: Some("2024-01-01T00:00:00Z".to_string()),
name: Some("Test Account".to_string()),
display_name: Some("Test Display Name".to_string()),
product: Some("Test Product".to_string()),
cash_account_type: Some("CACC".to_string()),
}
}
fn create_test_transaction(id: &str, date: &str) -> Transaction {
Transaction {
transaction_id: Some(id.to_string()),
entry_reference: None,
end_to_end_id: None,
mandate_id: None,
check_id: None,
creditor_id: None,
booking_date: Some(date.to_string()),
value_date: None,
booking_date_time: None,
value_date_time: None,
transaction_amount: gocardless_client::models::TransactionAmount {
amount: "100.00".to_string(),
currency: "EUR".to_string(),
},
currency_exchange: None,
creditor_name: Some("Test Creditor".to_string()),
creditor_account: None,
ultimate_creditor: None,
debtor_name: None,
debtor_account: None,
ultimate_debtor: None,
remittance_information_unstructured: Some("Test payment".to_string()),
remittance_information_unstructured_array: None,
remittance_information_structured: None,
remittance_information_structured_array: None,
additional_information: None,
purpose_code: None,
bank_transaction_code: None,
proprietary_bank_transaction_code: None,
internal_transaction_id: None,
}
}
#[tokio::test]
async fn test_get_account_status_with_data() {
// Setup
let config = create_test_config_with_suffix("-with-data");
let _ = std::fs::remove_dir_all(&config.cache.directory); // Clean up any existing test data
// Create a mock client (we won't actually use it for this test)
let client =
gocardless_client::client::GoCardlessClient::new("https://test.com", "test", "test")
.unwrap();
let adapter = GoCardlessAdapter::new(client, config.clone());
// Add test accounts to the cache
let mut account_cache = adapter.cache.lock().await;
account_cache.insert(CachedAccount::GoCardless(Box::new(create_test_gc_account(
"acc1",
"DE12345678901234567890",
))));
account_cache.insert(CachedAccount::GoCardless(Box::new(create_test_gc_account(
"acc2",
"DE09876543210987654321",
))));
account_cache.save();
// Create transaction caches with data
let encryption = Encryption::new(config.cache.key.clone());
let mut cache1 = AccountTransactionCache::new(
"acc1".to_string(),
config.cache.directory.clone(),
encryption.clone(),
);
cache1.store_transactions(
chrono::NaiveDate::from_ymd_opt(2024, 1, 1).unwrap(),
chrono::NaiveDate::from_ymd_opt(2024, 1, 31).unwrap(),
vec![create_test_transaction("tx1", "2024-01-15")],
);
cache1.save().unwrap();
let mut cache2 = AccountTransactionCache::new(
"acc2".to_string(),
config.cache.directory.clone(),
encryption.clone(),
);
cache2.store_transactions(
chrono::NaiveDate::from_ymd_opt(2024, 2, 1).unwrap(),
chrono::NaiveDate::from_ymd_opt(2024, 2, 28).unwrap(),
vec![
create_test_transaction("tx2", "2024-02-10"),
create_test_transaction("tx3", "2024-02-20"),
],
);
cache2.save().unwrap();
drop(account_cache); // Release the lock
// Test
let statuses = adapter.get_account_status().await.unwrap();
// Verify
assert_eq!(statuses.len(), 2);
// Find status for acc1
let status1 = statuses.iter().find(|s| s.account_id == "acc1").unwrap();
assert_eq!(status1.iban, "DE12345678901234567890");
assert_eq!(status1.transaction_count, 1);
assert_eq!(status1.status, "synced");
assert_eq!(
status1.last_sync_date,
Some(chrono::NaiveDate::from_ymd_opt(2024, 1, 31).unwrap())
);
// Find status for acc2
let status2 = statuses.iter().find(|s| s.account_id == "acc2").unwrap();
assert_eq!(status2.iban, "DE09876543210987654321");
assert_eq!(status2.transaction_count, 2);
assert_eq!(status2.status, "synced");
assert_eq!(
status2.last_sync_date,
Some(chrono::NaiveDate::from_ymd_opt(2024, 2, 28).unwrap())
);
// Cleanup
let _ = std::fs::remove_dir_all(&config.cache.directory);
}
#[tokio::test]
async fn test_get_account_status_no_transaction_cache() {
// Setup
let config = create_test_config();
let _ = std::fs::remove_dir_all(&config.cache.directory);
let client =
gocardless_client::client::GoCardlessClient::new("https://test.com", "test", "test")
.unwrap();
let adapter = GoCardlessAdapter::new(client, config.clone());
// Add test account to the cache but don't create transaction cache
let mut account_cache = adapter.cache.lock().await;
account_cache.insert(CachedAccount::GoCardless(Box::new(create_test_gc_account(
"acc_no_cache",
"DE11111111111111111111",
))));
account_cache.save();
drop(account_cache);
// Test
let statuses = adapter.get_account_status().await.unwrap();
// Verify
assert_eq!(statuses.len(), 1);
let status = &statuses[0];
assert_eq!(status.account_id, "acc_no_cache");
assert_eq!(status.iban, "DE11111111111111111111");
assert_eq!(status.transaction_count, 0);
assert_eq!(status.status, "pending");
assert_eq!(status.last_sync_date, None);
// Cleanup
let _ = std::fs::remove_dir_all(&config.cache.directory);
}
#[tokio::test]
async fn test_get_account_status_empty() {
// Setup
let config = create_test_config_with_suffix("-empty");
let _ = std::fs::remove_dir_all(&config.cache.directory);
let client =
gocardless_client::client::GoCardlessClient::new("https://test.com", "test", "test")
.unwrap();
let adapter = GoCardlessAdapter::new(client, config.clone());
// Don't add any accounts to cache
// Test
let statuses = adapter.get_account_status().await.unwrap();
// Verify
assert_eq!(statuses.len(), 0);
// Cleanup
let _ = std::fs::remove_dir_all(&config.cache.directory);
}
} }

View File

@@ -0,0 +1,173 @@
//! # Encryption Module
//!
//! Provides AES-GCM encryption for sensitive cache data using PBKDF2 key derivation.
//!
//! ## Security Considerations
//!
//! - **Algorithm**: AES-GCM (Authenticated Encryption) with 256-bit keys
//! - **Key Derivation**: PBKDF2 with 200,000 iterations for brute-force resistance
//! - **Salt**: Random 16-byte salt per encryption (prepended to ciphertext)
//! - **Nonce**: Random 96-bit nonce per encryption (prepended to ciphertext)
//! - **Key Source**: Environment variable `BANKS2FF_CACHE_KEY`
//!
//! ## Data Format
//!
//! Encrypted data format: `[salt(16)][nonce(12)][ciphertext]`
//!
//! ## Security Guarantees
//!
//! - **Confidentiality**: AES-GCM encryption protects data at rest
//! - **Integrity**: GCM authentication prevents tampering
//! - **Forward Security**: Unique salt/nonce per encryption prevents rainbow tables
//! - **Key Security**: PBKDF2 slows brute-force attacks
//!
//! ## Performance
//!
//! - Encryption: ~10-50μs for typical cache payloads
//! - Key derivation: ~50-100ms (computed once per operation)
//! - Memory: Minimal additional overhead
use aes_gcm::{Aes256Gcm, Key, Nonce};
use aes_gcm::aead::{Aead, KeyInit};
use pbkdf2::pbkdf2_hmac;
use rand::RngCore;
use sha2::Sha256;
use std::env;
use anyhow::{anyhow, Result};
const KEY_LEN: usize = 32; // 256-bit key
const NONCE_LEN: usize = 12; // 96-bit nonce for AES-GCM
const SALT_LEN: usize = 16; // 128-bit salt for PBKDF2
pub struct Encryption;
impl Encryption {
/// Derive encryption key from environment variable and salt
pub fn derive_key(password: &str, salt: &[u8]) -> Key<Aes256Gcm> {
let mut key = [0u8; KEY_LEN];
pbkdf2_hmac::<Sha256>(password.as_bytes(), salt, 200_000, &mut key);
key.into()
}
/// Get password from environment variable
fn get_password() -> Result<String> {
env::var("BANKS2FF_CACHE_KEY")
.map_err(|_| anyhow!("BANKS2FF_CACHE_KEY environment variable not set"))
}
/// Encrypt data using AES-GCM
pub fn encrypt(data: &[u8]) -> Result<Vec<u8>> {
let password = Self::get_password()?;
// Generate random salt
let mut salt = [0u8; SALT_LEN];
rand::thread_rng().fill_bytes(&mut salt);
let key = Self::derive_key(&password, &salt);
let cipher = Aes256Gcm::new(&key);
// Generate random nonce
let mut nonce_bytes = [0u8; NONCE_LEN];
rand::thread_rng().fill_bytes(&mut nonce_bytes);
let nonce = Nonce::from_slice(&nonce_bytes);
// Encrypt
let ciphertext = cipher.encrypt(nonce, data)
.map_err(|e| anyhow!("Encryption failed: {}", e))?;
// Prepend salt and nonce to ciphertext: [salt(16)][nonce(12)][ciphertext]
let mut result = salt.to_vec();
result.extend(nonce_bytes);
result.extend(ciphertext);
Ok(result)
}
/// Decrypt data using AES-GCM
pub fn decrypt(encrypted_data: &[u8]) -> Result<Vec<u8>> {
let min_len = SALT_LEN + NONCE_LEN;
if encrypted_data.len() < min_len {
return Err(anyhow!("Encrypted data too short"));
}
let password = Self::get_password()?;
// Extract salt, nonce and ciphertext: [salt(16)][nonce(12)][ciphertext]
let salt = &encrypted_data[..SALT_LEN];
let nonce = Nonce::from_slice(&encrypted_data[SALT_LEN..min_len]);
let ciphertext = &encrypted_data[min_len..];
let key = Self::derive_key(&password, salt);
let cipher = Aes256Gcm::new(&key);
// Decrypt
cipher.decrypt(nonce, ciphertext)
.map_err(|e| anyhow!("Decryption failed: {}", e))
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::env;
#[test]
fn test_encrypt_decrypt_round_trip() {
// Set test environment variable
env::set_var("BANKS2FF_CACHE_KEY", "test-key-for-encryption");
let original_data = b"Hello, World! This is test data.";
// Encrypt
let encrypted = Encryption::encrypt(original_data).expect("Encryption should succeed");
// Ensure env var is still set for decryption
env::set_var("BANKS2FF_CACHE_KEY", "test-key-for-encryption");
// Decrypt
let decrypted = Encryption::decrypt(&encrypted).expect("Decryption should succeed");
// Verify
assert_eq!(original_data.to_vec(), decrypted);
assert_ne!(original_data.to_vec(), encrypted);
}
#[test]
fn test_encrypt_decrypt_different_keys() {
env::set_var("BANKS2FF_CACHE_KEY", "key1");
let data = b"Test data";
let encrypted = Encryption::encrypt(data).unwrap();
env::set_var("BANKS2FF_CACHE_KEY", "key2");
let result = Encryption::decrypt(&encrypted);
assert!(result.is_err(), "Should fail with different key");
}
#[test]
fn test_missing_env_var() {
// Save current value and restore after test
let original_value = env::var("BANKS2FF_CACHE_KEY").ok();
env::remove_var("BANKS2FF_CACHE_KEY");
let result = Encryption::get_password();
assert!(result.is_err(), "Should fail without env var");
// Restore original value
if let Some(val) = original_value {
env::set_var("BANKS2FF_CACHE_KEY", val);
}
}
#[test]
fn test_small_data() {
// Set env var multiple times to ensure it's available
env::set_var("BANKS2FF_CACHE_KEY", "test-key");
let data = b"{}"; // Minimal JSON object
env::set_var("BANKS2FF_CACHE_KEY", "test-key");
let encrypted = Encryption::encrypt(data).unwrap();
env::set_var("BANKS2FF_CACHE_KEY", "test-key");
let decrypted = Encryption::decrypt(&encrypted).unwrap();
assert_eq!(data.to_vec(), decrypted);
}
}

View File

@@ -1,19 +1,15 @@
use crate::core::models::BankTransaction;
use anyhow::Result;
use gocardless_client::models::Transaction;
use rust_decimal::prelude::Signed;
use rust_decimal::Decimal; use rust_decimal::Decimal;
use rust_decimal::prelude::Signed;
use std::str::FromStr; use std::str::FromStr;
use anyhow::Result;
use crate::core::models::BankTransaction;
use gocardless_client::models::Transaction;
pub fn map_transaction(tx: Transaction) -> Result<BankTransaction> { pub fn map_transaction(tx: Transaction) -> Result<BankTransaction> {
let internal_id = tx let internal_id = tx.transaction_id
.transaction_id
.or(tx.internal_transaction_id)
.ok_or_else(|| anyhow::anyhow!("Transaction ID missing"))?; .ok_or_else(|| anyhow::anyhow!("Transaction ID missing"))?;
let date_str = tx let date_str = tx.booking_date.or(tx.value_date)
.booking_date
.or(tx.value_date)
.ok_or_else(|| anyhow::anyhow!("Transaction date missing"))?; .ok_or_else(|| anyhow::anyhow!("Transaction date missing"))?;
let date = chrono::NaiveDate::parse_from_str(&date_str, "%Y-%m-%d")?; let date = chrono::NaiveDate::parse_from_str(&date_str, "%Y-%m-%d")?;
@@ -27,9 +23,7 @@ pub fn map_transaction(tx: Transaction) -> Result<BankTransaction> {
if let Some(exchanges) = tx.currency_exchange { if let Some(exchanges) = tx.currency_exchange {
if let Some(exchange) = exchanges.first() { if let Some(exchange) = exchanges.first() {
if let (Some(source_curr), Some(rate_str)) = if let (Some(source_curr), Some(rate_str)) = (&exchange.source_currency, &exchange.exchange_rate) {
(&exchange.source_currency, &exchange.exchange_rate)
{
foreign_currency = Some(source_curr.clone()); foreign_currency = Some(source_curr.clone());
if let Ok(rate) = Decimal::from_str(rate_str) { if let Ok(rate) = Decimal::from_str(rate_str) {
let calc = amount.abs() * rate; let calc = amount.abs() * rate;
@@ -48,8 +42,7 @@ pub fn map_transaction(tx: Transaction) -> Result<BankTransaction> {
} }
// Fallback for description: Remittance Unstructured -> Debtor/Creditor Name -> "Unknown" // Fallback for description: Remittance Unstructured -> Debtor/Creditor Name -> "Unknown"
let description = tx let description = tx.remittance_information_unstructured
.remittance_information_unstructured
.or(tx.creditor_name.clone()) .or(tx.creditor_name.clone())
.or(tx.debtor_name.clone()) .or(tx.debtor_name.clone())
.unwrap_or_else(|| "Unknown Transaction".to_string()); .unwrap_or_else(|| "Unknown Transaction".to_string());
@@ -63,20 +56,14 @@ pub fn map_transaction(tx: Transaction) -> Result<BankTransaction> {
foreign_currency, foreign_currency,
description, description,
counterparty_name: tx.creditor_name.or(tx.debtor_name), counterparty_name: tx.creditor_name.or(tx.debtor_name),
counterparty_iban: tx counterparty_iban: tx.creditor_account.and_then(|a| a.iban).or(tx.debtor_account.and_then(|a| a.iban)),
.creditor_account
.and_then(|a| a.iban)
.or(tx.debtor_account.and_then(|a| a.iban)),
}) })
} }
fn validate_amount(amount: &Decimal) -> Result<()> { fn validate_amount(amount: &Decimal) -> Result<()> {
let abs = amount.abs(); let abs = amount.abs();
if abs > Decimal::new(1_000_000_000, 0) { if abs > Decimal::new(1_000_000_000, 0) {
return Err(anyhow::anyhow!( return Err(anyhow::anyhow!("Amount exceeds reasonable bounds: {}", amount));
"Amount exceeds reasonable bounds: {}",
amount
));
} }
if abs == Decimal::ZERO { if abs == Decimal::ZERO {
return Err(anyhow::anyhow!("Amount cannot be zero")); return Err(anyhow::anyhow!("Amount cannot be zero"));
@@ -86,16 +73,10 @@ fn validate_amount(amount: &Decimal) -> Result<()> {
fn validate_currency(currency: &str) -> Result<()> { fn validate_currency(currency: &str) -> Result<()> {
if currency.len() != 3 { if currency.len() != 3 {
return Err(anyhow::anyhow!( return Err(anyhow::anyhow!("Invalid currency code length: {}", currency));
"Invalid currency code length: {}",
currency
));
} }
if !currency.chars().all(|c| c.is_ascii_uppercase()) { if !currency.chars().all(|c| c.is_ascii_uppercase()) {
return Err(anyhow::anyhow!( return Err(anyhow::anyhow!("Invalid currency code format: {}", currency));
"Invalid currency code format: {}",
currency
));
} }
Ok(()) Ok(())
} }
@@ -103,21 +84,14 @@ fn validate_currency(currency: &str) -> Result<()> {
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
use gocardless_client::models::{CurrencyExchange, TransactionAmount}; use gocardless_client::models::{TransactionAmount, CurrencyExchange};
#[test] #[test]
fn test_map_normal_transaction() { fn test_map_normal_transaction() {
let t = Transaction { let t = Transaction {
transaction_id: Some("123".into()), transaction_id: Some("123".into()),
entry_reference: None,
end_to_end_id: None,
mandate_id: None,
check_id: None,
creditor_id: None,
booking_date: Some("2023-01-01".into()), booking_date: Some("2023-01-01".into()),
value_date: None, value_date: None,
booking_date_time: None,
value_date_time: None,
transaction_amount: TransactionAmount { transaction_amount: TransactionAmount {
amount: "100.50".into(), amount: "100.50".into(),
currency: "EUR".into(), currency: "EUR".into(),
@@ -125,19 +99,10 @@ mod tests {
currency_exchange: None, currency_exchange: None,
creditor_name: Some("Shop".into()), creditor_name: Some("Shop".into()),
creditor_account: None, creditor_account: None,
ultimate_creditor: None,
debtor_name: None, debtor_name: None,
debtor_account: None, debtor_account: None,
ultimate_debtor: None,
remittance_information_unstructured: Some("Groceries".into()), remittance_information_unstructured: Some("Groceries".into()),
remittance_information_unstructured_array: None,
remittance_information_structured: None,
remittance_information_structured_array: None,
additional_information: None,
purpose_code: None,
bank_transaction_code: None,
proprietary_bank_transaction_code: None, proprietary_bank_transaction_code: None,
internal_transaction_id: None,
}; };
let res = map_transaction(t).unwrap(); let res = map_transaction(t).unwrap();
@@ -152,15 +117,8 @@ mod tests {
fn test_map_multicurrency_transaction() { fn test_map_multicurrency_transaction() {
let t = Transaction { let t = Transaction {
transaction_id: Some("124".into()), transaction_id: Some("124".into()),
entry_reference: None,
end_to_end_id: None,
mandate_id: None,
check_id: None,
creditor_id: None,
booking_date: Some("2023-01-02".into()), booking_date: Some("2023-01-02".into()),
value_date: None, value_date: None,
booking_date_time: None,
value_date_time: None,
transaction_amount: TransactionAmount { transaction_amount: TransactionAmount {
amount: "-10.00".into(), amount: "-10.00".into(),
currency: "EUR".into(), currency: "EUR".into(),
@@ -173,19 +131,10 @@ mod tests {
}]), }]),
creditor_name: Some("US Shop".into()), creditor_name: Some("US Shop".into()),
creditor_account: None, creditor_account: None,
ultimate_creditor: None,
debtor_name: None, debtor_name: None,
debtor_account: None, debtor_account: None,
ultimate_debtor: None,
remittance_information_unstructured: None, remittance_information_unstructured: None,
remittance_information_unstructured_array: None,
remittance_information_structured: None,
remittance_information_structured_array: None,
additional_information: None,
purpose_code: None,
bank_transaction_code: None,
proprietary_bank_transaction_code: None, proprietary_bank_transaction_code: None,
internal_transaction_id: None,
}; };
let res = map_transaction(t).unwrap(); let res = map_transaction(t).unwrap();
@@ -212,6 +161,8 @@ mod tests {
assert!(validate_amount(&amount).is_err()); assert!(validate_amount(&amount).is_err());
} }
#[test] #[test]
fn test_validate_currency_invalid_length() { fn test_validate_currency_invalid_length() {
assert!(validate_currency("EU").is_err()); assert!(validate_currency("EU").is_err());
@@ -234,15 +185,8 @@ mod tests {
fn test_map_transaction_invalid_amount() { fn test_map_transaction_invalid_amount() {
let t = Transaction { let t = Transaction {
transaction_id: Some("125".into()), transaction_id: Some("125".into()),
entry_reference: None,
end_to_end_id: None,
mandate_id: None,
check_id: None,
creditor_id: None,
booking_date: Some("2023-01-03".into()), booking_date: Some("2023-01-03".into()),
value_date: None, value_date: None,
booking_date_time: None,
value_date_time: None,
transaction_amount: TransactionAmount { transaction_amount: TransactionAmount {
amount: "0.00".into(), amount: "0.00".into(),
currency: "EUR".into(), currency: "EUR".into(),
@@ -250,19 +194,10 @@ mod tests {
currency_exchange: None, currency_exchange: None,
creditor_name: Some("Test".into()), creditor_name: Some("Test".into()),
creditor_account: None, creditor_account: None,
ultimate_creditor: None,
debtor_name: None, debtor_name: None,
debtor_account: None, debtor_account: None,
ultimate_debtor: None,
remittance_information_unstructured: None, remittance_information_unstructured: None,
remittance_information_unstructured_array: None,
remittance_information_structured: None,
remittance_information_structured_array: None,
additional_information: None,
purpose_code: None,
bank_transaction_code: None,
proprietary_bank_transaction_code: None, proprietary_bank_transaction_code: None,
internal_transaction_id: None,
}; };
assert!(map_transaction(t).is_err()); assert!(map_transaction(t).is_err());
@@ -272,15 +207,8 @@ mod tests {
fn test_map_transaction_invalid_currency() { fn test_map_transaction_invalid_currency() {
let t = Transaction { let t = Transaction {
transaction_id: Some("126".into()), transaction_id: Some("126".into()),
entry_reference: None,
end_to_end_id: None,
mandate_id: None,
check_id: None,
creditor_id: None,
booking_date: Some("2023-01-04".into()), booking_date: Some("2023-01-04".into()),
value_date: None, value_date: None,
booking_date_time: None,
value_date_time: None,
transaction_amount: TransactionAmount { transaction_amount: TransactionAmount {
amount: "100.00".into(), amount: "100.00".into(),
currency: "euro".into(), currency: "euro".into(),
@@ -288,19 +216,10 @@ mod tests {
currency_exchange: None, currency_exchange: None,
creditor_name: Some("Test".into()), creditor_name: Some("Test".into()),
creditor_account: None, creditor_account: None,
ultimate_creditor: None,
debtor_name: None, debtor_name: None,
debtor_account: None, debtor_account: None,
ultimate_debtor: None,
remittance_information_unstructured: None, remittance_information_unstructured: None,
remittance_information_unstructured_array: None,
remittance_information_structured: None,
remittance_information_structured_array: None,
additional_information: None,
purpose_code: None,
bank_transaction_code: None,
proprietary_bank_transaction_code: None, proprietary_bank_transaction_code: None,
internal_transaction_id: None,
}; };
assert!(map_transaction(t).is_err()); assert!(map_transaction(t).is_err());
@@ -310,15 +229,8 @@ mod tests {
fn test_map_transaction_invalid_foreign_amount() { fn test_map_transaction_invalid_foreign_amount() {
let t = Transaction { let t = Transaction {
transaction_id: Some("127".into()), transaction_id: Some("127".into()),
entry_reference: None,
end_to_end_id: None,
mandate_id: None,
check_id: None,
creditor_id: None,
booking_date: Some("2023-01-05".into()), booking_date: Some("2023-01-05".into()),
value_date: None, value_date: None,
booking_date_time: None,
value_date_time: None,
transaction_amount: TransactionAmount { transaction_amount: TransactionAmount {
amount: "-10.00".into(), amount: "-10.00".into(),
currency: "EUR".into(), currency: "EUR".into(),
@@ -331,19 +243,10 @@ mod tests {
}]), }]),
creditor_name: Some("Test".into()), creditor_name: Some("Test".into()),
creditor_account: None, creditor_account: None,
ultimate_creditor: None,
debtor_name: None, debtor_name: None,
debtor_account: None, debtor_account: None,
ultimate_debtor: None,
remittance_information_unstructured: None, remittance_information_unstructured: None,
remittance_information_unstructured_array: None,
remittance_information_structured: None,
remittance_information_structured_array: None,
additional_information: None,
purpose_code: None,
bank_transaction_code: None,
proprietary_bank_transaction_code: None, proprietary_bank_transaction_code: None,
internal_transaction_id: None,
}; };
assert!(map_transaction(t).is_err()); assert!(map_transaction(t).is_err());
@@ -353,15 +256,8 @@ mod tests {
fn test_map_transaction_invalid_foreign_currency() { fn test_map_transaction_invalid_foreign_currency() {
let t = Transaction { let t = Transaction {
transaction_id: Some("128".into()), transaction_id: Some("128".into()),
entry_reference: None,
end_to_end_id: None,
mandate_id: None,
check_id: None,
creditor_id: None,
booking_date: Some("2023-01-06".into()), booking_date: Some("2023-01-06".into()),
value_date: None, value_date: None,
booking_date_time: None,
value_date_time: None,
transaction_amount: TransactionAmount { transaction_amount: TransactionAmount {
amount: "-10.00".into(), amount: "-10.00".into(),
currency: "EUR".into(), currency: "EUR".into(),
@@ -374,19 +270,10 @@ mod tests {
}]), }]),
creditor_name: Some("Test".into()), creditor_name: Some("Test".into()),
creditor_account: None, creditor_account: None,
ultimate_creditor: None,
debtor_name: None, debtor_name: None,
debtor_account: None, debtor_account: None,
ultimate_debtor: None,
remittance_information_unstructured: None, remittance_information_unstructured: None,
remittance_information_unstructured_array: None,
remittance_information_structured: None,
remittance_information_structured_array: None,
additional_information: None,
purpose_code: None,
bank_transaction_code: None,
proprietary_bank_transaction_code: None, proprietary_bank_transaction_code: None,
internal_transaction_id: None,
}; };
assert!(map_transaction(t).is_err()); assert!(map_transaction(t).is_err());

View File

@@ -1,3 +1,5 @@
pub mod client; pub mod client;
pub mod mapper; pub mod mapper;
pub mod cache;
pub mod encryption;
pub mod transaction_cache; pub mod transaction_cache;

View File

@@ -1,16 +1,16 @@
use crate::core::encryption::Encryption; use chrono::{NaiveDate, Days};
use anyhow::Result;
use chrono::{Days, NaiveDate};
use gocardless_client::models::Transaction;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use std::path::Path; use std::path::Path;
use std::collections::HashSet;
use anyhow::Result;
use crate::adapters::gocardless::encryption::Encryption;
use gocardless_client::models::Transaction;
use rand;
#[derive(Debug, Clone)] #[derive(Serialize, Deserialize, Debug, Clone)]
pub struct AccountTransactionCache { pub struct AccountTransactionCache {
pub account_id: String, pub account_id: String,
pub ranges: Vec<CachedRange>, pub ranges: Vec<CachedRange>,
cache_dir: String,
encryption: Encryption,
} }
#[derive(Serialize, Deserialize, Debug, Clone)] #[derive(Serialize, Deserialize, Debug, Clone)]
@@ -20,65 +20,44 @@ pub struct CachedRange {
pub transactions: Vec<Transaction>, pub transactions: Vec<Transaction>,
} }
#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct AccountTransactionCacheData {
pub account_id: String,
pub ranges: Vec<CachedRange>,
}
impl AccountTransactionCache { impl AccountTransactionCache {
/// Create new cache with directory and encryption
pub fn new(account_id: String, cache_dir: String, encryption: Encryption) -> Self {
Self {
account_id,
cache_dir,
encryption,
ranges: Vec::new(),
}
}
/// Get cache file path for an account /// Get cache file path for an account
fn get_cache_path(&self, account_id: &str) -> String { fn get_cache_path(account_id: &str) -> String {
format!("{}/transactions/{}.enc", self.cache_dir, account_id) let cache_dir = std::env::var("BANKS2FF_CACHE_DIR").unwrap_or_else(|_| "data/cache".to_string());
format!("{}/transactions/{}.enc", cache_dir, account_id)
} }
/// Load cache from disk /// Load cache from disk
pub fn load(account_id: &str, cache_dir: String, encryption: Encryption) -> Result<Self> { pub fn load(account_id: &str) -> Result<Self> {
let path = format!("{}/transactions/{}.enc", cache_dir, account_id); let path = Self::get_cache_path(account_id);
if !Path::new(&path).exists() { if !Path::new(&path).exists() {
// Return empty cache if file doesn't exist // Return empty cache if file doesn't exist
return Ok(Self::new(account_id.to_string(), cache_dir, encryption)); return Ok(Self {
account_id: account_id.to_string(),
ranges: Vec::new(),
});
} }
// Read encrypted data // Read encrypted data
let encrypted_data = std::fs::read(&path)?; let encrypted_data = std::fs::read(&path)?;
let json_data = encryption.decrypt(&encrypted_data)?; let json_data = Encryption::decrypt(&encrypted_data)?;
// Deserialize // Deserialize
let cache_data: AccountTransactionCacheData = serde_json::from_slice(&json_data)?; let cache: Self = serde_json::from_slice(&json_data)?;
Ok(Self { Ok(cache)
account_id: cache_data.account_id,
ranges: cache_data.ranges,
cache_dir,
encryption,
})
} }
/// Save cache to disk /// Save cache to disk
pub fn save(&self) -> Result<()> { pub fn save(&self) -> Result<()> {
// Serialize to JSON (only the data fields) // Serialize to JSON
let cache_data = AccountTransactionCacheData { let json_data = serde_json::to_vec(self)?;
account_id: self.account_id.clone(),
ranges: self.ranges.clone(),
};
let json_data = serde_json::to_vec(&cache_data)?;
// Encrypt // Encrypt
let encrypted_data = self.encryption.encrypt(&json_data)?; let encrypted_data = Encryption::encrypt(&json_data)?;
// Write to file (create directory if needed) // Write to file (create directory if needed)
let path = self.get_cache_path(&self.account_id); let path = Self::get_cache_path(&self.account_id);
if let Some(parent) = std::path::Path::new(&path).parent() { if let Some(parent) = std::path::Path::new(&path).parent() {
std::fs::create_dir_all(parent)?; std::fs::create_dir_all(parent)?;
} }
@@ -93,9 +72,7 @@ impl AccountTransactionCache {
if Self::ranges_overlap(range.start_date, range.end_date, start, end) { if Self::ranges_overlap(range.start_date, range.end_date, start, end) {
for tx in &range.transactions { for tx in &range.transactions {
if let Some(booking_date_str) = &tx.booking_date { if let Some(booking_date_str) = &tx.booking_date {
if let Ok(booking_date) = if let Ok(booking_date) = NaiveDate::parse_from_str(booking_date_str, "%Y-%m-%d") {
NaiveDate::parse_from_str(booking_date_str, "%Y-%m-%d")
{
if booking_date >= start && booking_date <= end { if booking_date >= start && booking_date <= end {
result.push(tx.clone()); result.push(tx.clone());
} }
@@ -108,13 +85,8 @@ impl AccountTransactionCache {
} }
/// Get uncovered date ranges within requested period /// Get uncovered date ranges within requested period
pub fn get_uncovered_ranges( pub fn get_uncovered_ranges(&self, start: NaiveDate, end: NaiveDate) -> Vec<(NaiveDate, NaiveDate)> {
&self, let mut covered_periods: Vec<(NaiveDate, NaiveDate)> = self.ranges
start: NaiveDate,
end: NaiveDate,
) -> Vec<(NaiveDate, NaiveDate)> {
let mut covered_periods: Vec<(NaiveDate, NaiveDate)> = self
.ranges
.iter() .iter()
.filter_map(|range| { .filter_map(|range| {
if Self::ranges_overlap(range.start_date, range.end_date, start, end) { if Self::ranges_overlap(range.start_date, range.end_date, start, end) {
@@ -164,12 +136,7 @@ impl AccountTransactionCache {
} }
/// Store transactions for a date range, merging with existing cache /// Store transactions for a date range, merging with existing cache
pub fn store_transactions( pub fn store_transactions(&mut self, start: NaiveDate, end: NaiveDate, mut transactions: Vec<Transaction>) {
&mut self,
start: NaiveDate,
end: NaiveDate,
mut transactions: Vec<Transaction>,
) {
Self::deduplicate_transactions(&mut transactions); Self::deduplicate_transactions(&mut transactions);
let new_range = CachedRange { let new_range = CachedRange {
start_date: start, start_date: start,
@@ -186,12 +153,7 @@ impl AccountTransactionCache {
let mut remaining = Vec::new(); let mut remaining = Vec::new();
for range in &self.ranges { for range in &self.ranges {
if Self::ranges_overlap_or_adjacent( if Self::ranges_overlap_or_adjacent(range.start_date, range.end_date, new_range.start_date, new_range.end_date) {
range.start_date,
range.end_date,
new_range.start_date,
new_range.end_date,
) {
to_merge.push(range.clone()); to_merge.push(range.clone());
} else { } else {
remaining.push(range.clone()); remaining.push(range.clone());
@@ -209,25 +171,15 @@ impl AccountTransactionCache {
} }
/// Check if two date ranges overlap /// Check if two date ranges overlap
fn ranges_overlap( fn ranges_overlap(start1: NaiveDate, end1: NaiveDate, start2: NaiveDate, end2: NaiveDate) -> bool {
start1: NaiveDate,
end1: NaiveDate,
start2: NaiveDate,
end2: NaiveDate,
) -> bool {
start1 <= end2 && start2 <= end1 start1 <= end2 && start2 <= end1
} }
/// Check if two date ranges overlap or are adjacent /// Check if two date ranges overlap or are adjacent
fn ranges_overlap_or_adjacent( fn ranges_overlap_or_adjacent(start1: NaiveDate, end1: NaiveDate, start2: NaiveDate, end2: NaiveDate) -> bool {
start1: NaiveDate, Self::ranges_overlap(start1, end1, start2, end2) ||
end1: NaiveDate, (end1 + Days::new(1)) == start2 ||
start2: NaiveDate, (end2 + Days::new(1)) == start1
end2: NaiveDate,
) -> bool {
Self::ranges_overlap(start1, end1, start2, end2)
|| (end1 + Days::new(1)) == start2
|| (end2 + Days::new(1)) == start1
} }
/// Merge a list of ranges into minimal set /// Merge a list of ranges into minimal set
@@ -244,12 +196,7 @@ impl AccountTransactionCache {
let mut current = sorted[0].clone(); let mut current = sorted[0].clone();
for range in sorted.into_iter().skip(1) { for range in sorted.into_iter().skip(1) {
if Self::ranges_overlap_or_adjacent( if Self::ranges_overlap_or_adjacent(current.start_date, current.end_date, range.start_date, range.end_date) {
current.start_date,
current.end_date,
range.start_date,
range.end_date,
) {
// Merge // Merge
current.start_date = current.start_date.min(range.start_date); current.start_date = current.start_date.min(range.start_date);
current.end_date = current.end_date.max(range.end_date); current.end_date = current.end_date.max(range.end_date);
@@ -282,13 +229,11 @@ impl AccountTransactionCache {
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
use std::env;
use chrono::NaiveDate; use chrono::NaiveDate;
fn create_unique_key(prefix: &str) -> String { fn setup_test_env(test_name: &str) -> String {
format!("{}-{}", prefix, rand::random::<u64>()) env::set_var("BANKS2FF_CACHE_KEY", "test-cache-key");
}
fn setup_test_dir(test_name: &str) -> String {
// Use a unique cache directory for each test to avoid interference // Use a unique cache directory for each test to avoid interference
// Include random component and timestamp for true parallelism safety // Include random component and timestamp for true parallelism safety
let random_suffix = rand::random::<u64>(); let random_suffix = rand::random::<u64>();
@@ -296,10 +241,9 @@ mod tests {
.duration_since(std::time::UNIX_EPOCH) .duration_since(std::time::UNIX_EPOCH)
.unwrap() .unwrap()
.as_nanos(); .as_nanos();
format!( let cache_dir = format!("tmp/test-cache-{}-{}-{}", test_name, random_suffix, timestamp);
"tmp/test-cache-{}-{}-{}", env::set_var("BANKS2FF_CACHE_DIR", cache_dir.clone());
test_name, random_suffix, timestamp cache_dir
)
} }
fn cleanup_test_dir(cache_dir: &str) { fn cleanup_test_dir(cache_dir: &str) {
@@ -319,12 +263,12 @@ mod tests {
} }
} }
#[test] #[test]
fn test_load_nonexistent_cache() { fn test_load_nonexistent_cache() {
let cache_dir = setup_test_dir("nonexistent"); let cache_dir = setup_test_env("nonexistent");
let encryption = Encryption::new(create_unique_key("test-key")); let cache = AccountTransactionCache::load("nonexistent").unwrap();
let cache =
AccountTransactionCache::load("nonexistent", cache_dir.clone(), encryption).unwrap();
assert_eq!(cache.account_id, "nonexistent"); assert_eq!(cache.account_id, "nonexistent");
assert!(cache.ranges.is_empty()); assert!(cache.ranges.is_empty());
cleanup_test_dir(&cache_dir); cleanup_test_dir(&cache_dir);
@@ -332,24 +276,24 @@ mod tests {
#[test] #[test]
fn test_save_and_load_empty_cache() { fn test_save_and_load_empty_cache() {
let cache_dir = setup_test_dir("empty"); let cache_dir = setup_test_env("empty");
let encryption_key = create_unique_key("test-key");
let encryption = Encryption::new(encryption_key.clone());
let cache = AccountTransactionCache::new( let cache = AccountTransactionCache {
"test_account_empty".to_string(), account_id: "test_account_empty".to_string(),
cache_dir.clone(), ranges: Vec::new(),
encryption, };
);
// Ensure env vars are set before save
env::set_var("BANKS2FF_CACHE_KEY", "test-cache-key");
// Ensure env vars are set before save
env::set_var("BANKS2FF_CACHE_KEY", "test-cache-key");
// Save // Save
cache.save().expect("Save should succeed"); cache.save().expect("Save should succeed");
// Ensure env vars are set before load
env::set_var("BANKS2FF_CACHE_KEY", "test-cache-key");
// Load // Load
let encryption = Encryption::new(encryption_key); let loaded = AccountTransactionCache::load("test_account_empty").expect("Load should succeed");
let loaded =
AccountTransactionCache::load("test_account_empty", cache_dir.clone(), encryption)
.expect("Load should succeed");
assert_eq!(loaded.account_id, "test_account_empty"); assert_eq!(loaded.account_id, "test_account_empty");
assert!(loaded.ranges.is_empty()); assert!(loaded.ranges.is_empty());
@@ -359,20 +303,12 @@ mod tests {
#[test] #[test]
fn test_save_and_load_with_data() { fn test_save_and_load_with_data() {
let cache_dir = setup_test_dir("data"); let cache_dir = setup_test_env("data");
let encryption_key = create_unique_key("test-key");
let transaction = Transaction { let transaction = Transaction {
transaction_id: Some("test-tx-1".to_string()), transaction_id: Some("test-tx-1".to_string()),
entry_reference: None,
end_to_end_id: None,
mandate_id: None,
check_id: None,
creditor_id: None,
booking_date: Some("2024-01-01".to_string()), booking_date: Some("2024-01-01".to_string()),
value_date: None, value_date: None,
booking_date_time: None,
value_date_time: None,
transaction_amount: gocardless_client::models::TransactionAmount { transaction_amount: gocardless_client::models::TransactionAmount {
amount: "100.00".to_string(), amount: "100.00".to_string(),
currency: "EUR".to_string(), currency: "EUR".to_string(),
@@ -380,19 +316,10 @@ mod tests {
currency_exchange: None, currency_exchange: None,
creditor_name: Some("Test Creditor".to_string()), creditor_name: Some("Test Creditor".to_string()),
creditor_account: None, creditor_account: None,
ultimate_creditor: None,
debtor_name: None, debtor_name: None,
debtor_account: None, debtor_account: None,
ultimate_debtor: None,
remittance_information_unstructured: Some("Test payment".to_string()), remittance_information_unstructured: Some("Test payment".to_string()),
remittance_information_unstructured_array: None,
remittance_information_structured: None,
remittance_information_structured_array: None,
additional_information: None,
purpose_code: None,
bank_transaction_code: None,
proprietary_bank_transaction_code: None, proprietary_bank_transaction_code: None,
internal_transaction_id: None,
}; };
let range = CachedRange { let range = CachedRange {
@@ -401,62 +328,57 @@ mod tests {
transactions: vec![transaction], transactions: vec![transaction],
}; };
let encryption = Encryption::new(encryption_key.clone()); let cache = AccountTransactionCache {
let mut cache = AccountTransactionCache::new( account_id: "test_account_data".to_string(),
"test_account_data".to_string(), ranges: vec![range],
cache_dir.clone(), };
encryption,
);
cache.ranges = vec![range];
// Ensure env vars are set before save
env::set_var("BANKS2FF_CACHE_KEY", "test-cache-key");
// Save // Save
cache.save().expect("Save should succeed"); cache.save().expect("Save should succeed");
// Ensure env vars are set before load
env::set_var("BANKS2FF_CACHE_KEY", "test-cache-key");
// Load // Load
let encryption = Encryption::new(encryption_key); let loaded = AccountTransactionCache::load("test_account_data").expect("Load should succeed");
let loaded =
AccountTransactionCache::load("test_account_data", cache_dir.clone(), encryption)
.expect("Load should succeed");
assert_eq!(loaded.account_id, "test_account_data"); assert_eq!(loaded.account_id, "test_account_data");
assert_eq!(loaded.ranges.len(), 1); assert_eq!(loaded.ranges.len(), 1);
assert_eq!(loaded.ranges[0].transactions.len(), 1); assert_eq!(loaded.ranges[0].transactions.len(), 1);
assert_eq!( assert_eq!(loaded.ranges[0].transactions[0].transaction_id, Some("test-tx-1".to_string()));
loaded.ranges[0].transactions[0].transaction_id,
Some("test-tx-1".to_string())
);
cleanup_test_dir(&cache_dir); cleanup_test_dir(&cache_dir);
} }
#[test] #[test]
fn test_save_load_different_accounts() { fn test_save_load_different_accounts() {
let cache_dir = setup_test_dir("different_accounts"); let cache_dir = setup_test_env("different_accounts");
let encryption_key_a = create_unique_key("test-key-a");
let encryption_key_b = create_unique_key("test-key-b");
// Save cache for account A // Save cache for account A
let encryption_a = Encryption::new(encryption_key_a.clone()); env::set_var("BANKS2FF_CACHE_KEY", "test-cache-key");
let cache_a = let cache_a = AccountTransactionCache {
AccountTransactionCache::new("account_a".to_string(), cache_dir.clone(), encryption_a); account_id: "account_a".to_string(),
ranges: Vec::new(),
};
cache_a.save().unwrap(); cache_a.save().unwrap();
// Save cache for account B // Save cache for account B
let encryption_b = Encryption::new(encryption_key_b.clone()); env::set_var("BANKS2FF_CACHE_KEY", "test-cache-key");
let cache_b = let cache_b = AccountTransactionCache {
AccountTransactionCache::new("account_b".to_string(), cache_dir.clone(), encryption_b); account_id: "account_b".to_string(),
ranges: Vec::new(),
};
cache_b.save().unwrap(); cache_b.save().unwrap();
// Load account A // Load account A
let encryption_a = Encryption::new(encryption_key_a); env::set_var("BANKS2FF_CACHE_KEY", "test-cache-key");
let loaded_a = let loaded_a = AccountTransactionCache::load("account_a").unwrap();
AccountTransactionCache::load("account_a", cache_dir.clone(), encryption_a).unwrap();
assert_eq!(loaded_a.account_id, "account_a"); assert_eq!(loaded_a.account_id, "account_a");
// Load account B // Load account B
let encryption_b = Encryption::new(encryption_key_b); env::set_var("BANKS2FF_CACHE_KEY", "test-cache-key");
let loaded_b = let loaded_b = AccountTransactionCache::load("account_b").unwrap();
AccountTransactionCache::load("account_b", cache_dir.clone(), encryption_b).unwrap();
assert_eq!(loaded_b.account_id, "account_b"); assert_eq!(loaded_b.account_id, "account_b");
cleanup_test_dir(&cache_dir); cleanup_test_dir(&cache_dir);
@@ -464,9 +386,10 @@ mod tests {
#[test] #[test]
fn test_get_uncovered_ranges_no_cache() { fn test_get_uncovered_ranges_no_cache() {
let encryption = Encryption::new(create_unique_key("test-key")); let cache = AccountTransactionCache {
let cache = account_id: "test".to_string(),
AccountTransactionCache::new("test".to_string(), "test_cache".to_string(), encryption); ranges: Vec::new(),
};
let start = NaiveDate::from_ymd_opt(2024, 1, 1).unwrap(); let start = NaiveDate::from_ymd_opt(2024, 1, 1).unwrap();
let end = NaiveDate::from_ymd_opt(2024, 1, 31).unwrap(); let end = NaiveDate::from_ymd_opt(2024, 1, 31).unwrap();
let uncovered = cache.get_uncovered_ranges(start, end); let uncovered = cache.get_uncovered_ranges(start, end);
@@ -480,10 +403,10 @@ mod tests {
end_date: NaiveDate::from_ymd_opt(2024, 1, 31).unwrap(), end_date: NaiveDate::from_ymd_opt(2024, 1, 31).unwrap(),
transactions: Vec::new(), transactions: Vec::new(),
}; };
let encryption = Encryption::new(create_unique_key("test-key")); let cache = AccountTransactionCache {
let mut cache = account_id: "test".to_string(),
AccountTransactionCache::new("test".to_string(), "test_cache".to_string(), encryption); ranges: vec![range],
cache.ranges = vec![range]; };
let start = NaiveDate::from_ymd_opt(2024, 1, 1).unwrap(); let start = NaiveDate::from_ymd_opt(2024, 1, 1).unwrap();
let end = NaiveDate::from_ymd_opt(2024, 1, 31).unwrap(); let end = NaiveDate::from_ymd_opt(2024, 1, 31).unwrap();
let uncovered = cache.get_uncovered_ranges(start, end); let uncovered = cache.get_uncovered_ranges(start, end);
@@ -497,48 +420,30 @@ mod tests {
end_date: NaiveDate::from_ymd_opt(2024, 1, 20).unwrap(), end_date: NaiveDate::from_ymd_opt(2024, 1, 20).unwrap(),
transactions: Vec::new(), transactions: Vec::new(),
}; };
let encryption = Encryption::new(create_unique_key("test-key")); let cache = AccountTransactionCache {
let mut cache = account_id: "test".to_string(),
AccountTransactionCache::new("test".to_string(), "test_cache".to_string(), encryption); ranges: vec![range],
cache.ranges = vec![range]; };
let start = NaiveDate::from_ymd_opt(2024, 1, 1).unwrap(); let start = NaiveDate::from_ymd_opt(2024, 1, 1).unwrap();
let end = NaiveDate::from_ymd_opt(2024, 1, 31).unwrap(); let end = NaiveDate::from_ymd_opt(2024, 1, 31).unwrap();
let uncovered = cache.get_uncovered_ranges(start, end); let uncovered = cache.get_uncovered_ranges(start, end);
assert_eq!(uncovered.len(), 2); assert_eq!(uncovered.len(), 2);
assert_eq!( assert_eq!(uncovered[0], (NaiveDate::from_ymd_opt(2024, 1, 1).unwrap(), NaiveDate::from_ymd_opt(2024, 1, 9).unwrap()));
uncovered[0], assert_eq!(uncovered[1], (NaiveDate::from_ymd_opt(2024, 1, 21).unwrap(), NaiveDate::from_ymd_opt(2024, 1, 31).unwrap()));
(
NaiveDate::from_ymd_opt(2024, 1, 1).unwrap(),
NaiveDate::from_ymd_opt(2024, 1, 9).unwrap()
)
);
assert_eq!(
uncovered[1],
(
NaiveDate::from_ymd_opt(2024, 1, 21).unwrap(),
NaiveDate::from_ymd_opt(2024, 1, 31).unwrap()
)
);
} }
#[test] #[test]
fn test_store_transactions_and_merge() { fn test_store_transactions_and_merge() {
let encryption = Encryption::new(create_unique_key("test-key")); let mut cache = AccountTransactionCache {
let mut cache = account_id: "test".to_string(),
AccountTransactionCache::new("test".to_string(), "test_cache".to_string(), encryption); ranges: Vec::new(),
};
let start1 = NaiveDate::from_ymd_opt(2024, 1, 1).unwrap(); let start1 = NaiveDate::from_ymd_opt(2024, 1, 1).unwrap();
let end1 = NaiveDate::from_ymd_opt(2024, 1, 10).unwrap(); let end1 = NaiveDate::from_ymd_opt(2024, 1, 10).unwrap();
let tx1 = Transaction { let tx1 = Transaction {
transaction_id: Some("tx1".to_string()), transaction_id: Some("tx1".to_string()),
entry_reference: None,
end_to_end_id: None,
mandate_id: None,
check_id: None,
creditor_id: None,
booking_date: Some("2024-01-05".to_string()), booking_date: Some("2024-01-05".to_string()),
value_date: None, value_date: None,
booking_date_time: None,
value_date_time: None,
transaction_amount: gocardless_client::models::TransactionAmount { transaction_amount: gocardless_client::models::TransactionAmount {
amount: "100.00".to_string(), amount: "100.00".to_string(),
currency: "EUR".to_string(), currency: "EUR".to_string(),
@@ -546,19 +451,10 @@ mod tests {
currency_exchange: None, currency_exchange: None,
creditor_name: Some("Creditor".to_string()), creditor_name: Some("Creditor".to_string()),
creditor_account: None, creditor_account: None,
ultimate_creditor: None,
debtor_name: None, debtor_name: None,
debtor_account: None, debtor_account: None,
ultimate_debtor: None,
remittance_information_unstructured: Some("Payment".to_string()), remittance_information_unstructured: Some("Payment".to_string()),
remittance_information_unstructured_array: None,
remittance_information_structured: None,
remittance_information_structured_array: None,
additional_information: None,
purpose_code: None,
bank_transaction_code: None,
proprietary_bank_transaction_code: None, proprietary_bank_transaction_code: None,
internal_transaction_id: None,
}; };
cache.store_transactions(start1, end1, vec![tx1]); cache.store_transactions(start1, end1, vec![tx1]);
@@ -572,15 +468,8 @@ mod tests {
let end2 = NaiveDate::from_ymd_opt(2024, 1, 15).unwrap(); let end2 = NaiveDate::from_ymd_opt(2024, 1, 15).unwrap();
let tx2 = Transaction { let tx2 = Transaction {
transaction_id: Some("tx2".to_string()), transaction_id: Some("tx2".to_string()),
entry_reference: None,
end_to_end_id: None,
mandate_id: None,
check_id: None,
creditor_id: None,
booking_date: Some("2024-01-12".to_string()), booking_date: Some("2024-01-12".to_string()),
value_date: None, value_date: None,
booking_date_time: None,
value_date_time: None,
transaction_amount: gocardless_client::models::TransactionAmount { transaction_amount: gocardless_client::models::TransactionAmount {
amount: "200.00".to_string(), amount: "200.00".to_string(),
currency: "EUR".to_string(), currency: "EUR".to_string(),
@@ -588,19 +477,10 @@ mod tests {
currency_exchange: None, currency_exchange: None,
creditor_name: Some("Creditor2".to_string()), creditor_name: Some("Creditor2".to_string()),
creditor_account: None, creditor_account: None,
ultimate_creditor: None,
debtor_name: None, debtor_name: None,
debtor_account: None, debtor_account: None,
ultimate_debtor: None,
remittance_information_unstructured: Some("Payment2".to_string()), remittance_information_unstructured: Some("Payment2".to_string()),
remittance_information_unstructured_array: None,
remittance_information_structured: None,
remittance_information_structured_array: None,
additional_information: None,
purpose_code: None,
bank_transaction_code: None,
proprietary_bank_transaction_code: None, proprietary_bank_transaction_code: None,
internal_transaction_id: None,
}; };
cache.store_transactions(start2, end2, vec![tx2]); cache.store_transactions(start2, end2, vec![tx2]);
@@ -613,22 +493,16 @@ mod tests {
#[test] #[test]
fn test_transaction_deduplication() { fn test_transaction_deduplication() {
let encryption = Encryption::new(create_unique_key("test-key")); let mut cache = AccountTransactionCache {
let mut cache = account_id: "test".to_string(),
AccountTransactionCache::new("test".to_string(), "test_cache".to_string(), encryption); ranges: Vec::new(),
};
let start = NaiveDate::from_ymd_opt(2024, 1, 1).unwrap(); let start = NaiveDate::from_ymd_opt(2024, 1, 1).unwrap();
let end = NaiveDate::from_ymd_opt(2024, 1, 10).unwrap(); let end = NaiveDate::from_ymd_opt(2024, 1, 10).unwrap();
let tx1 = Transaction { let tx1 = Transaction {
transaction_id: Some("tx1".to_string()), transaction_id: Some("dup".to_string()),
entry_reference: None,
end_to_end_id: None,
mandate_id: None,
check_id: None,
creditor_id: None,
booking_date: Some("2024-01-05".to_string()), booking_date: Some("2024-01-05".to_string()),
value_date: None, value_date: None,
booking_date_time: None,
value_date_time: None,
transaction_amount: gocardless_client::models::TransactionAmount { transaction_amount: gocardless_client::models::TransactionAmount {
amount: "100.00".to_string(), amount: "100.00".to_string(),
currency: "EUR".to_string(), currency: "EUR".to_string(),
@@ -636,19 +510,10 @@ mod tests {
currency_exchange: None, currency_exchange: None,
creditor_name: Some("Creditor".to_string()), creditor_name: Some("Creditor".to_string()),
creditor_account: None, creditor_account: None,
ultimate_creditor: None,
debtor_name: None, debtor_name: None,
debtor_account: None, debtor_account: None,
ultimate_debtor: None,
remittance_information_unstructured: Some("Payment".to_string()), remittance_information_unstructured: Some("Payment".to_string()),
remittance_information_unstructured_array: None,
remittance_information_structured: None,
remittance_information_structured_array: None,
additional_information: None,
purpose_code: None,
bank_transaction_code: None,
proprietary_bank_transaction_code: None, proprietary_bank_transaction_code: None,
internal_transaction_id: None,
}; };
let tx2 = tx1.clone(); // Duplicate let tx2 = tx1.clone(); // Duplicate
cache.store_transactions(start, end, vec![tx1, tx2]); cache.store_transactions(start, end, vec![tx1, tx2]);
@@ -660,15 +525,8 @@ mod tests {
fn test_get_cached_transactions() { fn test_get_cached_transactions() {
let tx1 = Transaction { let tx1 = Transaction {
transaction_id: Some("tx1".to_string()), transaction_id: Some("tx1".to_string()),
entry_reference: None,
end_to_end_id: None,
mandate_id: None,
check_id: None,
creditor_id: None,
booking_date: Some("2024-01-05".to_string()), booking_date: Some("2024-01-05".to_string()),
value_date: None, value_date: None,
booking_date_time: None,
value_date_time: None,
transaction_amount: gocardless_client::models::TransactionAmount { transaction_amount: gocardless_client::models::TransactionAmount {
amount: "100.00".to_string(), amount: "100.00".to_string(),
currency: "EUR".to_string(), currency: "EUR".to_string(),
@@ -676,29 +534,20 @@ mod tests {
currency_exchange: None, currency_exchange: None,
creditor_name: Some("Creditor".to_string()), creditor_name: Some("Creditor".to_string()),
creditor_account: None, creditor_account: None,
ultimate_creditor: None,
debtor_name: None, debtor_name: None,
debtor_account: None, debtor_account: None,
ultimate_debtor: None,
remittance_information_unstructured: Some("Payment".to_string()), remittance_information_unstructured: Some("Payment".to_string()),
remittance_information_unstructured_array: None,
remittance_information_structured: None,
remittance_information_structured_array: None,
additional_information: None,
purpose_code: None,
bank_transaction_code: None,
proprietary_bank_transaction_code: None, proprietary_bank_transaction_code: None,
internal_transaction_id: None,
}; };
let range = CachedRange { let range = CachedRange {
start_date: NaiveDate::from_ymd_opt(2024, 1, 1).unwrap(), start_date: NaiveDate::from_ymd_opt(2024, 1, 1).unwrap(),
end_date: NaiveDate::from_ymd_opt(2024, 1, 31).unwrap(), end_date: NaiveDate::from_ymd_opt(2024, 1, 31).unwrap(),
transactions: vec![tx1], transactions: vec![tx1],
}; };
let encryption = Encryption::new(create_unique_key("test-key")); let cache = AccountTransactionCache {
let mut cache = account_id: "test".to_string(),
AccountTransactionCache::new("test".to_string(), "test_cache".to_string(), encryption); ranges: vec![range],
cache.ranges = vec![range]; };
let start = NaiveDate::from_ymd_opt(2024, 1, 1).unwrap(); let start = NaiveDate::from_ymd_opt(2024, 1, 1).unwrap();
let end = NaiveDate::from_ymd_opt(2024, 1, 10).unwrap(); let end = NaiveDate::from_ymd_opt(2024, 1, 10).unwrap();
let cached = cache.get_cached_transactions(start, end); let cached = cache.get_cached_transactions(start, end);

View File

@@ -1,155 +0,0 @@
use crate::core::cache::AccountCache;
use crate::core::models::{AccountStatus, AccountSummary, CacheInfo, TransactionInfo};
use comfy_table::{presets::UTF8_FULL, Table};
pub enum OutputFormat {
Table,
}
pub trait Formattable {
fn to_table(&self, account_cache: Option<&AccountCache>) -> Table;
}
pub fn print_list_output<T: Formattable>(
data: Vec<T>,
format: &OutputFormat,
account_cache: Option<&AccountCache>,
) {
if data.is_empty() {
println!("No data available");
return;
}
match format {
OutputFormat::Table => {
for item in data {
println!("{}", item.to_table(account_cache));
}
}
}
}
// Implement Formattable for the model structs
impl Formattable for AccountSummary {
fn to_table(&self, _account_cache: Option<&AccountCache>) -> Table {
let mut table = Table::new();
table.load_preset(UTF8_FULL);
table.set_header(vec!["Name", "IBAN", "Currency"]);
let name = self.name.as_deref().unwrap_or("");
table.add_row(vec![
name.to_string(),
mask_iban(&self.iban),
self.currency.clone(),
]);
table
}
}
impl Formattable for AccountStatus {
fn to_table(&self, account_cache: Option<&AccountCache>) -> Table {
let mut table = Table::new();
table.load_preset(UTF8_FULL);
table.set_header(vec![
"Account",
"IBAN",
"Last Sync",
"Transaction Count",
"Status",
]);
let display_name = if let Some(cache) = account_cache {
cache
.get_display_name(&self.account_id)
.unwrap_or_else(|| self.account_id.clone())
} else {
self.account_id.clone()
};
table.add_row(vec![
display_name,
mask_iban(&self.iban),
self.last_sync_date
.map(|d| d.to_string())
.unwrap_or_else(|| "Never".to_string()),
self.transaction_count.to_string(),
self.status.clone(),
]);
table
}
}
impl Formattable for TransactionInfo {
fn to_table(&self, _account_cache: Option<&AccountCache>) -> Table {
let mut table = Table::new();
table.load_preset(UTF8_FULL);
table.set_header(vec![
"Account ID",
"Total Transactions",
"Date Range",
"Last Updated",
]);
let date_range = self
.date_range
.map(|(start, end)| format!("{} to {}", start, end))
.unwrap_or_else(|| "N/A".to_string());
table.add_row(vec![
self.account_id.clone(),
self.total_count.to_string(),
date_range,
self.last_updated
.map(|d| d.to_string())
.unwrap_or_else(|| "Never".to_string()),
]);
table
}
}
impl Formattable for CacheInfo {
fn to_table(&self, _account_cache: Option<&AccountCache>) -> Table {
let mut table = Table::new();
table.load_preset(UTF8_FULL);
table.set_header(vec![
"Account ID",
"Cache Type",
"Entry Count",
"Size (bytes)",
"Last Updated",
]);
table.add_row(vec![
self.account_id.as_deref().unwrap_or("Global").to_string(),
self.cache_type.clone(),
self.entry_count.to_string(),
self.total_size_bytes.to_string(),
self.last_updated
.map(|d| d.to_string())
.unwrap_or_else(|| "Never".to_string()),
]);
table
}
}
fn mask_iban(iban: &str) -> String {
if iban.len() <= 4 {
iban.to_string()
} else {
let country_code = &iban[0..2];
let last_four = &iban[iban.len() - 4..];
if country_code == "NL" && iban.len() >= 12 {
// NL: show first 2 (CC) + next 6 + mask + last 4
let next_six = &iban[2..8];
let mask_length = iban.len() - 12; // 2 + 6 + 4 = 12, so mask_length = total - 12
format!(
"{}{}{}{}",
country_code,
next_six,
"*".repeat(mask_length),
last_four
)
} else {
// Other countries: show first 2 + mask + last 4
let mask_length = iban.len() - 6; // 2 + 4 = 6
format!("{}{}{}", country_code, "*".repeat(mask_length), last_four)
}
}
}

View File

@@ -1,3 +0,0 @@
pub mod formatters;
pub mod setup;
pub mod tables;

View File

@@ -1,55 +0,0 @@
use crate::adapters::firefly::client::FireflyAdapter;
use crate::adapters::gocardless::client::GoCardlessAdapter;
use crate::core::config::Config;
use crate::debug::DebugLogger;
use anyhow::Result;
use firefly_client::client::FireflyClient;
use gocardless_client::client::GoCardlessClient;
use reqwest_middleware::ClientBuilder;
pub struct AppContext {
pub source: GoCardlessAdapter,
pub destination: FireflyAdapter,
}
impl AppContext {
pub async fn new(config: Config, debug: bool) -> Result<Self> {
// Clients
let gc_client = if debug {
let client = ClientBuilder::new(reqwest::Client::new())
.with(DebugLogger::new("gocardless"))
.build();
GoCardlessClient::with_client(
&config.gocardless.url,
&config.gocardless.secret_id,
&config.gocardless.secret_key,
Some(client),
)?
} else {
GoCardlessClient::new(
&config.gocardless.url,
&config.gocardless.secret_id,
&config.gocardless.secret_key,
)?
};
let ff_client = if debug {
let client = ClientBuilder::new(reqwest::Client::new())
.with(DebugLogger::new("firefly"))
.build();
FireflyClient::with_client(&config.firefly.url, &config.firefly.api_key, Some(client))?
} else {
FireflyClient::new(&config.firefly.url, &config.firefly.api_key)?
};
// Adapters
let source = GoCardlessAdapter::new(gc_client, config.clone());
let destination = FireflyAdapter::new(ff_client, config);
Ok(Self {
source,
destination,
})
}
}

View File

@@ -1,99 +0,0 @@
use crate::core::cache::AccountCache;
use crate::core::models::{AccountStatus, AccountSummary};
use comfy_table::{presets::UTF8_FULL, Table};
pub fn print_accounts_table(accounts: &[AccountSummary]) {
let mut table = Table::new();
table.load_preset(UTF8_FULL);
table.set_header(vec!["Name", "IBAN", "Currency"]);
for account in accounts {
let name = account.name.as_deref().unwrap_or("");
table.add_row(vec![
name.to_string(),
mask_iban(&account.iban),
account.currency.clone(),
]);
}
println!("{}", table);
}
pub fn print_links_table(
links: &[crate::core::linking::AccountLink],
account_cache: &crate::core::cache::AccountCache,
) {
let mut table = Table::new();
table.load_preset(UTF8_FULL);
table.set_header(vec!["Source Account", "Destination Account", "Auto-Linked"]);
for link in links {
let source_name = account_cache
.get_display_name(&link.source_account_id)
.unwrap_or_else(|| format!("Account {}", &link.source_account_id));
let dest_name = account_cache
.get_display_name(&link.dest_account_id)
.unwrap_or_else(|| format!("Account {}", &link.dest_account_id));
let auto_linked = if link.auto_linked { "Yes" } else { "No" };
table.add_row(vec![source_name, dest_name, auto_linked.to_string()]);
}
println!("{}", table);
}
pub fn print_account_status_table(statuses: &[AccountStatus], account_cache: &AccountCache) {
let mut table = Table::new();
table.load_preset(UTF8_FULL);
table.set_header(vec![
"Account",
"IBAN",
"Last Sync",
"Transaction Count",
"Status",
]);
for status in statuses {
let display_name = account_cache
.get_display_name(&status.account_id)
.unwrap_or_else(|| status.account_id.clone());
table.add_row(vec![
display_name,
mask_iban(&status.iban),
status
.last_sync_date
.map(|d| d.to_string())
.unwrap_or_else(|| "Never".to_string()),
status.transaction_count.to_string(),
status.status.clone(),
]);
}
println!("{}", table);
}
pub fn mask_iban(iban: &str) -> String {
if iban.len() <= 4 {
iban.to_string()
} else {
let country_code = &iban[0..2];
let last_four = &iban[iban.len() - 4..];
if country_code == "NL" && iban.len() >= 12 {
// NL: show first 2 (CC) + next 6 + mask + last 4
let next_six = &iban[2..8];
let mask_length = iban.len() - 12; // 2 + 6 + 4 = 12, so mask_length = total - 12
format!(
"{}{}{}{}",
country_code,
next_six,
"*".repeat(mask_length),
last_four
)
} else {
// Other countries: show first 2 + mask + last 4
let mask_length = iban.len() - 6; // 2 + 4 = 6
format!("{}{}{}", country_code, "*".repeat(mask_length), last_four)
}
}
}

View File

@@ -1,513 +0,0 @@
use crate::core::cache::{AccountCache, CachedAccount};
use crate::core::linking::LinkStore;
use crate::core::models::{Account, AccountData};
use clap::Subcommand;
pub fn handle_interactive_link_creation(
link_store: &mut LinkStore,
account_cache: &AccountCache,
) -> anyhow::Result<()> {
// Get unlinked GoCardless accounts
let gocardless_accounts = get_gocardless_accounts(account_cache);
let unlinked_sources: Vec<_> = gocardless_accounts
.iter()
.filter(|acc| {
!link_store
.find_links_by_source(acc.id())
.iter()
.any(|link| link.dest_adapter_type == "firefly")
})
.collect();
if unlinked_sources.is_empty() {
println!("No unlinked source accounts found. All GoCardless accounts are already linked to Firefly III.");
return Ok(());
}
// Create selection items for dialoguer
let source_items: Vec<String> = unlinked_sources
.iter()
.map(|account| {
let display_name = account
.display_name()
.unwrap_or_else(|| account.id().to_string());
display_name.to_string()
})
.collect();
// Add cancel option
let mut items = source_items.clone();
items.push("Cancel".to_string());
// Prompt user to select source account
let source_selection =
match dialoguer::Select::with_theme(&dialoguer::theme::ColorfulTheme::default())
.with_prompt("Select a source account to link")
.items(&items)
.default(0)
.interact()
{
Ok(selection) => selection,
Err(_) => {
// Non-interactive environment (e.g., tests, scripts)
println!("Interactive mode not available in this environment.");
println!("Use: banks2ff accounts link create <source> <destination>");
return Ok(());
}
};
if source_selection == items.len() - 1 {
// User selected "Cancel"
println!("Operation cancelled.");
return Ok(());
}
let selected_source = &unlinked_sources[source_selection];
handle_source_selection(link_store, account_cache, selected_source.id().to_string())?;
Ok(())
}
pub fn handle_single_arg_link_creation(
link_store: &mut LinkStore,
account_cache: &AccountCache,
arg: &str,
) -> anyhow::Result<()> {
// Try to find account by ID, name, or IBAN
let matched_account = find_account_by_identifier(account_cache, arg);
match matched_account {
Some((account_id, adapter_type)) => {
if adapter_type == "gocardless" {
// It's a source account - show available destinations
handle_source_selection(link_store, account_cache, account_id)
} else {
// It's a destination account - show available sources
handle_destination_selection(link_store, account_cache, account_id)
}
}
None => {
println!("No account found matching '{}'.", arg);
println!("Try using an account ID, name, or IBAN pattern.");
println!("Run 'banks2ff accounts list' to see available accounts.");
Ok(())
}
}
}
pub fn handle_direct_link_creation(
link_store: &mut LinkStore,
account_cache: &AccountCache,
source_arg: &str,
dest_arg: &str,
) -> anyhow::Result<()> {
let source_match = find_account_by_identifier(account_cache, source_arg);
let dest_match = find_account_by_identifier(account_cache, dest_arg);
match (source_match, dest_match) {
(Some((source_id, source_adapter)), Some((dest_id, dest_adapter))) => {
if source_adapter != "gocardless" {
println!(
"Error: Source must be a GoCardless account, got {} account.",
source_adapter
);
return Ok(());
}
if dest_adapter != "firefly" {
println!(
"Error: Destination must be a Firefly III account, got {} account.",
dest_adapter
);
return Ok(());
}
create_link(
link_store,
account_cache,
&source_id,
&dest_id,
&dest_adapter,
)
}
(None, _) => {
println!("Source account '{}' not found.", source_arg);
Ok(())
}
(_, None) => {
println!("Destination account '{}' not found.", dest_arg);
Ok(())
}
}
}
pub fn find_account_by_identifier(
account_cache: &AccountCache,
identifier: &str,
) -> Option<(String, String)> {
// First try exact ID match
if let Some(adapter_type) = account_cache.get_adapter_type(identifier) {
return Some((identifier.to_string(), adapter_type.to_string()));
}
// Then try name/IBAN matching
for (id, account) in &account_cache.accounts {
if let Some(display_name) = account.display_name() {
if display_name
.to_lowercase()
.contains(&identifier.to_lowercase())
{
let adapter_type = if matches!(account, CachedAccount::GoCardless(_)) {
"gocardless"
} else {
"firefly"
};
return Some((id.clone(), adapter_type.to_string()));
}
}
if let Some(iban) = account.iban() {
if iban.contains(identifier) {
let adapter_type = if matches!(account, CachedAccount::GoCardless(_)) {
"gocardless"
} else {
"firefly"
};
return Some((id.clone(), adapter_type.to_string()));
}
}
}
None
}
pub fn handle_source_selection(
link_store: &mut LinkStore,
account_cache: &AccountCache,
source_id: String,
) -> anyhow::Result<()> {
// Check if source is already linked to firefly
if let Some(existing_link) = link_store.find_link_by_source_and_dest_type(&source_id, "firefly")
{
let dest_name = account_cache
.get_display_name(&existing_link.dest_account_id)
.unwrap_or_else(|| existing_link.dest_account_id.clone());
println!(
"Source account '{}' is already linked to '{}'.",
account_cache
.get_display_name(&source_id)
.unwrap_or_else(|| source_id.clone()),
dest_name
);
return Ok(());
}
// Get available Firefly destinations
let firefly_accounts = get_firefly_accounts(account_cache);
if firefly_accounts.is_empty() {
println!("No Firefly III accounts found. Run sync first.");
return Ok(());
}
// Create selection items for dialoguer
let dest_items: Vec<String> = firefly_accounts
.iter()
.map(|account| {
let display_name = account
.display_name()
.unwrap_or_else(|| account.id().to_string());
display_name.to_string()
})
.collect();
// Add cancel option
let mut items = dest_items.clone();
items.push("Cancel".to_string());
// Prompt user to select destination account
let source_name = account_cache
.get_display_name(&source_id)
.unwrap_or_else(|| source_id.clone());
let dest_selection =
match dialoguer::Select::with_theme(&dialoguer::theme::ColorfulTheme::default())
.with_prompt(format!(
"Select a destination account for '{}'",
source_name
))
.items(&items)
.default(0)
.interact()
{
Ok(selection) => selection,
Err(_) => {
// Non-interactive environment (e.g., tests, scripts)
println!("Interactive mode not available in this environment.");
println!("Use: banks2ff accounts link create <source> <destination>");
return Ok(());
}
};
if dest_selection == items.len() - 1 {
// User selected "Cancel"
println!("Operation cancelled.");
return Ok(());
}
let selected_dest = &firefly_accounts[dest_selection];
create_link(
link_store,
account_cache,
&source_id,
selected_dest.id(),
"firefly",
)?;
Ok(())
}
pub fn handle_destination_selection(
link_store: &mut LinkStore,
account_cache: &AccountCache,
dest_id: String,
) -> anyhow::Result<()> {
// Get available GoCardless sources that aren't already linked to this destination
let gocardless_accounts = get_gocardless_accounts(account_cache);
let available_sources: Vec<_> = gocardless_accounts
.iter()
.filter(|acc| {
!link_store
.find_links_by_source(acc.id())
.iter()
.any(|link| link.dest_account_id == dest_id)
})
.collect();
if available_sources.is_empty() {
println!(
"No available source accounts found that can link to '{}'.",
account_cache
.get_display_name(&dest_id)
.unwrap_or_else(|| dest_id.clone())
);
return Ok(());
}
// Create selection items for dialoguer
let source_items: Vec<String> = available_sources
.iter()
.map(|account| {
let display_name = account
.display_name()
.unwrap_or_else(|| account.id().to_string());
display_name.to_string()
})
.collect();
// Add cancel option
let mut items = source_items.clone();
items.push("Cancel".to_string());
// Prompt user to select source account
let dest_name = account_cache
.get_display_name(&dest_id)
.unwrap_or_else(|| dest_id.clone());
let source_selection =
match dialoguer::Select::with_theme(&dialoguer::theme::ColorfulTheme::default())
.with_prompt(format!(
"Select a source account to link to '{}'",
dest_name
))
.items(&items)
.default(0)
.interact()
{
Ok(selection) => selection,
Err(_) => {
// Non-interactive environment (e.g., tests, scripts)
println!("Interactive mode not available in this environment.");
println!("Use: banks2ff accounts link create <source> <destination>");
return Ok(());
}
};
if source_selection == items.len() - 1 {
// User selected "Cancel"
println!("Operation cancelled.");
return Ok(());
}
let selected_source = &available_sources[source_selection];
create_link(
link_store,
account_cache,
selected_source.id(),
&dest_id,
"firefly",
)?;
Ok(())
}
pub fn create_link(
link_store: &mut LinkStore,
account_cache: &AccountCache,
source_id: &str,
dest_id: &str,
dest_adapter_type: &str,
) -> anyhow::Result<()> {
let source_acc = account_cache.get_account(source_id);
let dest_acc = account_cache.get_account(dest_id);
if let (Some(src), Some(dst)) = (source_acc, dest_acc) {
let src_minimal = Account {
id: src.id().to_string(),
name: Some(src.id().to_string()),
iban: src.iban().map(|s| s.to_string()),
currency: "EUR".to_string(),
};
let dst_minimal = Account {
id: dst.id().to_string(),
name: Some(dst.id().to_string()),
iban: dst.iban().map(|s| s.to_string()),
currency: "EUR".to_string(),
};
match link_store.add_link(
&src_minimal,
&dst_minimal,
"gocardless",
dest_adapter_type,
false,
) {
Ok(true) => {
link_store.save()?;
let src_display = account_cache
.get_display_name(source_id)
.unwrap_or_else(|| source_id.to_string());
let dst_display = account_cache
.get_display_name(dest_id)
.unwrap_or_else(|| dest_id.to_string());
println!("Created link between {} and {}", src_display, dst_display);
}
Ok(false) => {
let src_display = account_cache
.get_display_name(source_id)
.unwrap_or_else(|| source_id.to_string());
let dst_display = account_cache
.get_display_name(dest_id)
.unwrap_or_else(|| dest_id.to_string());
println!(
"Link between {} and {} already exists",
src_display, dst_display
);
}
Err(e) => {
println!("Cannot create link: {}", e);
}
}
} else {
println!("Account not found in cache. Run sync first.");
}
Ok(())
}
pub fn get_gocardless_accounts(account_cache: &AccountCache) -> Vec<&dyn AccountData> {
account_cache
.accounts
.values()
.filter_map(|acc| match acc {
CachedAccount::GoCardless(gc_acc) => Some(gc_acc.as_ref() as &dyn AccountData),
_ => None,
})
.collect()
}
pub fn get_firefly_accounts(account_cache: &AccountCache) -> Vec<&dyn AccountData> {
account_cache
.accounts
.values()
.filter_map(|acc| match acc {
CachedAccount::Firefly(ff_acc) => Some(ff_acc.as_ref() as &dyn AccountData),
_ => None,
})
.collect()
}
// Command handlers
use crate::cli::tables::print_links_table;
use crate::core::config::Config;
use crate::core::encryption::Encryption;
#[derive(Subcommand, Debug)]
pub enum LinkCommands {
/// List all account links
List,
/// Create a new account link
Create {
/// Source account identifier (ID, IBAN, or name). If omitted, interactive mode is used.
source_account: Option<String>,
/// Destination account identifier (ID, IBAN, or name). Required if source is provided.
dest_account: Option<String>,
},
}
pub async fn handle_link_commands(config: Config, subcommand: LinkCommands) -> anyhow::Result<()> {
match subcommand {
LinkCommands::List => {
handle_link_list(config).await?;
}
LinkCommands::Create {
source_account,
dest_account,
} => {
handle_link_create(config, source_account, dest_account).await?;
}
}
Ok(())
}
pub async fn handle_link_list(config: Config) -> anyhow::Result<()> {
let encryption = Encryption::new(config.cache.key.clone());
let link_store = LinkStore::load(config.cache.directory.clone());
let account_cache =
crate::core::cache::AccountCache::load(config.cache.directory.clone(), encryption);
if link_store.links.is_empty() {
println!("No account links found.");
} else {
print_links_table(&link_store.links, &account_cache);
}
Ok(())
}
pub async fn handle_link_create(
config: Config,
source_account: Option<String>,
dest_account: Option<String>,
) -> anyhow::Result<()> {
let encryption = Encryption::new(config.cache.key.clone());
let mut link_store = LinkStore::load(config.cache.directory.clone());
let account_cache =
crate::core::cache::AccountCache::load(config.cache.directory.clone(), encryption);
match (source_account, dest_account) {
(None, None) => {
// Interactive mode
handle_interactive_link_creation(&mut link_store, &account_cache)?;
}
(Some(source), None) => {
// Single argument - try to resolve as source or destination
handle_single_arg_link_creation(&mut link_store, &account_cache, &source)?;
}
(Some(source), Some(dest)) => {
// Two arguments - direct linking
handle_direct_link_creation(&mut link_store, &account_cache, &source, &dest)?;
}
(None, Some(_)) => {
println!("Error: Cannot specify destination without source. Use 'banks2ff accounts link create <source> <destination>' or interactive mode.");
}
}
Ok(())
}

View File

@@ -1,91 +0,0 @@
use crate::cli::setup::AppContext;
use crate::cli::tables::print_accounts_table;
use crate::core::config::Config;
use crate::core::ports::{TransactionDestination, TransactionSource};
use tracing::warn;
pub async fn handle_list(config: Config, filter: Option<String>) -> anyhow::Result<()> {
let context = AppContext::new(config, false).await?;
// Validate filter parameter
let show_gocardless = match filter.as_deref() {
Some("gocardless") => true,
Some("firefly") => false,
None => true, // Show both by default
Some(invalid) => {
anyhow::bail!(
"Invalid filter '{}'. Use 'gocardless', 'firefly', or omit for all.",
invalid
);
}
};
let show_firefly = match filter.as_deref() {
Some("gocardless") => false,
Some("firefly") => true,
None => true, // Show both by default
Some(_) => unreachable!(), // Already validated above
};
// Get GoCardless accounts if needed
let gocardless_accounts = if show_gocardless {
match context.source.list_accounts().await {
Ok(mut accounts) => {
accounts.sort_by(|a, b| {
a.name
.as_deref()
.unwrap_or("")
.cmp(b.name.as_deref().unwrap_or(""))
});
accounts
}
Err(e) => {
warn!("Failed to list GoCardless accounts: {}", e);
Vec::new()
}
}
} else {
Vec::new()
};
// Get Firefly III accounts if needed
let firefly_accounts = if show_firefly {
match context.destination.list_accounts().await {
Ok(mut accounts) => {
accounts.sort_by(|a, b| {
a.name
.as_deref()
.unwrap_or("")
.cmp(b.name.as_deref().unwrap_or(""))
});
accounts
}
Err(e) => {
warn!("Failed to list Firefly III accounts: {}", e);
Vec::new()
}
}
} else {
Vec::new()
};
if gocardless_accounts.is_empty() && firefly_accounts.is_empty() {
println!("No accounts found. Run 'banks2ff sync gocardless firefly' first to discover and cache account data.");
} else {
// Print GoCardless accounts
if !gocardless_accounts.is_empty() {
println!("GoCardless Accounts ({}):", gocardless_accounts.len());
print_accounts_table(&gocardless_accounts);
}
// Print Firefly III accounts
if !firefly_accounts.is_empty() {
if !gocardless_accounts.is_empty() {
println!(); // Add spacing between tables
}
println!("Firefly III Accounts ({}):", firefly_accounts.len());
print_accounts_table(&firefly_accounts);
}
}
Ok(())
}

View File

@@ -1,43 +0,0 @@
pub mod link;
pub mod list;
pub mod status;
use crate::core::config::Config;
use clap::Subcommand;
use link::handle_link_commands;
use link::LinkCommands;
#[derive(Subcommand, Debug)]
pub enum AccountCommands {
/// Manage account links between sources and destinations
Link {
#[command(subcommand)]
subcommand: LinkCommands,
},
/// List all accounts
List {
/// Filter by adapter type: 'gocardless' or 'firefly', or omit for all
filter: Option<String>,
},
/// Show account status
Status,
}
pub async fn handle_accounts(config: Config, subcommand: AccountCommands) -> anyhow::Result<()> {
match subcommand {
AccountCommands::Link {
subcommand: link_sub,
} => {
handle_link_commands(config.clone(), link_sub).await?;
}
AccountCommands::List { filter } => {
list::handle_list(config, filter).await?;
}
AccountCommands::Status => {
status::handle_status(config).await?;
}
}
Ok(())
}

View File

@@ -1,21 +0,0 @@
use crate::cli::setup::AppContext;
use crate::cli::tables::print_account_status_table;
use crate::core::config::Config;
use crate::core::encryption::Encryption;
use crate::core::ports::TransactionSource;
pub async fn handle_status(config: Config) -> anyhow::Result<()> {
let context = AppContext::new(config.clone(), false).await?;
let encryption = Encryption::new(config.cache.key.clone());
let account_cache =
crate::core::cache::AccountCache::load(config.cache.directory.clone(), encryption);
let status = context.source.get_account_status().await?;
if status.is_empty() {
println!("No account status available. Run 'banks2ff sync gocardless firefly' first to sync transactions and build status data.");
} else {
print_account_status_table(&status, &account_cache);
}
Ok(())
}

View File

@@ -1,17 +0,0 @@
use crate::core::adapters::{get_available_destinations, get_available_sources};
pub async fn handle_sources() -> anyhow::Result<()> {
println!("Available sources:");
for source in get_available_sources() {
println!(" {} - {}", source.id, source.description);
}
Ok(())
}
pub async fn handle_destinations() -> anyhow::Result<()> {
println!("Available destinations:");
for destination in get_available_destinations() {
println!(" {} - {}", destination.id, destination.description);
}
Ok(())
}

View File

@@ -1,4 +0,0 @@
pub mod accounts;
pub mod list;
pub mod sync;
pub mod transactions;

View File

@@ -1,88 +0,0 @@
use crate::cli::setup::AppContext;
use crate::core::adapters::{
get_available_destinations, get_available_sources, is_valid_destination, is_valid_source,
};
use crate::core::config::Config;
use crate::core::sync::run_sync;
use chrono::NaiveDate;
use tracing::{error, info};
pub async fn handle_sync(
config: Config,
debug: bool,
source: String,
destination: String,
start: Option<NaiveDate>,
end: Option<NaiveDate>,
dry_run: bool,
) -> anyhow::Result<()> {
// Validate source
if !is_valid_source(&source) {
let available = get_available_sources()
.iter()
.map(|s| s.id)
.collect::<Vec<_>>()
.join(", ");
anyhow::bail!(
"Unknown source '{}'. Available sources: {}",
source,
available
);
}
// Validate destination
if !is_valid_destination(&destination) {
let available = get_available_destinations()
.iter()
.map(|d| d.id)
.collect::<Vec<_>>()
.join(", ");
anyhow::bail!(
"Unknown destination '{}'. Available destinations: {}",
destination,
available
);
}
// For now, only support gocardless -> firefly
if source != "gocardless" {
anyhow::bail!("Only 'gocardless' source is currently supported (implementation pending)");
}
if destination != "firefly" {
anyhow::bail!("Only 'firefly' destination is currently supported (implementation pending)");
}
let context = AppContext::new(config.clone(), debug).await?;
// Run sync
match run_sync(
context.source,
context.destination,
config,
start,
end,
dry_run,
)
.await
{
Ok(result) => {
info!("Sync completed successfully.");
info!(
"Accounts processed: {}, skipped (expired): {}, skipped (errors): {}",
result.accounts_processed,
result.accounts_skipped_expired,
result.accounts_skipped_errors
);
info!(
"Transactions - Created: {}, Healed: {}, Duplicates: {}, Errors: {}",
result.ingest.created,
result.ingest.healed,
result.ingest.duplicates,
result.ingest.errors
);
}
Err(e) => error!("Sync failed: {}", e),
}
Ok(())
}

View File

@@ -1,24 +0,0 @@
use crate::cli::formatters::{print_list_output, OutputFormat};
use crate::cli::setup::AppContext;
use crate::core::config::Config;
use crate::core::encryption::Encryption;
use crate::core::ports::TransactionSource;
pub async fn handle_cache_status(config: Config) -> anyhow::Result<()> {
let context = AppContext::new(config.clone(), false).await?;
let format = OutputFormat::Table; // TODO: Add --json flag
// Load account cache for display name resolution
let encryption = Encryption::new(config.cache.key.clone());
let account_cache =
crate::core::cache::AccountCache::load(config.cache.directory.clone(), encryption);
let cache_info = context.source.get_cache_info().await?;
if cache_info.is_empty() {
println!("No cache data available. Run 'banks2ff sync gocardless firefly' first to populate caches.");
} else {
print_list_output(cache_info, &format, Some(&account_cache));
}
Ok(())
}

View File

@@ -1,7 +0,0 @@
use crate::core::config::Config;
pub async fn handle_clear_cache(_config: Config) -> anyhow::Result<()> {
// TODO: Implement cache clearing
println!("Cache clearing not yet implemented");
Ok(())
}

View File

@@ -1,24 +0,0 @@
use crate::cli::formatters::{print_list_output, OutputFormat};
use crate::cli::setup::AppContext;
use crate::core::config::Config;
use crate::core::encryption::Encryption;
use crate::core::ports::TransactionSource;
pub async fn handle_list(config: Config, account_id: String) -> anyhow::Result<()> {
let context = AppContext::new(config.clone(), false).await?;
let format = OutputFormat::Table; // TODO: Add --json flag
// Load account cache for display name resolution
let encryption = Encryption::new(config.cache.key.clone());
let account_cache =
crate::core::cache::AccountCache::load(config.cache.directory.clone(), encryption);
let info = context.source.get_transaction_info(&account_id).await?;
if info.total_count == 0 {
println!("No transaction data found for account {}. Run 'banks2ff sync gocardless firefly' first to sync transactions.", account_id);
} else {
print_list_output(vec![info], &format, Some(&account_cache));
}
Ok(())
}

View File

@@ -1,41 +0,0 @@
pub mod cache;
pub mod clear;
pub mod list;
use crate::core::config::Config;
use clap::Subcommand;
use self::cache::handle_cache_status;
use self::clear::handle_clear_cache;
use self::list::handle_list as handle_transaction_list;
#[derive(Subcommand, Debug)]
pub enum TransactionCommands {
/// List transactions for an account
List {
/// Account ID to list transactions for
account_id: String,
},
/// Show cache status
CacheStatus,
/// Clear transaction cache
ClearCache,
}
pub async fn handle_transactions(
config: Config,
subcommand: TransactionCommands,
) -> anyhow::Result<()> {
match subcommand {
TransactionCommands::List { account_id } => {
handle_transaction_list(config, account_id).await?;
}
TransactionCommands::CacheStatus => {
handle_cache_status(config).await?;
}
TransactionCommands::ClearCache => {
handle_clear_cache(config).await?;
}
}
Ok(())
}

View File

@@ -1,70 +0,0 @@
#[derive(Debug, Clone)]
pub struct AdapterInfo {
pub id: &'static str,
pub description: &'static str,
}
pub fn get_available_sources() -> Vec<AdapterInfo> {
vec![AdapterInfo {
id: "gocardless",
description: "GoCardless Bank Account Data API",
}]
}
pub fn get_available_destinations() -> Vec<AdapterInfo> {
vec![AdapterInfo {
id: "firefly",
description: "Firefly III personal finance manager",
}]
}
pub fn is_valid_source(source: &str) -> bool {
get_available_sources().iter().any(|s| s.id == source)
}
pub fn is_valid_destination(destination: &str) -> bool {
get_available_destinations()
.iter()
.any(|d| d.id == destination)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_get_available_sources() {
let sources = get_available_sources();
assert_eq!(sources.len(), 1);
assert_eq!(sources[0].id, "gocardless");
assert_eq!(sources[0].description, "GoCardless Bank Account Data API");
}
#[test]
fn test_get_available_destinations() {
let destinations = get_available_destinations();
assert_eq!(destinations.len(), 1);
assert_eq!(destinations[0].id, "firefly");
assert_eq!(
destinations[0].description,
"Firefly III personal finance manager"
);
}
#[test]
fn test_is_valid_source() {
assert!(is_valid_source("gocardless"));
assert!(!is_valid_source("csv")); // Not implemented yet
assert!(!is_valid_source("camt053")); // Not implemented yet
assert!(!is_valid_source("mt940")); // Not implemented yet
assert!(!is_valid_source("invalid"));
assert!(!is_valid_source(""));
}
#[test]
fn test_is_valid_destination() {
assert!(is_valid_destination("firefly"));
assert!(!is_valid_destination("invalid"));
assert!(!is_valid_destination("gocardless"));
}
}

View File

@@ -1,296 +0,0 @@
use crate::core::encryption::Encryption;
use crate::core::models::AccountData;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::fs;
use std::path::Path;
use tracing::warn;
#[derive(Debug, Serialize, Deserialize, Clone)]
pub enum CachedAccount {
GoCardless(Box<GoCardlessAccount>),
Firefly(Box<FireflyAccount>),
}
#[derive(Debug, Serialize, Deserialize, Clone)]
pub struct GoCardlessAccount {
pub id: String,
pub iban: Option<String>,
pub name: Option<String>, // From AccountDetail.name
pub display_name: Option<String>, // From AccountDetail.displayName
pub owner_name: Option<String>, // From Account.owner_name
pub status: Option<String>, // From Account.status
pub institution_id: Option<String>, // From Account.institution_id
pub created: Option<String>, // From Account.created
pub last_accessed: Option<String>, // From Account.last_accessed
pub product: Option<String>, // From AccountDetail.product
pub cash_account_type: Option<String>, // From AccountDetail.cashAccountType
}
#[derive(Debug, Serialize, Deserialize, Clone)]
pub struct FireflyAccount {
pub id: String,
pub name: String, // From Account.name
pub account_type: String, // From Account.type
pub iban: Option<String>, // From Account.iban
pub active: Option<bool>, // From Account.active
pub order: Option<i32>, // From Account.order
pub created_at: Option<String>, // From Account.created_at
pub account_role: Option<String>, // From Account.account_role
pub object_group_id: Option<String>, // From Account.object_group_id
pub object_group_title: Option<String>, // From Account.object_group_title
pub object_group_order: Option<i32>, // From Account.object_group_order
pub currency_id: Option<String>, // From Account.currency_id
pub currency_name: Option<String>, // From Account.currency_name
pub currency_code: Option<String>, // From Account.currency_code
pub currency_symbol: Option<String>, // From Account.currency_symbol
pub currency_decimal_places: Option<i32>, // From Account.currency_decimal_places
pub primary_currency_id: Option<String>, // From Account.primary_currency_id
pub primary_currency_name: Option<String>, // From Account.primary_currency_name
pub primary_currency_code: Option<String>, // From Account.primary_currency_code
pub primary_currency_symbol: Option<String>, // From Account.primary_currency_symbol
pub primary_currency_decimal_places: Option<i32>, // From Account.primary_currency_decimal_places
pub opening_balance: Option<String>, // From Account.opening_balance
pub pc_opening_balance: Option<String>, // From Account.pc_opening_balance
pub debt_amount: Option<String>, // From Account.debt_amount
pub pc_debt_amount: Option<String>, // From Account.pc_debt_amount
pub notes: Option<String>, // From Account.notes
pub monthly_payment_date: Option<String>, // From Account.monthly_payment_date
pub credit_card_type: Option<String>, // From Account.credit_card_type
pub account_number: Option<String>, // From Account.account_number
pub bic: Option<String>, // From Account.bic
pub opening_balance_date: Option<String>, // From Account.opening_balance_date
pub liability_type: Option<String>, // From Account.liability_type
pub liability_direction: Option<String>, // From Account.liability_direction
pub interest: Option<String>, // From Account.interest
pub interest_period: Option<String>, // From Account.interest_period
pub include_net_worth: Option<bool>, // From Account.include_net_worth
pub longitude: Option<f64>, // From Account.longitude
pub latitude: Option<f64>, // From Account.latitude
pub zoom_level: Option<i32>, // From Account.zoom_level
pub last_activity: Option<String>, // From Account.last_activity
}
impl crate::core::models::AccountData for CachedAccount {
fn id(&self) -> &str {
match self {
CachedAccount::GoCardless(acc) => &acc.id,
CachedAccount::Firefly(acc) => &acc.id,
}
}
fn iban(&self) -> Option<&str> {
match self {
CachedAccount::GoCardless(acc) => acc.iban.as_deref(),
CachedAccount::Firefly(acc) => acc.iban.as_deref(),
}
}
fn display_name(&self) -> Option<String> {
match self {
CachedAccount::GoCardless(acc) => acc
.display_name
.clone()
.or_else(|| acc.name.clone())
.or_else(|| {
acc.owner_name
.as_ref()
.map(|owner| format!("{} Account", owner))
})
.or_else(|| {
acc.iban.as_ref().map(|iban| {
if iban.len() > 4 {
format!("{}****{}", &iban[..4], &iban[iban.len() - 4..])
} else {
iban.to_string()
}
})
}),
CachedAccount::Firefly(acc) => Some(acc.name.clone()),
}
}
}
impl AccountData for GoCardlessAccount {
fn id(&self) -> &str {
&self.id
}
fn iban(&self) -> Option<&str> {
self.iban.as_deref()
}
fn display_name(&self) -> Option<String> {
// Priority: display_name > name > owner_name > masked IBAN
let base_name = self
.display_name
.clone()
.or_else(|| self.name.clone())
.or_else(|| {
self.owner_name
.as_ref()
.map(|owner| format!("{} Account", owner))
})
.or_else(|| {
self.iban.as_ref().map(|iban| {
if iban.len() > 4 {
format!("{}****{}", &iban[..4], &iban[iban.len() - 4..])
} else {
iban.to_string()
}
})
});
// For GoCardless accounts, append institution if available
if let (Some(name), Some(institution_id)) = (&base_name, &self.institution_id) {
Some(format!("{} ({})", name, institution_id))
} else {
base_name
}
}
}
impl AccountData for FireflyAccount {
fn id(&self) -> &str {
&self.id
}
fn iban(&self) -> Option<&str> {
self.iban.as_deref()
}
fn display_name(&self) -> Option<String> {
// Priority: name > iban > None (will fallback to "Account <id>")
if !self.name.is_empty() {
Some(self.name.clone())
} else {
self.iban.as_ref().map(|iban| {
if iban.len() > 4 {
format!("{}****{}", &iban[..4], &iban[iban.len() - 4..])
} else {
iban.to_string()
}
})
}
}
}
#[derive(Debug)]
pub struct AccountCache {
/// Map of Account ID -> Full Account Data
pub accounts: HashMap<String, CachedAccount>,
/// Cache directory path
cache_dir: String,
/// Encryption instance
encryption: Encryption,
}
#[derive(Serialize, Deserialize, Debug, Default)]
pub struct AccountCacheData {
/// Map of Account ID -> Full Account Data
pub accounts: HashMap<String, CachedAccount>,
}
impl Default for AccountCache {
fn default() -> Self {
// This should not be used in practice, but provide a dummy implementation
Self {
accounts: HashMap::new(),
cache_dir: String::new(),
encryption: Encryption::new(String::new()), // Dummy key
}
}
}
impl AccountCache {
/// Create new AccountCache with directory and encryption
pub fn new(cache_dir: String, encryption: Encryption) -> Self {
Self {
accounts: HashMap::new(),
cache_dir,
encryption,
}
}
fn get_path(&self) -> String {
format!("{}/accounts.enc", self.cache_dir)
}
pub fn load(cache_dir: String, encryption: Encryption) -> Self {
let path = format!("{}/accounts.enc", cache_dir);
if Path::new(&path).exists() {
match fs::read(&path) {
Ok(encrypted_data) => match encryption.decrypt(&encrypted_data) {
Ok(json_data) => match serde_json::from_slice::<AccountCacheData>(&json_data) {
Ok(cache_data) => {
return Self {
accounts: cache_data.accounts,
cache_dir,
encryption,
}
}
Err(e) => warn!("Failed to parse cache file: {}", e),
},
Err(e) => warn!("Failed to decrypt cache file: {}", e),
},
Err(e) => warn!("Failed to read cache file: {}", e),
}
}
Self::new(cache_dir, encryption)
}
pub fn save(&self) {
let path = self.get_path();
if let Some(parent) = std::path::Path::new(&path).parent() {
if let Err(e) = std::fs::create_dir_all(parent) {
warn!(
"Failed to create cache folder '{}': {}",
parent.display(),
e
);
}
}
match serde_json::to_vec(&AccountCacheData {
accounts: self.accounts.clone(),
}) {
Ok(json_data) => match self.encryption.encrypt(&json_data) {
Ok(encrypted_data) => {
if let Err(e) = fs::write(&path, encrypted_data) {
warn!("Failed to write cache file: {}", e);
}
}
Err(e) => warn!("Failed to encrypt cache: {}", e),
},
Err(e) => warn!("Failed to serialize cache: {}", e),
}
}
pub fn get_account(&self, account_id: &str) -> Option<&CachedAccount> {
self.accounts.get(account_id)
}
pub fn get_account_data(&self, account_id: &str) -> Option<&dyn AccountData> {
match self.accounts.get(account_id)? {
CachedAccount::GoCardless(acc) => Some(acc.as_ref() as &dyn AccountData),
CachedAccount::Firefly(acc) => Some(acc.as_ref() as &dyn AccountData),
}
}
pub fn get_display_name(&self, account_id: &str) -> Option<String> {
self.get_account_data(account_id)?.display_name()
}
pub fn get_adapter_type(&self, account_id: &str) -> Option<&str> {
match self.accounts.get(account_id)? {
CachedAccount::GoCardless(_) => Some("gocardless"),
CachedAccount::Firefly(_) => Some("firefly"),
}
}
pub fn insert(&mut self, account: CachedAccount) {
let account_id = account.id().to_string();
self.accounts.insert(account_id, account);
}
}

View File

@@ -1,241 +0,0 @@
//! Configuration management for banks2ff
//!
//! Provides centralized configuration loading from environment variables
//! with type-safe configuration structures.
use anyhow::{anyhow, Result};
use std::env;
/// Main application configuration
#[derive(Clone)]
pub struct Config {
pub gocardless: GoCardlessConfig,
pub firefly: FireflyConfig,
pub cache: CacheConfig,
pub logging: LoggingConfig,
}
/// GoCardless API configuration
#[derive(Clone)]
pub struct GoCardlessConfig {
pub url: String,
pub secret_id: String,
pub secret_key: String,
}
/// Firefly III API configuration
#[derive(Clone)]
pub struct FireflyConfig {
pub url: String,
pub api_key: String,
}
/// Cache configuration
#[derive(Clone)]
pub struct CacheConfig {
pub key: String,
pub directory: String,
}
/// Logging configuration
#[derive(Clone)]
pub struct LoggingConfig {
pub level: String,
}
impl Config {
/// Load configuration from environment variables
pub fn from_env() -> Result<Self> {
let gocardless = GoCardlessConfig::from_env()?;
let firefly = FireflyConfig::from_env()?;
let cache = CacheConfig::from_env()?;
let logging = LoggingConfig::from_env()?;
Ok(Self {
gocardless,
firefly,
cache,
logging,
})
}
}
impl GoCardlessConfig {
pub fn from_env() -> Result<Self> {
let url = env::var("GOCARDLESS_URL")
.unwrap_or_else(|_| "https://bankaccountdata.gocardless.com".to_string());
let secret_id = env::var("GOCARDLESS_ID")
.map_err(|_| anyhow!("GOCARDLESS_ID environment variable not set"))?;
let secret_key = env::var("GOCARDLESS_KEY")
.map_err(|_| anyhow!("GOCARDLESS_KEY environment variable not set"))?;
Ok(Self {
url,
secret_id,
secret_key,
})
}
}
impl FireflyConfig {
pub fn from_env() -> Result<Self> {
let url = env::var("FIREFLY_III_URL")
.map_err(|_| anyhow!("FIREFLY_III_URL environment variable not set"))?;
let api_key = env::var("FIREFLY_III_API_KEY")
.map_err(|_| anyhow!("FIREFLY_III_API_KEY environment variable not set"))?;
Ok(Self { url, api_key })
}
}
impl CacheConfig {
pub fn from_env() -> Result<Self> {
let key = env::var("BANKS2FF_CACHE_KEY")
.map_err(|_| anyhow!("BANKS2FF_CACHE_KEY environment variable not set"))?;
let directory = env::var("BANKS2FF_CACHE_DIR").unwrap_or_else(|_| "data/cache".to_string());
Ok(Self { key, directory })
}
}
impl LoggingConfig {
pub fn from_env() -> Result<Self> {
let level = env::var("RUST_LOG").unwrap_or_else(|_| "warn".to_string());
Ok(Self { level })
}
}
#[cfg(test)]
mod tests {
use super::*;
use temp_env::with_vars;
#[test]
fn test_gocardless_config_from_env() {
with_vars(
[
("GOCARDLESS_ID", Some("test-id")),
("GOCARDLESS_KEY", Some("test-key")),
("GOCARDLESS_URL", Some("https://test.example.com")),
],
|| {
let config = GoCardlessConfig::from_env().unwrap();
assert_eq!(config.secret_id, "test-id");
assert_eq!(config.secret_key, "test-key");
assert_eq!(config.url, "https://test.example.com");
},
);
}
#[test]
fn test_gocardless_config_default_url() {
with_vars(
[
("GOCARDLESS_ID", Some("test-id")),
("GOCARDLESS_KEY", Some("test-key")),
("GOCARDLESS_URL", None),
],
|| {
let config = GoCardlessConfig::from_env().unwrap();
assert_eq!(config.url, "https://bankaccountdata.gocardless.com");
},
);
}
#[test]
fn test_gocardless_config_missing_id() {
with_vars(
[
("GOCARDLESS_ID", None),
("GOCARDLESS_KEY", Some("test-key")),
],
|| {
let result = GoCardlessConfig::from_env();
assert!(result.is_err());
},
);
}
#[test]
fn test_firefly_config_from_env() {
with_vars(
[
("FIREFLY_III_URL", Some("https://firefly.test.com")),
("FIREFLY_III_API_KEY", Some("test-api-key")),
],
|| {
let config = FireflyConfig::from_env().unwrap();
assert_eq!(config.url, "https://firefly.test.com");
assert_eq!(config.api_key, "test-api-key");
},
);
}
#[test]
fn test_cache_config_from_env() {
with_vars(
[
("BANKS2FF_CACHE_KEY", Some("test-cache-key")),
("BANKS2FF_CACHE_DIR", Some("/tmp/test-cache")),
],
|| {
let config = CacheConfig::from_env().unwrap();
assert_eq!(config.key, "test-cache-key");
assert_eq!(config.directory, "/tmp/test-cache");
},
);
}
#[test]
fn test_cache_config_default_directory() {
with_vars(
[
("BANKS2FF_CACHE_DIR", None),
("BANKS2FF_CACHE_KEY", Some("test-cache-key")),
],
|| {
let config = CacheConfig::from_env().unwrap();
assert_eq!(config.directory, "data/cache");
},
);
}
#[test]
fn test_logging_config_from_env() {
with_vars([("RUST_LOG", Some("debug"))], || {
let config = LoggingConfig::from_env().unwrap();
assert_eq!(config.level, "debug");
});
}
#[test]
fn test_logging_config_default() {
with_vars([("RUST_LOG", None::<&str>)], || {
let config = LoggingConfig::from_env().unwrap();
assert_eq!(config.level, "warn");
});
}
#[test]
fn test_full_config_from_env() {
with_vars(
[
("GOCARDLESS_ID", Some("test-id")),
("GOCARDLESS_KEY", Some("test-key")),
("FIREFLY_III_URL", Some("https://firefly.test.com")),
("FIREFLY_III_API_KEY", Some("test-api-key")),
("BANKS2FF_CACHE_KEY", Some("test-cache-key")),
("RUST_LOG", Some("info")),
],
|| {
let config = Config::from_env().unwrap();
assert_eq!(config.gocardless.secret_id, "test-id");
assert_eq!(config.gocardless.secret_key, "test-key");
assert_eq!(config.firefly.url, "https://firefly.test.com");
assert_eq!(config.firefly.api_key, "test-api-key");
assert_eq!(config.cache.key, "test-cache-key");
assert_eq!(config.logging.level, "info");
},
);
}
}

View File

@@ -1,192 +0,0 @@
//! # Encryption Module
//!
//! Provides AES-GCM encryption for sensitive cache data using hybrid key derivation.
//!
//! ## Security Considerations
//!
//! - **Algorithm**: AES-GCM (Authenticated Encryption) with 256-bit keys
//! - **Key Derivation**: PBKDF2 (50k iterations) for master key + HKDF for per-operation keys
//! - **Salt**: Random 16-byte salt per encryption (prepended to ciphertext)
//! - **Nonce**: Random 96-bit nonce per encryption (prepended to ciphertext)
//! - **Key Source**: Environment variable `BANKS2FF_CACHE_KEY`
//!
//! ## Data Format (Version 1)
//!
//! Encrypted data format: `[magic(4:"B2FF")][version(1)][salt(16)][nonce(12)][ciphertext]`
//!
//! ## Security Guarantees
//!
//! - **Confidentiality**: AES-GCM encryption protects data at rest
//! - **Integrity**: GCM authentication prevents tampering
//! - **Forward Security**: Unique salt/nonce per encryption prevents rainbow tables
//! - **Key Security**: PBKDF2 + HKDF provides strong key derivation
//!
//! ## Performance
//!
//! - Encryption: ~10-50μs for typical cache payloads
//! - Key derivation: ~5-10ms master key (once per session) + ~1μs per operation
//! - Memory: Minimal additional overhead
use aes_gcm::aead::{Aead, KeyInit};
use aes_gcm::{Aes256Gcm, Key, Nonce};
use anyhow::{anyhow, Result};
use hkdf::Hkdf;
use pbkdf2::pbkdf2_hmac;
use rand::RngCore;
use sha2::Sha256;
const MAGIC: &[u8] = b"B2FF";
const VERSION_1: u8 = 1;
const KEY_LEN: usize = 32; // 256-bit key
const NONCE_LEN: usize = 12; // 96-bit nonce for AES-GCM
const SALT_LEN: usize = 16; // 128-bit salt for HKDF
const MASTER_SALT: &[u8] = b"Banks2FF_MasterSalt"; // Fixed salt for master key
#[derive(Debug, Clone)]
pub struct Encryption {
master_key: Key<Aes256Gcm>,
}
impl Encryption {
/// Create new Encryption instance with cache key
pub fn new(cache_key: String) -> Self {
let master_key = Self::derive_master_key(&cache_key);
Self { master_key }
}
/// Derive master key from password using PBKDF2
fn derive_master_key(password: &str) -> Key<Aes256Gcm> {
let mut key = [0u8; KEY_LEN];
pbkdf2_hmac::<Sha256>(password.as_bytes(), MASTER_SALT, 50_000, &mut key);
key.into()
}
/// Derive operation key from master key and salt using HKDF
fn derive_operation_key(master_key: &Key<Aes256Gcm>, salt: &[u8]) -> Key<Aes256Gcm> {
let hkdf = Hkdf::<Sha256>::new(Some(salt), master_key);
let mut okm = [0u8; KEY_LEN];
hkdf.expand(b"banks2ff-operation-key", &mut okm)
.expect("HKDF expand failed");
okm.into()
}
/// Encrypt data using AES-GCM (Version 1 format)
pub fn encrypt(&self, data: &[u8]) -> Result<Vec<u8>> {
// Generate random operation salt
let mut salt = [0u8; SALT_LEN];
rand::thread_rng().fill_bytes(&mut salt);
let key = Self::derive_operation_key(&self.master_key, &salt);
let cipher = Aes256Gcm::new(&key);
// Generate random nonce
let mut nonce_bytes = [0u8; NONCE_LEN];
rand::thread_rng().fill_bytes(&mut nonce_bytes);
let nonce = Nonce::from_slice(&nonce_bytes);
// Encrypt
let ciphertext = cipher
.encrypt(nonce, data)
.map_err(|e| anyhow!("Encryption failed: {}", e))?;
// Format: [magic(4)][version(1)][salt(16)][nonce(12)][ciphertext]
let mut result = MAGIC.to_vec();
result.push(VERSION_1);
result.extend(salt);
result.extend(nonce_bytes);
result.extend(ciphertext);
Ok(result)
}
/// Decrypt data using AES-GCM (Version 1 format)
pub fn decrypt(&self, encrypted_data: &[u8]) -> Result<Vec<u8>> {
let header_len = MAGIC.len() + 1 + SALT_LEN + NONCE_LEN;
if encrypted_data.len() < header_len {
return Err(anyhow!("Encrypted data too short"));
}
// Verify magic and version: [magic(4)][version(1)][salt(16)][nonce(12)][ciphertext]
if &encrypted_data[0..MAGIC.len()] != MAGIC {
return Err(anyhow!(
"Invalid encrypted data format - missing magic bytes"
));
}
if encrypted_data[MAGIC.len()] != VERSION_1 {
return Err(anyhow!("Unsupported encryption version"));
}
let salt_start = MAGIC.len() + 1;
let nonce_start = salt_start + SALT_LEN;
let ciphertext_start = nonce_start + NONCE_LEN;
let salt = &encrypted_data[salt_start..nonce_start];
let nonce = Nonce::from_slice(&encrypted_data[nonce_start..ciphertext_start]);
let ciphertext = &encrypted_data[ciphertext_start..];
let key = Self::derive_operation_key(&self.master_key, salt);
let cipher = Aes256Gcm::new(&key);
// Decrypt
cipher
.decrypt(nonce, ciphertext)
.map_err(|e| anyhow!("Decryption failed: {}", e))
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_encrypt_decrypt_round_trip() {
let encryption = Encryption::new("test-key-for-encryption".to_string());
let original_data = b"Hello, World! This is test data.";
// Encrypt
let encrypted = encryption
.encrypt(original_data)
.expect("Encryption should succeed");
// Decrypt
let decrypted = encryption
.decrypt(&encrypted)
.expect("Decryption should succeed");
// Verify
assert_eq!(original_data.to_vec(), decrypted);
assert_ne!(original_data.to_vec(), encrypted);
}
#[test]
fn test_encrypt_decrypt_different_keys() {
let encryption1 = Encryption::new("key1".to_string());
let encryption2 = Encryption::new("key2".to_string());
let data = b"Test data";
let encrypted = encryption1.encrypt(data).unwrap();
let result = encryption2.decrypt(&encrypted);
assert!(result.is_err(), "Should fail with different key");
}
#[test]
fn test_encryption_creation() {
let encryption = Encryption::new("test-key".to_string());
// Encryption now stores master_key, not password
// Test that it can encrypt/decrypt
let data = b"test";
let encrypted = encryption.encrypt(data).unwrap();
let decrypted = encryption.decrypt(&encrypted).unwrap();
assert_eq!(data.to_vec(), decrypted);
}
#[test]
fn test_small_data() {
let encryption = Encryption::new("test-key".to_string());
let data = b"{}"; // Minimal JSON object
let encrypted = encryption.encrypt(data).unwrap();
let decrypted = encryption.decrypt(&encrypted).unwrap();
assert_eq!(data.to_vec(), decrypted);
}
}

View File

@@ -1,295 +0,0 @@
use crate::core::models::Account;
use anyhow::Result;
use serde::{Deserialize, Serialize};
use std::fs;
use std::path::Path;
use tracing::warn;
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AccountLink {
pub source_account_id: String,
pub dest_account_id: String,
#[serde(default = "default_source_adapter_type")]
pub source_adapter_type: String, // e.g., "gocardless", "other_source"
#[serde(default = "default_dest_adapter_type")]
pub dest_adapter_type: String, // e.g., "firefly", "other_destination"
pub auto_linked: bool,
}
fn default_source_adapter_type() -> String {
"gocardless".to_string()
}
fn default_dest_adapter_type() -> String {
"firefly".to_string()
}
#[derive(Debug, Serialize, Deserialize, Default)]
pub struct LinkStore {
pub links: Vec<AccountLink>,
#[serde(skip)]
cache_dir: String,
}
impl LinkStore {
/// Create new LinkStore with cache directory
pub fn new(cache_dir: String) -> Self {
Self {
links: Vec::new(),
cache_dir,
}
}
fn get_path(&self) -> String {
format!("{}/links.json", self.cache_dir)
}
pub fn load(cache_dir: String) -> Self {
let path = format!("{}/links.json", cache_dir);
if Path::new(&path).exists() {
match fs::read_to_string(&path) {
Ok(content) => match serde_json::from_str::<LinkStore>(&content) {
Ok(mut store) => {
store.cache_dir = cache_dir;
return store;
}
Err(e) => warn!("Failed to parse link store: {}", e),
},
Err(e) => warn!("Failed to read link store: {}", e),
}
}
Self::new(cache_dir)
}
pub fn save(&self) -> Result<()> {
let path = self.get_path();
if let Some(parent) = std::path::Path::new(&path).parent() {
std::fs::create_dir_all(parent)?;
}
let content = serde_json::to_string_pretty(self)?;
fs::write(path, content)?;
Ok(())
}
pub fn add_link(
&mut self,
source_account: &Account,
dest_account: &Account,
source_adapter_type: &str,
dest_adapter_type: &str,
auto_linked: bool,
) -> Result<bool, String> {
// Check if link already exists (exact same source-dest pair)
if self.links.iter().any(|l| {
l.source_account_id == source_account.id && l.dest_account_id == dest_account.id
}) {
return Ok(false); // Link already exists
}
// Check if source account is already linked to a DIFFERENT destination of this adapter type
if let Some(existing_link) = self.links.iter().find(|l| {
l.source_account_id == source_account.id
&& l.dest_adapter_type == dest_adapter_type
&& l.dest_account_id != dest_account.id
}) {
return Err(format!(
"Source account '{}' is already linked to destination '{}' of type '{}'. Unlink first to create a new link.",
source_account.id, existing_link.dest_account_id, dest_adapter_type
));
}
let link = AccountLink {
source_account_id: source_account.id.clone(),
dest_account_id: dest_account.id.clone(),
source_adapter_type: source_adapter_type.to_string(),
dest_adapter_type: dest_adapter_type.to_string(),
auto_linked,
};
self.links.push(link);
Ok(true)
}
pub fn find_link_by_source(&self, source_id: &str) -> Option<&AccountLink> {
self.links.iter().find(|l| l.source_account_id == source_id)
}
pub fn find_links_by_source(&self, source_id: &str) -> Vec<&AccountLink> {
self.links
.iter()
.filter(|l| l.source_account_id == source_id)
.collect()
}
pub fn find_link_by_source_and_dest_type(
&self,
source_id: &str,
dest_adapter_type: &str,
) -> Option<&AccountLink> {
self.links
.iter()
.find(|l| l.source_account_id == source_id && l.dest_adapter_type == dest_adapter_type)
}
}
pub fn auto_link_accounts(
source_accounts: &[Account],
dest_accounts: &[Account],
) -> Vec<(usize, usize)> {
let mut links = Vec::new();
for (i, source) in source_accounts.iter().enumerate() {
for (j, dest) in dest_accounts.iter().enumerate() {
if source.iban == dest.iban
&& source.iban.as_ref().map(|s| !s.is_empty()).unwrap_or(false)
{
links.push((i, j));
break; // First match
}
}
}
// Could add name similarity matching here
links
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_add_link_prevents_duplicates() {
let mut store = LinkStore::default();
let src = Account {
id: "src1".to_string(),
name: Some("Source Account".to_string()),
iban: Some("NL01".to_string()),
currency: "EUR".to_string(),
};
let dest = Account {
id: "dest1".to_string(),
name: Some("Destination Account".to_string()),
iban: Some("NL01".to_string()),
currency: "EUR".to_string(),
};
// First call should create a link
let first_result = store.add_link(&src, &dest, "gocardless", "firefly", true);
assert!(first_result.is_ok());
assert!(first_result.unwrap());
// Second call should not create a duplicate
let second_result = store.add_link(&src, &dest, "gocardless", "firefly", true);
assert!(second_result.is_ok());
assert!(!second_result.unwrap());
assert_eq!(store.links.len(), 1); // Still only one link
}
#[test]
fn test_add_link_allows_different_accounts() {
let mut store = LinkStore::default();
let src1 = Account {
id: "src1".to_string(),
name: Some("Source Account 1".to_string()),
iban: Some("NL01".to_string()),
currency: "EUR".to_string(),
};
let dest1 = Account {
id: "dest1".to_string(),
name: Some("Destination Account 1".to_string()),
iban: Some("NL01".to_string()),
currency: "EUR".to_string(),
};
let dest2 = Account {
id: "dest2".to_string(),
name: Some("Destination Account 2".to_string()),
iban: Some("NL02".to_string()),
currency: "EUR".to_string(),
};
// Link src1 to dest1 (firefly)
let result1 = store.add_link(&src1, &dest1, "gocardless", "firefly", false);
assert!(result1.is_ok());
assert!(result1.unwrap());
// Try to link src1 to dest2 (same adapter type) - should fail
let result2 = store.add_link(&src1, &dest2, "gocardless", "firefly", false);
assert!(result2.is_err());
assert!(result2.unwrap_err().contains("already linked"));
assert_eq!(store.links.len(), 1); // Still only one link
}
fn setup_test_dir(test_name: &str) -> String {
// Use a unique cache directory for each test to avoid interference
// Include random component and timestamp for true parallelism safety
let random_suffix = rand::random::<u64>();
let timestamp = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos();
format!(
"tmp/test-links-{}-{}-{}",
test_name, random_suffix, timestamp
)
}
fn cleanup_test_dir(cache_dir: &str) {
// Wait a bit longer to ensure all file operations are complete
std::thread::sleep(std::time::Duration::from_millis(50));
// Try multiple times in case of temporary file locks
for _ in 0..5 {
if std::path::Path::new(cache_dir).exists() {
if std::fs::remove_dir_all(cache_dir).is_ok() {
break;
}
} else {
break; // Directory already gone
}
std::thread::sleep(std::time::Duration::from_millis(10));
}
}
#[test]
fn test_load_links_store() {
let cache_dir = setup_test_dir("load");
// Create JSON in the correct format
let json_content = r#"{
"links": [
{
"source_account_id": "src1",
"dest_account_id": "dest1",
"source_adapter_type": "gocardless",
"dest_adapter_type": "firefly",
"auto_linked": true
},
{
"source_account_id": "src2",
"dest_account_id": "dest2",
"source_adapter_type": "gocardless",
"dest_adapter_type": "firefly",
"auto_linked": false
}
]
}"#;
// Create directory and write file
std::fs::create_dir_all(&cache_dir).unwrap();
std::fs::write(format!("{}/links.json", cache_dir), json_content).unwrap();
// Load should work and set cache_dir
let store = LinkStore::load(cache_dir.clone());
assert_eq!(store.links.len(), 2);
assert_eq!(store.links[0].source_account_id, "src1");
assert_eq!(store.links[0].dest_account_id, "dest1");
assert_eq!(store.links[0].source_adapter_type, "gocardless");
assert_eq!(store.links[0].dest_adapter_type, "firefly");
assert!(store.links[0].auto_linked);
assert_eq!(store.links[1].source_account_id, "src2");
assert_eq!(store.links[1].dest_account_id, "dest2");
assert_eq!(store.links[1].source_adapter_type, "gocardless");
assert_eq!(store.links[1].dest_adapter_type, "firefly");
assert!(!store.links[1].auto_linked);
assert_eq!(store.cache_dir, cache_dir);
cleanup_test_dir(&cache_dir);
}
}

View File

@@ -1,8 +1,3 @@
pub mod adapters;
pub mod cache;
pub mod config;
pub mod encryption;
pub mod linking;
pub mod models; pub mod models;
pub mod ports; pub mod ports;
pub mod sync; pub mod sync;

View File

@@ -1,6 +1,5 @@
use chrono::NaiveDate;
use rust_decimal::Decimal; use rust_decimal::Decimal;
use serde::Serialize; use chrono::NaiveDate;
use std::fmt; use std::fmt;
use thiserror::Error; use thiserror::Error;
@@ -33,29 +32,19 @@ impl fmt::Debug for BankTransaction {
.field("date", &self.date) .field("date", &self.date)
.field("amount", &"[REDACTED]") .field("amount", &"[REDACTED]")
.field("currency", &self.currency) .field("currency", &self.currency)
.field( .field("foreign_amount", &self.foreign_amount.as_ref().map(|_| "[REDACTED]"))
"foreign_amount",
&self.foreign_amount.as_ref().map(|_| "[REDACTED]"),
)
.field("foreign_currency", &self.foreign_currency) .field("foreign_currency", &self.foreign_currency)
.field("description", &"[REDACTED]") .field("description", &"[REDACTED]")
.field( .field("counterparty_name", &self.counterparty_name.as_ref().map(|_| "[REDACTED]"))
"counterparty_name", .field("counterparty_iban", &self.counterparty_iban.as_ref().map(|_| "[REDACTED]"))
&self.counterparty_name.as_ref().map(|_| "[REDACTED]"),
)
.field(
"counterparty_iban",
&self.counterparty_iban.as_ref().map(|_| "[REDACTED]"),
)
.finish() .finish()
} }
} }
#[derive(Clone, PartialEq, serde::Serialize, serde::Deserialize)] #[derive(Clone, PartialEq)]
pub struct Account { pub struct Account {
pub id: String, pub id: String,
pub name: Option<String>, // Account display name pub iban: String,
pub iban: Option<String>, // IBAN may not be available for all accounts
pub currency: String, pub currency: String,
} }
@@ -69,13 +58,6 @@ impl fmt::Debug for Account {
} }
} }
/// Common interface for account data from different sources
pub trait AccountData {
fn id(&self) -> &str;
fn iban(&self) -> Option<&str>;
fn display_name(&self) -> Option<String>;
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
@@ -112,8 +94,7 @@ mod tests {
fn test_account_debug_masks_iban() { fn test_account_debug_masks_iban() {
let account = Account { let account = Account {
id: "123".to_string(), id: "123".to_string(),
name: Some("Test Account".to_string()), iban: "DE1234567890".to_string(),
iban: Some("DE1234567890".to_string()),
currency: "EUR".to_string(), currency: "EUR".to_string(),
}; };
@@ -125,40 +106,6 @@ mod tests {
} }
} }
#[derive(Clone, Debug, Serialize)]
pub struct AccountSummary {
pub id: String,
pub name: Option<String>,
pub iban: String,
pub currency: String,
}
#[derive(Clone, Debug, Serialize)]
pub struct AccountStatus {
pub account_id: String,
pub iban: String,
pub last_sync_date: Option<NaiveDate>,
pub transaction_count: usize,
pub status: String, // e.g., "synced", "pending", "error"
}
#[derive(Clone, Debug, Serialize)]
pub struct TransactionInfo {
pub account_id: String,
pub total_count: usize,
pub date_range: Option<(NaiveDate, NaiveDate)>,
pub last_updated: Option<NaiveDate>,
}
#[derive(Clone, Debug, Serialize)]
pub struct CacheInfo {
pub account_id: Option<String>, // None for global, Some for per-account
pub cache_type: String, // e.g., "account", "transaction"
pub entry_count: usize,
pub total_size_bytes: usize,
pub last_updated: Option<NaiveDate>,
}
#[derive(Error, Debug)] #[derive(Error, Debug)]
pub enum SyncError { pub enum SyncError {
#[error("End User Agreement {agreement_id} has expired")] #[error("End User Agreement {agreement_id} has expired")]

View File

@@ -1,11 +1,9 @@
use crate::core::models::{
Account, AccountStatus, AccountSummary, BankTransaction, CacheInfo, TransactionInfo,
};
use anyhow::Result;
use async_trait::async_trait; use async_trait::async_trait;
use chrono::NaiveDate; use chrono::NaiveDate;
use anyhow::Result;
#[cfg(test)] #[cfg(test)]
use mockall::automock; use mockall::automock;
use crate::core::models::{BankTransaction, Account};
#[derive(Debug, Default)] #[derive(Debug, Default)]
pub struct IngestResult { pub struct IngestResult {
@@ -20,21 +18,7 @@ pub struct IngestResult {
pub trait TransactionSource: Send + Sync { pub trait TransactionSource: Send + Sync {
/// Fetch accounts. Optionally filter by a list of wanted IBANs to save requests. /// Fetch accounts. Optionally filter by a list of wanted IBANs to save requests.
async fn get_accounts(&self, wanted_ibans: Option<Vec<String>>) -> Result<Vec<Account>>; async fn get_accounts(&self, wanted_ibans: Option<Vec<String>>) -> Result<Vec<Account>>;
async fn get_transactions( async fn get_transactions(&self, account_id: &str, start: NaiveDate, end: NaiveDate) -> Result<Vec<BankTransaction>>;
&self,
account_id: &str,
start: NaiveDate,
end: NaiveDate,
) -> Result<Vec<BankTransaction>>;
/// Inspection methods for CLI
async fn list_accounts(&self) -> Result<Vec<AccountSummary>>;
async fn get_account_status(&self) -> Result<Vec<AccountStatus>>;
async fn get_transaction_info(&self, account_id: &str) -> Result<TransactionInfo>;
async fn get_cache_info(&self) -> Result<Vec<CacheInfo>>;
/// Account discovery for linking
async fn discover_accounts(&self) -> Result<Vec<Account>>;
} }
// Blanket implementation for references // Blanket implementation for references
@@ -44,34 +28,9 @@ impl<T: TransactionSource> TransactionSource for &T {
(**self).get_accounts(wanted_ibans).await (**self).get_accounts(wanted_ibans).await
} }
async fn get_transactions( async fn get_transactions(&self, account_id: &str, start: NaiveDate, end: NaiveDate) -> Result<Vec<BankTransaction>> {
&self,
account_id: &str,
start: NaiveDate,
end: NaiveDate,
) -> Result<Vec<BankTransaction>> {
(**self).get_transactions(account_id, start, end).await (**self).get_transactions(account_id, start, end).await
} }
async fn list_accounts(&self) -> Result<Vec<AccountSummary>> {
(**self).list_accounts().await
}
async fn get_account_status(&self) -> Result<Vec<AccountStatus>> {
(**self).get_account_status().await
}
async fn get_transaction_info(&self, account_id: &str) -> Result<TransactionInfo> {
(**self).get_transaction_info(account_id).await
}
async fn get_cache_info(&self) -> Result<Vec<CacheInfo>> {
(**self).get_cache_info().await
}
async fn discover_accounts(&self) -> Result<Vec<Account>> {
(**self).discover_accounts().await
}
} }
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
@@ -83,35 +42,33 @@ pub struct TransactionMatch {
#[cfg_attr(test, automock)] #[cfg_attr(test, automock)]
#[async_trait] #[async_trait]
pub trait TransactionDestination: Send + Sync { pub trait TransactionDestination: Send + Sync {
async fn resolve_account_id(&self, iban: &str) -> Result<Option<String>>;
/// Get list of all active asset account IBANs to drive the sync
async fn get_active_account_ibans(&self) -> Result<Vec<String>>;
// New granular methods for Healer Logic // New granular methods for Healer Logic
async fn get_last_transaction_date(&self, account_id: &str) -> Result<Option<NaiveDate>>; async fn get_last_transaction_date(&self, account_id: &str) -> Result<Option<NaiveDate>>;
async fn find_transaction( async fn find_transaction(&self, account_id: &str, transaction: &BankTransaction) -> Result<Option<TransactionMatch>>;
&self,
account_id: &str,
transaction: &BankTransaction,
) -> Result<Option<TransactionMatch>>;
async fn create_transaction(&self, account_id: &str, tx: &BankTransaction) -> Result<()>; async fn create_transaction(&self, account_id: &str, tx: &BankTransaction) -> Result<()>;
async fn update_transaction_external_id(&self, id: &str, external_id: &str) -> Result<()>; async fn update_transaction_external_id(&self, id: &str, external_id: &str) -> Result<()>;
/// Account discovery for linking
async fn discover_accounts(&self) -> Result<Vec<Account>>;
/// Inspection methods for CLI
async fn list_accounts(&self) -> Result<Vec<AccountSummary>>;
} }
// Blanket implementation for references // Blanket implementation for references
#[async_trait] #[async_trait]
impl<T: TransactionDestination> TransactionDestination for &T { impl<T: TransactionDestination> TransactionDestination for &T {
async fn resolve_account_id(&self, iban: &str) -> Result<Option<String>> {
(**self).resolve_account_id(iban).await
}
async fn get_active_account_ibans(&self) -> Result<Vec<String>> {
(**self).get_active_account_ibans().await
}
async fn get_last_transaction_date(&self, account_id: &str) -> Result<Option<NaiveDate>> { async fn get_last_transaction_date(&self, account_id: &str) -> Result<Option<NaiveDate>> {
(**self).get_last_transaction_date(account_id).await (**self).get_last_transaction_date(account_id).await
} }
async fn find_transaction( async fn find_transaction(&self, account_id: &str, transaction: &BankTransaction) -> Result<Option<TransactionMatch>> {
&self,
account_id: &str,
transaction: &BankTransaction,
) -> Result<Option<TransactionMatch>> {
(**self).find_transaction(account_id, transaction).await (**self).find_transaction(account_id, transaction).await
} }
@@ -120,16 +77,6 @@ impl<T: TransactionDestination> TransactionDestination for &T {
} }
async fn update_transaction_external_id(&self, id: &str, external_id: &str) -> Result<()> { async fn update_transaction_external_id(&self, id: &str, external_id: &str) -> Result<()> {
(**self) (**self).update_transaction_external_id(id, external_id).await
.update_transaction_external_id(id, external_id)
.await
}
async fn discover_accounts(&self) -> Result<Vec<Account>> {
(**self).discover_accounts().await
}
async fn list_accounts(&self) -> Result<Vec<AccountSummary>> {
(**self).list_accounts().await
} }
} }

View File

@@ -1,10 +1,9 @@
use crate::core::config::Config;
use crate::core::linking::{auto_link_accounts, LinkStore};
use crate::core::models::{Account, SyncError};
use crate::core::ports::{IngestResult, TransactionDestination, TransactionSource};
use anyhow::Result; use anyhow::Result;
use chrono::{Local, NaiveDate}; use tracing::{info, warn, instrument};
use tracing::{info, instrument, warn}; use crate::core::ports::{IngestResult, TransactionSource, TransactionDestination};
use crate::core::models::{SyncError, Account};
use chrono::{NaiveDate, Local};
#[derive(Debug, Default)] #[derive(Debug, Default)]
pub struct SyncResult { pub struct SyncResult {
@@ -14,108 +13,48 @@ pub struct SyncResult {
pub accounts_skipped_errors: usize, pub accounts_skipped_errors: usize,
} }
#[instrument(skip(source, destination, config))] #[instrument(skip(source, destination))]
pub async fn run_sync( pub async fn run_sync(
source: impl TransactionSource, source: impl TransactionSource,
destination: impl TransactionDestination, destination: impl TransactionDestination,
config: Config,
cli_start_date: Option<NaiveDate>, cli_start_date: Option<NaiveDate>,
cli_end_date: Option<NaiveDate>, cli_end_date: Option<NaiveDate>,
dry_run: bool, dry_run: bool,
) -> Result<SyncResult> { ) -> Result<SyncResult> {
info!("Starting synchronization..."); info!("Starting synchronization...");
// Discover all accounts from both source and destination // Optimization: Get active Firefly IBANs first
let all_source_accounts = source let wanted_ibans = destination.get_active_account_ibans().await.map_err(SyncError::DestinationError)?;
.discover_accounts() info!("Syncing {} active accounts from Firefly III", wanted_ibans.len());
.await
.map_err(SyncError::SourceError)?;
let all_dest_accounts = destination
.discover_accounts()
.await
.map_err(SyncError::DestinationError)?;
info!(
"Discovered {} source accounts and {} destination accounts",
all_source_accounts.len(),
all_dest_accounts.len()
);
// Accounts are cached by their respective adapters during discover_accounts let accounts = source.get_accounts(Some(wanted_ibans)).await.map_err(SyncError::SourceError)?;
info!("Found {} accounts from source", accounts.len());
let mut link_store = LinkStore::load(config.cache.directory.clone());
// Auto-link accounts based on IBAN
let links = auto_link_accounts(&all_source_accounts, &all_dest_accounts);
for (src_idx, dest_idx) in links {
let src = &all_source_accounts[src_idx];
let dest = &all_dest_accounts[dest_idx];
match link_store.add_link(src, dest, "gocardless", "firefly", true) {
Ok(true) => {
info!("Created new account link: {} -> {}", src.id, dest.id);
}
Ok(false) => {
info!("Account link already exists: {} -> {}", src.id, dest.id);
}
Err(e) => {
warn!("Failed to create account link: {}", e);
}
}
}
link_store.save().map_err(SyncError::SourceError)?;
// Get all matched accounts (those with existing links)
let mut accounts_to_sync = Vec::new();
for source_account in &all_source_accounts {
if link_store.find_link_by_source(&source_account.id).is_some() {
accounts_to_sync.push(source_account.clone());
}
}
info!(
"Found {} accounts with existing links to sync",
accounts_to_sync.len()
);
// Default end date is Yesterday // Default end date is Yesterday
let end_date = let end_date = cli_end_date.unwrap_or_else(|| Local::now().date_naive() - chrono::Duration::days(1));
cli_end_date.unwrap_or_else(|| Local::now().date_naive() - chrono::Duration::days(1));
let mut result = SyncResult::default(); let mut result = SyncResult::default();
for account in accounts_to_sync { for account in accounts {
let span = tracing::info_span!("sync_account", account_id = %account.id); let span = tracing::info_span!("sync_account", account_id = %account.id);
let _enter = span.enter(); let _enter = span.enter();
info!("Processing account..."); info!("Processing account...");
// Process account with error handling // Process account with error handling
match process_single_account( match process_single_account(&source, &destination, &account, cli_start_date, end_date, dry_run).await {
&source,
&destination,
&account,
&link_store,
cli_start_date,
end_date,
dry_run,
)
.await
{
Ok(stats) => { Ok(stats) => {
result.accounts_processed += 1; result.accounts_processed += 1;
result.ingest.created += stats.created; result.ingest.created += stats.created;
result.ingest.healed += stats.healed; result.ingest.healed += stats.healed;
result.ingest.duplicates += stats.duplicates; result.ingest.duplicates += stats.duplicates;
result.ingest.errors += stats.errors; result.ingest.errors += stats.errors;
info!( info!("Account {} sync complete. Created: {}, Healed: {}, Duplicates: {}, Errors: {}",
"Account {} sync complete. Created: {}, Healed: {}, Duplicates: {}, Errors: {}", account.id, stats.created, stats.healed, stats.duplicates, stats.errors);
account.id, stats.created, stats.healed, stats.duplicates, stats.errors
);
} }
Err(SyncError::AgreementExpired { agreement_id }) => { Err(SyncError::AgreementExpired { agreement_id }) => {
result.accounts_skipped_expired += 1; result.accounts_skipped_expired += 1;
warn!( warn!("Account {} skipped - associated agreement {} has expired", account.id, agreement_id);
"Account {} skipped - associated agreement {} has expired",
account.id, agreement_id
);
} }
Err(SyncError::AccountSkipped { account_id, reason }) => { Err(SyncError::AccountSkipped { account_id, reason }) => {
result.accounts_skipped_errors += 1; result.accounts_skipped_errors += 1;
@@ -128,14 +67,10 @@ pub async fn run_sync(
} }
} }
info!( info!("Synchronization finished. Processed: {}, Skipped (expired): {}, Skipped (errors): {}",
"Synchronization finished. Processed: {}, Skipped (expired): {}, Skipped (errors): {}", result.accounts_processed, result.accounts_skipped_expired, result.accounts_skipped_errors);
result.accounts_processed, result.accounts_skipped_expired, result.accounts_skipped_errors info!("Total transactions - Created: {}, Healed: {}, Duplicates: {}, Errors: {}",
); result.ingest.created, result.ingest.healed, result.ingest.duplicates, result.ingest.errors);
info!(
"Total transactions - Created: {}, Healed: {}, Duplicates: {}, Errors: {}",
result.ingest.created, result.ingest.healed, result.ingest.duplicates, result.ingest.errors
);
Ok(result) Ok(result)
} }
@@ -144,19 +79,17 @@ async fn process_single_account(
source: &impl TransactionSource, source: &impl TransactionSource,
destination: &impl TransactionDestination, destination: &impl TransactionDestination,
account: &Account, account: &Account,
link_store: &LinkStore,
cli_start_date: Option<NaiveDate>, cli_start_date: Option<NaiveDate>,
end_date: NaiveDate, end_date: NaiveDate,
dry_run: bool, dry_run: bool,
) -> Result<IngestResult, SyncError> { ) -> Result<IngestResult, SyncError> {
let link_opt = link_store.find_link_by_source(&account.id); let dest_id_opt = destination.resolve_account_id(&account.iban).await.map_err(SyncError::DestinationError)?;
let Some(link) = link_opt else { let Some(dest_id) = dest_id_opt else {
return Err(SyncError::AccountSkipped { return Err(SyncError::AccountSkipped {
account_id: account.id.clone(), account_id: account.id.clone(),
reason: "No link found to destination account".to_string(), reason: "Not found in destination".to_string(),
}); });
}; };
let dest_id = link.dest_account_id.clone();
info!("Resolved destination ID: {}", dest_id); info!("Resolved destination ID: {}", dest_id);
@@ -165,34 +98,24 @@ async fn process_single_account(
d d
} else { } else {
// Default: Latest transaction date + 1 day // Default: Latest transaction date + 1 day
match destination match destination.get_last_transaction_date(&dest_id).await.map_err(SyncError::DestinationError)? {
.get_last_transaction_date(&dest_id)
.await
.map_err(SyncError::DestinationError)?
{
Some(last_date) => last_date + chrono::Duration::days(1), Some(last_date) => last_date + chrono::Duration::days(1),
None => { None => {
// If no transaction exists in Firefly, we assume this is a fresh sync. // If no transaction exists in Firefly, we assume this is a fresh sync.
// Default to syncing last 30 days. // Default to syncing last 30 days.
end_date - chrono::Duration::days(30) end_date - chrono::Duration::days(30)
} },
} }
}; };
if start_date > end_date { if start_date > end_date {
info!( info!("Start date {} is after end date {}. Nothing to sync.", start_date, end_date);
"Start date {} is after end date {}. Nothing to sync.",
start_date, end_date
);
return Ok(IngestResult::default()); return Ok(IngestResult::default());
} }
info!("Syncing interval: {} to {}", start_date, end_date); info!("Syncing interval: {} to {}", start_date, end_date);
let transactions = match source let transactions = match source.get_transactions(&account.id, start_date, end_date).await {
.get_transactions(&account.id, start_date, end_date)
.await
{
Ok(txns) => txns, Ok(txns) => txns,
Err(e) => { Err(e) => {
let err_str = e.to_string(); let err_str = e.to_string();
@@ -217,11 +140,7 @@ async fn process_single_account(
// Healer Logic Loop // Healer Logic Loop
for tx in transactions { for tx in transactions {
// 1. Check if it exists // 1. Check if it exists
match destination match destination.find_transaction(&dest_id, &tx).await.map_err(SyncError::DestinationError)? {
.find_transaction(&dest_id, &tx)
.await
.map_err(SyncError::DestinationError)?
{
Some(existing) => { Some(existing) => {
if existing.has_external_id { if existing.has_external_id {
// Already synced properly // Already synced properly
@@ -229,20 +148,11 @@ async fn process_single_account(
} else { } else {
// Found "naked" transaction -> Heal it // Found "naked" transaction -> Heal it
if dry_run { if dry_run {
info!( info!("[DRY RUN] Would heal transaction {} (Firefly ID: {})", tx.internal_id, existing.id);
"[DRY RUN] Would heal transaction {} (Firefly ID: {})",
tx.internal_id, existing.id
);
stats.healed += 1; stats.healed += 1;
} else { } else {
info!( info!("Healing transaction {} (Firefly ID: {})", tx.internal_id, existing.id);
"Healing transaction {} (Firefly ID: {})", if let Err(e) = destination.update_transaction_external_id(&existing.id, &tx.internal_id).await {
tx.internal_id, existing.id
);
if let Err(e) = destination
.update_transaction_external_id(&existing.id, &tx.internal_id)
.await
{
tracing::error!("Failed to heal transaction: {}", e); tracing::error!("Failed to heal transaction: {}", e);
stats.errors += 1; stats.errors += 1;
} else { } else {
@@ -250,13 +160,14 @@ async fn process_single_account(
} }
} }
} }
} },
None => { None => {
// New transaction // New transaction
if dry_run { if dry_run {
info!("[DRY RUN] Would create transaction {}", tx.internal_id); info!("[DRY RUN] Would create transaction {}", tx.internal_id);
stats.created += 1; stats.created += 1;
} else if let Err(e) = destination.create_transaction(&dest_id, &tx).await { } else {
if let Err(e) = destination.create_transaction(&dest_id, &tx).await {
// Firefly might still reject it as duplicate if hash matches, even if we didn't find it via heuristic // Firefly might still reject it as duplicate if hash matches, even if we didn't find it via heuristic
// (unlikely if heuristic is good, but possible) // (unlikely if heuristic is good, but possible)
let err_str = e.to_string(); let err_str = e.to_string();
@@ -273,6 +184,7 @@ async fn process_single_account(
} }
} }
} }
}
Ok(stats) Ok(stats)
} }
@@ -280,38 +192,10 @@ async fn process_single_account(
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
use crate::core::config::{ use crate::core::ports::{MockTransactionSource, MockTransactionDestination, TransactionMatch};
CacheConfig, Config, FireflyConfig, GoCardlessConfig, LoggingConfig,
};
use crate::core::models::{Account, BankTransaction}; use crate::core::models::{Account, BankTransaction};
use crate::core::ports::{MockTransactionDestination, MockTransactionSource, TransactionMatch};
use mockall::predicate::*;
use rust_decimal::Decimal; use rust_decimal::Decimal;
use mockall::predicate::*;
fn create_unique_key(prefix: &str) -> String {
format!("{}-{}", prefix, rand::random::<u64>())
}
fn create_test_config(temp_dir: &str) -> Config {
Config {
gocardless: GoCardlessConfig {
url: "https://bankaccountdata.gocardless.com".to_string(),
secret_id: create_unique_key("gocardless-id"),
secret_key: create_unique_key("gocardless-key"),
},
firefly: FireflyConfig {
url: "https://firefly.test.com".to_string(),
api_key: create_unique_key("firefly-api-key"),
},
cache: CacheConfig {
key: create_unique_key("cache-key"),
directory: temp_dir.to_string(),
},
logging: LoggingConfig {
level: "warn".to_string(),
},
}
}
#[tokio::test] #[tokio::test]
async fn test_sync_flow_create_new() { async fn test_sync_flow_create_new() {
@@ -319,26 +203,13 @@ mod tests {
let mut dest = MockTransactionDestination::new(); let mut dest = MockTransactionDestination::new();
// Source setup // Source setup
source source.expect_get_accounts()
.expect_get_accounts()
.with(always()) // Match any argument .with(always()) // Match any argument
.returning(|_| { .returning(|_| Ok(vec![Account {
Ok(vec![Account {
id: "src_1".to_string(), id: "src_1".to_string(),
name: Some("Test Account".to_string()), iban: "NL01".to_string(),
iban: Some("NL01".to_string()),
currency: "EUR".to_string(), currency: "EUR".to_string(),
}]) }]));
});
source.expect_discover_accounts().returning(|| {
Ok(vec![Account {
id: "src_1".to_string(),
name: Some("Test Account".to_string()),
iban: Some("NL01".to_string()),
currency: "EUR".to_string(),
}])
});
let tx = BankTransaction { let tx = BankTransaction {
internal_id: "tx1".into(), internal_id: "tx1".into(),
@@ -353,19 +224,15 @@ mod tests {
}; };
let tx_clone = tx.clone(); let tx_clone = tx.clone();
source source.expect_get_transactions()
.expect_get_transactions()
.returning(move |_, _, _| Ok(vec![tx.clone()])); .returning(move |_, _, _| Ok(vec![tx.clone()]));
// Destination setup // Destination setup
dest.expect_discover_accounts().returning(|| { dest.expect_get_active_account_ibans()
Ok(vec![Account { .returning(|| Ok(vec!["NL01".to_string()]));
id: "dest_1".to_string(),
name: Some("Savings Account".to_string()), dest.expect_resolve_account_id()
iban: Some("NL01".to_string()), .returning(|_| Ok(Some("dest_1".into())));
currency: "EUR".to_string(),
}])
});
dest.expect_get_last_transaction_date() dest.expect_get_last_transaction_date()
.returning(|_| Ok(Some(NaiveDate::from_ymd_opt(2022, 12, 31).unwrap()))); .returning(|_| Ok(Some(NaiveDate::from_ymd_opt(2022, 12, 31).unwrap())));
@@ -382,13 +249,8 @@ mod tests {
.returning(|_, _| Ok(())); .returning(|_, _| Ok(()));
// Execution // Execution
let temp_dir = format!("tmp/test-sync-{}", rand::random::<u64>()); let res = run_sync(&source, &dest, None, None, false).await;
let config = create_test_config(&temp_dir);
let res = run_sync(&source, &dest, config, None, None, false).await;
assert!(res.is_ok()); assert!(res.is_ok());
// Cleanup
let _ = std::fs::remove_dir_all(&temp_dir);
} }
#[tokio::test] #[tokio::test]
@@ -396,35 +258,20 @@ mod tests {
let mut source = MockTransactionSource::new(); let mut source = MockTransactionSource::new();
let mut dest = MockTransactionDestination::new(); let mut dest = MockTransactionDestination::new();
dest.expect_discover_accounts().returning(|| { dest.expect_get_active_account_ibans()
Ok(vec![Account { .returning(|| Ok(vec!["NL01".to_string()]));
id: "dest_1".to_string(),
name: Some("Savings Account".to_string()),
iban: Some("NL01".to_string()),
currency: "EUR".to_string(),
}])
});
source.expect_get_accounts().with(always()).returning(|_| { source.expect_get_accounts()
Ok(vec![Account { .with(always())
.returning(|_| Ok(vec![Account {
id: "src_1".to_string(), id: "src_1".to_string(),
name: Some("Test Account".to_string()), iban: "NL01".to_string(),
iban: Some("NL01".to_string()),
currency: "EUR".to_string(), currency: "EUR".to_string(),
}]) }]));
});
source.expect_discover_accounts().returning(|| { source.expect_get_transactions()
Ok(vec![Account { .returning(|_, _, _| Ok(vec![
id: "src_1".to_string(), BankTransaction {
name: Some("Test Account".to_string()),
iban: Some("NL01".to_string()),
currency: "EUR".to_string(),
}])
});
source.expect_get_transactions().returning(|_, _, _| {
Ok(vec![BankTransaction {
internal_id: "tx1".into(), internal_id: "tx1".into(),
date: NaiveDate::from_ymd_opt(2023, 1, 1).unwrap(), date: NaiveDate::from_ymd_opt(2023, 1, 1).unwrap(),
amount: Decimal::new(100, 0), amount: Decimal::new(100, 0),
@@ -434,19 +281,19 @@ mod tests {
description: "Test".into(), description: "Test".into(),
counterparty_name: None, counterparty_name: None,
counterparty_iban: None, counterparty_iban: None,
}]) }
}); ]));
dest.expect_get_last_transaction_date() dest.expect_resolve_account_id().returning(|_| Ok(Some("dest_1".into())));
.returning(|_| Ok(Some(NaiveDate::from_ymd_opt(2022, 12, 31).unwrap()))); dest.expect_get_last_transaction_date().returning(|_| Ok(Some(NaiveDate::from_ymd_opt(2022, 12, 31).unwrap())));
// 1. Find -> Some(No External ID) // 1. Find -> Some(No External ID)
dest.expect_find_transaction().times(1).returning(|_, _| { dest.expect_find_transaction()
Ok(Some(TransactionMatch { .times(1)
.returning(|_, _| Ok(Some(TransactionMatch {
id: "ff_tx_1".to_string(), id: "ff_tx_1".to_string(),
has_external_id: false, has_external_id: false,
})) })));
});
// 2. Update -> Ok // 2. Update -> Ok
dest.expect_update_transaction_external_id() dest.expect_update_transaction_external_id()
@@ -454,13 +301,8 @@ mod tests {
.times(1) .times(1)
.returning(|_, _| Ok(())); .returning(|_, _| Ok(()));
let temp_dir = format!("tmp/test-sync-heal-{}", rand::random::<u64>()); let res = run_sync(&source, &dest, None, None, false).await;
let config = create_test_config(&temp_dir);
let res = run_sync(&source, &dest, config, None, None, false).await;
assert!(res.is_ok()); assert!(res.is_ok());
// Cleanup
let _ = std::fs::remove_dir_all(&temp_dir);
} }
#[tokio::test] #[tokio::test]
@@ -468,32 +310,16 @@ mod tests {
let mut source = MockTransactionSource::new(); let mut source = MockTransactionSource::new();
let mut dest = MockTransactionDestination::new(); let mut dest = MockTransactionDestination::new();
dest.expect_discover_accounts().returning(|| { dest.expect_get_active_account_ibans()
Ok(vec![Account { .returning(|| Ok(vec!["NL01".to_string()]));
id: "dest_1".to_string(),
name: Some("Savings Account".to_string()),
iban: Some("NL01".to_string()),
currency: "EUR".to_string(),
}])
});
source.expect_get_accounts().with(always()).returning(|_| { source.expect_get_accounts()
Ok(vec![Account { .with(always())
.returning(|_| Ok(vec![Account {
id: "src_1".to_string(), id: "src_1".to_string(),
name: Some("Test Account".to_string()), iban: "NL01".to_string(),
iban: Some("NL01".to_string()),
currency: "EUR".to_string(), currency: "EUR".to_string(),
}]) }]));
});
source.expect_discover_accounts().returning(|| {
Ok(vec![Account {
id: "src_1".to_string(),
name: Some("Test Account".to_string()),
iban: Some("NL01".to_string()),
currency: "EUR".to_string(),
}])
});
let tx = BankTransaction { let tx = BankTransaction {
internal_id: "tx1".into(), internal_id: "tx1".into(),
@@ -507,26 +333,21 @@ mod tests {
counterparty_iban: None, counterparty_iban: None,
}; };
source source.expect_get_transactions()
.expect_get_transactions()
.returning(move |_, _, _| Ok(vec![tx.clone()])); .returning(move |_, _, _| Ok(vec![tx.clone()]));
dest.expect_get_last_transaction_date() dest.expect_resolve_account_id().returning(|_| Ok(Some("dest_1".into())));
.returning(|_| Ok(Some(NaiveDate::from_ymd_opt(2022, 12, 31).unwrap()))); dest.expect_get_last_transaction_date().returning(|_| Ok(Some(NaiveDate::from_ymd_opt(2022, 12, 31).unwrap())));
// 1. Find -> None (New transaction) // 1. Find -> None (New transaction)
dest.expect_find_transaction().returning(|_, _| Ok(None)); dest.expect_find_transaction()
.returning(|_, _| Ok(None));
// 2. Create -> NEVER Called (Dry Run) // 2. Create -> NEVER Called (Dry Run)
dest.expect_create_transaction().never(); dest.expect_create_transaction().never();
dest.expect_update_transaction_external_id().never(); dest.expect_update_transaction_external_id().never();
let temp_dir = format!("tmp/test-sync-dry-run-{}", rand::random::<u64>()); let res = run_sync(source, dest, None, None, true).await;
let config = create_test_config(&temp_dir);
let res = run_sync(source, dest, config, None, None, true).await;
assert!(res.is_ok()); assert!(res.is_ok());
// Cleanup
let _ = std::fs::remove_dir_all(&temp_dir);
} }
} }

View File

@@ -1,11 +1,11 @@
use chrono::Utc;
use hyper::Body;
use reqwest::{Request, Response};
use reqwest_middleware::{Middleware, Next}; use reqwest_middleware::{Middleware, Next};
use task_local_extensions::Extensions;
use reqwest::{Request, Response};
use std::sync::atomic::{AtomicU64, Ordering};
use std::fs; use std::fs;
use std::path::Path; use std::path::Path;
use std::sync::atomic::{AtomicU64, Ordering}; use chrono::Utc;
use task_local_extensions::Extensions; use hyper::Body;
static REQUEST_COUNTER: AtomicU64 = AtomicU64::new(0); static REQUEST_COUNTER: AtomicU64 = AtomicU64::new(0);
@@ -51,11 +51,7 @@ impl Middleware for DebugLogger {
log_content.push_str("# Request:\n"); log_content.push_str("# Request:\n");
log_content.push_str(&format!("{} {} HTTP/1.1\n", req.method(), req.url())); log_content.push_str(&format!("{} {} HTTP/1.1\n", req.method(), req.url()));
for (key, value) in req.headers() { for (key, value) in req.headers() {
log_content.push_str(&format!( log_content.push_str(&format!("{}: {}\n", key, value.to_str().unwrap_or("[INVALID]")));
"{}: {}\n",
key,
value.to_str().unwrap_or("[INVALID]")
));
} }
if let Some(body) = req.body() { if let Some(body) = req.body() {
if let Some(bytes) = body.as_bytes() { if let Some(bytes) = body.as_bytes() {
@@ -74,26 +70,13 @@ impl Middleware for DebugLogger {
// Response // Response
log_content.push_str("# Response:\n"); log_content.push_str("# Response:\n");
log_content.push_str(&format!( log_content.push_str(&format!("HTTP/1.1 {} {}\n", status.as_u16(), status.canonical_reason().unwrap_or("Unknown")));
"HTTP/1.1 {} {}\n",
status.as_u16(),
status.canonical_reason().unwrap_or("Unknown")
));
for (key, value) in &headers { for (key, value) in &headers {
log_content.push_str(&format!( log_content.push_str(&format!("{}: {}\n", key, value.to_str().unwrap_or("[INVALID]")));
"{}: {}\n",
key,
value.to_str().unwrap_or("[INVALID]")
));
} }
// Read body // Read body
let body_bytes = response.bytes().await.map_err(|e| { let body_bytes = response.bytes().await.map_err(|e| reqwest_middleware::Error::Middleware(anyhow::anyhow!("Failed to read response body: {}", e)))?;
reqwest_middleware::Error::Middleware(anyhow::anyhow!(
"Failed to read response body: {}",
e
))
})?;
let body_str = String::from_utf8_lossy(&body_bytes); let body_str = String::from_utf8_lossy(&body_bytes);
log_content.push_str(&format!("\n{}", body_str)); log_content.push_str(&format!("\n{}", body_str));
@@ -103,7 +86,9 @@ impl Middleware for DebugLogger {
} }
// Reconstruct response // Reconstruct response
let mut builder = http::Response::builder().status(status).version(version); let mut builder = http::Response::builder()
.status(status)
.version(version);
for (key, value) in &headers { for (key, value) in &headers {
builder = builder.header(key, value); builder = builder.header(key, value);
} }

View File

@@ -1,16 +1,18 @@
mod adapters; mod adapters;
mod cli;
mod commands;
mod core; mod core;
mod debug; mod debug;
use crate::commands::accounts::AccountCommands; use clap::Parser;
use crate::commands::sync::handle_sync; use tracing::{info, error};
use crate::commands::transactions::TransactionCommands; use crate::adapters::gocardless::client::GoCardlessAdapter;
use crate::core::config::Config; use crate::adapters::firefly::client::FireflyAdapter;
use crate::core::sync::run_sync;
use crate::debug::DebugLogger;
use gocardless_client::client::GoCardlessClient;
use firefly_client::client::FireflyClient;
use reqwest_middleware::ClientBuilder;
use std::env;
use chrono::NaiveDate; use chrono::NaiveDate;
use clap::{Parser, Subcommand};
use tracing::info;
#[derive(Parser, Debug)] #[derive(Parser, Debug)]
#[command(author, version, about, long_about = None)] #[command(author, version, about, long_about = None)]
@@ -19,6 +21,14 @@ struct Args {
#[arg(short, long)] #[arg(short, long)]
config: Option<String>, config: Option<String>,
/// Start date for synchronization (YYYY-MM-DD). Defaults to last transaction date + 1.
#[arg(short, long)]
start: Option<NaiveDate>,
/// End date for synchronization (YYYY-MM-DD). Defaults to yesterday.
#[arg(short, long)]
end: Option<NaiveDate>,
/// Dry run mode: Do not create or update transactions in Firefly III. /// Dry run mode: Do not create or update transactions in Firefly III.
#[arg(long, default_value_t = false)] #[arg(long, default_value_t = false)]
dry_run: bool, dry_run: bool,
@@ -26,130 +36,67 @@ struct Args {
/// Enable debug logging of HTTP requests/responses to ./debug_logs/ /// Enable debug logging of HTTP requests/responses to ./debug_logs/
#[arg(long, default_value_t = false)] #[arg(long, default_value_t = false)]
debug: bool, debug: bool,
#[command(subcommand)]
command: Commands,
}
#[derive(Subcommand, Debug)]
enum Commands {
/// Synchronize transactions between source and destination
Sync {
/// Source type (gocardless, csv, camt053, mt940)
source: String,
/// Destination type (firefly)
destination: String,
/// Start date for synchronization (YYYY-MM-DD)
#[arg(short, long)]
start: Option<NaiveDate>,
/// End date for synchronization (YYYY-MM-DD)
#[arg(short, long)]
end: Option<NaiveDate>,
},
/// Manage accounts and linking
Accounts {
#[command(subcommand)]
subcommand: AccountCommands,
},
/// Manage transactions and cache
Transactions {
#[command(subcommand)]
subcommand: TransactionCommands,
},
/// List all available source types
Sources,
/// List all available destination types
Destinations,
} }
#[tokio::main] #[tokio::main]
async fn main() -> anyhow::Result<()> { async fn main() -> anyhow::Result<()> {
// Load environment variables first // Initialize logging
tracing_subscriber::fmt()
.with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
.init();
// Load environment variables
dotenvy::dotenv().ok(); dotenvy::dotenv().ok();
// Load configuration
let config = Config::from_env()?;
let args = Args::parse(); let args = Args::parse();
// Initialize logging based on command type
// For sync command, show INFO logs by default (but allow RUST_LOG override)
// For other commands, only show warnings/errors by default (but allow RUST_LOG override)
let default_level = match args.command {
Commands::Sync { .. } => "info",
_ => "warn",
};
let log_level = config
.logging
.level
.parse()
.unwrap_or(tracing_subscriber::EnvFilter::new(default_level));
tracing_subscriber::fmt().with_env_filter(log_level).init();
info!("Starting banks2ff..."); info!("Starting banks2ff...");
if args.dry_run { if args.dry_run {
info!("DRY RUN MODE ENABLED: No changes will be made to Firefly III."); info!("DRY RUN MODE ENABLED: No changes will be made to Firefly III.");
} }
match args.command { // Config Load
Commands::Sync { let gc_url = env::var("GOCARDLESS_URL").unwrap_or_else(|_| "https://bankaccountdata.gocardless.com".to_string());
source, let gc_id = env::var("GOCARDLESS_ID").expect("GOCARDLESS_ID not set");
destination, let gc_key = env::var("GOCARDLESS_KEY").expect("GOCARDLESS_KEY not set");
start,
end,
} => {
handle_sync(
config,
args.debug,
source,
destination,
start,
end,
args.dry_run,
)
.await?;
}
Commands::Sources => { let ff_url = env::var("FIREFLY_III_URL").expect("FIREFLY_III_URL not set");
commands::list::handle_sources().await?; let ff_key = env::var("FIREFLY_III_API_KEY").expect("FIREFLY_III_API_KEY not set");
}
Commands::Destinations => {
commands::list::handle_destinations().await?;
}
Commands::Accounts { subcommand } => { // Clients
commands::accounts::handle_accounts(config, subcommand).await?; let gc_client = if args.debug {
} let client = ClientBuilder::new(reqwest::Client::new())
.with(DebugLogger::new("gocardless"))
.build();
GoCardlessClient::with_client(&gc_url, &gc_id, &gc_key, Some(client))?
} else {
GoCardlessClient::new(&gc_url, &gc_id, &gc_key)?
};
Commands::Transactions { subcommand } => { let ff_client = if args.debug {
commands::transactions::handle_transactions(config, subcommand).await?; let client = ClientBuilder::new(reqwest::Client::new())
.with(DebugLogger::new("firefly"))
.build();
FireflyClient::with_client(&ff_url, &ff_key, Some(client))?
} else {
FireflyClient::new(&ff_url, &ff_key)?
};
// Adapters
let source = GoCardlessAdapter::new(gc_client);
let destination = FireflyAdapter::new(ff_client);
// Run
match run_sync(source, destination, args.start, args.end, args.dry_run).await {
Ok(result) => {
info!("Sync completed successfully.");
info!("Accounts processed: {}, skipped (expired): {}, skipped (errors): {}",
result.accounts_processed, result.accounts_skipped_expired, result.accounts_skipped_errors);
info!("Transactions - Created: {}, Healed: {}, Duplicates: {}, Errors: {}",
result.ingest.created, result.ingest.healed, result.ingest.duplicates, result.ingest.errors);
} }
Err(e) => error!("Sync failed: {}", e),
} }
Ok(()) Ok(())
} }
#[cfg(test)]
mod tests {
use crate::cli::tables::mask_iban;
#[test]
fn test_mask_iban_short() {
assert_eq!(mask_iban("123"), "123");
}
#[test]
fn test_mask_iban_long() {
assert_eq!(mask_iban("NL12ABCD1234567890"), "NL12ABCD******7890");
}
#[test]
fn test_mask_iban_other_country() {
assert_eq!(mask_iban("DE1234567890123456"), "DE************3456");
}
}

View File

@@ -10,19 +10,9 @@ Banks2FF implements a **Hexagonal (Ports & Adapters) Architecture** to synchroni
banks2ff/ banks2ff/
├── banks2ff/ # Main CLI application ├── banks2ff/ # Main CLI application
│ └── src/ │ └── src/
│ ├── commands/ # Command handlers
│ │ ├── accounts/ # Account management commands
│ │ │ ├── mod.rs # Account commands dispatch
│ │ │ ├── link.rs # Account linking logic
│ │ │ ├── list.rs # Account listing functionality
│ │ │ └── status.rs # Account status functionality
│ │ ├── transactions/ # Transaction management commands
│ │ ├── list.rs # Source/destination listing
│ │ └── sync.rs # Sync command handler
│ ├── cli/ # CLI utilities and formatting
│ ├── core/ # Domain logic and models │ ├── core/ # Domain logic and models
│ ├── adapters/ # External service integrations │ ├── adapters/ # External service integrations
│ └── main.rs # CLI entry point and command dispatch │ └── main.rs # CLI entry point
├── firefly-client/ # Firefly III API client library ├── firefly-client/ # Firefly III API client library
├── gocardless-client/ # GoCardless API client library ├── gocardless-client/ # GoCardless API client library
└── docs/ # Architecture documentation └── docs/ # Architecture documentation
@@ -59,15 +49,7 @@ banks2ff/
- `client.rs`: Wrapper for Firefly client for transaction storage - `client.rs`: Wrapper for Firefly client for transaction storage
- Maps domain models to Firefly API format - Maps domain models to Firefly API format
### 3. Command Handlers (`banks2ff/src/commands/`) ### 3. API Clients
The CLI commands are organized into focused modules:
- **sync.rs**: Handles transaction synchronization between sources and destinations
- **accounts/**: Account management including linking, listing, and status
- **transactions/**: Transaction inspection, caching, and cache management
- **list.rs**: Simple listing of available sources and destinations
### 4. API Clients
Both clients are hand-crafted using `reqwest`: Both clients are hand-crafted using `reqwest`:
- Strongly-typed DTOs for compile-time safety - Strongly-typed DTOs for compile-time safety

View File

@@ -6,7 +6,7 @@ GOCARDLESS_KEY=
GOCARDLESS_ID= GOCARDLESS_ID=
# Required: Generate a secure random key (32+ characters recommended) # Required: Generate a secure random key (32+ characters recommended)
# Linux/macOS: tr -dc [:alnum:] < /dev/urandom | head -c 32 # Linux/macOS: od -vAn -N32 -tx1 /dev/urandom | tr -d ' '
# Windows PowerShell: [Convert]::ToBase64String((1..32 | ForEach-Object { Get-Random -Minimum 0 -Maximum 256 })) # Windows PowerShell: [Convert]::ToBase64String((1..32 | ForEach-Object { Get-Random -Minimum 0 -Maximum 256 }))
# Or use any password manager to generate a strong random string # Or use any password manager to generate a strong random string
BANKS2FF_CACHE_KEY= BANKS2FF_CACHE_KEY=

View File

@@ -1,9 +1,9 @@
use crate::models::{AccountArray, TransactionArray, TransactionStore, TransactionUpdate};
use reqwest::Url; use reqwest::Url;
use reqwest_middleware::ClientWithMiddleware; use reqwest_middleware::ClientWithMiddleware;
use serde::de::DeserializeOwned; use serde::de::DeserializeOwned;
use thiserror::Error; use thiserror::Error;
use tracing::instrument; use tracing::instrument;
use crate::models::{AccountArray, TransactionStore, TransactionArray, TransactionUpdate};
#[derive(Error, Debug)] #[derive(Error, Debug)]
pub enum FireflyError { pub enum FireflyError {
@@ -28,16 +28,10 @@ impl FireflyClient {
Self::with_client(base_url, access_token, None) Self::with_client(base_url, access_token, None)
} }
pub fn with_client( pub fn with_client(base_url: &str, access_token: &str, client: Option<ClientWithMiddleware>) -> Result<Self, FireflyError> {
base_url: &str,
access_token: &str,
client: Option<ClientWithMiddleware>,
) -> Result<Self, FireflyError> {
Ok(Self { Ok(Self {
base_url: Url::parse(base_url)?, base_url: Url::parse(base_url)?,
client: client.unwrap_or_else(|| { client: client.unwrap_or_else(|| reqwest_middleware::ClientBuilder::new(reqwest::Client::new()).build()),
reqwest_middleware::ClientBuilder::new(reqwest::Client::new()).build()
}),
access_token: access_token.to_string(), access_token: access_token.to_string(),
}) })
} }
@@ -45,7 +39,8 @@ impl FireflyClient {
#[instrument(skip(self))] #[instrument(skip(self))]
pub async fn get_accounts(&self, _iban: &str) -> Result<AccountArray, FireflyError> { pub async fn get_accounts(&self, _iban: &str) -> Result<AccountArray, FireflyError> {
let mut url = self.base_url.join("/api/v1/accounts")?; let mut url = self.base_url.join("/api/v1/accounts")?;
url.query_pairs_mut().append_pair("type", "asset"); url.query_pairs_mut()
.append_pair("type", "asset");
self.get_authenticated(url).await self.get_authenticated(url).await
} }
@@ -62,15 +57,10 @@ impl FireflyClient {
} }
#[instrument(skip(self, transaction))] #[instrument(skip(self, transaction))]
pub async fn store_transaction( pub async fn store_transaction(&self, transaction: TransactionStore) -> Result<(), FireflyError> {
&self,
transaction: TransactionStore,
) -> Result<(), FireflyError> {
let url = self.base_url.join("/api/v1/transactions")?; let url = self.base_url.join("/api/v1/transactions")?;
let response = self let response = self.client.post(url)
.client
.post(url)
.bearer_auth(&self.access_token) .bearer_auth(&self.access_token)
.header("accept", "application/json") .header("accept", "application/json")
.json(&transaction) .json(&transaction)
@@ -80,25 +70,15 @@ impl FireflyClient {
if !response.status().is_success() { if !response.status().is_success() {
let status = response.status(); let status = response.status();
let text = response.text().await?; let text = response.text().await?;
return Err(FireflyError::ApiError(format!( return Err(FireflyError::ApiError(format!("Store Transaction Failed {}: {}", status, text)));
"Store Transaction Failed {}: {}",
status, text
)));
} }
Ok(()) Ok(())
} }
#[instrument(skip(self))] #[instrument(skip(self))]
pub async fn list_account_transactions( pub async fn list_account_transactions(&self, account_id: &str, start: Option<&str>, end: Option<&str>) -> Result<TransactionArray, FireflyError> {
&self, let mut url = self.base_url.join(&format!("/api/v1/accounts/{}/transactions", account_id))?;
account_id: &str,
start: Option<&str>,
end: Option<&str>,
) -> Result<TransactionArray, FireflyError> {
let mut url = self
.base_url
.join(&format!("/api/v1/accounts/{}/transactions", account_id))?;
{ {
let mut pairs = url.query_pairs_mut(); let mut pairs = url.query_pairs_mut();
if let Some(s) = start { if let Some(s) = start {
@@ -115,18 +95,10 @@ impl FireflyClient {
} }
#[instrument(skip(self, update))] #[instrument(skip(self, update))]
pub async fn update_transaction( pub async fn update_transaction(&self, id: &str, update: TransactionUpdate) -> Result<(), FireflyError> {
&self, let url = self.base_url.join(&format!("/api/v1/transactions/{}", id))?;
id: &str,
update: TransactionUpdate,
) -> Result<(), FireflyError> {
let url = self
.base_url
.join(&format!("/api/v1/transactions/{}", id))?;
let response = self let response = self.client.put(url)
.client
.put(url)
.bearer_auth(&self.access_token) .bearer_auth(&self.access_token)
.header("accept", "application/json") .header("accept", "application/json")
.json(&update) .json(&update)
@@ -136,19 +108,14 @@ impl FireflyClient {
if !response.status().is_success() { if !response.status().is_success() {
let status = response.status(); let status = response.status();
let text = response.text().await?; let text = response.text().await?;
return Err(FireflyError::ApiError(format!( return Err(FireflyError::ApiError(format!("Update Transaction Failed {}: {}", status, text)));
"Update Transaction Failed {}: {}",
status, text
)));
} }
Ok(()) Ok(())
} }
async fn get_authenticated<T: DeserializeOwned>(&self, url: Url) -> Result<T, FireflyError> { async fn get_authenticated<T: DeserializeOwned>(&self, url: Url) -> Result<T, FireflyError> {
let response = self let response = self.client.get(url)
.client
.get(url)
.bearer_auth(&self.access_token) .bearer_auth(&self.access_token)
.header("accept", "application/json") .header("accept", "application/json")
.send() .send()
@@ -157,10 +124,7 @@ impl FireflyClient {
if !response.status().is_success() { if !response.status().is_success() {
let status = response.status(); let status = response.status();
let text = response.text().await?; let text = response.text().await?;
return Err(FireflyError::ApiError(format!( return Err(FireflyError::ApiError(format!("API request failed {}: {}", status, text)));
"API request failed {}: {}",
status, text
)));
} }
let data = response.json().await?; let data = response.json().await?;

View File

@@ -13,42 +13,6 @@ pub struct Account {
#[serde(rename = "type")] #[serde(rename = "type")]
pub account_type: String, pub account_type: String,
pub active: Option<bool>, pub active: Option<bool>,
pub order: Option<i32>,
pub created_at: Option<String>,
pub updated_at: Option<String>,
pub account_role: Option<String>,
pub object_group_id: Option<String>,
pub object_group_title: Option<String>,
pub object_group_order: Option<i32>,
pub currency_id: Option<String>,
pub currency_name: Option<String>,
pub currency_code: Option<String>,
pub currency_symbol: Option<String>,
pub currency_decimal_places: Option<i32>,
pub primary_currency_id: Option<String>,
pub primary_currency_name: Option<String>,
pub primary_currency_code: Option<String>,
pub primary_currency_symbol: Option<String>,
pub primary_currency_decimal_places: Option<i32>,
pub opening_balance: Option<String>,
pub pc_opening_balance: Option<String>,
pub debt_amount: Option<String>,
pub pc_debt_amount: Option<String>,
pub notes: Option<String>,
pub monthly_payment_date: Option<String>,
pub credit_card_type: Option<String>,
pub account_number: Option<String>,
pub bic: Option<String>,
pub opening_balance_date: Option<String>,
pub liability_type: Option<String>,
pub liability_direction: Option<String>,
pub interest: Option<String>,
pub interest_period: Option<String>,
pub include_net_worth: Option<bool>,
pub longitude: Option<f64>,
pub latitude: Option<f64>,
pub zoom_level: Option<i32>,
pub last_activity: Option<String>,
} }
#[derive(Debug, Clone, Serialize, Deserialize)] #[derive(Debug, Clone, Serialize, Deserialize)]

View File

@@ -1,8 +1,8 @@
use firefly_client::client::FireflyClient; use firefly_client::client::FireflyClient;
use firefly_client::models::{TransactionSplitStore, TransactionStore}; use firefly_client::models::{TransactionStore, TransactionSplitStore};
use std::fs; use wiremock::matchers::{method, path, header};
use wiremock::matchers::{header, method, path};
use wiremock::{Mock, MockServer, ResponseTemplate}; use wiremock::{Mock, MockServer, ResponseTemplate};
use std::fs;
#[tokio::test] #[tokio::test]
async fn test_search_accounts() { async fn test_search_accounts() {
@@ -21,10 +21,7 @@ async fn test_search_accounts() {
assert_eq!(accounts.data.len(), 1); assert_eq!(accounts.data.len(), 1);
assert_eq!(accounts.data[0].attributes.name, "Checking Account"); assert_eq!(accounts.data[0].attributes.name, "Checking Account");
assert_eq!( assert_eq!(accounts.data[0].attributes.iban.as_deref(), Some("NL01BANK0123456789"));
accounts.data[0].attributes.iban.as_deref(),
Some("NL01BANK0123456789")
);
} }
#[tokio::test] #[tokio::test]

View File

@@ -1,12 +1,9 @@
use crate::models::{
Account, AccountDetail, EndUserAgreement, PaginatedResponse, Requisition, TokenResponse,
TransactionsResponse,
};
use reqwest::Url; use reqwest::Url;
use reqwest_middleware::ClientWithMiddleware; use reqwest_middleware::ClientWithMiddleware;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use thiserror::Error; use thiserror::Error;
use tracing::{debug, instrument}; use tracing::{debug, instrument};
use crate::models::{TokenResponse, PaginatedResponse, Requisition, Account, TransactionsResponse, EndUserAgreement};
#[derive(Error, Debug)] #[derive(Error, Debug)]
pub enum GoCardlessError { pub enum GoCardlessError {
@@ -42,17 +39,10 @@ impl GoCardlessClient {
Self::with_client(base_url, secret_id, secret_key, None) Self::with_client(base_url, secret_id, secret_key, None)
} }
pub fn with_client( pub fn with_client(base_url: &str, secret_id: &str, secret_key: &str, client: Option<ClientWithMiddleware>) -> Result<Self, GoCardlessError> {
base_url: &str,
secret_id: &str,
secret_key: &str,
client: Option<ClientWithMiddleware>,
) -> Result<Self, GoCardlessError> {
Ok(Self { Ok(Self {
base_url: Url::parse(base_url)?, base_url: Url::parse(base_url)?,
client: client.unwrap_or_else(|| { client: client.unwrap_or_else(|| reqwest_middleware::ClientBuilder::new(reqwest::Client::new()).build()),
reqwest_middleware::ClientBuilder::new(reqwest::Client::new()).build()
}),
secret_id: secret_id.to_string(), secret_id: secret_id.to_string(),
secret_key: secret_key.to_string(), secret_key: secret_key.to_string(),
access_token: None, access_token: None,
@@ -77,47 +67,40 @@ impl GoCardlessClient {
}; };
debug!("Requesting new access token"); debug!("Requesting new access token");
let response = self.client.post(url).json(&body).send().await?; let response = self.client.post(url)
.json(&body)
.send()
.await?;
if !response.status().is_success() { if !response.status().is_success() {
let status = response.status(); let status = response.status();
let text = response.text().await?; let text = response.text().await?;
return Err(GoCardlessError::ApiError(format!( return Err(GoCardlessError::ApiError(format!("Token request failed {}: {}", status, text)));
"Token request failed {}: {}",
status, text
)));
} }
let token_resp: TokenResponse = response.json().await?; let token_resp: TokenResponse = response.json().await?;
self.access_token = Some(token_resp.access); self.access_token = Some(token_resp.access);
self.access_expires_at = self.access_expires_at = Some(chrono::Utc::now() + chrono::Duration::seconds(token_resp.access_expires as i64));
Some(chrono::Utc::now() + chrono::Duration::seconds(token_resp.access_expires as i64));
debug!("Access token obtained"); debug!("Access token obtained");
Ok(()) Ok(())
} }
#[instrument(skip(self))] #[instrument(skip(self))]
pub async fn get_requisitions( pub async fn get_requisitions(&self) -> Result<PaginatedResponse<Requisition>, GoCardlessError> {
&self,
) -> Result<PaginatedResponse<Requisition>, GoCardlessError> {
let url = self.base_url.join("/api/v2/requisitions/")?; let url = self.base_url.join("/api/v2/requisitions/")?;
self.get_authenticated(url).await self.get_authenticated(url).await
} }
#[instrument(skip(self))] #[instrument(skip(self))]
pub async fn get_agreements( pub async fn get_agreements(&self) -> Result<PaginatedResponse<EndUserAgreement>, GoCardlessError> {
&self,
) -> Result<PaginatedResponse<EndUserAgreement>, GoCardlessError> {
let url = self.base_url.join("/api/v2/agreements/enduser/")?; let url = self.base_url.join("/api/v2/agreements/enduser/")?;
self.get_authenticated(url).await self.get_authenticated(url).await
} }
#[instrument(skip(self))] #[instrument(skip(self))]
pub async fn get_agreement(&self, id: &str) -> Result<EndUserAgreement, GoCardlessError> { pub async fn get_agreement(&self, id: &str) -> Result<EndUserAgreement, GoCardlessError> {
let url = self let url = self.base_url.join(&format!("/api/v2/agreements/enduser/{}/", id))?;
.base_url
.join(&format!("/api/v2/agreements/enduser/{}/", id))?;
self.get_authenticated(url).await self.get_authenticated(url).await
} }
@@ -149,23 +132,8 @@ impl GoCardlessClient {
} }
#[instrument(skip(self))] #[instrument(skip(self))]
pub async fn get_account_details(&self, id: &str) -> Result<AccountDetail, GoCardlessError> { pub async fn get_transactions(&self, account_id: &str, date_from: Option<&str>, date_to: Option<&str>) -> Result<TransactionsResponse, GoCardlessError> {
let url = self let mut url = self.base_url.join(&format!("/api/v2/accounts/{}/transactions/", account_id))?;
.base_url
.join(&format!("/api/v2/accounts/{}/details/", id))?;
self.get_authenticated(url).await
}
#[instrument(skip(self))]
pub async fn get_transactions(
&self,
account_id: &str,
date_from: Option<&str>,
date_to: Option<&str>,
) -> Result<TransactionsResponse, GoCardlessError> {
let mut url = self
.base_url
.join(&format!("/api/v2/accounts/{}/transactions/", account_id))?;
{ {
let mut pairs = url.query_pairs_mut(); let mut pairs = url.query_pairs_mut();
@@ -180,17 +148,10 @@ impl GoCardlessClient {
self.get_authenticated(url).await self.get_authenticated(url).await
} }
async fn get_authenticated<T: for<'de> Deserialize<'de>>( async fn get_authenticated<T: for<'de> Deserialize<'de>>(&self, url: Url) -> Result<T, GoCardlessError> {
&self, let token = self.access_token.as_ref().ok_or(GoCardlessError::ApiError("No access token available. Call obtain_access_token() first.".into()))?;
url: Url,
) -> Result<T, GoCardlessError> {
let token = self.access_token.as_ref().ok_or(GoCardlessError::ApiError(
"No access token available. Call obtain_access_token() first.".into(),
))?;
let response = self let response = self.client.get(url)
.client
.get(url)
.bearer_auth(token) .bearer_auth(token)
.header("accept", "application/json") .header("accept", "application/json")
.send() .send()
@@ -199,10 +160,7 @@ impl GoCardlessClient {
if !response.status().is_success() { if !response.status().is_success() {
let status = response.status(); let status = response.status();
let text = response.text().await?; let text = response.text().await?;
return Err(GoCardlessError::ApiError(format!( return Err(GoCardlessError::ApiError(format!("API request failed {}: {}", status, text)));
"API request failed {}: {}",
status, text
)));
} }
let data = response.json().await?; let data = response.json().await?;

View File

@@ -42,33 +42,6 @@ pub struct Account {
pub iban: Option<String>, pub iban: Option<String>,
pub institution_id: Option<String>, pub institution_id: Option<String>,
pub status: Option<String>, pub status: Option<String>,
pub owner_name: Option<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AccountDetail {
pub account: DetailSchema,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DetailSchema {
pub resource_id: Option<String>,
pub iban: Option<String>,
pub bban: Option<String>,
pub pan: Option<String>,
pub masked_pan: Option<String>,
pub msisdn: Option<String>,
pub currency: Option<String>,
pub owner_name: Option<String>,
pub name: Option<String>,
pub display_name: Option<String>,
pub product: Option<String>,
pub cash_account_type: Option<String>,
pub status: Option<String>,
pub bic: Option<String>,
pub linked_accounts: Option<String>,
pub usage: Option<String>,
pub details: Option<String>,
} }
#[derive(Debug, Clone, Serialize, Deserialize)] #[derive(Debug, Clone, Serialize, Deserialize)]
@@ -86,24 +59,10 @@ pub struct TransactionBookedPending {
pub struct Transaction { pub struct Transaction {
#[serde(rename = "transactionId")] #[serde(rename = "transactionId")]
pub transaction_id: Option<String>, pub transaction_id: Option<String>,
#[serde(rename = "entryReference")]
pub entry_reference: Option<String>,
#[serde(rename = "endToEndId")]
pub end_to_end_id: Option<String>,
#[serde(rename = "mandateId")]
pub mandate_id: Option<String>,
#[serde(rename = "checkId")]
pub check_id: Option<String>,
#[serde(rename = "creditorId")]
pub creditor_id: Option<String>,
#[serde(rename = "bookingDate")] #[serde(rename = "bookingDate")]
pub booking_date: Option<String>, pub booking_date: Option<String>,
#[serde(rename = "valueDate")] #[serde(rename = "valueDate")]
pub value_date: Option<String>, pub value_date: Option<String>,
#[serde(rename = "bookingDateTime")]
pub booking_date_time: Option<String>,
#[serde(rename = "valueDateTime")]
pub value_date_time: Option<String>,
#[serde(rename = "transactionAmount")] #[serde(rename = "transactionAmount")]
pub transaction_amount: TransactionAmount, pub transaction_amount: TransactionAmount,
#[serde(rename = "currencyExchange")] #[serde(rename = "currencyExchange")]
@@ -112,32 +71,14 @@ pub struct Transaction {
pub creditor_name: Option<String>, pub creditor_name: Option<String>,
#[serde(rename = "creditorAccount")] #[serde(rename = "creditorAccount")]
pub creditor_account: Option<AccountDetails>, pub creditor_account: Option<AccountDetails>,
#[serde(rename = "ultimateCreditor")]
pub ultimate_creditor: Option<String>,
#[serde(rename = "debtorName")] #[serde(rename = "debtorName")]
pub debtor_name: Option<String>, pub debtor_name: Option<String>,
#[serde(rename = "debtorAccount")] #[serde(rename = "debtorAccount")]
pub debtor_account: Option<AccountDetails>, pub debtor_account: Option<AccountDetails>,
#[serde(rename = "ultimateDebtor")]
pub ultimate_debtor: Option<String>,
#[serde(rename = "remittanceInformationUnstructured")] #[serde(rename = "remittanceInformationUnstructured")]
pub remittance_information_unstructured: Option<String>, pub remittance_information_unstructured: Option<String>,
#[serde(rename = "remittanceInformationUnstructuredArray")]
pub remittance_information_unstructured_array: Option<Vec<String>>,
#[serde(rename = "remittanceInformationStructured")]
pub remittance_information_structured: Option<String>,
#[serde(rename = "remittanceInformationStructuredArray")]
pub remittance_information_structured_array: Option<Vec<String>>,
#[serde(rename = "additionalInformation")]
pub additional_information: Option<String>,
#[serde(rename = "purposeCode")]
pub purpose_code: Option<String>,
#[serde(rename = "bankTransactionCode")]
pub bank_transaction_code: Option<String>,
#[serde(rename = "proprietaryBankTransactionCode")] #[serde(rename = "proprietaryBankTransactionCode")]
pub proprietary_bank_transaction_code: Option<String>, pub proprietary_bank_transaction_code: Option<String>,
#[serde(rename = "internalTransactionId")]
pub internal_transaction_id: Option<String>,
} }
#[derive(Debug, Clone, Serialize, Deserialize)] #[derive(Debug, Clone, Serialize, Deserialize)]
@@ -161,10 +102,4 @@ pub struct CurrencyExchange {
#[derive(Debug, Clone, Serialize, Deserialize)] #[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AccountDetails { pub struct AccountDetails {
pub iban: Option<String>, pub iban: Option<String>,
pub bban: Option<String>,
pub pan: Option<String>,
#[serde(rename = "maskedPan")]
pub masked_pan: Option<String>,
pub msisdn: Option<String>,
pub currency: Option<String>,
} }

View File

@@ -1,300 +0,0 @@
# CLI Refactor Plan: Decoupling for Multi-Source Financial Sync
## Overview
This document outlines a phased plan to refactor the `banks2ff` CLI from a tightly coupled, single-purpose sync tool into a modular, multi-source financial synchronization application. The refactor maintains the existing hexagonal architecture while enabling inspection of accounts, transactions, and sync status, support for multiple data sources (GoCardless, CSV, CAMT.053, MT940), and preparation for web API exposure.
## Goals
- **Decouple CLI Architecture**: Separate CLI logic from core business logic to enable multiple entry points (CLI, web API)
- **Retain Sync Functionality**: Keep existing sync as primary subcommand with backward compatibility
- **Add Financial Entity Management**: Enable viewing/managing accounts, transactions, and sync status
- **Support Multiple Sources/Destinations**: Implement pluggable adapters for different data sources and destinations
- **Prepare for Web API**: Ensure core logic returns serializable data structures
- **Maintain Security**: Preserve financial data masking and compliance protocols
- **Follow Best Practices**: Adhere to Rust idioms, error handling, testing, and project guidelines
## Revised CLI Structure
```bash
banks2ff [OPTIONS] <COMMAND>
OPTIONS:
--config <FILE> Path to config file
--dry-run Preview changes without applying
--debug Enable debug logging (advanced users)
COMMANDS:
sync <SOURCE> <DESTINATION> [OPTIONS]
Synchronize transactions between source and destination
--start <DATE> Start date (YYYY-MM-DD)
--end <DATE> End date (YYYY-MM-DD)
sources List all available source types
destinations List all available destination types
help Show help
```
## Implementation Phases
### Phase 1: CLI Structure Refactor ✅ COMPLETED
**Objective**: Establish new subcommand architecture while preserving existing sync functionality.
**Steps:**
1. ✅ Refactor `main.rs` to use `clap::Subcommand` with nested enums for commands and subcommands
2. ✅ Extract environment loading and client initialization into a `cli::setup` module
3. ✅ Update argument parsing to handle source/destination as positional arguments
4. ✅ Implement basic command dispatch logic with placeholder handlers
5. ✅ Ensure backward compatibility for existing sync usage
**Testing:**
- ✅ Unit tests for new CLI argument parsing
- ✅ Integration tests verifying existing sync command works unchanged
- ✅ Mock tests for new subcommand structure
**Implementation Details:**
- Created `cli/` module with `setup.rs` containing `AppContext` for client initialization
- Implemented subcommand structure: `sync`, `accounts`, `transactions`, `status`, `sources`, `destinations`
- Added dynamic adapter registry in `core::adapters.rs` for discoverability and validation
- Implemented comprehensive input validation with helpful error messages
- Added conditional logging (INFO for sync, WARN for interactive commands)
- All placeholder commands log appropriate messages for future implementation
- Maintained all existing sync functionality and flags
### Phase 2: Core Port Extensions ✅ COMPLETED
**Objective**: Extend ports and adapters to support inspection capabilities.
**Steps:**
1. ✅ Add inspection methods to `TransactionSource` and `TransactionDestination` traits:
- `list_accounts()`: Return account summaries
- `get_account_status()`: Return sync status for accounts
- `get_transaction_info()`: Return transaction metadata
- `get_cache_info()`: Return caching status
2. ✅ Update existing adapters (GoCardless, Firefly) to implement new methods
3. ✅ Define serializable response structs in `core::models` for inspection data
4. ✅ Ensure all new methods handle errors gracefully with `anyhow`
**Testing:**
- Unit tests for trait implementations on existing adapters
- Mock tests for new inspection methods
- Integration tests verifying data serialization
**Implementation Details:**
- Added `AccountSummary`, `AccountStatus`, `TransactionInfo`, and `CacheInfo` structs with `Serialize` and `Debug` traits
- Extended both `TransactionSource` and `TransactionDestination` traits with inspection methods
- Implemented methods in `GoCardlessAdapter` using existing client calls and cache data
- Implemented methods in `FireflyAdapter` using existing client calls
- All code formatted with `cargo fmt` and linted with `cargo clippy`
- Existing tests pass; new methods compile but not yet tested due to CLI not implemented
### Phase 3: Account Linking and Management ✅ COMPLETED
**Objective**: Implement comprehensive account linking between sources and destinations to enable reliable sync, with auto-linking where possible and manual overrides.
**Steps:**
1. ✅ Create `core::linking` module with data structures:
- `AccountLink`: Links source account ID to destination account ID with metadata
- `LinkStore`: Persistent storage for links, aliases, and account registries
- Auto-linking logic (IBAN/name similarity scoring)
2. ✅ Extend adapters with account discovery:
- `TransactionSource::discover_accounts()`: Full account list without filtering
- `TransactionDestination::discover_accounts()`: Full account list
3. ✅ Implement linking management:
- Auto-link on sync/account discovery (IBAN/name matches)
- CLI commands: `banks2ff accounts link list`, `banks2ff accounts link create <source_account> <dest_account>`, `banks2ff accounts link delete <link_id>`
- Alias support: `banks2ff accounts alias set <link_id> <alias>`, `banks2ff accounts alias update <link_id> <new_alias>`
4. ✅ Integrate with sync:
- Always discover accounts during sync and update stores
- Use links in `run_sync()` instead of IBAN-only matching
- Handle unlinked accounts (skip with warning or prompt for manual linking)
5. ✅ Update CLI help text:
- Explain linking process in `banks2ff accounts --help`
- Note that sync auto-discovers and attempts linking
**Testing:**
- Unit tests for auto-linking algorithms
- Integration tests for various account scenarios (IBAN matches, name matches, no matches)
- Persistence tests for link store
- CLI tests for link management commands
**Implementation Details:**
- Created `core::linking` with `LinkStore` using nested `HashMap`s for organized storage by adapter type
- Extended traits with `discover_accounts()` and implemented in GoCardless/Firefly adapters
- Integrated account discovery and auto-linking into `run_sync()` with persistent storage
- Added CLI commands under `banks2ff accounts link` with full CRUD operations and alias support
- Updated README with new account linking feature, examples, and troubleshooting
### Phase 4: CLI Output and Formatting ✅ COMPLETED
**Objective**: Implement user-friendly output for inspection commands.
**Steps:**
1. ✅ Create `cli::formatters` module for consistent output formatting
2. ✅ Implement table-based display for accounts and transactions
3. ✅ Add JSON output option for programmatic use
4. ✅ Ensure sensitive data masking in all outputs
5. Add progress indicators for long-running operations (pending)
6. ✅ Implement `accounts` command with `list` and `status` subcommands
7. ✅ Implement `transactions` command with `list`, `cache-status`, and `clear-cache` subcommands
8. ✅ Add account and transaction inspection methods to adapter traits
**Testing:**
- Unit tests for formatter functions
- Integration tests for CLI output with sample data
- Accessibility tests for output readability
- Unit tests for new command implementations
- Integration tests for account/transaction inspection
**Implementation Details:**
- Created `cli::formatters` module with `Formattable` trait and table formatting using `comfy-table`
- Implemented table display for `AccountSummary`, `AccountStatus`, `TransactionInfo`, and `CacheInfo` structs
- Added IBAN masking (showing only last 4 characters) for privacy
- Updated CLI structure with new `accounts` and `transactions` commands
- Added `print_list_output` function for displaying collections of data
- All code formatted with `cargo fmt` and linted with `cargo clippy`
### Phase 5: Status and Cache Management
**Objective**: Implement status overview and cache management commands.
**Steps:**
1. Implement `status` command aggregating data from all adapters
2. Add cache inspection and clearing functionality to `transactions cache-status` and `transactions clear-cache`
3. Create status models for sync health metrics
4. Integrate with existing debug logging infrastructure
**Testing:**
- Unit tests for status aggregation logic
- Integration tests for cache operations
- Mock tests for status data collection
### Phase 6: Sync Logic Updates
**Objective**: Make sync logic adapter-agnostic and reusable.
**Steps:**
1. Modify `core::sync::run_sync()` to accept source/destination traits instead of concrete types
2. Update sync result structures to include inspection data
3. Refactor account processing to work with any `TransactionSource`
4. Ensure dry-run mode works with all adapter types
**Testing:**
- Unit tests for sync logic with mock adapters
- Integration tests with different source/destination combinations
- Regression tests ensuring existing functionality unchanged
### Phase 7: Adapter Factory Implementation
**Objective**: Enable dynamic adapter instantiation for multiple sources/destinations.
**Steps:**
1. Create `core::adapter_factory` module with factory functions
2. Implement source factory supporting "gocardless", "csv", "camt053", "mt940"
3. Implement destination factory supporting "firefly" (extensible for others)
4. Add configuration structs for adapter-specific settings
5. Integrate factory into CLI setup logic
**Testing:**
- Unit tests for factory functions with valid/invalid inputs
- Mock tests for adapter creation
- Integration tests with real configurations
### Phase 8: Integration and Validation
**Objective**: Ensure all components work together and prepare for web API.
**Steps:**
1. Full integration testing across all source/destination combinations
2. Performance testing with realistic data volumes
3. Documentation updates in `docs/architecture.md`
4. Code review against project guidelines
5. Update `AGENTS.md` with new development patterns
**Testing:**
- End-to-end tests for complete workflows
- Load tests for sync operations
- Security audits for data handling
- Compatibility tests with existing configurations
### Phase 9.5: Command Handler Extraction ✅ COMPLETED
**Objective**: Extract command handling logic from main.rs into dedicated modules for better maintainability and separation of concerns.
**Steps:**
1. ✅ Create `commands/` module structure with submodules for each command group
2. ✅ Extract table printing utilities to `cli/tables.rs`
3. ✅ Move command handlers to appropriate modules:
- `commands/sync.rs`: Sync command logic
- `commands/accounts/`: Account management (link, list, status)
- `commands/transactions/`: Transaction operations (list, cache, clear)
- `commands/list.rs`: Source/destination listing
4. ✅ Update main.rs to dispatch to new command modules
5. ✅ Remove extracted functions from main.rs, reducing it from 1049 to ~150 lines
6. ✅ Update documentation to reflect new structure
**Implementation Details:**
- Created hierarchical module structure with focused responsibilities
- Maintained all existing functionality and CLI interface
- Improved code organization and testability
- Updated architecture documentation with new module structure
**Testing:**
- All existing tests pass
- CLI functionality preserved
- Code formatting and linting applied
### Phase 10: File-Based Source Adapters
**Objective**: Implement adapters for file-based transaction sources.
**Steps:**
1. Create `adapters::csv` module implementing `TransactionSource`
- Parse CSV files with configurable column mappings
- Implement caching similar to GoCardless adapter
- Add inspection methods for file status and transaction counts
2. Create `adapters::camt053` and `adapters::mt940` modules
- Parse respective financial file formats
- Implement transaction mapping and validation
- Add format-specific caching and inspection
3. Update `adapter_factory` to instantiate file adapters with file paths
**Testing:**
- Unit tests for file parsing with sample data
- Mock tests for adapter implementations
- Integration tests with fixture files from `tests/fixtures/`
- Performance tests for large file handling
## Architecture Considerations
- **Hexagonal Architecture**: Maintain separation between core business logic, ports, and adapters
- **Error Handling**: Use `thiserror` for domain errors, `anyhow` for application errors
- **Async Programming**: Leverage `tokio` for concurrent operations where beneficial
- **Testing Strategy**: Combine unit tests, integration tests, and mocks using `mockall`
- **Dependencies**: Add new crates only if necessary, preferring workspace dependencies
- **Code Organization**: Keep modules focused and single-responsibility
- **Performance**: Implement caching and batching for file-based sources
## Security and Compliance Notes
- **Financial Data Masking**: Never expose amounts, IBANs, or personal data in logs/outputs
- **Input Validation**: Validate all external data before processing
- **Error Messages**: Avoid sensitive information in error responses
- **Audit Trail**: Maintain structured logging for operations
- **Compliance**: Ensure GDPR/privacy compliance for financial data handling
## Success Criteria
- All existing sync functionality preserved
- New commands work with all supported sources/destinations
- Core logic remains adapter-agnostic
- Comprehensive test coverage maintained
- Performance meets or exceeds current benchmarks
- Architecture supports future web API development</content>
<parameter name="filePath">specs/cli-refactor-plan.md

View File

@@ -1,13 +1,12 @@
# Encrypted Transaction Caching Implementation Plan # Encrypted Transaction Caching Implementation Plan
## Overview ## Overview
High-performance encrypted caching for GoCardless transactions to minimize API calls against rate limits (4 reqs/day per account). Uses optimized hybrid encryption with PBKDF2 master key derivation and HKDF per-operation keys. Implement encrypted caching for GoCardless transactions to minimize API calls against the extremely low rate limits (4 reqs/day per account). Cache raw transaction data with automatic range merging and deduplication.
## Architecture ## Architecture
- **Location**: `banks2ff/src/adapters/gocardless/` - **Location**: `banks2ff/src/adapters/gocardless/`
- **Storage**: `data/cache/` directory - **Storage**: `data/cache/` directory
- **Encryption**: AES-GCM with hybrid key derivation (PBKDF2 + HKDF) - **Encryption**: AES-GCM for disk storage only
- **Performance**: Single PBKDF2 derivation per adapter instance
- **No API Client Changes**: All caching logic in adapter layer - **No API Client Changes**: All caching logic in adapter layer
## Components to Create ## Components to Create
@@ -123,9 +122,8 @@ struct CachedRange {
### Encryption Scope ### Encryption Scope
- **In Memory**: Plain structs (no performance overhead) - **In Memory**: Plain structs (no performance overhead)
- **On Disk**: Full AES-GCM encryption with hybrid key derivation - **On Disk**: Full AES-GCM encryption
- **Key Source**: Environment variable `BANKS2FF_CACHE_KEY` - **Key Source**: Environment variable `BANKS2FF_CACHE_KEY`
- **Performance**: Single PBKDF2 derivation per adapter instance
### Range Merging Strategy ### Range Merging Strategy
- **Overlap Detection**: Check date range intersections - **Overlap Detection**: Check date range intersections
@@ -140,17 +138,15 @@ struct CachedRange {
## Dependencies to Add ## Dependencies to Add
- `aes-gcm`: For encryption - `aes-gcm`: For encryption
- `pbkdf2`: For master key derivation - `pbkdf2`: For key derivation
- `hkdf`: For per-operation key derivation
- `rand`: For encryption nonces - `rand`: For encryption nonces
## Security Considerations ## Security Considerations
- **Encryption**: AES-GCM with 256-bit keys and hybrid derivation (PBKDF2 50k + HKDF) - **Encryption**: AES-GCM with 256-bit keys and PBKDF2 (200,000 iterations)
- **Salt Security**: Fixed master salt + random operation salts - **Salt Security**: Random 16-byte salt per encryption (prepended to ciphertext)
- **Key Management**: Environment variable `BANKS2FF_CACHE_KEY` required - **Key Management**: Environment variable `BANKS2FF_CACHE_KEY` required
- **Data Protection**: Financial data encrypted at rest, no sensitive data in logs - **Data Protection**: Financial data encrypted at rest, no sensitive data in logs
- **Authentication**: GCM provides integrity protection against tampering - **Authentication**: GCM provides integrity protection against tampering
- **Performance**: ~10-50μs per cache operation vs 50-100ms previously
- **Forward Security**: Unique salt/nonce prevents rainbow table attacks - **Forward Security**: Unique salt/nonce prevents rainbow table attacks
## Performance Expectations ## Performance Expectations
@@ -266,12 +262,13 @@ struct CachedRange {
- **Disk I/O**: Encrypted storage with minimal overhead for persistence - **Disk I/O**: Encrypted storage with minimal overhead for persistence
### Security Validation ### Security Validation
- **Encryption**: All cache operations use AES-GCM with hybrid PBKDF2+HKDF key derivation - **Encryption**: All cache operations use AES-GCM with PBKDF2 key derivation
- **Data Integrity**: GCM authentication prevents tampering detection - **Data Integrity**: GCM authentication prevents tampering detection
- **Key Security**: 50k iteration PBKDF2 master key + HKDF per-operation keys - **Key Security**: 200,000 iteration PBKDF2 with random salt per operation
- **No Sensitive Data**: Financial amounts masked in logs, secure at-rest storage - **No Sensitive Data**: Financial amounts masked in logs, secure at-rest storage
### Final Status ### Final Status
- **All Phases Completed**: Core infrastructure, range management, adapter integration, and testing - **All Phases Completed**: Core infrastructure, range management, adapter integration, and testing
- **Production Ready**: High-performance encrypted caching reduces API calls by 99% - **Production Ready**: Encrypted caching reduces API calls by 99% while maintaining security
- **Maintainable**: Clean architecture with comprehensive test coverage - **Maintainable**: Clean architecture with comprehensive test coverage