jjkiers/banks2ff
1
0
Fork 0
Code Issues Pull Requests Actions Packages Releases Activity

Implement logic

Browse Source
This commit is contained in:
Jacob Kiers
2025-11-19 21:18:37 +00:00
parent 4b5dc6f59a
commit fcd59b7fc5
31 changed files with 4802 additions and 139 deletions
Download Patch File Download Diff File Expand all files Collapse all files

226
specs/planning.md
View File

@@ -1,129 +1,137 @@
# Implementation Plan: Bank2FF Refactoring
## 1. Objective
Refactor the `bank2ff` application from a prototype script into a robust, testable, and observable production-grade CLI tool. The application must synchronize bank transactions from GoCardless to Firefly III, ensuring:
- **Clean Architecture**: Decoupling of business logic from API clients.
- **Testability**: Ability to unit test core logic without external dependencies.
- **Multi-Currency Support**: Accurate capturing of foreign amounts and currencies.
- **Idempotency**: Preventing duplicate transactions in Firefly III.
Refactor the `bank2ff` application from a prototype script into a robust, testable, and observable production-grade CLI tool. The application must synchronize bank transactions from GoCardless to Firefly III.
## 2. Architecture: Hexagonal (Ports & Adapters)
**Key Constraints:**
- **Multi-Crate Workspace**: Separate crates for the CLI application and the API clients.
- **Hand-Crafted Clients**: No autogenerated code. Custom, strongly-typed clients for better UX.
- **Clean Architecture**: Hexagonal architecture within the main application.
- **Multi-Currency**: Accurate handling of foreign amounts.
- **Test-Driven**: Every component must be testable from the start.
- **Observability**: Structured logging (tracing) throughout the stack.
- **Healer Strategy**: Detect and heal historical duplicates that lack external IDs.
- **Dry Run**: Safe mode to preview changes.
- **Rate Limit Handling**: Smart caching and graceful skipping to respect 4 requests/day limits.
The application will be structured into three distinct layers:
1. **Core (Domain)**: Pure Rust, no external API dependencies. Defines the `BankTransaction` model and the `Ports` (traits) for interacting with the world.
2. **Adapters**: Implementations of the Ports.
- `GoCardlessAdapter`: Implements `TransactionSource`.
- `FireflyAdapter`: Implements `TransactionDestination`.
3. **Application**: Configuration, CLI parsing, and wiring (`main.rs`).
## 2. Architecture
### Directory Structure
### Workspace Structure
The project uses a Cargo Workspace with three members:
1. `gocardless-client`: A reusable library crate wrapping the GoCardless Bank Account Data API v2.
2. `firefly-client`: A reusable library crate wrapping the Firefly III API v6.4.4.
3. `banks2ff`: The main CLI application containing the Domain Core and Adapters that use the client libraries.
### Directory Layout
```text
bank2ff/src/
├── core/
│ ├── mod.rs
│ ├── models.rs # Domain entities (BankTransaction, Account)
│ ├── ports.rs # Traits (TransactionSource, TransactionDestination)
│ └── sync.rs # Core business logic (The "Use Case")
├── adapters/
│ ├── mod.rs
│ ├── gocardless/
│ │ ├── mod.rs
│ │ ├── client.rs # Wrapper around generated client (Auth/RateLimits)
│ │ └── mapper.rs # Logic to map API response -> Domain Model
│ └── firefly/
│ │ ├── mod.rs
│ │ └── client.rs # Implementation of TransactionDestination
└── main.rs # Entry point, wiring, and config loading
root/
├── Cargo.toml # Workspace definition
├── gocardless-client/ # Crate 1
│ ├── Cargo.toml
│ └── src/
│ ├── lib.rs
│ ├── client.rs # Reqwest client logic
│ ├── models.rs # Request/Response DTOs
│ └── tests/ # Unit/Integration tests with mocks
├── firefly-client/ # Crate 2
│ ├── Cargo.toml
│ └── src/
│ ├── lib.rs
│ ├── client.rs # Reqwest client logic
│ ├── models.rs # Request/Response DTOs
│ └── tests/ # Unit/Integration tests with mocks
└── banks2ff/ # Crate 3 (Main App)
├── Cargo.toml
└── src/
├── main.rs
├── core/ # Domain
│ ├── models.rs
│ ├── ports.rs # Traits (Mockable)
│ ├── sync.rs # Logic (Tested with mocks)
│ └── tests/ # Unit tests for logic
└── adapters/ # Integration Layers
├── gocardless/
│ ├── client.rs # Uses gocardless-client & Cache
│ ├── cache.rs # JSON Cache for Account details
│ └── mapper.rs
└── firefly/
└── client.rs # Uses firefly-client
```
## 3. Core definitions
## 3. Observability Strategy
- **Tracing**: All crates will use the `tracing` crate.
- **Spans**:
- `client` crates: Create spans for every HTTP request (method, URL).
- `banks2ff` adapter: Create spans for "Fetching transactions".
- `banks2ff` sync: Create a span per account synchronization.
- **Context**: IDs (account, transaction) must be attached to log events.
### `src/core/models.rs`
The domain model must support multi-currency data.
## 4. Testing Strategy
- **Client Crates**:
- Use `wiremock` to mock the HTTP server.
- Test parsing of *real* JSON responses (saved in `tests/fixtures/`).
- Verify correct request construction (Headers, Auth, Body).
- **Core (`banks2ff`)**:
- Use `mockall` to mock `TransactionSource` and `TransactionDestination` traits.
- Unit test `sync::run_sync` logic (filtering, flow control) without any I/O.
- **Adapters (`banks2ff`)**:
- Test mapping logic (Client DTO -> Domain Model) using unit tests with fixtures.
```rust
pub struct BankTransaction {
pub internal_id: String, // Source ID (GoCardless transactionId)
pub date: NaiveDate, // Booking date
pub amount: Decimal, // Amount in account currency
pub currency: String, // Account currency code (e.g., EUR)
pub foreign_amount: Option<Decimal>, // Original amount (if currency exchange occurred)
pub foreign_currency: Option<String>,// Original currency code
pub description: String,
pub counterparty_name: Option<String>,
pub counterparty_iban: Option<String>,
}
```
## 5. Implementation Steps
### `src/core/ports.rs`
Traits to decouple the architecture.
### Phase 1: Infrastructure & Workspace
- [x] **Setup**: Initialize `gocardless-client` and `firefly-client` crates. Update root `Cargo.toml`.
- [x] **Dependencies**: Add `reqwest`, `serde`, `thiserror`, `url`, `tracing`.
- [x] **Test Deps**: Add `wiremock`, `tokio-test`, `serde_json` (dev-dependencies).
```rust
#[async_trait]
pub trait TransactionSource: Send + Sync {
async fn get_accounts(&self) -> Result<Vec<Account>>;
async fn get_transactions(&self, account_id: &str, start: NaiveDate, end: NaiveDate) -> Result<Vec<BankTransaction>>;
}
### Phase 2: Core (`banks2ff`)
- [x] **Definitions**: Implement `models.rs` and `ports.rs` in `banks2ff`.
- [x] **Mocks**: Add `mockall` attribute to ports for easier testing.
#[async_trait]
pub trait TransactionDestination: Send + Sync {
async fn resolve_account_id(&self, iban: &str) -> Result<Option<String>>;
async fn ingest_transactions(&self, account_id: &str, transactions: Vec<BankTransaction>) -> Result<IngestResult>;
}
```
### Phase 3: GoCardless Client Crate
- [x] **Models**: Define DTOs in `gocardless-client/src/models.rs`.
- [x] **Fixtures**: Create `tests/fixtures/gc_transactions.json` (real example data).
- [x] **Client**: Implement `GoCardlessClient`.
- [x] **Tests**: Write `tests/client_test.rs` using `wiremock` to serve the fixture and verify the client parses it correctly.
## 4. Implementation Steps
### Phase 4: GoCardless Adapter (`banks2ff`)
- [x] **Implementation**: Implement `TransactionSource`.
- [x] **Logic**: Handle **Multi-Currency** (inspect `currencyExchange`).
- [x] **Tests**: Unit test the *mapping logic* specifically. Input: GC Client DTO. Output: Domain Model. Assert foreign amounts are correct.
- [x] **Optimization**: Implement "Firefly Leading" strategy (only fetch wanted accounts).
- [x] **Optimization**: Implement Account Cache & Rate Limit handling.
### Phase 1: Infrastructure & Core
1. **Setup**: Create the directory structure.
2. **Dependencies**: Add `anyhow`, `thiserror`, `tracing`, `tracing-subscriber`, `async-trait`, `rust_decimal`, `chrono`.
3. **Core**: Implement `models.rs` and `ports.rs`.
### Phase 5: Firefly Client Crate
- [x] **Models**: Define DTOs in `firefly-client/src/models.rs`.
- [x] **Fixtures**: Create `tests/fixtures/ff_store_transaction.json`.
- [x] **Client**: Implement `FireflyClient`.
- [x] **Tests**: Write `tests/client_test.rs` using `wiremock` to verify auth headers and body serialization.
### Phase 2: GoCardless Adapter (Source)
1. **Token Management**: Create a wrapper struct that holds the `gocardless_bankaccount_data_api` configuration and manages the Access/Refresh token lifecycle. It should check validity before every request.
2. **Mapping Logic**:
- Map `transactionAmount.amount` -> `BankTransaction.amount`.
- **Multi-Currency**: Inspect `currencyExchange`.
- If present, map `sourceCurrency` -> `foreign_currency`.
- Calculate `foreign_amount` using `exchangeRate` if `instructedAmount` is missing.
- Map `transactionId` -> `internal_id`.
3. **Trait Implementation**: Implement `TransactionSource`.
### Phase 6: Firefly Adapter (`banks2ff`)
- [x] **Implementation**: Implement `TransactionDestination`.
- [x] **Logic**: Set `external_id`, handle Credit/Debit swap.
- [x] **Tests**: Unit test mapping logic. Verify `external_id` is populated.
- [x] **Update**: Refactor for "Healer" strategy (split `ingest` into `find`, `create`, `update`).
### Phase 3: Firefly Adapter (Destination)
1. **Client Wrapper**: Initialize `firefly_iii_api` client with API Key.
2. **Resolution**: Implement `resolve_account_id` by querying Firefly accounts by IBAN (using the search/list endpoint).
3. **Ingestion**:
- Map `BankTransaction` to `TransactionSplitStore`.
- **Crucial**: Set `external_id` = `BankTransaction.internal_id`.
- Handle Credits vs Debits (Swap Source/Destination logic).
- Call `store_transaction` endpoint.
- Handle 422/409 errors gracefully (log duplicates).
### Phase 7: Synchronization Engine
- [x] **Logic**: Implement `banks2ff::core::sync::run_sync` with "Healer" logic.
- Check Destination for existing transaction (Windowed Search).
- If found without ID: Heal (Update).
- If found with ID: Skip.
- If not found: Create.
- [x] **Smart Defaults**: Implement default start date (Last Firefly Date + 1) and end date (Yesterday).
- [x] **Tests**: Update unit tests for the new flow.
### Phase 4: Synchronization Engine
1. Implement `src/core/sync.rs`.
2. Logic:
- Fetch accounts from Source.
- For each account, find Destination ID.
- Fetch transactions (default: last 30 days).
- Filter/Process (optional).
- Ingest to Destination.
- Log statistics using `tracing`.
### Phase 8: Wiring & CLI
- [x] **CLI**: Add `-s/--start` and `-e/--end` arguments.
- [x] **CLI**: Add `--dry-run` argument.
- [x] **Wiring**: Pass these arguments to the sync engine.
- [x] **Observability**: Initialize `tracing_subscriber` with env filter.
- [x] **Config**: Load Env vars.
### Phase 5: Wiring
1. Refactor `main.rs`.
2. Initialize `tracing` for structured logging.
3. Load Config (Env variables).
4. Instantiate Adapters.
5. Run the Sync Engine.
## 5. Multi-Currency Handling Specifics
- **GoCardless Spec**: The `currencyExchange` array contains the details.
- **Logic**:
- If `currencyExchange` is not null/empty:
- `foreign_currency` = `currencyExchange[0].sourceCurrency`
- `foreign_amount` = `amount` * `currencyExchange[0].exchangeRate` (Validation: check if `unitCurrency` affects this).
- Ensure `Decimal` precision is handled correctly.
## 6. Testability & Observability
- **Tests**: Write a unit test for `core/sync.rs` using Mock structs for Source/Destination to verify the flow logic.
- **Logs**: Use `tracing::info!` for high-level progress ("Synced Account X") and `tracing::debug!` for details ("Transaction Y mapped to Z").
## 6. Multi-Currency Logic
- **GoCardless Adapter**:
- `foreign_currency` = `currencyExchange[0].sourceCurrency`
- `foreign_amount` = `amount` * `currencyExchange[0].exchangeRate`
- Test this math explicitly in Phase 4 tests.