138 lines
6.7 KiB
Markdown
138 lines
6.7 KiB
Markdown
# Implementation Plan: Bank2FF Refactoring
|
|
|
|
## 1. Objective
|
|
Refactor the `bank2ff` application from a prototype script into a robust, testable, and observable production-grade CLI tool. The application must synchronize bank transactions from GoCardless to Firefly III.
|
|
|
|
**Key Constraints:**
|
|
- **Multi-Crate Workspace**: Separate crates for the CLI application and the API clients.
|
|
- **Hand-Crafted Clients**: No autogenerated code. Custom, strongly-typed clients for better UX.
|
|
- **Clean Architecture**: Hexagonal architecture within the main application.
|
|
- **Multi-Currency**: Accurate handling of foreign amounts.
|
|
- **Test-Driven**: Every component must be testable from the start.
|
|
- **Observability**: Structured logging (tracing) throughout the stack.
|
|
- **Healer Strategy**: Detect and heal historical duplicates that lack external IDs.
|
|
- **Dry Run**: Safe mode to preview changes.
|
|
- **Rate Limit Handling**: Smart caching and graceful skipping to respect 4 requests/day limits.
|
|
|
|
## 2. Architecture
|
|
|
|
### Workspace Structure
|
|
The project uses a Cargo Workspace with three members:
|
|
|
|
1. `gocardless-client`: A reusable library crate wrapping the GoCardless Bank Account Data API v2.
|
|
2. `firefly-client`: A reusable library crate wrapping the Firefly III API v6.4.4.
|
|
3. `banks2ff`: The main CLI application containing the Domain Core and Adapters that use the client libraries.
|
|
|
|
### Directory Layout
|
|
```text
|
|
root/
|
|
├── Cargo.toml # Workspace definition
|
|
├── gocardless-client/ # Crate 1
|
|
│ ├── Cargo.toml
|
|
│ └── src/
|
|
│ ├── lib.rs
|
|
│ ├── client.rs # Reqwest client logic
|
|
│ ├── models.rs # Request/Response DTOs
|
|
│ └── tests/ # Unit/Integration tests with mocks
|
|
├── firefly-client/ # Crate 2
|
|
│ ├── Cargo.toml
|
|
│ └── src/
|
|
│ ├── lib.rs
|
|
│ ├── client.rs # Reqwest client logic
|
|
│ ├── models.rs # Request/Response DTOs
|
|
│ └── tests/ # Unit/Integration tests with mocks
|
|
└── banks2ff/ # Crate 3 (Main App)
|
|
├── Cargo.toml
|
|
└── src/
|
|
├── main.rs
|
|
├── core/ # Domain
|
|
│ ├── models.rs
|
|
│ ├── ports.rs # Traits (Mockable)
|
|
│ ├── sync.rs # Logic (Tested with mocks)
|
|
│ └── tests/ # Unit tests for logic
|
|
└── adapters/ # Integration Layers
|
|
├── gocardless/
|
|
│ ├── client.rs # Uses gocardless-client & Cache
|
|
│ ├── cache.rs # JSON Cache for Account details
|
|
│ └── mapper.rs
|
|
└── firefly/
|
|
└── client.rs # Uses firefly-client
|
|
```
|
|
|
|
## 3. Observability Strategy
|
|
- **Tracing**: All crates will use the `tracing` crate.
|
|
- **Spans**:
|
|
- `client` crates: Create spans for every HTTP request (method, URL).
|
|
- `banks2ff` adapter: Create spans for "Fetching transactions".
|
|
- `banks2ff` sync: Create a span per account synchronization.
|
|
- **Context**: IDs (account, transaction) must be attached to log events.
|
|
|
|
## 4. Testing Strategy
|
|
- **Client Crates**:
|
|
- Use `wiremock` to mock the HTTP server.
|
|
- Test parsing of *real* JSON responses (saved in `tests/fixtures/`).
|
|
- Verify correct request construction (Headers, Auth, Body).
|
|
- **Core (`banks2ff`)**:
|
|
- Use `mockall` to mock `TransactionSource` and `TransactionDestination` traits.
|
|
- Unit test `sync::run_sync` logic (filtering, flow control) without any I/O.
|
|
- **Adapters (`banks2ff`)**:
|
|
- Test mapping logic (Client DTO -> Domain Model) using unit tests with fixtures.
|
|
|
|
## 5. Implementation Steps
|
|
|
|
### Phase 1: Infrastructure & Workspace
|
|
- [x] **Setup**: Initialize `gocardless-client` and `firefly-client` crates. Update root `Cargo.toml`.
|
|
- [x] **Dependencies**: Add `reqwest`, `serde`, `thiserror`, `url`, `tracing`.
|
|
- [x] **Test Deps**: Add `wiremock`, `tokio-test`, `serde_json` (dev-dependencies).
|
|
|
|
### Phase 2: Core (`banks2ff`)
|
|
- [x] **Definitions**: Implement `models.rs` and `ports.rs` in `banks2ff`.
|
|
- [x] **Mocks**: Add `mockall` attribute to ports for easier testing.
|
|
|
|
### Phase 3: GoCardless Client Crate
|
|
- [x] **Models**: Define DTOs in `gocardless-client/src/models.rs`.
|
|
- [x] **Fixtures**: Create `tests/fixtures/gc_transactions.json` (real example data).
|
|
- [x] **Client**: Implement `GoCardlessClient`.
|
|
- [x] **Tests**: Write `tests/client_test.rs` using `wiremock` to serve the fixture and verify the client parses it correctly.
|
|
|
|
### Phase 4: GoCardless Adapter (`banks2ff`)
|
|
- [x] **Implementation**: Implement `TransactionSource`.
|
|
- [x] **Logic**: Handle **Multi-Currency** (inspect `currencyExchange`).
|
|
- [x] **Tests**: Unit test the *mapping logic* specifically. Input: GC Client DTO. Output: Domain Model. Assert foreign amounts are correct.
|
|
- [x] **Optimization**: Implement "Firefly Leading" strategy (only fetch wanted accounts).
|
|
- [x] **Optimization**: Implement Account Cache & Rate Limit handling.
|
|
|
|
### Phase 5: Firefly Client Crate
|
|
- [x] **Models**: Define DTOs in `firefly-client/src/models.rs`.
|
|
- [x] **Fixtures**: Create `tests/fixtures/ff_store_transaction.json`.
|
|
- [x] **Client**: Implement `FireflyClient`.
|
|
- [x] **Tests**: Write `tests/client_test.rs` using `wiremock` to verify auth headers and body serialization.
|
|
|
|
### Phase 6: Firefly Adapter (`banks2ff`)
|
|
- [x] **Implementation**: Implement `TransactionDestination`.
|
|
- [x] **Logic**: Set `external_id`, handle Credit/Debit swap.
|
|
- [x] **Tests**: Unit test mapping logic. Verify `external_id` is populated.
|
|
- [x] **Update**: Refactor for "Healer" strategy (split `ingest` into `find`, `create`, `update`).
|
|
|
|
### Phase 7: Synchronization Engine
|
|
- [x] **Logic**: Implement `banks2ff::core::sync::run_sync` with "Healer" logic.
|
|
- Check Destination for existing transaction (Windowed Search).
|
|
- If found without ID: Heal (Update).
|
|
- If found with ID: Skip.
|
|
- If not found: Create.
|
|
- [x] **Smart Defaults**: Implement default start date (Last Firefly Date + 1) and end date (Yesterday).
|
|
- [x] **Tests**: Update unit tests for the new flow.
|
|
|
|
### Phase 8: Wiring & CLI
|
|
- [x] **CLI**: Add `-s/--start` and `-e/--end` arguments.
|
|
- [x] **CLI**: Add `--dry-run` argument.
|
|
- [x] **Wiring**: Pass these arguments to the sync engine.
|
|
- [x] **Observability**: Initialize `tracing_subscriber` with env filter.
|
|
- [x] **Config**: Load Env vars.
|
|
|
|
## 6. Multi-Currency Logic
|
|
- **GoCardless Adapter**:
|
|
- `foreign_currency` = `currencyExchange[0].sourceCurrency`
|
|
- `foreign_amount` = `amount` * `currencyExchange[0].exchangeRate`
|
|
- Test this math explicitly in Phase 4 tests.
|