banks2ff/specs/planning.md

Implementation Plan: Bank2FF Refactoring

1. Objective

Refactor the bank2ff application from a prototype script into a robust, testable, and observable production-grade CLI tool. The application must synchronize bank transactions from GoCardless to Firefly III.

Key Constraints:

• Multi-Crate Workspace: Separate crates for the CLI application and the API clients.
• Hand-Crafted Clients: No autogenerated code. Custom, strongly-typed clients for better UX.
• Clean Architecture: Hexagonal architecture within the main application.
• Multi-Currency: Accurate handling of foreign amounts.
• Test-Driven: Every component must be testable from the start.
• Observability: Structured logging (tracing) throughout the stack.
• Healer Strategy: Detect and heal historical duplicates that lack external IDs.
• Dry Run: Safe mode to preview changes.
• Rate Limit Handling: Smart caching and graceful skipping to respect 4 requests/day limits.
• Robust Agreement Handling: Gracefully handle expired GoCardless EUAs without failing entire sync.

2. Architecture

Workspace Structure

The project uses a Cargo Workspace with three members:

1. gocardless-client: A reusable library crate wrapping the GoCardless Bank Account Data API v2.
2. firefly-client: A reusable library crate wrapping the Firefly III API v6.4.4.
3. banks2ff: The main CLI application containing the Domain Core and Adapters that use the client libraries.

Directory Layout

root/
├── Cargo.toml                  # Workspace definition
├── gocardless-client/          # Crate 1
│   ├── Cargo.toml
│   └── src/
│       ├── lib.rs
│       ├── client.rs           # Reqwest client logic
│       ├── models.rs           # Request/Response DTOs
│       └── tests/              # Unit/Integration tests with mocks
├── firefly-client/             # Crate 2
│   ├── Cargo.toml
│   └── src/
│       ├── lib.rs
│       ├── client.rs           # Reqwest client logic
│       ├── models.rs           # Request/Response DTOs
│       └── tests/              # Unit/Integration tests with mocks
└── banks2ff/                   # Crate 3 (Main App)
    ├── Cargo.toml
    └── src/
        ├── main.rs
        ├── core/               # Domain
        │   ├── models.rs
        │   ├── ports.rs        # Traits (Mockable)
        │   ├── sync.rs         # Logic (Tested with mocks)
        │   └── tests/          # Unit tests for logic
        └── adapters/           # Integration Layers
            ├── gocardless/
            │   ├── client.rs   # Uses gocardless-client & Cache
            │   ├── cache.rs    # JSON Cache for Account details
            │   └── mapper.rs
            └── firefly/
                └── client.rs   # Uses firefly-client

3. Observability Strategy

• Tracing: All crates will use the tracing crate.
• Spans:
  • client crates: Create spans for every HTTP request (method, URL).
  • banks2ff adapter: Create spans for "Fetching transactions".
  • banks2ff sync: Create a span per account synchronization.
• Context: IDs (account, transaction) must be attached to log events.

4. Testing Strategy

• Client Crates:
  • Use wiremock to mock the HTTP server.
  • Test parsing of real JSON responses (saved in tests/fixtures/).
  • Verify correct request construction (Headers, Auth, Body).
• Core (banks2ff):
  • Use mockall to mock TransactionSource and TransactionDestination traits.
  • Unit test sync::run_sync logic (filtering, flow control) without any I/O.
• Adapters (banks2ff):
  • Test mapping logic (Client DTO -> Domain Model) using unit tests with fixtures.

5. Implementation Steps

Phase 1: Infrastructure & Workspace

• Setup: Initialize gocardless-client and firefly-client crates. Update root Cargo.toml.
• Dependencies: Add reqwest, serde, thiserror, url, tracing.
• Test Deps: Add wiremock, tokio-test, serde_json (dev-dependencies).

Phase 2: Core (banks2ff)

• Definitions: Implement models.rs and ports.rs in banks2ff.
• Mocks: Add mockall attribute to ports for easier testing.

Phase 3: GoCardless Client Crate

• Models: Define DTOs in gocardless-client/src/models.rs.
• Fixtures: Create tests/fixtures/gc_transactions.json (real example data).
• Client: Implement GoCardlessClient.
• Tests: Write tests/client_test.rs using wiremock to serve the fixture and verify the client parses it correctly.

Phase 4: GoCardless Adapter (banks2ff)

• Implementation: Implement TransactionSource.
• Logic: Handle Multi-Currency (inspect currencyExchange).
• Tests: Unit test the mapping logic specifically. Input: GC Client DTO. Output: Domain Model. Assert foreign amounts are correct.
• Optimization: Implement "Firefly Leading" strategy (only fetch wanted accounts).
• Optimization: Implement Account Cache & Rate Limit handling.

Phase 5: Firefly Client Crate

• Models: Define DTOs in firefly-client/src/models.rs.
• Fixtures: Create tests/fixtures/ff_store_transaction.json.
• Client: Implement FireflyClient.
• Tests: Write tests/client_test.rs using wiremock to verify auth headers and body serialization.

Phase 6: Firefly Adapter (banks2ff)

• Implementation: Implement TransactionDestination.
• Logic: Set external_id, handle Credit/Debit swap.
• Tests: Unit test mapping logic. Verify external_id is populated.
• Update: Refactor for "Healer" strategy (split ingest into find, create, update).

Phase 7: Synchronization Engine

• Logic: Implement banks2ff::core::sync::run_sync with "Healer" logic.
  • Check Destination for existing transaction (Windowed Search).
  • If found without ID: Heal (Update).
  • If found with ID: Skip.
  • If not found: Create.
• Smart Defaults: Implement default start date (Last Firefly Date + 1) and end date (Yesterday).
• Tests: Update unit tests for the new flow.

Phase 8: Wiring & CLI

• CLI: Add -s/--start and -e/--end arguments.
• CLI: Add --dry-run argument.
• Wiring: Pass these arguments to the sync engine.
• Observability: Initialize tracing_subscriber with env filter.
• Config: Load Env vars.

6. Multi-Currency Logic

• GoCardless Adapter:
  • foreign_currency = currencyExchange[0].sourceCurrency
  • foreign_amount = amount * currencyExchange[0].exchangeRate
  • Test this math explicitly in Phase 4 tests.