Completely replace implementation #1

Manually merged
jjkiers merged 12 commits from push-wtvsvxromnno into master 2025-11-29 00:25:36 +00:00
31 changed files with 4802 additions and 139 deletions
Showing only changes of commit 9a5c6d0f68 - Show all commits

178
AGENTS.md Normal file
View File

@@ -0,0 +1,178 @@
# Banks2FF Development Guide
## Project Purpose
Banks2FF is a Rust CLI tool that synchronizes bank transactions from GoCardless Bank Account Data API to Firefly III personal finance manager. It implements a hexagonal architecture for clean separation of concerns and comprehensive testing.
## 🚨 CRITICAL: Financial Data Security
### **ABSOLUTE REQUIREMENT: Financial Data Masking**
**NEVER** expose, log, or display raw financial information including:
- Transaction amounts
- Account balances
- IBANs or account numbers
- Transaction descriptions
- Personal identifiers
- API keys or tokens
### **Compliance Protocol for Debugging**
When debugging financial data issues:
1. **Create Anonymized Test Scripts**: Write small, focused scripts that extract only the necessary data structure information
2. **Use Mock Data**: Replace real financial values with placeholder data
3. **Validate Structure, Not Values**: Focus on data structure integrity, not actual financial content
4. **Sanitize All Outputs**: Ensure any debugging output masks sensitive information
```rust
// ✅ GOOD: Structure validation with mock data
fn validate_transaction_structure() {
let mock_tx = BankTransaction {
amount: Decimal::new(12345, 2), // Mock amount
currency: "EUR".to_string(),
// ... other fields with mock data
};
// Validate structure only
}
// ❌ BAD: Exposing real financial data
fn debug_real_transactions(transactions: Vec<BankTransaction>) {
for tx in transactions {
println!("Real amount: {}", tx.amount); // SECURITY VIOLATION
}
}
```
## Rust Development Best Practices
### Error Handling
- **Use `thiserror`** for domain-specific error types in core modules
- **Use `anyhow`** for application-level error context and propagation
- **Never use `panic!`** in production code - handle errors gracefully
- **Implement `From` traits** for error type conversions
```rust
// Core domain errors
#[derive(Error, Debug)]
pub enum SyncError {
#[error("Failed to fetch transactions from source: {0}")]
SourceError(#[from] anyhow::Error),
#[error("Failed to store transaction: {0}")]
DestinationError(#[from] anyhow::Error),
}
```
### Async Programming
- **Use `tokio`** as the async runtime (workspace dependency)
- **Prefer `async-trait`** for trait methods that need to be async
- **Handle cancellation** properly with `select!` or `tokio::time::timeout`
- **Use `?` operator** for error propagation in async functions
### Testing Strategy
- **Unit Tests**: Test pure functions and business logic in isolation
- **Integration Tests**: Test adapter implementations with `wiremock`
- **Mock External Dependencies**: Use `mockall` for trait-based testing
- **Test Fixtures**: Store sample JSON responses in `tests/fixtures/`
```rust
#[cfg(test)]
mod tests {
use super::*;
use mockall::predicate::*;
#[tokio::test]
async fn test_sync_with_mock_source() {
let mut mock_source = MockTransactionSource::new();
// Setup mock expectations
// Test core logic
}
}
```
### Code Organization
- **Workspace Dependencies**: Define common dependencies in root `Cargo.toml`
- **Feature Flags**: Use features for optional functionality
- **Module Structure**: Keep modules focused and single-responsibility
- **Public API**: Minimize public surface area; prefer internal modules
### Dependencies and Patterns
**Key Workspace Dependencies:**
- `tokio`: Async runtime with full features
- `reqwest`: HTTP client with JSON support
- `serde`/`serde_json`: Serialization/deserialization
- `chrono`: Date/time handling with serde support
- `rust_decimal`: Precise decimal arithmetic for financial data
- `tracing`/`tracing-subscriber`: Structured logging
- `clap`: CLI argument parsing with derive macros
- `anyhow`/`thiserror`: Error handling
- `async-trait`: Async trait support
- `wiremock`: HTTP mocking for tests
- `mockall`: Runtime mocking for tests
## Development Workflow
### 1. Code Development
- Write code in appropriate modules following the hexagonal architecture
- Keep core business logic separate from external integrations
- Use workspace dependencies consistently
### 2. Testing
- Write tests alongside code in `#[cfg(test)]` modules
- Test both happy path and error conditions
- Use mock objects for external dependencies
- Ensure all tests pass: `cargo test --workspace`
### 3. Code Quality
- Follow Rust idioms and conventions
- Use `cargo fmt` for formatting
- Use `cargo clippy` for linting
- Ensure documentation for public APIs
### 4. Commit Standards
- Commit both code and tests together
- Write clear, descriptive commit messages
- Ensure the workspace compiles: `cargo build --workspace`
## Project Structure Guidelines
### Core Module (`banks2ff/src/core/`)
- **models.rs**: Domain entities (BankTransaction, Account)
- **ports.rs**: Trait definitions (TransactionSource, TransactionDestination)
- **sync.rs**: Business logic orchestration
### Adapters Module (`banks2ff/src/adapters/`)
- **gocardless/**: GoCardless API integration
- **firefly/**: Firefly III API integration
- Each adapter implements the appropriate port trait
### Client Libraries
- **gocardless-client/**: Standalone GoCardless API wrapper
- **firefly-client/**: Standalone Firefly III API wrapper
- Both use `reqwest` for HTTP communication
## Security Considerations
- **Never log sensitive data**: Use tracing filters to exclude financial information
- **Environment variables**: Store credentials in `.env` file (never in code)
- **Input validation**: Validate all external data before processing
- **Error messages**: Don't expose sensitive information in error messages
## Performance Considerations
- **Caching**: Use caching to reduce API calls (see GoCardlessAdapter)
- **Rate Limiting**: Handle 429 responses gracefully
- **Batch Processing**: Process transactions in reasonable batches
- **Async Concurrency**: Use `tokio` for concurrent operations where appropriate
## Observability
- **Structured Logging**: Use `tracing` with spans for operations
- **Error Context**: Provide context in error messages for debugging
- **Metrics**: Consider adding metrics for sync operations
- **Log Levels**: Use appropriate log levels (debug, info, warn, error)

2610
Cargo.lock generated Normal file

File diff suppressed because it is too large Load Diff

31
Cargo.toml Normal file
View File

@@ -0,0 +1,31 @@
[workspace]
members = [
"banks2ff",
"firefly-client",
"gocardless-client",
]
resolver = "2"
[workspace.package]
version = "0.1.0"
edition = "2021"
authors = ["Your Name <your.email@example.com>"]
[workspace.dependencies]
tokio = { version = "1.34", features = ["full"] }
anyhow = "1.0"
thiserror = "1.0"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
chrono = { version = "0.4", features = ["serde"] }
rust_decimal = { version = "1.33", features = ["serde-float"] }
async-trait = "0.1"
dotenvy = "0.15"
clap = { version = "4.4", features = ["derive", "env"] }
reqwest = { version = "0.11", features = ["json", "multipart"] }
url = "2.5"
wiremock = "0.5"
tokio-test = "0.4"
mockall = "0.11"

115
README.md
View File

@@ -1,30 +1,85 @@
# Bank2FF
Bank2FF is a tool that can retrieve bank transactions from Gocardless and
add them to Firefly III.
It contains autogenerated APIs for both Firefly III and for the
Gocardless Bank Account Data API.
## Usage
TBD
## Generating the API clients
These API clients are generated with the OpenAPI Generators for Rust.
These need Podman installed, and assume this command is run from the same
directory where this README.md file is located.
For Gocardless:
`podman run --rm -v ${PWD}:/local openapitools/openapi-generator-cli generate -g rust -o /local/gocardless-bankaccount-data-api -i 'https://bankaccountdata.gocardless.com/api/v2/swagger.json' --additional-properties=library=reqwest,packageName=gocardless-bankaccount-data-api,packageVersion=2.0.0,supportMiddleware=true,avoidBoxedModels=true`
For Firefly III:
If necessary, change the URL to the definition. If that is a new version, then also change the `packageVersion` parameter.
`podman run --rm -v ${PWD}:/local openapitools/openapi-generator-cli generate -g rust -o /local/firefly-iii-api -i 'https://api-docs.firefly-iii.org/firefly-iii-2.1.0-v1.yaml' --additional-properties=library=reqwest,packageName=firefly-iii-api,packageVersion=2.1.0,supportMiddleware=true,avoidBoxedModels=true`
# Banks2FF
A robust command-line tool to synchronize bank transactions from GoCardless (formerly Nordigen) to Firefly III.
## Architecture
This project is a Rust Workspace consisting of:
- `banks2ff`: The main CLI application (Hexagonal Architecture).
- `gocardless-client`: A hand-crafted, strongly-typed library for the GoCardless Bank Account Data API.
- `firefly-client`: A hand-crafted, strongly-typed library for the Firefly III API.
## Features
- **Multi-Currency Support**: Correctly handles foreign currency transactions by extracting exchange rate data.
- **Idempotency (Healer Mode)**:
- Detects duplicates using a windowed search (Date +/- 3 days, exact Amount).
- "Heals" historical transactions by updating them with the correct `external_id`.
- Skips transactions that already have a matching `external_id`.
- **Clean Architecture**: Decoupled core logic makes it reliable and testable.
- **Observability**: Structured logging via `tracing`.
- **Dry Run**: Preview changes without writing to Firefly III.
- **Rate Limit Protection**:
- Caches GoCardless account details to avoid unnecessary calls.
- Respects token expiry to minimize auth calls.
- Handles `429 Too Many Requests` gracefully by skipping affected accounts.
## Setup & Configuration
1. **Prerequisites**:
- Rust (latest stable)
- An account with GoCardless Bank Account Data (get your `secret_id` and `secret_key`).
- A running Firefly III instance (get your Personal Access Token).
2. **Environment Variables**:
Copy `env.example` to `.env` and fill in your details:
```bash
cp env.example .env
```
Required variables:
- `GOCARDLESS_ID`: Your GoCardless Secret ID.
- `GOCARDLESS_KEY`: Your GoCardless Secret Key.
- `FIREFLY_III_URL`: The base URL of your Firefly instance (e.g., `https://money.example.com`).
- `FIREFLY_III_API_KEY`: Your Personal Access Token.
Optional:
- `GOCARDLESS_URL`: Defaults to `https://bankaccountdata.gocardless.com`.
- `RUST_LOG`: Set log level (e.g., `info`, `debug`, `trace`).
## Testing
The project has a comprehensive test suite using `wiremock` for API clients and `mockall` for core logic.
To run all tests:
```bash
cargo test --workspace
```
## Usage
To run the synchronization:
```bash
# Run via cargo (defaults: Start = Last Firefly Date + 1, End = Yesterday)
cargo run -p banks2ff
# Dry Run (Read-only)
cargo run -p banks2ff -- --dry-run
# Custom Date Range
cargo run -p banks2ff -- --start 2023-01-01 --end 2023-01-31
```
## How it works
1. **Fetch**: Retrieves active accounts from GoCardless (filtered by those present in Firefly III to save requests).
2. **Match**: Resolves the destination account in Firefly III by matching the IBAN.
3. **Sync Window**: Determines the start date automatically by finding the latest transaction in Firefly for that account.
4. **Process**: For each transaction:
- **Search**: Checks Firefly for an existing transaction (matching Amount and Date +/- 3 days).
- **Heal**: If found but missing an `external_id`, it updates the transaction.
- **Skip**: If found and matches `external_id`, it skips.
- **Create**: If not found, it creates a new transaction.

27
banks2ff/Cargo.toml Normal file
View File

@@ -0,0 +1,27 @@
[package]
name = "banks2ff"
version.workspace = true
edition.workspace = true
authors.workspace = true
[dependencies]
tokio = { workspace = true }
anyhow = { workspace = true }
tracing = { workspace = true }
tracing-subscriber = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
chrono = { workspace = true }
rust_decimal = { workspace = true }
dotenvy = { workspace = true }
clap = { workspace = true }
# Core logic dependencies
async-trait = { workspace = true }
# API Client dependencies
firefly-client = { path = "../firefly-client" }
gocardless-client = { path = "../gocardless-client" }
[dev-dependencies]
mockall = { workspace = true }

View File

@@ -0,0 +1,188 @@
use async_trait::async_trait;
use anyhow::Result;
use tracing::instrument;
use crate::core::ports::{TransactionDestination, TransactionMatch};
use crate::core::models::BankTransaction;
use firefly_client::client::FireflyClient;
use firefly_client::models::{TransactionStore, TransactionSplitStore, TransactionUpdate, TransactionSplitUpdate};
use std::sync::Arc;
use tokio::sync::Mutex;
use rust_decimal::Decimal;
use std::str::FromStr;
use chrono::NaiveDate;
pub struct FireflyAdapter {
client: Arc<Mutex<FireflyClient>>,
}
impl FireflyAdapter {
pub fn new(client: FireflyClient) -> Self {
Self {
client: Arc::new(Mutex::new(client)),
}
}
}
#[async_trait]
impl TransactionDestination for FireflyAdapter {
#[instrument(skip(self))]
async fn resolve_account_id(&self, iban: &str) -> Result<Option<String>> {
let client = self.client.lock().await;
let accounts = client.search_accounts(iban).await?;
// Look for exact match on IBAN, ensuring account is active
for acc in accounts.data {
// Filter for active accounts only (default is usually active, but let's check if attribute exists)
// Note: The Firefly API spec v6.4.4 Account object has 'active' attribute as boolean.
let is_active = acc.attributes.active.unwrap_or(true);
if !is_active {
continue;
}
if let Some(acc_iban) = acc.attributes.iban {
if acc_iban.replace(" ", "") == iban.replace(" ", "") {
return Ok(Some(acc.id));
}
}
}
Ok(None)
}
#[instrument(skip(self))]
async fn get_active_account_ibans(&self) -> Result<Vec<String>> {
let client = self.client.lock().await;
// Get all asset accounts. Note: Pagination might be needed if user has > 50 accounts.
// For typical users, 50 is enough. If needed we can loop pages.
// The client `get_accounts` method hardcodes limit=default. We should probably expose a list_all method or loop here.
// For now, let's assume page 1 covers it or use search.
let accounts = client.get_accounts("").await?; // Argument ignored in current impl
let mut ibans = Vec::new();
for acc in accounts.data {
let is_active = acc.attributes.active.unwrap_or(true);
if is_active {
if let Some(iban) = acc.attributes.iban {
if !iban.is_empty() {
ibans.push(iban);
}
}
}
}
Ok(ibans)
}
#[instrument(skip(self))]
async fn get_last_transaction_date(&self, account_id: &str) -> Result<Option<NaiveDate>> {
let client = self.client.lock().await;
// Fetch latest 1 transaction
let tx_list = client.list_account_transactions(account_id, None, None).await?;
if let Some(first) = tx_list.data.first() {
if let Some(split) = first.attributes.transactions.first() {
// Format is usually YYYY-MM-DDT... or YYYY-MM-DD
let date_str = split.date.split('T').next().unwrap_or(&split.date);
if let Ok(date) = NaiveDate::parse_from_str(date_str, "%Y-%m-%d") {
return Ok(Some(date));
}
}
}
Ok(None)
}
#[instrument(skip(self))]
async fn find_transaction(&self, account_id: &str, tx: &BankTransaction) -> Result<Option<TransactionMatch>> {
let client = self.client.lock().await;
// Search window: +/- 3 days
let start_date = tx.date - chrono::Duration::days(3);
let end_date = tx.date + chrono::Duration::days(3);
let tx_list = client.list_account_transactions(
account_id,
Some(&start_date.format("%Y-%m-%d").to_string()),
Some(&end_date.format("%Y-%m-%d").to_string())
).await?;
// Filter logic
for existing_tx in tx_list.data {
for split in existing_tx.attributes.transactions {
// 1. Check Amount (exact match absolute value)
if let Ok(amount) = Decimal::from_str(&split.amount) {
if amount.abs() == tx.amount.abs() {
// 2. Check External ID
if let Some(ref ext_id) = split.external_id {
if ext_id == &tx.internal_id {
return Ok(Some(TransactionMatch {
id: existing_tx.id.clone(),
has_external_id: true,
}));
}
} else {
// 3. "Naked" transaction match (Heuristic)
// If currency matches
if let Some(ref code) = split.currency_code {
if code != &tx.currency {
continue;
}
}
return Ok(Some(TransactionMatch {
id: existing_tx.id.clone(),
has_external_id: false,
}));
}
}
}
}
}
Ok(None)
}
#[instrument(skip(self))]
async fn create_transaction(&self, account_id: &str, tx: &BankTransaction) -> Result<()> {
let client = self.client.lock().await;
// Map to Firefly Transaction
let is_credit = tx.amount.is_sign_positive();
let transaction_type = if is_credit { "deposit" } else { "withdrawal" };
let split = TransactionSplitStore {
transaction_type: transaction_type.to_string(),
date: tx.date.format("%Y-%m-%d").to_string(),
amount: tx.amount.abs().to_string(),
description: tx.description.clone(),
source_id: if !is_credit { Some(account_id.to_string()) } else { None },
source_name: if is_credit { tx.counterparty_name.clone().or(Some("Unknown Sender".to_string())) } else { None },
destination_id: if is_credit { Some(account_id.to_string()) } else { None },
destination_name: if !is_credit { tx.counterparty_name.clone().or(Some("Unknown Recipient".to_string())) } else { None },
currency_code: Some(tx.currency.clone()),
foreign_amount: tx.foreign_amount.map(|d| d.abs().to_string()),
foreign_currency_code: tx.foreign_currency.clone(),
external_id: Some(tx.internal_id.clone()),
};
let store = TransactionStore {
transactions: vec![split],
apply_rules: Some(true),
fire_webhooks: Some(true),
error_if_duplicate_hash: Some(true),
};
client.store_transaction(store).await.map_err(|e| e.into())
}
#[instrument(skip(self))]
async fn update_transaction_external_id(&self, id: &str, external_id: &str) -> Result<()> {
let client = self.client.lock().await;
let update = TransactionUpdate {
transactions: vec![TransactionSplitUpdate {
external_id: Some(external_id.to_string()),
}],
};
client.update_transaction(id, update).await.map_err(|e| e.into())
}
}

View File

@@ -0,0 +1 @@
pub mod client;

View File

@@ -0,0 +1,51 @@
use std::collections::HashMap;
use std::fs;
use std::path::Path;
use serde::{Deserialize, Serialize};
use tracing::warn;
#[derive(Debug, Serialize, Deserialize, Default)]
pub struct AccountCache {
/// Map of Account ID -> IBAN
pub accounts: HashMap<String, String>,
}
impl AccountCache {
fn get_path() -> String {
".banks2ff-cache.json".to_string()
}
pub fn load() -> Self {
let path = Self::get_path();
if Path::new(&path).exists() {
match fs::read_to_string(&path) {
Ok(content) => match serde_json::from_str(&content) {
Ok(cache) => return cache,
Err(e) => warn!("Failed to parse cache file: {}", e),
},
Err(e) => warn!("Failed to read cache file: {}", e),
}
}
Self::default()
}
pub fn save(&self) {
let path = Self::get_path();
match serde_json::to_string_pretty(self) {
Ok(content) => {
if let Err(e) = fs::write(&path, content) {
warn!("Failed to write cache file: {}", e);
}
},
Err(e) => warn!("Failed to serialize cache: {}", e),
}
}
pub fn get_iban(&self, account_id: &str) -> Option<String> {
self.accounts.get(account_id).cloned()
}
pub fn insert(&mut self, account_id: String, iban: String) {
self.accounts.insert(account_id, iban);
}
}

View File

@@ -0,0 +1,152 @@
use async_trait::async_trait;
use chrono::NaiveDate;
use anyhow::Result;
use tracing::{info, instrument, warn};
use crate::core::ports::TransactionSource;
use crate::core::models::{Account, BankTransaction};
use crate::adapters::gocardless::mapper::map_transaction;
use crate::adapters::gocardless::cache::AccountCache;
use gocardless_client::client::GoCardlessClient;
use std::sync::Arc;
use tokio::sync::Mutex;
pub struct GoCardlessAdapter {
client: Arc<Mutex<GoCardlessClient>>,
cache: Arc<Mutex<AccountCache>>,
}
impl GoCardlessAdapter {
pub fn new(client: GoCardlessClient) -> Self {
Self {
client: Arc::new(Mutex::new(client)),
cache: Arc::new(Mutex::new(AccountCache::load())),
}
}
}
#[async_trait]
impl TransactionSource for GoCardlessAdapter {
#[instrument(skip(self))]
async fn get_accounts(&self, wanted_ibans: Option<Vec<String>>) -> Result<Vec<Account>> {
let mut client = self.client.lock().await;
let mut cache = self.cache.lock().await;
// Ensure token
client.obtain_access_token().await?;
let requisitions = client.get_requisitions().await?;
let mut accounts = Vec::new();
// Build a hashset of wanted IBANs if provided, for faster lookup
let wanted_set = wanted_ibans.map(|list| {
list.into_iter()
.map(|i| i.replace(" ", ""))
.collect::<std::collections::HashSet<_>>()
});
let mut found_count = 0;
let target_count = wanted_set.as_ref().map(|s| s.len()).unwrap_or(0);
for req in requisitions.results {
// Optimization: Only process Linked requisitions to avoid 401/403 on expired ones
if req.status != "LN" {
continue;
}
if let Some(req_accounts) = req.accounts {
for acc_id in req_accounts {
// 1. Check Cache
let mut iban_opt = cache.get_iban(&acc_id);
// 2. Fetch if missing
if iban_opt.is_none() {
match client.get_account(&acc_id).await {
Ok(details) => {
let new_iban = details.iban.unwrap_or_default();
cache.insert(acc_id.clone(), new_iban.clone());
cache.save();
iban_opt = Some(new_iban);
},
Err(e) => {
// If rate limit hit here, we might want to skip this account and continue?
// But get_account is critical to identify the account.
// If we fail here, we can't match.
warn!("Failed to fetch details for account {}: {}", acc_id, e);
continue;
}
}
}
let iban = iban_opt.unwrap_or_default();
let mut keep = true;
if let Some(ref wanted) = wanted_set {
if !wanted.contains(&iban.replace(" ", "")) {
keep = false;
} else {
found_count += 1;
}
}
if keep {
accounts.push(Account {
id: acc_id,
iban,
currency: "EUR".to_string(),
});
}
// Optimization: Stop if we found all wanted accounts
if let Some(_) = wanted_set {
if found_count >= target_count && target_count > 0 {
info!("Found all {} wanted accounts. Stopping search.", target_count);
return Ok(accounts);
}
}
}
}
}
info!("Found {} matching accounts in GoCardless", accounts.len());
Ok(accounts)
}
#[instrument(skip(self))]
async fn get_transactions(&self, account_id: &str, start: NaiveDate, end: NaiveDate) -> Result<Vec<BankTransaction>> {
let mut client = self.client.lock().await;
client.obtain_access_token().await?;
let response_result = client.get_transactions(
account_id,
Some(&start.to_string()),
Some(&end.to_string())
).await;
match response_result {
Ok(response) => {
let mut transactions = Vec::new();
for tx in response.transactions.booked {
match map_transaction(tx) {
Ok(t) => transactions.push(t),
Err(e) => tracing::error!("Failed to map transaction: {}", e),
}
}
info!("Fetched {} transactions for account {}", transactions.len(), account_id);
Ok(transactions)
},
Err(e) => {
// Handle 429 specifically?
let err_str = e.to_string();
if err_str.contains("429") {
warn!("Rate limit reached for account {}. Skipping.", account_id);
// Return empty list implies "no transactions found", which is safe for sync loop (it just won't sync this account).
// Or we could return an error if we want to stop?
// Returning empty list allows other accounts to potentially proceed if limits are per-account (which GC says they are!)
return Ok(vec![]);
}
Err(e.into())
}
}
}
}

View File

@@ -0,0 +1,147 @@
use rust_decimal::Decimal;
use rust_decimal::prelude::Signed;
use std::str::FromStr;
use anyhow::Result;
use crate::core::models::BankTransaction;
use gocardless_client::models::Transaction;
pub fn map_transaction(tx: Transaction) -> Result<BankTransaction> {
let internal_id = tx.transaction_id
.ok_or_else(|| anyhow::anyhow!("Transaction ID missing"))?;
let date_str = tx.booking_date.or(tx.value_date)
.ok_or_else(|| anyhow::anyhow!("Transaction date missing"))?;
let date = chrono::NaiveDate::parse_from_str(&date_str, "%Y-%m-%d")?;
let amount = Decimal::from_str(&tx.transaction_amount.amount)?;
let currency = tx.transaction_amount.currency;
let mut foreign_amount = None;
let mut foreign_currency = None;
if let Some(exchanges) = tx.currency_exchange {
if let Some(exchange) = exchanges.first() {
if let (Some(source_curr), Some(rate_str)) = (&exchange.source_currency, &exchange.exchange_rate) {
foreign_currency = Some(source_curr.clone());
if let Ok(rate) = Decimal::from_str(rate_str) {
// If instructedAmount is not available (it's not in our DTO yet), we calculate it.
// But wait, normally instructedAmount is the foreign amount.
// If we don't have it, we estimate: foreign = amount * rate?
// Actually usually: Base (Account) Amount = Foreign Amount / Rate OR Foreign * Rate
// If I have 100 EUR and rate is 1.10 USD/EUR -> 110 USD.
// Let's check the GoCardless spec definition of exchangeRate.
// "exchangeRate": "Factor used to convert an amount from one currency into another. This reflects the price at which the acquirer has bought the currency."
// Without strict direction, simple multiplication is risky.
// ideally we should have `instructedAmount` or `unitCurrency` logic.
// For now, let's assume: foreign_amount = amount * rate is NOT always correct.
// BUT, usually `sourceCurrency` is the original currency.
// If I spent 10 USD, and my account is EUR.
// sourceCurrency: USD. targetCurrency: EUR.
// transactionAmount: -9.00 EUR.
// exchangeRate: ???
// Let's implement a safe calculation or just store what we have.
// Actually, simply multiplying might be wrong if the rate is inverted.
// Let's verify with unit tests if we had real data, but for now let's use the logic:
// foreign_amount = amount * rate (if rate > 0) or amount / rate ?
// Let's look at the example in my plan: "foreign_amount = amount * currencyExchange[0].exchangeRate"
// I will stick to that plan, but wrap it in a safe calculation.
let calc = amount.abs() * rate; // Usually rate is positive.
// We preserve the sign of the transaction amount for the foreign amount.
let sign = amount.signum();
foreign_amount = Some(calc * sign);
}
}
}
}
// Fallback for description: Remittance Unstructured -> Debtor/Creditor Name -> "Unknown"
let description = tx.remittance_information_unstructured
.or(tx.creditor_name.clone())
.or(tx.debtor_name.clone())
.unwrap_or_else(|| "Unknown Transaction".to_string());
Ok(BankTransaction {
internal_id,
date,
amount,
currency,
foreign_amount,
foreign_currency,
description,
counterparty_name: tx.creditor_name.or(tx.debtor_name),
counterparty_iban: tx.creditor_account.and_then(|a| a.iban).or(tx.debtor_account.and_then(|a| a.iban)),
})
}
#[cfg(test)]
mod tests {
use super::*;
use gocardless_client::models::{TransactionAmount, CurrencyExchange};
#[test]
fn test_map_normal_transaction() {
let t = Transaction {
transaction_id: Some("123".into()),
booking_date: Some("2023-01-01".into()),
value_date: None,
transaction_amount: TransactionAmount {
amount: "100.50".into(),
currency: "EUR".into(),
},
currency_exchange: None,
creditor_name: Some("Shop".into()),
creditor_account: None,
debtor_name: None,
debtor_account: None,
remittance_information_unstructured: Some("Groceries".into()),
proprietary_bank_transaction_code: None,
};
let res = map_transaction(t).unwrap();
assert_eq!(res.internal_id, "123");
assert_eq!(res.amount, Decimal::new(10050, 2));
assert_eq!(res.currency, "EUR");
assert_eq!(res.foreign_amount, None);
assert_eq!(res.description, "Groceries");
}
#[test]
fn test_map_multicurrency_transaction() {
let t = Transaction {
transaction_id: Some("124".into()),
booking_date: Some("2023-01-02".into()),
value_date: None,
transaction_amount: TransactionAmount {
amount: "-10.00".into(),
currency: "EUR".into(),
},
currency_exchange: Some(vec![CurrencyExchange {
source_currency: Some("USD".into()),
exchange_rate: Some("1.10".into()),
unit_currency: None,
target_currency: Some("EUR".into()),
}]),
creditor_name: Some("US Shop".into()),
creditor_account: None,
debtor_name: None,
debtor_account: None,
remittance_information_unstructured: None,
proprietary_bank_transaction_code: None,
};
let res = map_transaction(t).unwrap();
assert_eq!(res.internal_id, "124");
assert_eq!(res.amount, Decimal::new(-1000, 2));
assert_eq!(res.foreign_currency, Some("USD".to_string()));
// 10.00 * 1.10 = 11.00. Sign should be preserved (-11.00)
assert_eq!(res.foreign_amount, Some(Decimal::new(-1100, 2)));
// Description fallback to creditor name
assert_eq!(res.description, "US Shop");
}
}

View File

@@ -0,0 +1,3 @@
pub mod client;
pub mod mapper;
pub mod cache;

View File

@@ -0,0 +1,2 @@
pub mod firefly;
pub mod gocardless;

3
banks2ff/src/core/mod.rs Normal file
View File

@@ -0,0 +1,3 @@
pub mod models;
pub mod ports;
pub mod sync;

View File

@@ -0,0 +1,31 @@
use rust_decimal::Decimal;
use chrono::NaiveDate;
#[derive(Debug, Clone, PartialEq)]
pub struct BankTransaction {
/// Source ID (GoCardless transactionId)
pub internal_id: String,
/// Booking date
pub date: NaiveDate,
/// Amount in account currency
pub amount: Decimal,
/// Account currency code (e.g., EUR)
pub currency: String,
/// Original amount (if currency exchange occurred)
pub foreign_amount: Option<Decimal>,
/// Original currency code
pub foreign_currency: Option<String>,
/// Remittance info or description
pub description: String,
/// Counterparty name
pub counterparty_name: Option<String>,
/// Counterparty IBAN
pub counterparty_iban: Option<String>,
}
#[derive(Debug, Clone, PartialEq)]
pub struct Account {
pub id: String,
pub iban: String,
pub currency: String,
}

View File

@@ -0,0 +1,42 @@
use async_trait::async_trait;
use chrono::NaiveDate;
use anyhow::Result;
#[cfg(test)]
use mockall::automock;
use crate::core::models::{BankTransaction, Account};
#[derive(Debug, Default)]
pub struct IngestResult {
pub created: usize,
pub duplicates: usize,
pub errors: usize,
pub healed: usize,
}
#[cfg_attr(test, automock)]
#[async_trait]
pub trait TransactionSource: Send + Sync {
/// Fetch accounts. Optionally filter by a list of wanted IBANs to save requests.
async fn get_accounts(&self, wanted_ibans: Option<Vec<String>>) -> Result<Vec<Account>>;
async fn get_transactions(&self, account_id: &str, start: NaiveDate, end: NaiveDate) -> Result<Vec<BankTransaction>>;
}
#[derive(Debug, Clone)]
pub struct TransactionMatch {
pub id: String,
pub has_external_id: bool,
}
#[cfg_attr(test, automock)]
#[async_trait]
pub trait TransactionDestination: Send + Sync {
async fn resolve_account_id(&self, iban: &str) -> Result<Option<String>>;
/// Get list of all active asset account IBANs to drive the sync
async fn get_active_account_ibans(&self) -> Result<Vec<String>>;
// New granular methods for Healer Logic
async fn get_last_transaction_date(&self, account_id: &str) -> Result<Option<NaiveDate>>;
async fn find_transaction(&self, account_id: &str, transaction: &BankTransaction) -> Result<Option<TransactionMatch>>;
async fn create_transaction(&self, account_id: &str, tx: &BankTransaction) -> Result<()>;
async fn update_transaction_external_id(&self, id: &str, external_id: &str) -> Result<()>;
}

299
banks2ff/src/core/sync.rs Normal file
View File

@@ -0,0 +1,299 @@
use anyhow::Result;
use tracing::{info, warn, instrument};
use crate::core::ports::{TransactionSource, TransactionDestination, IngestResult};
use chrono::{NaiveDate, Local};
#[instrument(skip(source, destination))]
pub async fn run_sync<S, D>(
source: &S,
destination: &D,
cli_start_date: Option<NaiveDate>,
cli_end_date: Option<NaiveDate>,
dry_run: bool,
) -> Result<()>
where
S: TransactionSource,
D: TransactionDestination,
{
info!("Starting synchronization...");
// Optimization: Get active Firefly IBANs first
let wanted_ibans = destination.get_active_account_ibans().await?;
info!("Syncing {} active accounts from Firefly III", wanted_ibans.len());
let accounts = source.get_accounts(Some(wanted_ibans)).await?;
// Default end date is Yesterday
let end_date = cli_end_date.unwrap_or_else(|| Local::now().date_naive() - chrono::Duration::days(1));
for account in accounts {
let span = tracing::info_span!("sync_account", iban = %account.iban);
let _enter = span.enter();
info!("Processing account...");
let dest_id_opt = destination.resolve_account_id(&account.iban).await?;
let Some(dest_id) = dest_id_opt else {
warn!("Account {} not found in destination. Skipping.", account.iban);
continue;
};
info!("Resolved destination ID: {}", dest_id);
// Determine Start Date
let start_date = if let Some(d) = cli_start_date {
d
} else {
// Default: Latest transaction date + 1 day
match destination.get_last_transaction_date(&dest_id).await? {
Some(last_date) => last_date + chrono::Duration::days(1),
None => {
// If no transaction exists in Firefly, we assume this is a fresh sync.
// Default to syncing last 30 days.
end_date - chrono::Duration::days(30)
},
}
};
if start_date > end_date {
info!("Start date {} is after end date {}. Nothing to sync.", start_date, end_date);
continue;
}
info!("Syncing interval: {} to {}", start_date, end_date);
// Optimization: Only use active accounts is already filtered in resolve_account_id
// However, GoCardless requisitions can expire.
// We should check if we can optimize the GoCardless fetching side.
// But currently get_transactions takes an account_id.
let transactions = source.get_transactions(&account.id, start_date, end_date).await?;
if transactions.is_empty() {
info!("No transactions found for period.");
continue;
}
info!("Fetched {} transactions from source.", transactions.len());
let mut stats = IngestResult::default();
// Healer Logic Loop
for tx in transactions {
// 1. Check if it exists
match destination.find_transaction(&dest_id, &tx).await? {
Some(existing) => {
if existing.has_external_id {
// Already synced properly
stats.duplicates += 1;
} else {
// Found "naked" transaction -> Heal it
if dry_run {
info!("[DRY RUN] Would heal transaction {} (Firefly ID: {})", tx.internal_id, existing.id);
stats.healed += 1;
} else {
info!("Healing transaction {} (Firefly ID: {})", tx.internal_id, existing.id);
if let Err(e) = destination.update_transaction_external_id(&existing.id, &tx.internal_id).await {
tracing::error!("Failed to heal transaction: {}", e);
stats.errors += 1;
} else {
stats.healed += 1;
}
}
}
},
None => {
// New transaction
if dry_run {
info!("[DRY RUN] Would create transaction {}", tx.internal_id);
stats.created += 1;
} else {
if let Err(e) = destination.create_transaction(&dest_id, &tx).await {
// Firefly might still reject it as duplicate if hash matches, even if we didn't find it via heuristic
// (unlikely if heuristic is good, but possible)
let err_str = e.to_string();
if err_str.contains("422") || err_str.contains("Duplicate") {
warn!("Duplicate rejected by Firefly: {}", tx.internal_id);
stats.duplicates += 1;
} else {
tracing::error!("Failed to create transaction: {}", e);
stats.errors += 1;
}
} else {
stats.created += 1;
}
}
}
}
}
info!("Sync complete. Created: {}, Healed: {}, Duplicates: {}, Errors: {}",
stats.created, stats.healed, stats.duplicates, stats.errors);
}
info!("Synchronization finished.");
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use crate::core::ports::{MockTransactionSource, MockTransactionDestination, TransactionMatch};
use crate::core::models::{Account, BankTransaction};
use rust_decimal::Decimal;
use mockall::predicate::*;
#[tokio::test]
async fn test_sync_flow_create_new() {
let mut source = MockTransactionSource::new();
let mut dest = MockTransactionDestination::new();
// Source setup
source.expect_get_accounts()
.with(always()) // Match any argument
.returning(|_| Ok(vec![Account {
id: "src_1".to_string(),
iban: "NL01".to_string(),
currency: "EUR".to_string(),
}]));
let tx = BankTransaction {
internal_id: "tx1".into(),
date: NaiveDate::from_ymd_opt(2023, 1, 1).unwrap(),
amount: Decimal::new(100, 0),
currency: "EUR".into(),
foreign_amount: None,
foreign_currency: None,
description: "Test".into(),
counterparty_name: None,
counterparty_iban: None,
};
let tx_clone = tx.clone();
source.expect_get_transactions()
.returning(move |_, _, _| Ok(vec![tx.clone()]));
// Destination setup
dest.expect_get_active_account_ibans()
.returning(|| Ok(vec!["NL01".to_string()]));
dest.expect_resolve_account_id()
.returning(|_| Ok(Some("dest_1".into())));
dest.expect_get_last_transaction_date()
.returning(|_| Ok(Some(NaiveDate::from_ymd_opt(2022, 12, 31).unwrap())));
// 1. Find -> None
dest.expect_find_transaction()
.times(1)
.returning(|_, _| Ok(None));
// 2. Create -> Ok
dest.expect_create_transaction()
.with(eq("dest_1"), eq(tx_clone))
.times(1)
.returning(|_, _| Ok(()));
// Execution
let res = run_sync(&source, &dest, None, None, false).await;
assert!(res.is_ok());
}
#[tokio::test]
async fn test_sync_flow_heal_existing() {
let mut source = MockTransactionSource::new();
let mut dest = MockTransactionDestination::new();
dest.expect_get_active_account_ibans()
.returning(|| Ok(vec!["NL01".to_string()]));
source.expect_get_accounts()
.with(always())
.returning(|_| Ok(vec![Account {
id: "src_1".to_string(),
iban: "NL01".to_string(),
currency: "EUR".to_string(),
}]));
source.expect_get_transactions()
.returning(|_, _, _| Ok(vec![
BankTransaction {
internal_id: "tx1".into(),
date: NaiveDate::from_ymd_opt(2023, 1, 1).unwrap(),
amount: Decimal::new(100, 0),
currency: "EUR".into(),
foreign_amount: None,
foreign_currency: None,
description: "Test".into(),
counterparty_name: None,
counterparty_iban: None,
}
]));
dest.expect_resolve_account_id().returning(|_| Ok(Some("dest_1".into())));
dest.expect_get_last_transaction_date().returning(|_| Ok(Some(NaiveDate::from_ymd_opt(2022, 12, 31).unwrap())));
// 1. Find -> Some(No External ID)
dest.expect_find_transaction()
.times(1)
.returning(|_, _| Ok(Some(TransactionMatch {
id: "ff_tx_1".to_string(),
has_external_id: false,
})));
// 2. Update -> Ok
dest.expect_update_transaction_external_id()
.with(eq("ff_tx_1"), eq("tx1"))
.times(1)
.returning(|_, _| Ok(()));
let res = run_sync(&source, &dest, None, None, false).await;
assert!(res.is_ok());
}
#[tokio::test]
async fn test_sync_flow_dry_run() {
let mut source = MockTransactionSource::new();
let mut dest = MockTransactionDestination::new();
dest.expect_get_active_account_ibans()
.returning(|| Ok(vec!["NL01".to_string()]));
source.expect_get_accounts()
.with(always())
.returning(|_| Ok(vec![Account {
id: "src_1".to_string(),
iban: "NL01".to_string(),
currency: "EUR".to_string(),
}]));
let tx = BankTransaction {
internal_id: "tx1".into(),
date: NaiveDate::from_ymd_opt(2023, 1, 1).unwrap(),
amount: Decimal::new(100, 0),
currency: "EUR".into(),
foreign_amount: None,
foreign_currency: None,
description: "Test".into(),
counterparty_name: None,
counterparty_iban: None,
};
source.expect_get_transactions()
.returning(move |_, _, _| Ok(vec![tx.clone()]));
dest.expect_resolve_account_id().returning(|_| Ok(Some("dest_1".into())));
dest.expect_get_last_transaction_date().returning(|_| Ok(Some(NaiveDate::from_ymd_opt(2022, 12, 31).unwrap())));
// 1. Find -> None (New transaction)
dest.expect_find_transaction()
.returning(|_, _| Ok(None));
// 2. Create -> NEVER Called (Dry Run)
dest.expect_create_transaction().never();
dest.expect_update_transaction_external_id().never();
let res = run_sync(&source, &dest, None, None, true).await;
assert!(res.is_ok());
}
}

74
banks2ff/src/main.rs Normal file
View File

@@ -0,0 +1,74 @@
mod adapters;
mod core;
use clap::Parser;
use tracing::{info, error};
use crate::adapters::gocardless::client::GoCardlessAdapter;
use crate::adapters::firefly::client::FireflyAdapter;
use crate::core::sync::run_sync;
use gocardless_client::client::GoCardlessClient;
use firefly_client::client::FireflyClient;
use std::env;
use chrono::NaiveDate;
#[derive(Parser, Debug)]
#[command(author, version, about, long_about = None)]
struct Args {
/// Path to configuration file (optional)
#[arg(short, long)]
config: Option<String>,
/// Start date for synchronization (YYYY-MM-DD). Defaults to last transaction date + 1.
#[arg(short, long)]
start: Option<NaiveDate>,
/// End date for synchronization (YYYY-MM-DD). Defaults to yesterday.
#[arg(short, long)]
end: Option<NaiveDate>,
/// Dry run mode: Do not create or update transactions in Firefly III.
#[arg(long, default_value_t = false)]
dry_run: bool,
}
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// Initialize logging
tracing_subscriber::fmt()
.with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
.init();
// Load environment variables
dotenvy::dotenv().ok();
let args = Args::parse();
info!("Starting banks2ff...");
if args.dry_run {
info!("DRY RUN MODE ENABLED: No changes will be made to Firefly III.");
}
// Config Load
let gc_url = env::var("GOCARDLESS_URL").unwrap_or_else(|_| "https://bankaccountdata.gocardless.com".to_string());
let gc_id = env::var("GOCARDLESS_ID").expect("GOCARDLESS_ID not set");
let gc_key = env::var("GOCARDLESS_KEY").expect("GOCARDLESS_KEY not set");
let ff_url = env::var("FIREFLY_III_URL").expect("FIREFLY_III_URL not set");
let ff_key = env::var("FIREFLY_III_API_KEY").expect("FIREFLY_III_API_KEY not set");
// Clients
let gc_client = GoCardlessClient::new(&gc_url, &gc_id, &gc_key)?;
let ff_client = FireflyClient::new(&ff_url, &ff_key)?;
// Adapters
let source = GoCardlessAdapter::new(gc_client);
let destination = FireflyAdapter::new(ff_client);
// Run
match run_sync(&source, &destination, args.start, args.end, args.dry_run).await {
Ok(_) => info!("Sync completed successfully."),
Err(e) => error!("Sync failed: {}", e),
}
Ok(())
}

114
docs/architecture.md Normal file
View File

@@ -0,0 +1,114 @@
# Architecture Documentation
## Overview
Banks2FF implements a **Hexagonal (Ports & Adapters) Architecture** to synchronize bank transactions from GoCardless to Firefly III. This architecture separates business logic from external concerns, making the system testable and maintainable.
## Workspace Structure
```
banks2ff/
├── banks2ff/ # Main CLI application
│ └── src/
│ ├── core/ # Domain logic and models
│ ├── adapters/ # External service integrations
│ └── main.rs # CLI entry point
├── firefly-client/ # Firefly III API client library
├── gocardless-client/ # GoCardless API client library
└── docs/ # Architecture documentation
```
## Core Components
### 1. Domain Core (`banks2ff/src/core/`)
**models.rs**: Defines domain entities
- `BankTransaction`: Core transaction model with multi-currency support
- `Account`: Bank account representation
- Supports `foreign_amount` and `foreign_currency` for international transactions
**ports.rs**: Defines abstraction traits
- `TransactionSource`: Interface for fetching transactions (implemented by GoCardless adapter)
- `TransactionDestination`: Interface for storing transactions (implemented by Firefly adapter)
- Traits are mockable for isolated testing
**sync.rs**: Synchronization engine
- `run_sync()`: Orchestrates the entire sync process
- Implements "Healer" strategy for idempotency
- Smart date range calculation (Last Transaction Date + 1 to Yesterday)
### 2. Adapters (`banks2ff/src/adapters/`)
**gocardless/**: GoCardless integration
- `client.rs`: Wrapper for GoCardless client with token management
- `mapper.rs`: Converts GoCardless API responses to domain models
- `cache.rs`: Caches account mappings to reduce API calls
- Correctly handles multi-currency via `currencyExchange` array parsing
**firefly/**: Firefly III integration
- `client.rs`: Wrapper for Firefly client for transaction storage
- Maps domain models to Firefly API format
### 3. API Clients
Both clients are hand-crafted using `reqwest`:
- Strongly-typed DTOs for compile-time safety
- Custom error handling with `thiserror`
- Rate limit awareness and graceful degradation
## Synchronization Process
The "Healer" strategy ensures idempotency:
1. **Account Discovery**: Fetch active accounts from GoCardless
2. **Account Matching**: Match GoCardless accounts to Firefly asset accounts by IBAN
3. **Date Window**: Calculate sync range (Last Firefly transaction + 1 to Yesterday)
4. **Transaction Processing**:
- **Search**: Look for existing transaction using windowed heuristic (date ± 3 days, exact amount)
- **Heal**: If found without `external_id`, update with GoCardless transaction ID
- **Skip**: If found with matching `external_id`, ignore
- **Create**: If not found, create new transaction in Firefly
## Key Features
### Multi-Currency Support
- Parses `currencyExchange` array from GoCardless responses
- Calculates `foreign_amount = amount * exchange_rate`
- Maps to Firefly's `foreign_amount` and `foreign_currency_code` fields
### Rate Limit Management
- **Caching**: Stores `AccountId -> IBAN` mappings to reduce requisition calls
- **Token Reuse**: Maintains tokens until expiry to minimize auth requests
- **Graceful Handling**: Continues sync for other accounts when encountering 429 errors
### Idempotency
- GoCardless `transactionId` → Firefly `external_id` mapping
- Windowed duplicate detection prevents double-creation
- Historical transaction healing for pre-existing data
## Data Flow
```
GoCardless API → GoCardlessAdapter → TransactionSource → SyncEngine → TransactionDestination → FireflyAdapter → Firefly API
```
## Testing Strategy
- **Unit Tests**: Core logic with `mockall` for trait mocking
- **Integration Tests**: API clients with `wiremock` for HTTP mocking
- **Fixture Testing**: Real JSON responses for adapter mapping validation
- **Isolation**: Business logic tested without external dependencies
## Error Handling
- **Custom Errors**: `thiserror` for domain-specific error types
- **Propagation**: `anyhow` for error context across async boundaries
- **Graceful Degradation**: Rate limits and network issues don't crash entire sync
- **Structured Logging**: `tracing` for observability and debugging
## Configuration Management
- Environment variables loaded via `dotenvy`
- Workspace-level dependency management
- Feature flags for optional functionality
- Secure credential handling (no hardcoded secrets)

20
firefly-client/Cargo.toml Normal file
View File

@@ -0,0 +1,20 @@
[package]
name = "firefly-client"
version.workspace = true
edition.workspace = true
authors.workspace = true
[dependencies]
reqwest = { version = "0.11", default-features = false, features = ["json", "rustls-tls"] }
serde = { workspace = true }
serde_json = { workspace = true }
thiserror = { workspace = true }
tracing = { workspace = true }
url = { workspace = true }
chrono = { workspace = true }
rust_decimal = { workspace = true }
[dev-dependencies]
wiremock = { workspace = true }
tokio = { workspace = true }
tokio-test = { workspace = true }

View File

@@ -0,0 +1,126 @@
use reqwest::{Client, Url};
use serde::de::DeserializeOwned;
use thiserror::Error;
use tracing::instrument;
use crate::models::{AccountArray, TransactionStore, TransactionArray, TransactionUpdate};
#[derive(Error, Debug)]
pub enum FireflyError {
#[error("Request failed: {0}")]
RequestFailed(#[from] reqwest::Error),
#[error("API Error: {0}")]
ApiError(String),
#[error("URL Parse Error: {0}")]
UrlParseError(#[from] url::ParseError),
}
pub struct FireflyClient {
base_url: Url,
client: Client,
access_token: String,
}
impl FireflyClient {
pub fn new(base_url: &str, access_token: &str) -> Result<Self, FireflyError> {
Ok(Self {
base_url: Url::parse(base_url)?,
client: Client::new(),
access_token: access_token.to_string(),
})
}
#[instrument(skip(self))]
pub async fn get_accounts(&self, _iban: &str) -> Result<AccountArray, FireflyError> {
let mut url = self.base_url.join("/api/v1/accounts")?;
url.query_pairs_mut()
.append_pair("type", "asset");
self.get_authenticated(url).await
}
#[instrument(skip(self))]
pub async fn search_accounts(&self, query: &str) -> Result<AccountArray, FireflyError> {
let mut url = self.base_url.join("/api/v1/search/accounts")?;
url.query_pairs_mut()
.append_pair("query", query)
.append_pair("type", "asset")
.append_pair("field", "all");
self.get_authenticated(url).await
}
#[instrument(skip(self, transaction))]
pub async fn store_transaction(&self, transaction: TransactionStore) -> Result<(), FireflyError> {
let url = self.base_url.join("/api/v1/transactions")?;
let response = self.client.post(url)
.bearer_auth(&self.access_token)
.header("accept", "application/json")
.json(&transaction)
.send()
.await?;
if !response.status().is_success() {
let status = response.status();
let text = response.text().await?;
return Err(FireflyError::ApiError(format!("Store Transaction Failed {}: {}", status, text)));
}
Ok(())
}
#[instrument(skip(self))]
pub async fn list_account_transactions(&self, account_id: &str, start: Option<&str>, end: Option<&str>) -> Result<TransactionArray, FireflyError> {
let mut url = self.base_url.join(&format!("/api/v1/accounts/{}/transactions", account_id))?;
{
let mut pairs = url.query_pairs_mut();
if let Some(s) = start {
pairs.append_pair("start", s);
}
if let Some(e) = end {
pairs.append_pair("end", e);
}
// Limit to 50, could be higher but safer to page if needed. For heuristic checks 50 is usually plenty per day range.
pairs.append_pair("limit", "50");
}
self.get_authenticated(url).await
}
#[instrument(skip(self, update))]
pub async fn update_transaction(&self, id: &str, update: TransactionUpdate) -> Result<(), FireflyError> {
let url = self.base_url.join(&format!("/api/v1/transactions/{}", id))?;
let response = self.client.put(url)
.bearer_auth(&self.access_token)
.header("accept", "application/json")
.json(&update)
.send()
.await?;
if !response.status().is_success() {
let status = response.status();
let text = response.text().await?;
return Err(FireflyError::ApiError(format!("Update Transaction Failed {}: {}", status, text)));
}
Ok(())
}
async fn get_authenticated<T: DeserializeOwned>(&self, url: Url) -> Result<T, FireflyError> {
let response = self.client.get(url)
.bearer_auth(&self.access_token)
.header("accept", "application/json")
.send()
.await?;
if !response.status().is_success() {
let status = response.status();
let text = response.text().await?;
return Err(FireflyError::ApiError(format!("API request failed {}: {}", status, text)));
}
let data = response.json().await?;
Ok(data)
}
}

View File

@@ -0,0 +1,2 @@
pub mod client;
pub mod models;

View File

@@ -0,0 +1,81 @@
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AccountRead {
pub id: String,
pub attributes: Account,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Account {
pub name: String,
pub iban: Option<String>,
#[serde(rename = "type")]
pub account_type: String,
pub active: Option<bool>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AccountArray {
pub data: Vec<AccountRead>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TransactionRead {
pub id: String,
pub attributes: Transaction,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Transaction {
pub transactions: Vec<TransactionSplit>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TransactionSplit {
pub date: String,
pub amount: String,
pub description: String,
pub external_id: Option<String>,
pub currency_code: Option<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TransactionArray {
pub data: Vec<TransactionRead>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TransactionSplitStore {
#[serde(rename = "type")]
pub transaction_type: String,
pub date: String,
pub amount: String,
pub description: String,
pub source_id: Option<String>,
pub source_name: Option<String>,
pub destination_id: Option<String>,
pub destination_name: Option<String>,
pub currency_code: Option<String>,
pub foreign_amount: Option<String>,
pub foreign_currency_code: Option<String>,
pub external_id: Option<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TransactionStore {
pub transactions: Vec<TransactionSplitStore>,
pub apply_rules: Option<bool>,
pub fire_webhooks: Option<bool>,
pub error_if_duplicate_hash: Option<bool>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TransactionUpdate {
pub transactions: Vec<TransactionSplitUpdate>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TransactionSplitUpdate {
pub external_id: Option<String>,
}

View File

@@ -0,0 +1,62 @@
use firefly_client::client::FireflyClient;
use firefly_client::models::{TransactionStore, TransactionSplitStore};
use wiremock::matchers::{method, path, header};
use wiremock::{Mock, MockServer, ResponseTemplate};
use std::fs;
#[tokio::test]
async fn test_search_accounts() {
let mock_server = MockServer::start().await;
let fixture = fs::read_to_string("tests/fixtures/ff_accounts.json").unwrap();
Mock::given(method("GET"))
.and(path("/api/v1/search/accounts"))
.and(header("Authorization", "Bearer my-token"))
.respond_with(ResponseTemplate::new(200).set_body_string(fixture))
.mount(&mock_server)
.await;
let client = FireflyClient::new(&mock_server.uri(), "my-token").unwrap();
let accounts = client.search_accounts("NL01").await.unwrap();
assert_eq!(accounts.data.len(), 1);
assert_eq!(accounts.data[0].attributes.name, "Checking Account");
assert_eq!(accounts.data[0].attributes.iban.as_deref(), Some("NL01BANK0123456789"));
}
#[tokio::test]
async fn test_store_transaction() {
let mock_server = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/api/v1/transactions"))
.and(header("Authorization", "Bearer my-token"))
.respond_with(ResponseTemplate::new(200))
.mount(&mock_server)
.await;
let client = FireflyClient::new(&mock_server.uri(), "my-token").unwrap();
let tx = TransactionStore {
transactions: vec![TransactionSplitStore {
transaction_type: "withdrawal".to_string(),
date: "2023-01-01".to_string(),
amount: "10.00".to_string(),
description: "Test".to_string(),
source_id: Some("1".to_string()),
destination_name: Some("Shop".to_string()),
currency_code: None,
foreign_amount: None,
foreign_currency_code: None,
external_id: None,
source_name: None,
destination_id: None,
}],
apply_rules: None,
fire_webhooks: None,
error_if_duplicate_hash: None,
};
let result = client.store_transaction(tx).await;
assert!(result.is_ok());
}

View File

@@ -0,0 +1,22 @@
{
"data": [
{
"type": "accounts",
"id": "2",
"attributes": {
"name": "Checking Account",
"type": "asset",
"iban": "NL01BANK0123456789"
}
}
],
"meta": {
"pagination": {
"total": 1,
"count": 1,
"per_page": 20,
"current_page": 1,
"total_pages": 1
}
}
}

View File

@@ -0,0 +1,19 @@
[package]
name = "gocardless-client"
version.workspace = true
edition.workspace = true
authors.workspace = true
[dependencies]
reqwest = { version = "0.11", default-features = false, features = ["json", "rustls-tls"] }
serde = { workspace = true }
serde_json = { workspace = true }
thiserror = { workspace = true }
tracing = { workspace = true }
url = { workspace = true }
chrono = { workspace = true }
[dev-dependencies]
wiremock = { workspace = true }
tokio = { workspace = true }
tokio-test = { workspace = true }

View File

@@ -0,0 +1,129 @@
use reqwest::{Client, Url};
use serde::{Deserialize, Serialize};
use thiserror::Error;
use tracing::{debug, instrument};
use crate::models::{TokenResponse, PaginatedResponse, Requisition, Account, TransactionsResponse};
#[derive(Error, Debug)]
pub enum GoCardlessError {
#[error("Request failed: {0}")]
RequestFailed(#[from] reqwest::Error),
#[error("API Error: {0}")]
ApiError(String),
#[error("Serialization error: {0}")]
SerializationError(#[from] serde_json::Error),
#[error("URL Parse Error: {0}")]
UrlParseError(#[from] url::ParseError),
}
pub struct GoCardlessClient {
base_url: Url,
client: Client,
secret_id: String,
secret_key: String,
access_token: Option<String>,
access_expires_at: Option<chrono::DateTime<chrono::Utc>>,
}
#[derive(Serialize)]
struct TokenRequest<'a> {
secret_id: &'a str,
secret_key: &'a str,
}
impl GoCardlessClient {
pub fn new(base_url: &str, secret_id: &str, secret_key: &str) -> Result<Self, GoCardlessError> {
Ok(Self {
base_url: Url::parse(base_url)?,
client: Client::new(),
secret_id: secret_id.to_string(),
secret_key: secret_key.to_string(),
access_token: None,
access_expires_at: None,
})
}
#[instrument(skip(self))]
pub async fn obtain_access_token(&mut self) -> Result<(), GoCardlessError> {
// Check if current token is still valid (with 60s buffer)
if let Some(expires) = self.access_expires_at {
if chrono::Utc::now() < expires - chrono::Duration::seconds(60) {
debug!("Access token is still valid");
return Ok(());
}
}
let url = self.base_url.join("/api/v2/token/new/")?;
let body = TokenRequest {
secret_id: &self.secret_id,
secret_key: &self.secret_key,
};
debug!("Requesting new access token");
let response = self.client.post(url)
.json(&body)
.send()
.await?;
if !response.status().is_success() {
let status = response.status();
let text = response.text().await?;
return Err(GoCardlessError::ApiError(format!("Token request failed {}: {}", status, text)));
}
let token_resp: TokenResponse = response.json().await?;
self.access_token = Some(token_resp.access);
self.access_expires_at = Some(chrono::Utc::now() + chrono::Duration::seconds(token_resp.access_expires as i64));
debug!("Access token obtained");
Ok(())
}
#[instrument(skip(self))]
pub async fn get_requisitions(&self) -> Result<PaginatedResponse<Requisition>, GoCardlessError> {
let url = self.base_url.join("/api/v2/requisitions/")?;
self.get_authenticated(url).await
}
#[instrument(skip(self))]
pub async fn get_account(&self, id: &str) -> Result<Account, GoCardlessError> {
let url = self.base_url.join(&format!("/api/v2/accounts/{}/", id))?;
self.get_authenticated(url).await
}
#[instrument(skip(self))]
pub async fn get_transactions(&self, account_id: &str, date_from: Option<&str>, date_to: Option<&str>) -> Result<TransactionsResponse, GoCardlessError> {
let mut url = self.base_url.join(&format!("/api/v2/accounts/{}/transactions/", account_id))?;
{
let mut pairs = url.query_pairs_mut();
if let Some(from) = date_from {
pairs.append_pair("date_from", from);
}
if let Some(to) = date_to {
pairs.append_pair("date_to", to);
}
}
self.get_authenticated(url).await
}
async fn get_authenticated<T: for<'de> Deserialize<'de>>(&self, url: Url) -> Result<T, GoCardlessError> {
let token = self.access_token.as_ref().ok_or(GoCardlessError::ApiError("No access token available. Call obtain_access_token() first.".into()))?;
let response = self.client.get(url)
.bearer_auth(token)
.header("accept", "application/json")
.send()
.await?;
if !response.status().is_success() {
let status = response.status();
let text = response.text().await?;
return Err(GoCardlessError::ApiError(format!("API request failed {}: {}", status, text)));
}
let data = response.json().await?;
Ok(data)
}
}

View File

@@ -0,0 +1,2 @@
pub mod client;
pub mod models;

View File

@@ -0,0 +1,95 @@
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TokenResponse {
pub access: String,
pub access_expires: i32,
pub refresh: Option<String>,
pub refresh_expires: Option<i32>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Requisition {
pub id: String,
pub status: String,
pub accounts: Option<Vec<String>>,
pub reference: Option<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct PaginatedResponse<T> {
pub count: Option<i32>,
pub next: Option<String>,
pub previous: Option<String>,
pub results: Vec<T>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Account {
pub id: String,
pub created: Option<String>,
pub last_accessed: Option<String>,
pub iban: Option<String>,
pub institution_id: Option<String>,
pub status: Option<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TransactionsResponse {
pub transactions: TransactionBookedPending,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TransactionBookedPending {
pub booked: Vec<Transaction>,
pub pending: Option<Vec<Transaction>>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Transaction {
#[serde(rename = "transactionId")]
pub transaction_id: Option<String>,
#[serde(rename = "bookingDate")]
pub booking_date: Option<String>,
#[serde(rename = "valueDate")]
pub value_date: Option<String>,
#[serde(rename = "transactionAmount")]
pub transaction_amount: TransactionAmount,
#[serde(rename = "currencyExchange")]
pub currency_exchange: Option<Vec<CurrencyExchange>>,
#[serde(rename = "creditorName")]
pub creditor_name: Option<String>,
#[serde(rename = "creditorAccount")]
pub creditor_account: Option<AccountDetails>,
#[serde(rename = "debtorName")]
pub debtor_name: Option<String>,
#[serde(rename = "debtorAccount")]
pub debtor_account: Option<AccountDetails>,
#[serde(rename = "remittanceInformationUnstructured")]
pub remittance_information_unstructured: Option<String>,
#[serde(rename = "proprietaryBankTransactionCode")]
pub proprietary_bank_transaction_code: Option<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TransactionAmount {
pub amount: String,
pub currency: String,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CurrencyExchange {
#[serde(rename = "sourceCurrency")]
pub source_currency: Option<String>,
#[serde(rename = "exchangeRate")]
pub exchange_rate: Option<String>,
#[serde(rename = "unitCurrency")]
pub unit_currency: Option<String>,
#[serde(rename = "targetCurrency")]
pub target_currency: Option<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AccountDetails {
pub iban: Option<String>,
}

View File

@@ -0,0 +1,55 @@
use gocardless_client::client::GoCardlessClient;
use gocardless_client::models::TokenResponse;
use wiremock::matchers::{method, path};
use wiremock::{Mock, MockServer, ResponseTemplate};
use std::fs;
#[tokio::test]
async fn test_get_transactions_parsing() {
// 1. Setup WireMock
let mock_server = MockServer::start().await;
// Mock Token Endpoint
Mock::given(method("POST"))
.and(path("/api/v2/token/new/"))
.respond_with(ResponseTemplate::new(200).set_body_json(TokenResponse {
access: "fake_access_token".to_string(),
access_expires: 3600,
refresh: Some("fake_refresh".to_string()),
refresh_expires: Some(86400),
}))
.mount(&mock_server)
.await;
// Mock Transactions Endpoint
let fixture = fs::read_to_string("tests/fixtures/gc_transactions.json").unwrap();
Mock::given(method("GET"))
.and(path("/api/v2/accounts/ACC123/transactions/"))
.respond_with(ResponseTemplate::new(200).set_body_string(fixture))
.mount(&mock_server)
.await;
// 2. Run Client
let mut client = GoCardlessClient::new(&mock_server.uri(), "id", "key").unwrap();
client.obtain_access_token().await.unwrap();
let resp = client.get_transactions("ACC123", None, None).await.unwrap();
// 3. Assertions
assert_eq!(resp.transactions.booked.len(), 2);
let tx1 = &resp.transactions.booked[0];
assert_eq!(tx1.transaction_id.as_deref(), Some("TX123"));
assert_eq!(tx1.transaction_amount.amount, "100.00");
assert_eq!(tx1.transaction_amount.currency, "EUR");
let tx2 = &resp.transactions.booked[1];
assert_eq!(tx2.transaction_id.as_deref(), Some("TX124"));
assert_eq!(tx2.transaction_amount.amount, "-10.00");
// Verify Multi-Currency parsing
let exchange = tx2.currency_exchange.as_ref().unwrap();
assert_eq!(exchange[0].source_currency.as_deref(), Some("USD"));
assert_eq!(exchange[0].exchange_rate.as_deref(), Some("1.10"));
}

View File

@@ -0,0 +1,34 @@
{
"transactions": {
"booked": [
{
"transactionId": "TX123",
"bookingDate": "2023-10-01",
"transactionAmount": {
"amount": "100.00",
"currency": "EUR"
},
"debtorName": "John Doe",
"remittanceInformationUnstructured": "Payment for services"
},
{
"transactionId": "TX124",
"bookingDate": "2023-10-02",
"transactionAmount": {
"amount": "-10.00",
"currency": "EUR"
},
"currencyExchange": [
{
"sourceCurrency": "USD",
"exchangeRate": "1.10",
"targetCurrency": "EUR"
}
],
"creditorName": "US Store",
"remittanceInformationUnstructured": "US Purchase"
}
],
"pending": []
}
}

View File

@@ -1,129 +1,137 @@
# Implementation Plan: Bank2FF Refactoring
## 1. Objective
Refactor the `bank2ff` application from a prototype script into a robust, testable, and observable production-grade CLI tool. The application must synchronize bank transactions from GoCardless to Firefly III, ensuring:
- **Clean Architecture**: Decoupling of business logic from API clients.
- **Testability**: Ability to unit test core logic without external dependencies.
- **Multi-Currency Support**: Accurate capturing of foreign amounts and currencies.
- **Idempotency**: Preventing duplicate transactions in Firefly III.
Refactor the `bank2ff` application from a prototype script into a robust, testable, and observable production-grade CLI tool. The application must synchronize bank transactions from GoCardless to Firefly III.
## 2. Architecture: Hexagonal (Ports & Adapters)
**Key Constraints:**
- **Multi-Crate Workspace**: Separate crates for the CLI application and the API clients.
- **Hand-Crafted Clients**: No autogenerated code. Custom, strongly-typed clients for better UX.
- **Clean Architecture**: Hexagonal architecture within the main application.
- **Multi-Currency**: Accurate handling of foreign amounts.
- **Test-Driven**: Every component must be testable from the start.
- **Observability**: Structured logging (tracing) throughout the stack.
- **Healer Strategy**: Detect and heal historical duplicates that lack external IDs.
- **Dry Run**: Safe mode to preview changes.
- **Rate Limit Handling**: Smart caching and graceful skipping to respect 4 requests/day limits.
The application will be structured into three distinct layers:
1. **Core (Domain)**: Pure Rust, no external API dependencies. Defines the `BankTransaction` model and the `Ports` (traits) for interacting with the world.
2. **Adapters**: Implementations of the Ports.
- `GoCardlessAdapter`: Implements `TransactionSource`.
- `FireflyAdapter`: Implements `TransactionDestination`.
3. **Application**: Configuration, CLI parsing, and wiring (`main.rs`).
## 2. Architecture
### Directory Structure
### Workspace Structure
The project uses a Cargo Workspace with three members:
1. `gocardless-client`: A reusable library crate wrapping the GoCardless Bank Account Data API v2.
2. `firefly-client`: A reusable library crate wrapping the Firefly III API v6.4.4.
3. `banks2ff`: The main CLI application containing the Domain Core and Adapters that use the client libraries.
### Directory Layout
```text
bank2ff/src/
├── core/
│ ├── mod.rs
│ ├── models.rs # Domain entities (BankTransaction, Account)
── ports.rs # Traits (TransactionSource, TransactionDestination)
└── sync.rs # Core business logic (The "Use Case")
├── adapters/
│ ├── mod.rs
├── gocardless/
│ │ ├── mod.rs
│ ├── client.rs # Wrapper around generated client (Auth/RateLimits)
│ └── mapper.rs # Logic to map API response -> Domain Model
└── firefly/
├── mod.rs
── client.rs # Implementation of TransactionDestination
└── main.rs # Entry point, wiring, and config loading
root/
├── Cargo.toml # Workspace definition
├── gocardless-client/ # Crate 1
│ ├── Cargo.toml
── src/
├── lib.rs
│ ├── client.rs # Reqwest client logic
├── models.rs # Request/Response DTOs
└── tests/ # Unit/Integration tests with mocks
├── firefly-client/ # Crate 2
│ ├── Cargo.toml
│ └── src/
├── lib.rs
├── client.rs # Reqwest client logic
── models.rs # Request/Response DTOs
│ └── tests/ # Unit/Integration tests with mocks
└── banks2ff/ # Crate 3 (Main App)
├── Cargo.toml
└── src/
├── main.rs
├── core/ # Domain
│ ├── models.rs
│ ├── ports.rs # Traits (Mockable)
│ ├── sync.rs # Logic (Tested with mocks)
│ └── tests/ # Unit tests for logic
└── adapters/ # Integration Layers
├── gocardless/
│ ├── client.rs # Uses gocardless-client & Cache
│ ├── cache.rs # JSON Cache for Account details
│ └── mapper.rs
└── firefly/
└── client.rs # Uses firefly-client
```
## 3. Core definitions
## 3. Observability Strategy
- **Tracing**: All crates will use the `tracing` crate.
- **Spans**:
- `client` crates: Create spans for every HTTP request (method, URL).
- `banks2ff` adapter: Create spans for "Fetching transactions".
- `banks2ff` sync: Create a span per account synchronization.
- **Context**: IDs (account, transaction) must be attached to log events.
### `src/core/models.rs`
The domain model must support multi-currency data.
## 4. Testing Strategy
- **Client Crates**:
- Use `wiremock` to mock the HTTP server.
- Test parsing of *real* JSON responses (saved in `tests/fixtures/`).
- Verify correct request construction (Headers, Auth, Body).
- **Core (`banks2ff`)**:
- Use `mockall` to mock `TransactionSource` and `TransactionDestination` traits.
- Unit test `sync::run_sync` logic (filtering, flow control) without any I/O.
- **Adapters (`banks2ff`)**:
- Test mapping logic (Client DTO -> Domain Model) using unit tests with fixtures.
```rust
pub struct BankTransaction {
pub internal_id: String, // Source ID (GoCardless transactionId)
pub date: NaiveDate, // Booking date
pub amount: Decimal, // Amount in account currency
pub currency: String, // Account currency code (e.g., EUR)
pub foreign_amount: Option<Decimal>, // Original amount (if currency exchange occurred)
pub foreign_currency: Option<String>,// Original currency code
pub description: String,
pub counterparty_name: Option<String>,
pub counterparty_iban: Option<String>,
}
```
## 5. Implementation Steps
### `src/core/ports.rs`
Traits to decouple the architecture.
### Phase 1: Infrastructure & Workspace
- [x] **Setup**: Initialize `gocardless-client` and `firefly-client` crates. Update root `Cargo.toml`.
- [x] **Dependencies**: Add `reqwest`, `serde`, `thiserror`, `url`, `tracing`.
- [x] **Test Deps**: Add `wiremock`, `tokio-test`, `serde_json` (dev-dependencies).
```rust
#[async_trait]
pub trait TransactionSource: Send + Sync {
async fn get_accounts(&self) -> Result<Vec<Account>>;
async fn get_transactions(&self, account_id: &str, start: NaiveDate, end: NaiveDate) -> Result<Vec<BankTransaction>>;
}
### Phase 2: Core (`banks2ff`)
- [x] **Definitions**: Implement `models.rs` and `ports.rs` in `banks2ff`.
- [x] **Mocks**: Add `mockall` attribute to ports for easier testing.
#[async_trait]
pub trait TransactionDestination: Send + Sync {
async fn resolve_account_id(&self, iban: &str) -> Result<Option<String>>;
async fn ingest_transactions(&self, account_id: &str, transactions: Vec<BankTransaction>) -> Result<IngestResult>;
}
```
### Phase 3: GoCardless Client Crate
- [x] **Models**: Define DTOs in `gocardless-client/src/models.rs`.
- [x] **Fixtures**: Create `tests/fixtures/gc_transactions.json` (real example data).
- [x] **Client**: Implement `GoCardlessClient`.
- [x] **Tests**: Write `tests/client_test.rs` using `wiremock` to serve the fixture and verify the client parses it correctly.
## 4. Implementation Steps
### Phase 4: GoCardless Adapter (`banks2ff`)
- [x] **Implementation**: Implement `TransactionSource`.
- [x] **Logic**: Handle **Multi-Currency** (inspect `currencyExchange`).
- [x] **Tests**: Unit test the *mapping logic* specifically. Input: GC Client DTO. Output: Domain Model. Assert foreign amounts are correct.
- [x] **Optimization**: Implement "Firefly Leading" strategy (only fetch wanted accounts).
- [x] **Optimization**: Implement Account Cache & Rate Limit handling.
### Phase 1: Infrastructure & Core
1. **Setup**: Create the directory structure.
2. **Dependencies**: Add `anyhow`, `thiserror`, `tracing`, `tracing-subscriber`, `async-trait`, `rust_decimal`, `chrono`.
3. **Core**: Implement `models.rs` and `ports.rs`.
### Phase 5: Firefly Client Crate
- [x] **Models**: Define DTOs in `firefly-client/src/models.rs`.
- [x] **Fixtures**: Create `tests/fixtures/ff_store_transaction.json`.
- [x] **Client**: Implement `FireflyClient`.
- [x] **Tests**: Write `tests/client_test.rs` using `wiremock` to verify auth headers and body serialization.
### Phase 2: GoCardless Adapter (Source)
1. **Token Management**: Create a wrapper struct that holds the `gocardless_bankaccount_data_api` configuration and manages the Access/Refresh token lifecycle. It should check validity before every request.
2. **Mapping Logic**:
- Map `transactionAmount.amount` -> `BankTransaction.amount`.
- **Multi-Currency**: Inspect `currencyExchange`.
- If present, map `sourceCurrency` -> `foreign_currency`.
- Calculate `foreign_amount` using `exchangeRate` if `instructedAmount` is missing.
- Map `transactionId` -> `internal_id`.
3. **Trait Implementation**: Implement `TransactionSource`.
### Phase 6: Firefly Adapter (`banks2ff`)
- [x] **Implementation**: Implement `TransactionDestination`.
- [x] **Logic**: Set `external_id`, handle Credit/Debit swap.
- [x] **Tests**: Unit test mapping logic. Verify `external_id` is populated.
- [x] **Update**: Refactor for "Healer" strategy (split `ingest` into `find`, `create`, `update`).
### Phase 3: Firefly Adapter (Destination)
1. **Client Wrapper**: Initialize `firefly_iii_api` client with API Key.
2. **Resolution**: Implement `resolve_account_id` by querying Firefly accounts by IBAN (using the search/list endpoint).
3. **Ingestion**:
- Map `BankTransaction` to `TransactionSplitStore`.
- **Crucial**: Set `external_id` = `BankTransaction.internal_id`.
- Handle Credits vs Debits (Swap Source/Destination logic).
- Call `store_transaction` endpoint.
- Handle 422/409 errors gracefully (log duplicates).
### Phase 7: Synchronization Engine
- [x] **Logic**: Implement `banks2ff::core::sync::run_sync` with "Healer" logic.
- Check Destination for existing transaction (Windowed Search).
- If found without ID: Heal (Update).
- If found with ID: Skip.
- If not found: Create.
- [x] **Smart Defaults**: Implement default start date (Last Firefly Date + 1) and end date (Yesterday).
- [x] **Tests**: Update unit tests for the new flow.
### Phase 4: Synchronization Engine
1. Implement `src/core/sync.rs`.
2. Logic:
- Fetch accounts from Source.
- For each account, find Destination ID.
- Fetch transactions (default: last 30 days).
- Filter/Process (optional).
- Ingest to Destination.
- Log statistics using `tracing`.
### Phase 8: Wiring & CLI
- [x] **CLI**: Add `-s/--start` and `-e/--end` arguments.
- [x] **CLI**: Add `--dry-run` argument.
- [x] **Wiring**: Pass these arguments to the sync engine.
- [x] **Observability**: Initialize `tracing_subscriber` with env filter.
- [x] **Config**: Load Env vars.
### Phase 5: Wiring
1. Refactor `main.rs`.
2. Initialize `tracing` for structured logging.
3. Load Config (Env variables).
4. Instantiate Adapters.
5. Run the Sync Engine.
## 5. Multi-Currency Handling Specifics
- **GoCardless Spec**: The `currencyExchange` array contains the details.
- **Logic**:
- If `currencyExchange` is not null/empty:
- `foreign_currency` = `currencyExchange[0].sourceCurrency`
- `foreign_amount` = `amount` * `currencyExchange[0].exchangeRate` (Validation: check if `unitCurrency` affects this).
- Ensure `Decimal` precision is handled correctly.
## 6. Testability & Observability
- **Tests**: Write a unit test for `core/sync.rs` using Mock structs for Source/Destination to verify the flow logic.
- **Logs**: Use `tracing::info!` for high-level progress ("Synced Account X") and `tracing::debug!` for details ("Transaction Y mapped to Z").
## 6. Multi-Currency Logic
- **GoCardless Adapter**:
- `foreign_currency` = `currencyExchange[0].sourceCurrency`
- `foreign_amount` = `amount` * `currencyExchange[0].exchangeRate`
- Test this math explicitly in Phase 4 tests.