Completely replace implementation #1

Manually merged
jjkiers merged 12 commits from push-wtvsvxromnno into master 2025-11-29 00:25:36 +00:00
31 changed files with 4802 additions and 139 deletions
Showing only changes of commit 9a5c6d0f68 - Show all commits

178
AGENTS.md Normal file
View File

@@ -0,0 +1,178 @@
# Banks2FF Development Guide
## Project Purpose
Banks2FF is a Rust CLI tool that synchronizes bank transactions from GoCardless Bank Account Data API to Firefly III personal finance manager. It implements a hexagonal architecture for clean separation of concerns and comprehensive testing.
## 🚨 CRITICAL: Financial Data Security
### **ABSOLUTE REQUIREMENT: Financial Data Masking**
**NEVER** expose, log, or display raw financial information including:
- Transaction amounts
- Account balances
- IBANs or account numbers
- Transaction descriptions
- Personal identifiers
- API keys or tokens
### **Compliance Protocol for Debugging**
When debugging financial data issues:
1. **Create Anonymized Test Scripts**: Write small, focused scripts that extract only the necessary data structure information
2. **Use Mock Data**: Replace real financial values with placeholder data
3. **Validate Structure, Not Values**: Focus on data structure integrity, not actual financial content
4. **Sanitize All Outputs**: Ensure any debugging output masks sensitive information
```rust
// ✅ GOOD: Structure validation with mock data
fn validate_transaction_structure() {
let mock_tx = BankTransaction {
amount: Decimal::new(12345, 2), // Mock amount
currency: "EUR".to_string(),
// ... other fields with mock data
};
// Validate structure only
}
// ❌ BAD: Exposing real financial data
fn debug_real_transactions(transactions: Vec<BankTransaction>) {
for tx in transactions {
println!("Real amount: {}", tx.amount); // SECURITY VIOLATION
}
}
```
## Rust Development Best Practices
### Error Handling
- **Use `thiserror`** for domain-specific error types in core modules
- **Use `anyhow`** for application-level error context and propagation
- **Never use `panic!`** in production code - handle errors gracefully
- **Implement `From` traits** for error type conversions
```rust
// Core domain errors
#[derive(Error, Debug)]
pub enum SyncError {
#[error("Failed to fetch transactions from source: {0}")]
SourceError(#[from] anyhow::Error),
#[error("Failed to store transaction: {0}")]
DestinationError(#[from] anyhow::Error),
}
```
### Async Programming
- **Use `tokio`** as the async runtime (workspace dependency)
- **Prefer `async-trait`** for trait methods that need to be async
- **Handle cancellation** properly with `select!` or `tokio::time::timeout`
- **Use `?` operator** for error propagation in async functions
### Testing Strategy
- **Unit Tests**: Test pure functions and business logic in isolation
- **Integration Tests**: Test adapter implementations with `wiremock`
- **Mock External Dependencies**: Use `mockall` for trait-based testing
- **Test Fixtures**: Store sample JSON responses in `tests/fixtures/`
```rust
#[cfg(test)]
mod tests {
use super::*;
use mockall::predicate::*;
#[tokio::test]
async fn test_sync_with_mock_source() {
let mut mock_source = MockTransactionSource::new();
// Setup mock expectations
// Test core logic
}
}
```
### Code Organization
- **Workspace Dependencies**: Define common dependencies in root `Cargo.toml`
- **Feature Flags**: Use features for optional functionality
- **Module Structure**: Keep modules focused and single-responsibility
- **Public API**: Minimize public surface area; prefer internal modules
### Dependencies and Patterns
**Key Workspace Dependencies:**
- `tokio`: Async runtime with full features
- `reqwest`: HTTP client with JSON support
- `serde`/`serde_json`: Serialization/deserialization
- `chrono`: Date/time handling with serde support
- `rust_decimal`: Precise decimal arithmetic for financial data
- `tracing`/`tracing-subscriber`: Structured logging
- `clap`: CLI argument parsing with derive macros
- `anyhow`/`thiserror`: Error handling
- `async-trait`: Async trait support
- `wiremock`: HTTP mocking for tests
- `mockall`: Runtime mocking for tests
## Development Workflow
### 1. Code Development
- Write code in appropriate modules following the hexagonal architecture
- Keep core business logic separate from external integrations
- Use workspace dependencies consistently
### 2. Testing
- Write tests alongside code in `#[cfg(test)]` modules
- Test both happy path and error conditions
- Use mock objects for external dependencies
- Ensure all tests pass: `cargo test --workspace`
### 3. Code Quality
- Follow Rust idioms and conventions
- Use `cargo fmt` for formatting
- Use `cargo clippy` for linting
- Ensure documentation for public APIs
### 4. Commit Standards
- Commit both code and tests together
- Write clear, descriptive commit messages
- Ensure the workspace compiles: `cargo build --workspace`
## Project Structure Guidelines
### Core Module (`banks2ff/src/core/`)
- **models.rs**: Domain entities (BankTransaction, Account)
- **ports.rs**: Trait definitions (TransactionSource, TransactionDestination)
- **sync.rs**: Business logic orchestration
### Adapters Module (`banks2ff/src/adapters/`)
- **gocardless/**: GoCardless API integration
- **firefly/**: Firefly III API integration
- Each adapter implements the appropriate port trait
### Client Libraries
- **gocardless-client/**: Standalone GoCardless API wrapper
- **firefly-client/**: Standalone Firefly III API wrapper
- Both use `reqwest` for HTTP communication
## Security Considerations
- **Never log sensitive data**: Use tracing filters to exclude financial information
- **Environment variables**: Store credentials in `.env` file (never in code)
- **Input validation**: Validate all external data before processing
- **Error messages**: Don't expose sensitive information in error messages
## Performance Considerations
- **Caching**: Use caching to reduce API calls (see GoCardlessAdapter)
- **Rate Limiting**: Handle 429 responses gracefully
- **Batch Processing**: Process transactions in reasonable batches
- **Async Concurrency**: Use `tokio` for concurrent operations where appropriate
## Observability
- **Structured Logging**: Use `tracing` with spans for operations
- **Error Context**: Provide context in error messages for debugging
- **Metrics**: Consider adding metrics for sync operations
- **Log Levels**: Use appropriate log levels (debug, info, warn, error)

2610
Cargo.lock generated Normal file

File diff suppressed because it is too large Load Diff

31
Cargo.toml Normal file
View File

@@ -0,0 +1,31 @@
[workspace]
members = [
"banks2ff",
"firefly-client",
"gocardless-client",
]
resolver = "2"
[workspace.package]
version = "0.1.0"
edition = "2021"
authors = ["Your Name <your.email@example.com>"]
[workspace.dependencies]
tokio = { version = "1.34", features = ["full"] }
anyhow = "1.0"
thiserror = "1.0"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
chrono = { version = "0.4", features = ["serde"] }
rust_decimal = { version = "1.33", features = ["serde-float"] }
async-trait = "0.1"
dotenvy = "0.15"
clap = { version = "4.4", features = ["derive", "env"] }
reqwest = { version = "0.11", features = ["json", "multipart"] }
url = "2.5"
wiremock = "0.5"
tokio-test = "0.4"
mockall = "0.11"

115
README.md
View File

@@ -1,30 +1,85 @@
# Bank2FF # Banks2FF
Bank2FF is a tool that can retrieve bank transactions from Gocardless and A robust command-line tool to synchronize bank transactions from GoCardless (formerly Nordigen) to Firefly III.
add them to Firefly III.
## Architecture
It contains autogenerated APIs for both Firefly III and for the
Gocardless Bank Account Data API. This project is a Rust Workspace consisting of:
- `banks2ff`: The main CLI application (Hexagonal Architecture).
## Usage - `gocardless-client`: A hand-crafted, strongly-typed library for the GoCardless Bank Account Data API.
- `firefly-client`: A hand-crafted, strongly-typed library for the Firefly III API.
TBD
## Features
## Generating the API clients - **Multi-Currency Support**: Correctly handles foreign currency transactions by extracting exchange rate data.
- **Idempotency (Healer Mode)**:
These API clients are generated with the OpenAPI Generators for Rust. - Detects duplicates using a windowed search (Date +/- 3 days, exact Amount).
- "Heals" historical transactions by updating them with the correct `external_id`.
These need Podman installed, and assume this command is run from the same - Skips transactions that already have a matching `external_id`.
directory where this README.md file is located. - **Clean Architecture**: Decoupled core logic makes it reliable and testable.
- **Observability**: Structured logging via `tracing`.
For Gocardless: - **Dry Run**: Preview changes without writing to Firefly III.
- **Rate Limit Protection**:
`podman run --rm -v ${PWD}:/local openapitools/openapi-generator-cli generate -g rust -o /local/gocardless-bankaccount-data-api -i 'https://bankaccountdata.gocardless.com/api/v2/swagger.json' --additional-properties=library=reqwest,packageName=gocardless-bankaccount-data-api,packageVersion=2.0.0,supportMiddleware=true,avoidBoxedModels=true` - Caches GoCardless account details to avoid unnecessary calls.
- Respects token expiry to minimize auth calls.
- Handles `429 Too Many Requests` gracefully by skipping affected accounts.
For Firefly III:
## Setup & Configuration
If necessary, change the URL to the definition. If that is a new version, then also change the `packageVersion` parameter.
1. **Prerequisites**:
`podman run --rm -v ${PWD}:/local openapitools/openapi-generator-cli generate -g rust -o /local/firefly-iii-api -i 'https://api-docs.firefly-iii.org/firefly-iii-2.1.0-v1.yaml' --additional-properties=library=reqwest,packageName=firefly-iii-api,packageVersion=2.1.0,supportMiddleware=true,avoidBoxedModels=true` - Rust (latest stable)
- An account with GoCardless Bank Account Data (get your `secret_id` and `secret_key`).
- A running Firefly III instance (get your Personal Access Token).
2. **Environment Variables**:
Copy `env.example` to `.env` and fill in your details:
```bash
cp env.example .env
```
Required variables:
- `GOCARDLESS_ID`: Your GoCardless Secret ID.
- `GOCARDLESS_KEY`: Your GoCardless Secret Key.
- `FIREFLY_III_URL`: The base URL of your Firefly instance (e.g., `https://money.example.com`).
- `FIREFLY_III_API_KEY`: Your Personal Access Token.
Optional:
- `GOCARDLESS_URL`: Defaults to `https://bankaccountdata.gocardless.com`.
- `RUST_LOG`: Set log level (e.g., `info`, `debug`, `trace`).
## Testing
The project has a comprehensive test suite using `wiremock` for API clients and `mockall` for core logic.
To run all tests:
```bash
cargo test --workspace
```
## Usage
To run the synchronization:
```bash
# Run via cargo (defaults: Start = Last Firefly Date + 1, End = Yesterday)
cargo run -p banks2ff
# Dry Run (Read-only)
cargo run -p banks2ff -- --dry-run
# Custom Date Range
cargo run -p banks2ff -- --start 2023-01-01 --end 2023-01-31
```
## How it works
1. **Fetch**: Retrieves active accounts from GoCardless (filtered by those present in Firefly III to save requests).
2. **Match**: Resolves the destination account in Firefly III by matching the IBAN.
3. **Sync Window**: Determines the start date automatically by finding the latest transaction in Firefly for that account.
4. **Process**: For each transaction:
- **Search**: Checks Firefly for an existing transaction (matching Amount and Date +/- 3 days).
- **Heal**: If found but missing an `external_id`, it updates the transaction.
- **Skip**: If found and matches `external_id`, it skips.
- **Create**: If not found, it creates a new transaction.

27
banks2ff/Cargo.toml Normal file
View File

@@ -0,0 +1,27 @@
[package]
name = "banks2ff"
version.workspace = true
edition.workspace = true
authors.workspace = true
[dependencies]
tokio = { workspace = true }
anyhow = { workspace = true }
tracing = { workspace = true }
tracing-subscriber = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
chrono = { workspace = true }
rust_decimal = { workspace = true }
dotenvy = { workspace = true }
clap = { workspace = true }
# Core logic dependencies
async-trait = { workspace = true }
# API Client dependencies
firefly-client = { path = "../firefly-client" }
gocardless-client = { path = "../gocardless-client" }
[dev-dependencies]
mockall = { workspace = true }

View File

@@ -0,0 +1,188 @@
use async_trait::async_trait;
use anyhow::Result;
use tracing::instrument;
use crate::core::ports::{TransactionDestination, TransactionMatch};
use crate::core::models::BankTransaction;
use firefly_client::client::FireflyClient;
use firefly_client::models::{TransactionStore, TransactionSplitStore, TransactionUpdate, TransactionSplitUpdate};
use std::sync::Arc;
use tokio::sync::Mutex;
use rust_decimal::Decimal;
use std::str::FromStr;
use chrono::NaiveDate;
pub struct FireflyAdapter {
client: Arc<Mutex<FireflyClient>>,
}
impl FireflyAdapter {
pub fn new(client: FireflyClient) -> Self {
Self {
client: Arc::new(Mutex::new(client)),
}
}
}
#[async_trait]
impl TransactionDestination for FireflyAdapter {
#[instrument(skip(self))]
async fn resolve_account_id(&self, iban: &str) -> Result<Option<String>> {
let client = self.client.lock().await;
let accounts = client.search_accounts(iban).await?;
// Look for exact match on IBAN, ensuring account is active
for acc in accounts.data {
// Filter for active accounts only (default is usually active, but let's check if attribute exists)
// Note: The Firefly API spec v6.4.4 Account object has 'active' attribute as boolean.
let is_active = acc.attributes.active.unwrap_or(true);
if !is_active {
continue;
}
if let Some(acc_iban) = acc.attributes.iban {
if acc_iban.replace(" ", "") == iban.replace(" ", "") {
return Ok(Some(acc.id));
}
}
}
Ok(None)
}
#[instrument(skip(self))]
async fn get_active_account_ibans(&self) -> Result<Vec<String>> {
let client = self.client.lock().await;
// Get all asset accounts. Note: Pagination might be needed if user has > 50 accounts.
// For typical users, 50 is enough. If needed we can loop pages.
// The client `get_accounts` method hardcodes limit=default. We should probably expose a list_all method or loop here.
// For now, let's assume page 1 covers it or use search.
let accounts = client.get_accounts("").await?; // Argument ignored in current impl
let mut ibans = Vec::new();
for acc in accounts.data {
let is_active = acc.attributes.active.unwrap_or(true);
if is_active {
if let Some(iban) = acc.attributes.iban {
if !iban.is_empty() {
ibans.push(iban);
}
}
}
}
Ok(ibans)
}
#[instrument(skip(self))]
async fn get_last_transaction_date(&self, account_id: &str) -> Result<Option<NaiveDate>> {
let client = self.client.lock().await;
// Fetch latest 1 transaction
let tx_list = client.list_account_transactions(account_id, None, None).await?;
if let Some(first) = tx_list.data.first() {
if let Some(split) = first.attributes.transactions.first() {
// Format is usually YYYY-MM-DDT... or YYYY-MM-DD
let date_str = split.date.split('T').next().unwrap_or(&split.date);
if let Ok(date) = NaiveDate::parse_from_str(date_str, "%Y-%m-%d") {
return Ok(Some(date));
}
}
}
Ok(None)
}
#[instrument(skip(self))]
async fn find_transaction(&self, account_id: &str, tx: &BankTransaction) -> Result<Option<TransactionMatch>> {
let client = self.client.lock().await;
// Search window: +/- 3 days
let start_date = tx.date - chrono::Duration::days(3);
let end_date = tx.date + chrono::Duration::days(3);
let tx_list = client.list_account_transactions(
account_id,
Some(&start_date.format("%Y-%m-%d").to_string()),
Some(&end_date.format("%Y-%m-%d").to_string())
).await?;
// Filter logic
for existing_tx in tx_list.data {
for split in existing_tx.attributes.transactions {
// 1. Check Amount (exact match absolute value)
if let Ok(amount) = Decimal::from_str(&split.amount) {
if amount.abs() == tx.amount.abs() {
// 2. Check External ID
if let Some(ref ext_id) = split.external_id {
if ext_id == &tx.internal_id {
return Ok(Some(TransactionMatch {
id: existing_tx.id.clone(),
has_external_id: true,
}));
}
} else {
// 3. "Naked" transaction match (Heuristic)
// If currency matches
if let Some(ref code) = split.currency_code {
if code != &tx.currency {
continue;
}
}
return Ok(Some(TransactionMatch {
id: existing_tx.id.clone(),
has_external_id: false,
}));
}
}
}
}
}
Ok(None)
}
#[instrument(skip(self))]
async fn create_transaction(&self, account_id: &str, tx: &BankTransaction) -> Result<()> {
let client = self.client.lock().await;
// Map to Firefly Transaction
let is_credit = tx.amount.is_sign_positive();
let transaction_type = if is_credit { "deposit" } else { "withdrawal" };
let split = TransactionSplitStore {
transaction_type: transaction_type.to_string(),
date: tx.date.format("%Y-%m-%d").to_string(),
amount: tx.amount.abs().to_string(),
description: tx.description.clone(),
source_id: if !is_credit { Some(account_id.to_string()) } else { None },
source_name: if is_credit { tx.counterparty_name.clone().or(Some("Unknown Sender".to_string())) } else { None },
destination_id: if is_credit { Some(account_id.to_string()) } else { None },
destination_name: if !is_credit { tx.counterparty_name.clone().or(Some("Unknown Recipient".to_string())) } else { None },
currency_code: Some(tx.currency.clone()),
foreign_amount: tx.foreign_amount.map(|d| d.abs().to_string()),
foreign_currency_code: tx.foreign_currency.clone(),
external_id: Some(tx.internal_id.clone()),
};
let store = TransactionStore {
transactions: vec![split],
apply_rules: Some(true),
fire_webhooks: Some(true),
error_if_duplicate_hash: Some(true),
};
client.store_transaction(store).await.map_err(|e| e.into())
}
#[instrument(skip(self))]
async fn update_transaction_external_id(&self, id: &str, external_id: &str) -> Result<()> {
let client = self.client.lock().await;
let update = TransactionUpdate {
transactions: vec![TransactionSplitUpdate {
external_id: Some(external_id.to_string()),
}],
};
client.update_transaction(id, update).await.map_err(|e| e.into())
}
}

View File

@@ -0,0 +1 @@
pub mod client;

View File

@@ -0,0 +1,51 @@
use std::collections::HashMap;
use std::fs;
use std::path::Path;
use serde::{Deserialize, Serialize};
use tracing::warn;
#[derive(Debug, Serialize, Deserialize, Default)]
pub struct AccountCache {
/// Map of Account ID -> IBAN
pub accounts: HashMap<String, String>,
}
impl AccountCache {
fn get_path() -> String {
".banks2ff-cache.json".to_string()
}
pub fn load() -> Self {
let path = Self::get_path();
if Path::new(&path).exists() {
match fs::read_to_string(&path) {
Ok(content) => match serde_json::from_str(&content) {
Ok(cache) => return cache,
Err(e) => warn!("Failed to parse cache file: {}", e),
},
Err(e) => warn!("Failed to read cache file: {}", e),
}
}
Self::default()
}
pub fn save(&self) {
let path = Self::get_path();
match serde_json::to_string_pretty(self) {
Ok(content) => {
if let Err(e) = fs::write(&path, content) {
warn!("Failed to write cache file: {}", e);
}
},
Err(e) => warn!("Failed to serialize cache: {}", e),
}
}
pub fn get_iban(&self, account_id: &str) -> Option<String> {
self.accounts.get(account_id).cloned()
}
pub fn insert(&mut self, account_id: String, iban: String) {
self.accounts.insert(account_id, iban);
}
}

View File

@@ -0,0 +1,152 @@
use async_trait::async_trait;
use chrono::NaiveDate;
use anyhow::Result;
use tracing::{info, instrument, warn};
use crate::core::ports::TransactionSource;
use crate::core::models::{Account, BankTransaction};
use crate::adapters::gocardless::mapper::map_transaction;
use crate::adapters::gocardless::cache::AccountCache;
use gocardless_client::client::GoCardlessClient;
use std::sync::Arc;
use tokio::sync::Mutex;
pub struct GoCardlessAdapter {
client: Arc<Mutex<GoCardlessClient>>,
cache: Arc<Mutex<AccountCache>>,
}
impl GoCardlessAdapter {
pub fn new(client: GoCardlessClient) -> Self {
Self {
client: Arc::new(Mutex::new(client)),
cache: Arc::new(Mutex::new(AccountCache::load())),
}
}
}
#[async_trait]
impl TransactionSource for GoCardlessAdapter {
#[instrument(skip(self))]
async fn get_accounts(&self, wanted_ibans: Option<Vec<String>>) -> Result<Vec<Account>> {
let mut client = self.client.lock().await;
let mut cache = self.cache.lock().await;
// Ensure token
client.obtain_access_token().await?;
let requisitions = client.get_requisitions().await?;
let mut accounts = Vec::new();
// Build a hashset of wanted IBANs if provided, for faster lookup
let wanted_set = wanted_ibans.map(|list| {
list.into_iter()
.map(|i| i.replace(" ", ""))
.collect::<std::collections::HashSet<_>>()
});
let mut found_count = 0;
let target_count = wanted_set.as_ref().map(|s| s.len()).unwrap_or(0);
for req in requisitions.results {
// Optimization: Only process Linked requisitions to avoid 401/403 on expired ones
if req.status != "LN" {
continue;
}
if let Some(req_accounts) = req.accounts {
for acc_id in req_accounts {
// 1. Check Cache
let mut iban_opt = cache.get_iban(&acc_id);
// 2. Fetch if missing
if iban_opt.is_none() {
match client.get_account(&acc_id).await {
Ok(details) => {
let new_iban = details.iban.unwrap_or_default();
cache.insert(acc_id.clone(), new_iban.clone());
cache.save();
iban_opt = Some(new_iban);
},
Err(e) => {
// If rate limit hit here, we might want to skip this account and continue?
// But get_account is critical to identify the account.
// If we fail here, we can't match.
warn!("Failed to fetch details for account {}: {}", acc_id, e);
continue;
}
}
}
let iban = iban_opt.unwrap_or_default();
let mut keep = true;
if let Some(ref wanted) = wanted_set {
if !wanted.contains(&iban.replace(" ", "")) {
keep = false;
} else {
found_count += 1;
}
}
if keep {
accounts.push(Account {
id: acc_id,
iban,
currency: "EUR".to_string(),
});
}
// Optimization: Stop if we found all wanted accounts
if let Some(_) = wanted_set {
if found_count >= target_count && target_count > 0 {
info!("Found all {} wanted accounts. Stopping search.", target_count);
return Ok(accounts);
}
}
}
}
}
info!("Found {} matching accounts in GoCardless", accounts.len());
Ok(accounts)
}
#[instrument(skip(self))]
async fn get_transactions(&self, account_id: &str, start: NaiveDate, end: NaiveDate) -> Result<Vec<BankTransaction>> {
let mut client = self.client.lock().await;
client.obtain_access_token().await?;
let response_result = client.get_transactions(
account_id,
Some(&start.to_string()),
Some(&end.to_string())
).await;
match response_result {
Ok(response) => {
let mut transactions = Vec::new();
for tx in response.transactions.booked {
match map_transaction(tx) {
Ok(t) => transactions.push(t),
Err(e) => tracing::error!("Failed to map transaction: {}", e),
}
}
info!("Fetched {} transactions for account {}", transactions.len(), account_id);
Ok(transactions)
},
Err(e) => {
// Handle 429 specifically?
let err_str = e.to_string();
if err_str.contains("429") {
warn!("Rate limit reached for account {}. Skipping.", account_id);
// Return empty list implies "no transactions found", which is safe for sync loop (it just won't sync this account).
// Or we could return an error if we want to stop?
// Returning empty list allows other accounts to potentially proceed if limits are per-account (which GC says they are!)
return Ok(vec![]);
}
Err(e.into())
}
}
}
}

View File

@@ -0,0 +1,147 @@
use rust_decimal::Decimal;
use rust_decimal::prelude::Signed;
use std::str::FromStr;
use anyhow::Result;
use crate::core::models::BankTransaction;
use gocardless_client::models::Transaction;
pub fn map_transaction(tx: Transaction) -> Result<BankTransaction> {
let internal_id = tx.transaction_id
.ok_or_else(|| anyhow::anyhow!("Transaction ID missing"))?;
let date_str = tx.booking_date.or(tx.value_date)
.ok_or_else(|| anyhow::anyhow!("Transaction date missing"))?;
let date = chrono::NaiveDate::parse_from_str(&date_str, "%Y-%m-%d")?;
let amount = Decimal::from_str(&tx.transaction_amount.amount)?;
let currency = tx.transaction_amount.currency;
let mut foreign_amount = None;
let mut foreign_currency = None;
if let Some(exchanges) = tx.currency_exchange {
if let Some(exchange) = exchanges.first() {
if let (Some(source_curr), Some(rate_str)) = (&exchange.source_currency, &exchange.exchange_rate) {
foreign_currency = Some(source_curr.clone());
if let Ok(rate) = Decimal::from_str(rate_str) {
// If instructedAmount is not available (it's not in our DTO yet), we calculate it.
// But wait, normally instructedAmount is the foreign amount.
// If we don't have it, we estimate: foreign = amount * rate?
// Actually usually: Base (Account) Amount = Foreign Amount / Rate OR Foreign * Rate
// If I have 100 EUR and rate is 1.10 USD/EUR -> 110 USD.
// Let's check the GoCardless spec definition of exchangeRate.
// "exchangeRate": "Factor used to convert an amount from one currency into another. This reflects the price at which the acquirer has bought the currency."
// Without strict direction, simple multiplication is risky.
// ideally we should have `instructedAmount` or `unitCurrency` logic.
// For now, let's assume: foreign_amount = amount * rate is NOT always correct.
// BUT, usually `sourceCurrency` is the original currency.
// If I spent 10 USD, and my account is EUR.
// sourceCurrency: USD. targetCurrency: EUR.
// transactionAmount: -9.00 EUR.
// exchangeRate: ???
// Let's implement a safe calculation or just store what we have.
// Actually, simply multiplying might be wrong if the rate is inverted.
// Let's verify with unit tests if we had real data, but for now let's use the logic:
// foreign_amount = amount * rate (if rate > 0) or amount / rate ?
// Let's look at the example in my plan: "foreign_amount = amount * currencyExchange[0].exchangeRate"
// I will stick to that plan, but wrap it in a safe calculation.
let calc = amount.abs() * rate; // Usually rate is positive.
// We preserve the sign of the transaction amount for the foreign amount.
let sign = amount.signum();
foreign_amount = Some(calc * sign);
}
}
}
}
// Fallback for description: Remittance Unstructured -> Debtor/Creditor Name -> "Unknown"
let description = tx.remittance_information_unstructured
.or(tx.creditor_name.clone())
.or(tx.debtor_name.clone())
.unwrap_or_else(|| "Unknown Transaction".to_string());
Ok(BankTransaction {
internal_id,
date,
amount,
currency,
foreign_amount,
foreign_currency,
description,
counterparty_name: tx.creditor_name.or(tx.debtor_name),
counterparty_iban: tx.creditor_account.and_then(|a| a.iban).or(tx.debtor_account.and_then(|a| a.iban)),
})
}
#[cfg(test)]
mod tests {
use super::*;
use gocardless_client::models::{TransactionAmount, CurrencyExchange};
#[test]
fn test_map_normal_transaction() {
let t = Transaction {
transaction_id: Some("123".into()),
booking_date: Some("2023-01-01".into()),
value_date: None,
transaction_amount: TransactionAmount {
amount: "100.50".into(),
currency: "EUR".into(),
},
currency_exchange: None,
creditor_name: Some("Shop".into()),
creditor_account: None,
debtor_name: None,
debtor_account: None,
remittance_information_unstructured: Some("Groceries".into()),
proprietary_bank_transaction_code: None,
};
let res = map_transaction(t).unwrap();
assert_eq!(res.internal_id, "123");
assert_eq!(res.amount, Decimal::new(10050, 2));
assert_eq!(res.currency, "EUR");
assert_eq!(res.foreign_amount, None);
assert_eq!(res.description, "Groceries");
}
#[test]
fn test_map_multicurrency_transaction() {
let t = Transaction {
transaction_id: Some("124".into()),
booking_date: Some("2023-01-02".into()),
value_date: None,
transaction_amount: TransactionAmount {
amount: "-10.00".into(),
currency: "EUR".into(),
},
currency_exchange: Some(vec![CurrencyExchange {
source_currency: Some("USD".into()),
exchange_rate: Some("1.10".into()),
unit_currency: None,
target_currency: Some("EUR".into()),
}]),
creditor_name: Some("US Shop".into()),
creditor_account: None,
debtor_name: None,
debtor_account: None,
remittance_information_unstructured: None,
proprietary_bank_transaction_code: None,
};
let res = map_transaction(t).unwrap();
assert_eq!(res.internal_id, "124");
assert_eq!(res.amount, Decimal::new(-1000, 2));
assert_eq!(res.foreign_currency, Some("USD".to_string()));
// 10.00 * 1.10 = 11.00. Sign should be preserved (-11.00)
assert_eq!(res.foreign_amount, Some(Decimal::new(-1100, 2)));
// Description fallback to creditor name
assert_eq!(res.description, "US Shop");
}
}

View File

@@ -0,0 +1,3 @@
pub mod client;
pub mod mapper;
pub mod cache;

View File

@@ -0,0 +1,2 @@
pub mod firefly;
pub mod gocardless;

3
banks2ff/src/core/mod.rs Normal file
View File

@@ -0,0 +1,3 @@
pub mod models;
pub mod ports;
pub mod sync;

View File

@@ -0,0 +1,31 @@
use rust_decimal::Decimal;
use chrono::NaiveDate;
#[derive(Debug, Clone, PartialEq)]
pub struct BankTransaction {
/// Source ID (GoCardless transactionId)
pub internal_id: String,
/// Booking date
pub date: NaiveDate,
/// Amount in account currency
pub amount: Decimal,
/// Account currency code (e.g., EUR)
pub currency: String,
/// Original amount (if currency exchange occurred)
pub foreign_amount: Option<Decimal>,
/// Original currency code
pub foreign_currency: Option<String>,
/// Remittance info or description
pub description: String,
/// Counterparty name
pub counterparty_name: Option<String>,
/// Counterparty IBAN
pub counterparty_iban: Option<String>,
}
#[derive(Debug, Clone, PartialEq)]
pub struct Account {
pub id: String,
pub iban: String,
pub currency: String,
}

View File

@@ -0,0 +1,42 @@
use async_trait::async_trait;
use chrono::NaiveDate;
use anyhow::Result;
#[cfg(test)]
use mockall::automock;
use crate::core::models::{BankTransaction, Account};
#[derive(Debug, Default)]
pub struct IngestResult {
pub created: usize,
pub duplicates: usize,
pub errors: usize,
pub healed: usize,
}
#[cfg_attr(test, automock)]
#[async_trait]
pub trait TransactionSource: Send + Sync {
/// Fetch accounts. Optionally filter by a list of wanted IBANs to save requests.
async fn get_accounts(&self, wanted_ibans: Option<Vec<String>>) -> Result<Vec<Account>>;
async fn get_transactions(&self, account_id: &str, start: NaiveDate, end: NaiveDate) -> Result<Vec<BankTransaction>>;
}
#[derive(Debug, Clone)]
pub struct TransactionMatch {
pub id: String,
pub has_external_id: bool,
}
#[cfg_attr(test, automock)]
#[async_trait]
pub trait TransactionDestination: Send + Sync {
async fn resolve_account_id(&self, iban: &str) -> Result<Option<String>>;
/// Get list of all active asset account IBANs to drive the sync
async fn get_active_account_ibans(&self) -> Result<Vec<String>>;
// New granular methods for Healer Logic
async fn get_last_transaction_date(&self, account_id: &str) -> Result<Option<NaiveDate>>;
async fn find_transaction(&self, account_id: &str, transaction: &BankTransaction) -> Result<Option<TransactionMatch>>;
async fn create_transaction(&self, account_id: &str, tx: &BankTransaction) -> Result<()>;
async fn update_transaction_external_id(&self, id: &str, external_id: &str) -> Result<()>;
}

299
banks2ff/src/core/sync.rs Normal file
View File

@@ -0,0 +1,299 @@
use anyhow::Result;
use tracing::{info, warn, instrument};
use crate::core::ports::{TransactionSource, TransactionDestination, IngestResult};
use chrono::{NaiveDate, Local};
#[instrument(skip(source, destination))]
pub async fn run_sync<S, D>(
source: &S,
destination: &D,
cli_start_date: Option<NaiveDate>,
cli_end_date: Option<NaiveDate>,
dry_run: bool,
) -> Result<()>
where
S: TransactionSource,
D: TransactionDestination,
{
info!("Starting synchronization...");
// Optimization: Get active Firefly IBANs first
let wanted_ibans = destination.get_active_account_ibans().await?;
info!("Syncing {} active accounts from Firefly III", wanted_ibans.len());
let accounts = source.get_accounts(Some(wanted_ibans)).await?;
// Default end date is Yesterday
let end_date = cli_end_date.unwrap_or_else(|| Local::now().date_naive() - chrono::Duration::days(1));
for account in accounts {
let span = tracing::info_span!("sync_account", iban = %account.iban);
let _enter = span.enter();
info!("Processing account...");
let dest_id_opt = destination.resolve_account_id(&account.iban).await?;
let Some(dest_id) = dest_id_opt else {
warn!("Account {} not found in destination. Skipping.", account.iban);
continue;
};
info!("Resolved destination ID: {}", dest_id);
// Determine Start Date
let start_date = if let Some(d) = cli_start_date {
d
} else {
// Default: Latest transaction date + 1 day
match destination.get_last_transaction_date(&dest_id).await? {
Some(last_date) => last_date + chrono::Duration::days(1),
None => {
// If no transaction exists in Firefly, we assume this is a fresh sync.
// Default to syncing last 30 days.
end_date - chrono::Duration::days(30)
},
}
};
if start_date > end_date {
info!("Start date {} is after end date {}. Nothing to sync.", start_date, end_date);
continue;
}
info!("Syncing interval: {} to {}", start_date, end_date);
// Optimization: Only use active accounts is already filtered in resolve_account_id
// However, GoCardless requisitions can expire.
// We should check if we can optimize the GoCardless fetching side.
// But currently get_transactions takes an account_id.
let transactions = source.get_transactions(&account.id, start_date, end_date).await?;
if transactions.is_empty() {
info!("No transactions found for period.");
continue;
}
info!("Fetched {} transactions from source.", transactions.len());
let mut stats = IngestResult::default();
// Healer Logic Loop
for tx in transactions {
// 1. Check if it exists
match destination.find_transaction(&dest_id, &tx).await? {
Some(existing) => {
if existing.has_external_id {
// Already synced properly
stats.duplicates += 1;
} else {
// Found "naked" transaction -> Heal it
if dry_run {
info!("[DRY RUN] Would heal transaction {} (Firefly ID: {})", tx.internal_id, existing.id);
stats.healed += 1;
} else {
info!("Healing transaction {} (Firefly ID: {})", tx.internal_id, existing.id);
if let Err(e) = destination.update_transaction_external_id(&existing.id, &tx.internal_id).await {
tracing::error!("Failed to heal transaction: {}", e);
stats.errors += 1;
} else {
stats.healed += 1;
}
}
}
},
None => {
// New transaction
if dry_run {
info!("[DRY RUN] Would create transaction {}", tx.internal_id);
stats.created += 1;
} else {
if let Err(e) = destination.create_transaction(&dest_id, &tx).await {
// Firefly might still reject it as duplicate if hash matches, even if we didn't find it via heuristic
// (unlikely if heuristic is good, but possible)
let err_str = e.to_string();
if err_str.contains("422") || err_str.contains("Duplicate") {
warn!("Duplicate rejected by Firefly: {}", tx.internal_id);
stats.duplicates += 1;
} else {
tracing::error!("Failed to create transaction: {}", e);
stats.errors += 1;
}
} else {
stats.created += 1;
}
}
}
}
}
info!("Sync complete. Created: {}, Healed: {}, Duplicates: {}, Errors: {}",
stats.created, stats.healed, stats.duplicates, stats.errors);
}
info!("Synchronization finished.");
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use crate::core::ports::{MockTransactionSource, MockTransactionDestination, TransactionMatch};
use crate::core::models::{Account, BankTransaction};
use rust_decimal::Decimal;
use mockall::predicate::*;
#[tokio::test]
async fn test_sync_flow_create_new() {
let mut source = MockTransactionSource::new();
let mut dest = MockTransactionDestination::new();
// Source setup
source.expect_get_accounts()
.with(always()) // Match any argument
.returning(|_| Ok(vec![Account {
id: "src_1".to_string(),
iban: "NL01".to_string(),
currency: "EUR".to_string(),
}]));
let tx = BankTransaction {
internal_id: "tx1".into(),
date: NaiveDate::from_ymd_opt(2023, 1, 1).unwrap(),
amount: Decimal::new(100, 0),
currency: "EUR".into(),
foreign_amount: None,
foreign_currency: None,
description: "Test".into(),
counterparty_name: None,
counterparty_iban: None,
};
let tx_clone = tx.clone();
source.expect_get_transactions()
.returning(move |_, _, _| Ok(vec![tx.clone()]));
// Destination setup
dest.expect_get_active_account_ibans()
.returning(|| Ok(vec!["NL01".to_string()]));
dest.expect_resolve_account_id()
.returning(|_| Ok(Some("dest_1".into())));
dest.expect_get_last_transaction_date()
.returning(|_| Ok(Some(NaiveDate::from_ymd_opt(2022, 12, 31).unwrap())));
// 1. Find -> None
dest.expect_find_transaction()
.times(1)
.returning(|_, _| Ok(None));
// 2. Create -> Ok
dest.expect_create_transaction()
.with(eq("dest_1"), eq(tx_clone))
.times(1)
.returning(|_, _| Ok(()));
// Execution
let res = run_sync(&source, &dest, None, None, false).await;
assert!(res.is_ok());
}
#[tokio::test]
async fn test_sync_flow_heal_existing() {
let mut source = MockTransactionSource::new();
let mut dest = MockTransactionDestination::new();
dest.expect_get_active_account_ibans()
.returning(|| Ok(vec!["NL01".to_string()]));
source.expect_get_accounts()
.with(always())
.returning(|_| Ok(vec![Account {
id: "src_1".to_string(),
iban: "NL01".to_string(),
currency: "EUR".to_string(),
}]));
source.expect_get_transactions()
.returning(|_, _, _| Ok(vec![
BankTransaction {
internal_id: "tx1".into(),
date: NaiveDate::from_ymd_opt(2023, 1, 1).unwrap(),
amount: Decimal::new(100, 0),
currency: "EUR".into(),
foreign_amount: None,
foreign_currency: None,
description: "Test".into(),
counterparty_name: None,
counterparty_iban: None,
}
]));
dest.expect_resolve_account_id().returning(|_| Ok(Some("dest_1".into())));
dest.expect_get_last_transaction_date().returning(|_| Ok(Some(NaiveDate::from_ymd_opt(2022, 12, 31).unwrap())));
// 1. Find -> Some(No External ID)
dest.expect_find_transaction()
.times(1)
.returning(|_, _| Ok(Some(TransactionMatch {
id: "ff_tx_1".to_string(),
has_external_id: false,
})));
// 2. Update -> Ok
dest.expect_update_transaction_external_id()
.with(eq("ff_tx_1"), eq("tx1"))
.times(1)
.returning(|_, _| Ok(()));
let res = run_sync(&source, &dest, None, None, false).await;
assert!(res.is_ok());
}
#[tokio::test]
async fn test_sync_flow_dry_run() {
let mut source = MockTransactionSource::new();
let mut dest = MockTransactionDestination::new();
dest.expect_get_active_account_ibans()
.returning(|| Ok(vec!["NL01".to_string()]));
source.expect_get_accounts()
.with(always())
.returning(|_| Ok(vec![Account {
id: "src_1".to_string(),
iban: "NL01".to_string(),
currency: "EUR".to_string(),
}]));
let tx = BankTransaction {
internal_id: "tx1".into(),
date: NaiveDate::from_ymd_opt(2023, 1, 1).unwrap(),
amount: Decimal::new(100, 0),
currency: "EUR".into(),
foreign_amount: None,
foreign_currency: None,
description: "Test".into(),
counterparty_name: None,
counterparty_iban: None,
};
source.expect_get_transactions()
.returning(move |_, _, _| Ok(vec![tx.clone()]));
dest.expect_resolve_account_id().returning(|_| Ok(Some("dest_1".into())));
dest.expect_get_last_transaction_date().returning(|_| Ok(Some(NaiveDate::from_ymd_opt(2022, 12, 31).unwrap())));
// 1. Find -> None (New transaction)
dest.expect_find_transaction()
.returning(|_, _| Ok(None));
// 2. Create -> NEVER Called (Dry Run)
dest.expect_create_transaction().never();
dest.expect_update_transaction_external_id().never();
let res = run_sync(&source, &dest, None, None, true).await;
assert!(res.is_ok());
}
}

74
banks2ff/src/main.rs Normal file
View File

@@ -0,0 +1,74 @@
mod adapters;
mod core;
use clap::Parser;
use tracing::{info, error};
use crate::adapters::gocardless::client::GoCardlessAdapter;
use crate::adapters::firefly::client::FireflyAdapter;
use crate::core::sync::run_sync;
use gocardless_client::client::GoCardlessClient;
use firefly_client::client::FireflyClient;
use std::env;
use chrono::NaiveDate;
#[derive(Parser, Debug)]
#[command(author, version, about, long_about = None)]
struct Args {
/// Path to configuration file (optional)
#[arg(short, long)]
config: Option<String>,
/// Start date for synchronization (YYYY-MM-DD). Defaults to last transaction date + 1.
#[arg(short, long)]
start: Option<NaiveDate>,
/// End date for synchronization (YYYY-MM-DD). Defaults to yesterday.
#[arg(short, long)]
end: Option<NaiveDate>,
/// Dry run mode: Do not create or update transactions in Firefly III.
#[arg(long, default_value_t = false)]
dry_run: bool,
}
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// Initialize logging
tracing_subscriber::fmt()
.with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
.init();
// Load environment variables
dotenvy::dotenv().ok();
let args = Args::parse();
info!("Starting banks2ff...");
if args.dry_run {
info!("DRY RUN MODE ENABLED: No changes will be made to Firefly III.");
}
// Config Load
let gc_url = env::var("GOCARDLESS_URL").unwrap_or_else(|_| "https://bankaccountdata.gocardless.com".to_string());
let gc_id = env::var("GOCARDLESS_ID").expect("GOCARDLESS_ID not set");
let gc_key = env::var("GOCARDLESS_KEY").expect("GOCARDLESS_KEY not set");
let ff_url = env::var("FIREFLY_III_URL").expect("FIREFLY_III_URL not set");
let ff_key = env::var("FIREFLY_III_API_KEY").expect("FIREFLY_III_API_KEY not set");
// Clients
let gc_client = GoCardlessClient::new(&gc_url, &gc_id, &gc_key)?;
let ff_client = FireflyClient::new(&ff_url, &ff_key)?;
// Adapters
let source = GoCardlessAdapter::new(gc_client);
let destination = FireflyAdapter::new(ff_client);
// Run
match run_sync(&source, &destination, args.start, args.end, args.dry_run).await {
Ok(_) => info!("Sync completed successfully."),
Err(e) => error!("Sync failed: {}", e),
}
Ok(())
}

114
docs/architecture.md Normal file
View File

@@ -0,0 +1,114 @@
# Architecture Documentation
## Overview
Banks2FF implements a **Hexagonal (Ports & Adapters) Architecture** to synchronize bank transactions from GoCardless to Firefly III. This architecture separates business logic from external concerns, making the system testable and maintainable.
## Workspace Structure
```
banks2ff/
├── banks2ff/ # Main CLI application
│ └── src/
│ ├── core/ # Domain logic and models
│ ├── adapters/ # External service integrations
│ └── main.rs # CLI entry point
├── firefly-client/ # Firefly III API client library
├── gocardless-client/ # GoCardless API client library
└── docs/ # Architecture documentation
```
## Core Components
### 1. Domain Core (`banks2ff/src/core/`)
**models.rs**: Defines domain entities
- `BankTransaction`: Core transaction model with multi-currency support
- `Account`: Bank account representation
- Supports `foreign_amount` and `foreign_currency` for international transactions
**ports.rs**: Defines abstraction traits
- `TransactionSource`: Interface for fetching transactions (implemented by GoCardless adapter)
- `TransactionDestination`: Interface for storing transactions (implemented by Firefly adapter)
- Traits are mockable for isolated testing
**sync.rs**: Synchronization engine
- `run_sync()`: Orchestrates the entire sync process
- Implements "Healer" strategy for idempotency
- Smart date range calculation (Last Transaction Date + 1 to Yesterday)
### 2. Adapters (`banks2ff/src/adapters/`)
**gocardless/**: GoCardless integration
- `client.rs`: Wrapper for GoCardless client with token management
- `mapper.rs`: Converts GoCardless API responses to domain models
- `cache.rs`: Caches account mappings to reduce API calls
- Correctly handles multi-currency via `currencyExchange` array parsing
**firefly/**: Firefly III integration
- `client.rs`: Wrapper for Firefly client for transaction storage
- Maps domain models to Firefly API format
### 3. API Clients
Both clients are hand-crafted using `reqwest`:
- Strongly-typed DTOs for compile-time safety
- Custom error handling with `thiserror`
- Rate limit awareness and graceful degradation
## Synchronization Process
The "Healer" strategy ensures idempotency:
1. **Account Discovery**: Fetch active accounts from GoCardless
2. **Account Matching**: Match GoCardless accounts to Firefly asset accounts by IBAN
3. **Date Window**: Calculate sync range (Last Firefly transaction + 1 to Yesterday)
4. **Transaction Processing**:
- **Search**: Look for existing transaction using windowed heuristic (date ± 3 days, exact amount)
- **Heal**: If found without `external_id`, update with GoCardless transaction ID
- **Skip**: If found with matching `external_id`, ignore
- **Create**: If not found, create new transaction in Firefly
## Key Features
### Multi-Currency Support
- Parses `currencyExchange` array from GoCardless responses
- Calculates `foreign_amount = amount * exchange_rate`
- Maps to Firefly's `foreign_amount` and `foreign_currency_code` fields
### Rate Limit Management
- **Caching**: Stores `AccountId -> IBAN` mappings to reduce requisition calls
- **Token Reuse**: Maintains tokens until expiry to minimize auth requests
- **Graceful Handling**: Continues sync for other accounts when encountering 429 errors
### Idempotency
- GoCardless `transactionId` → Firefly `external_id` mapping
- Windowed duplicate detection prevents double-creation
- Historical transaction healing for pre-existing data
## Data Flow
```
GoCardless API → GoCardlessAdapter → TransactionSource → SyncEngine → TransactionDestination → FireflyAdapter → Firefly API
```
## Testing Strategy
- **Unit Tests**: Core logic with `mockall` for trait mocking
- **Integration Tests**: API clients with `wiremock` for HTTP mocking
- **Fixture Testing**: Real JSON responses for adapter mapping validation
- **Isolation**: Business logic tested without external dependencies
## Error Handling
- **Custom Errors**: `thiserror` for domain-specific error types
- **Propagation**: `anyhow` for error context across async boundaries
- **Graceful Degradation**: Rate limits and network issues don't crash entire sync
- **Structured Logging**: `tracing` for observability and debugging
## Configuration Management
- Environment variables loaded via `dotenvy`
- Workspace-level dependency management
- Feature flags for optional functionality
- Secure credential handling (no hardcoded secrets)

20
firefly-client/Cargo.toml Normal file
View File

@@ -0,0 +1,20 @@
[package]
name = "firefly-client"
version.workspace = true
edition.workspace = true
authors.workspace = true
[dependencies]
reqwest = { version = "0.11", default-features = false, features = ["json", "rustls-tls"] }
serde = { workspace = true }
serde_json = { workspace = true }
thiserror = { workspace = true }
tracing = { workspace = true }
url = { workspace = true }
chrono = { workspace = true }
rust_decimal = { workspace = true }
[dev-dependencies]
wiremock = { workspace = true }
tokio = { workspace = true }
tokio-test = { workspace = true }

View File

@@ -0,0 +1,126 @@
use reqwest::{Client, Url};
use serde::de::DeserializeOwned;
use thiserror::Error;
use tracing::instrument;
use crate::models::{AccountArray, TransactionStore, TransactionArray, TransactionUpdate};
#[derive(Error, Debug)]
pub enum FireflyError {
#[error("Request failed: {0}")]
RequestFailed(#[from] reqwest::Error),
#[error("API Error: {0}")]
ApiError(String),
#[error("URL Parse Error: {0}")]
UrlParseError(#[from] url::ParseError),
}
pub struct FireflyClient {
base_url: Url,
client: Client,
access_token: String,
}
impl FireflyClient {
pub fn new(base_url: &str, access_token: &str) -> Result<Self, FireflyError> {
Ok(Self {
base_url: Url::parse(base_url)?,
client: Client::new(),
access_token: access_token.to_string(),
})
}
#[instrument(skip(self))]
pub async fn get_accounts(&self, _iban: &str) -> Result<AccountArray, FireflyError> {
let mut url = self.base_url.join("/api/v1/accounts")?;
url.query_pairs_mut()
.append_pair("type", "asset");
self.get_authenticated(url).await
}
#[instrument(skip(self))]
pub async fn search_accounts(&self, query: &str) -> Result<AccountArray, FireflyError> {
let mut url = self.base_url.join("/api/v1/search/accounts")?;
url.query_pairs_mut()
.append_pair("query", query)
.append_pair("type", "asset")
.append_pair("field", "all");
self.get_authenticated(url).await
}
#[instrument(skip(self, transaction))]
pub async fn store_transaction(&self, transaction: TransactionStore) -> Result<(), FireflyError> {
let url = self.base_url.join("/api/v1/transactions")?;
let response = self.client.post(url)
.bearer_auth(&self.access_token)
.header("accept", "application/json")
.json(&transaction)
.send()
.await?;
if !response.status().is_success() {
let status = response.status();
let text = response.text().await?;
return Err(FireflyError::ApiError(format!("Store Transaction Failed {}: {}", status, text)));
}
Ok(())
}
#[instrument(skip(self))]
pub async fn list_account_transactions(&self, account_id: &str, start: Option<&str>, end: Option<&str>) -> Result<TransactionArray, FireflyError> {
let mut url = self.base_url.join(&format!("/api/v1/accounts/{}/transactions", account_id))?;
{
let mut pairs = url.query_pairs_mut();
if let Some(s) = start {
pairs.append_pair("start", s);
}
if let Some(e) = end {
pairs.append_pair("end", e);
}
// Limit to 50, could be higher but safer to page if needed. For heuristic checks 50 is usually plenty per day range.
pairs.append_pair("limit", "50");
}
self.get_authenticated(url).await
}
#[instrument(skip(self, update))]
pub async fn update_transaction(&self, id: &str, update: TransactionUpdate) -> Result<(), FireflyError> {
let url = self.base_url.join(&format!("/api/v1/transactions/{}", id))?;
let response = self.client.put(url)
.bearer_auth(&self.access_token)
.header("accept", "application/json")
.json(&update)
.send()
.await?;
if !response.status().is_success() {
let status = response.status();
let text = response.text().await?;
return Err(FireflyError::ApiError(format!("Update Transaction Failed {}: {}", status, text)));
}
Ok(())
}
async fn get_authenticated<T: DeserializeOwned>(&self, url: Url) -> Result<T, FireflyError> {
let response = self.client.get(url)
.bearer_auth(&self.access_token)
.header("accept", "application/json")
.send()
.await?;
if !response.status().is_success() {
let status = response.status();
let text = response.text().await?;
return Err(FireflyError::ApiError(format!("API request failed {}: {}", status, text)));
}
let data = response.json().await?;
Ok(data)
}
}

View File

@@ -0,0 +1,2 @@
pub mod client;
pub mod models;

View File

@@ -0,0 +1,81 @@
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AccountRead {
pub id: String,
pub attributes: Account,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Account {
pub name: String,
pub iban: Option<String>,
#[serde(rename = "type")]
pub account_type: String,
pub active: Option<bool>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AccountArray {
pub data: Vec<AccountRead>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TransactionRead {
pub id: String,
pub attributes: Transaction,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Transaction {
pub transactions: Vec<TransactionSplit>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TransactionSplit {
pub date: String,
pub amount: String,
pub description: String,
pub external_id: Option<String>,
pub currency_code: Option<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TransactionArray {
pub data: Vec<TransactionRead>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TransactionSplitStore {
#[serde(rename = "type")]
pub transaction_type: String,
pub date: String,
pub amount: String,
pub description: String,
pub source_id: Option<String>,
pub source_name: Option<String>,
pub destination_id: Option<String>,
pub destination_name: Option<String>,
pub currency_code: Option<String>,
pub foreign_amount: Option<String>,
pub foreign_currency_code: Option<String>,
pub external_id: Option<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TransactionStore {
pub transactions: Vec<TransactionSplitStore>,
pub apply_rules: Option<bool>,
pub fire_webhooks: Option<bool>,
pub error_if_duplicate_hash: Option<bool>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TransactionUpdate {
pub transactions: Vec<TransactionSplitUpdate>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TransactionSplitUpdate {
pub external_id: Option<String>,
}

View File

@@ -0,0 +1,62 @@
use firefly_client::client::FireflyClient;
use firefly_client::models::{TransactionStore, TransactionSplitStore};
use wiremock::matchers::{method, path, header};
use wiremock::{Mock, MockServer, ResponseTemplate};
use std::fs;
#[tokio::test]
async fn test_search_accounts() {
let mock_server = MockServer::start().await;
let fixture = fs::read_to_string("tests/fixtures/ff_accounts.json").unwrap();
Mock::given(method("GET"))
.and(path("/api/v1/search/accounts"))
.and(header("Authorization", "Bearer my-token"))
.respond_with(ResponseTemplate::new(200).set_body_string(fixture))
.mount(&mock_server)
.await;
let client = FireflyClient::new(&mock_server.uri(), "my-token").unwrap();
let accounts = client.search_accounts("NL01").await.unwrap();
assert_eq!(accounts.data.len(), 1);
assert_eq!(accounts.data[0].attributes.name, "Checking Account");
assert_eq!(accounts.data[0].attributes.iban.as_deref(), Some("NL01BANK0123456789"));
}
#[tokio::test]
async fn test_store_transaction() {
let mock_server = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/api/v1/transactions"))
.and(header("Authorization", "Bearer my-token"))
.respond_with(ResponseTemplate::new(200))
.mount(&mock_server)
.await;
let client = FireflyClient::new(&mock_server.uri(), "my-token").unwrap();
let tx = TransactionStore {
transactions: vec![TransactionSplitStore {
transaction_type: "withdrawal".to_string(),
date: "2023-01-01".to_string(),
amount: "10.00".to_string(),
description: "Test".to_string(),
source_id: Some("1".to_string()),
destination_name: Some("Shop".to_string()),
currency_code: None,
foreign_amount: None,
foreign_currency_code: None,
external_id: None,
source_name: None,
destination_id: None,
}],
apply_rules: None,
fire_webhooks: None,
error_if_duplicate_hash: None,
};
let result = client.store_transaction(tx).await;
assert!(result.is_ok());
}

View File

@@ -0,0 +1,22 @@
{
"data": [
{
"type": "accounts",
"id": "2",
"attributes": {
"name": "Checking Account",
"type": "asset",
"iban": "NL01BANK0123456789"
}
}
],
"meta": {
"pagination": {
"total": 1,
"count": 1,
"per_page": 20,
"current_page": 1,
"total_pages": 1
}
}
}

View File

@@ -0,0 +1,19 @@
[package]
name = "gocardless-client"
version.workspace = true
edition.workspace = true
authors.workspace = true
[dependencies]
reqwest = { version = "0.11", default-features = false, features = ["json", "rustls-tls"] }
serde = { workspace = true }
serde_json = { workspace = true }
thiserror = { workspace = true }
tracing = { workspace = true }
url = { workspace = true }
chrono = { workspace = true }
[dev-dependencies]
wiremock = { workspace = true }
tokio = { workspace = true }
tokio-test = { workspace = true }

View File

@@ -0,0 +1,129 @@
use reqwest::{Client, Url};
use serde::{Deserialize, Serialize};
use thiserror::Error;
use tracing::{debug, instrument};
use crate::models::{TokenResponse, PaginatedResponse, Requisition, Account, TransactionsResponse};
#[derive(Error, Debug)]
pub enum GoCardlessError {
#[error("Request failed: {0}")]
RequestFailed(#[from] reqwest::Error),
#[error("API Error: {0}")]
ApiError(String),
#[error("Serialization error: {0}")]
SerializationError(#[from] serde_json::Error),
#[error("URL Parse Error: {0}")]
UrlParseError(#[from] url::ParseError),
}
pub struct GoCardlessClient {
base_url: Url,
client: Client,
secret_id: String,
secret_key: String,
access_token: Option<String>,
access_expires_at: Option<chrono::DateTime<chrono::Utc>>,
}
#[derive(Serialize)]
struct TokenRequest<'a> {
secret_id: &'a str,
secret_key: &'a str,
}
impl GoCardlessClient {
pub fn new(base_url: &str, secret_id: &str, secret_key: &str) -> Result<Self, GoCardlessError> {
Ok(Self {
base_url: Url::parse(base_url)?,
client: Client::new(),
secret_id: secret_id.to_string(),
secret_key: secret_key.to_string(),
access_token: None,
access_expires_at: None,
})
}
#[instrument(skip(self))]
pub async fn obtain_access_token(&mut self) -> Result<(), GoCardlessError> {
// Check if current token is still valid (with 60s buffer)
if let Some(expires) = self.access_expires_at {
if chrono::Utc::now() < expires - chrono::Duration::seconds(60) {
debug!("Access token is still valid");
return Ok(());
}
}
let url = self.base_url.join("/api/v2/token/new/")?;
let body = TokenRequest {
secret_id: &self.secret_id,
secret_key: &self.secret_key,
};
debug!("Requesting new access token");
let response = self.client.post(url)
.json(&body)
.send()
.await?;
if !response.status().is_success() {
let status = response.status();
let text = response.text().await?;
return Err(GoCardlessError::ApiError(format!("Token request failed {}: {}", status, text)));
}
let token_resp: TokenResponse = response.json().await?;
self.access_token = Some(token_resp.access);
self.access_expires_at = Some(chrono::Utc::now() + chrono::Duration::seconds(token_resp.access_expires as i64));
debug!("Access token obtained");
Ok(())
}
#[instrument(skip(self))]
pub async fn get_requisitions(&self) -> Result<PaginatedResponse<Requisition>, GoCardlessError> {
let url = self.base_url.join("/api/v2/requisitions/")?;
self.get_authenticated(url).await
}
#[instrument(skip(self))]
pub async fn get_account(&self, id: &str) -> Result<Account, GoCardlessError> {
let url = self.base_url.join(&format!("/api/v2/accounts/{}/", id))?;
self.get_authenticated(url).await
}
#[instrument(skip(self))]
pub async fn get_transactions(&self, account_id: &str, date_from: Option<&str>, date_to: Option<&str>) -> Result<TransactionsResponse, GoCardlessError> {
let mut url = self.base_url.join(&format!("/api/v2/accounts/{}/transactions/", account_id))?;
{
let mut pairs = url.query_pairs_mut();
if let Some(from) = date_from {
pairs.append_pair("date_from", from);
}
if let Some(to) = date_to {
pairs.append_pair("date_to", to);
}
}
self.get_authenticated(url).await
}
async fn get_authenticated<T: for<'de> Deserialize<'de>>(&self, url: Url) -> Result<T, GoCardlessError> {
let token = self.access_token.as_ref().ok_or(GoCardlessError::ApiError("No access token available. Call obtain_access_token() first.".into()))?;
let response = self.client.get(url)
.bearer_auth(token)
.header("accept", "application/json")
.send()
.await?;
if !response.status().is_success() {
let status = response.status();
let text = response.text().await?;
return Err(GoCardlessError::ApiError(format!("API request failed {}: {}", status, text)));
}
let data = response.json().await?;
Ok(data)
}
}

View File

@@ -0,0 +1,2 @@
pub mod client;
pub mod models;

View File

@@ -0,0 +1,95 @@
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TokenResponse {
pub access: String,
pub access_expires: i32,
pub refresh: Option<String>,
pub refresh_expires: Option<i32>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Requisition {
pub id: String,
pub status: String,
pub accounts: Option<Vec<String>>,
pub reference: Option<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct PaginatedResponse<T> {
pub count: Option<i32>,
pub next: Option<String>,
pub previous: Option<String>,
pub results: Vec<T>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Account {
pub id: String,
pub created: Option<String>,
pub last_accessed: Option<String>,
pub iban: Option<String>,
pub institution_id: Option<String>,
pub status: Option<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TransactionsResponse {
pub transactions: TransactionBookedPending,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TransactionBookedPending {
pub booked: Vec<Transaction>,
pub pending: Option<Vec<Transaction>>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Transaction {
#[serde(rename = "transactionId")]
pub transaction_id: Option<String>,
#[serde(rename = "bookingDate")]
pub booking_date: Option<String>,
#[serde(rename = "valueDate")]
pub value_date: Option<String>,
#[serde(rename = "transactionAmount")]
pub transaction_amount: TransactionAmount,
#[serde(rename = "currencyExchange")]
pub currency_exchange: Option<Vec<CurrencyExchange>>,
#[serde(rename = "creditorName")]
pub creditor_name: Option<String>,
#[serde(rename = "creditorAccount")]
pub creditor_account: Option<AccountDetails>,
#[serde(rename = "debtorName")]
pub debtor_name: Option<String>,
#[serde(rename = "debtorAccount")]
pub debtor_account: Option<AccountDetails>,
#[serde(rename = "remittanceInformationUnstructured")]
pub remittance_information_unstructured: Option<String>,
#[serde(rename = "proprietaryBankTransactionCode")]
pub proprietary_bank_transaction_code: Option<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TransactionAmount {
pub amount: String,
pub currency: String,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CurrencyExchange {
#[serde(rename = "sourceCurrency")]
pub source_currency: Option<String>,
#[serde(rename = "exchangeRate")]
pub exchange_rate: Option<String>,
#[serde(rename = "unitCurrency")]
pub unit_currency: Option<String>,
#[serde(rename = "targetCurrency")]
pub target_currency: Option<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AccountDetails {
pub iban: Option<String>,
}

View File

@@ -0,0 +1,55 @@
use gocardless_client::client::GoCardlessClient;
use gocardless_client::models::TokenResponse;
use wiremock::matchers::{method, path};
use wiremock::{Mock, MockServer, ResponseTemplate};
use std::fs;
#[tokio::test]
async fn test_get_transactions_parsing() {
// 1. Setup WireMock
let mock_server = MockServer::start().await;
// Mock Token Endpoint
Mock::given(method("POST"))
.and(path("/api/v2/token/new/"))
.respond_with(ResponseTemplate::new(200).set_body_json(TokenResponse {
access: "fake_access_token".to_string(),
access_expires: 3600,
refresh: Some("fake_refresh".to_string()),
refresh_expires: Some(86400),
}))
.mount(&mock_server)
.await;
// Mock Transactions Endpoint
let fixture = fs::read_to_string("tests/fixtures/gc_transactions.json").unwrap();
Mock::given(method("GET"))
.and(path("/api/v2/accounts/ACC123/transactions/"))
.respond_with(ResponseTemplate::new(200).set_body_string(fixture))
.mount(&mock_server)
.await;
// 2. Run Client
let mut client = GoCardlessClient::new(&mock_server.uri(), "id", "key").unwrap();
client.obtain_access_token().await.unwrap();
let resp = client.get_transactions("ACC123", None, None).await.unwrap();
// 3. Assertions
assert_eq!(resp.transactions.booked.len(), 2);
let tx1 = &resp.transactions.booked[0];
assert_eq!(tx1.transaction_id.as_deref(), Some("TX123"));
assert_eq!(tx1.transaction_amount.amount, "100.00");
assert_eq!(tx1.transaction_amount.currency, "EUR");
let tx2 = &resp.transactions.booked[1];
assert_eq!(tx2.transaction_id.as_deref(), Some("TX124"));
assert_eq!(tx2.transaction_amount.amount, "-10.00");
// Verify Multi-Currency parsing
let exchange = tx2.currency_exchange.as_ref().unwrap();
assert_eq!(exchange[0].source_currency.as_deref(), Some("USD"));
assert_eq!(exchange[0].exchange_rate.as_deref(), Some("1.10"));
}

View File

@@ -0,0 +1,34 @@
{
"transactions": {
"booked": [
{
"transactionId": "TX123",
"bookingDate": "2023-10-01",
"transactionAmount": {
"amount": "100.00",
"currency": "EUR"
},
"debtorName": "John Doe",
"remittanceInformationUnstructured": "Payment for services"
},
{
"transactionId": "TX124",
"bookingDate": "2023-10-02",
"transactionAmount": {
"amount": "-10.00",
"currency": "EUR"
},
"currencyExchange": [
{
"sourceCurrency": "USD",
"exchangeRate": "1.10",
"targetCurrency": "EUR"
}
],
"creditorName": "US Store",
"remittanceInformationUnstructured": "US Purchase"
}
],
"pending": []
}
}

View File

@@ -1,129 +1,137 @@
# Implementation Plan: Bank2FF Refactoring # Implementation Plan: Bank2FF Refactoring
## 1. Objective ## 1. Objective
Refactor the `bank2ff` application from a prototype script into a robust, testable, and observable production-grade CLI tool. The application must synchronize bank transactions from GoCardless to Firefly III, ensuring: Refactor the `bank2ff` application from a prototype script into a robust, testable, and observable production-grade CLI tool. The application must synchronize bank transactions from GoCardless to Firefly III.
- **Clean Architecture**: Decoupling of business logic from API clients.
- **Testability**: Ability to unit test core logic without external dependencies.
- **Multi-Currency Support**: Accurate capturing of foreign amounts and currencies.
- **Idempotency**: Preventing duplicate transactions in Firefly III.
## 2. Architecture: Hexagonal (Ports & Adapters) **Key Constraints:**
- **Multi-Crate Workspace**: Separate crates for the CLI application and the API clients.
- **Hand-Crafted Clients**: No autogenerated code. Custom, strongly-typed clients for better UX.
- **Clean Architecture**: Hexagonal architecture within the main application.
- **Multi-Currency**: Accurate handling of foreign amounts.
- **Test-Driven**: Every component must be testable from the start.
- **Observability**: Structured logging (tracing) throughout the stack.
- **Healer Strategy**: Detect and heal historical duplicates that lack external IDs.
- **Dry Run**: Safe mode to preview changes.
- **Rate Limit Handling**: Smart caching and graceful skipping to respect 4 requests/day limits.
The application will be structured into three distinct layers: ## 2. Architecture
1. **Core (Domain)**: Pure Rust, no external API dependencies. Defines the `BankTransaction` model and the `Ports` (traits) for interacting with the world.
2. **Adapters**: Implementations of the Ports.
- `GoCardlessAdapter`: Implements `TransactionSource`.
- `FireflyAdapter`: Implements `TransactionDestination`.
3. **Application**: Configuration, CLI parsing, and wiring (`main.rs`).
### Directory Structure ### Workspace Structure
The project uses a Cargo Workspace with three members:
1. `gocardless-client`: A reusable library crate wrapping the GoCardless Bank Account Data API v2.
2. `firefly-client`: A reusable library crate wrapping the Firefly III API v6.4.4.
3. `banks2ff`: The main CLI application containing the Domain Core and Adapters that use the client libraries.
### Directory Layout
```text ```text
bank2ff/src/ root/
├── core/ ├── Cargo.toml # Workspace definition
│ ├── mod.rs ├── gocardless-client/ # Crate 1
│ ├── models.rs # Domain entities (BankTransaction, Account) │ ├── Cargo.toml
── ports.rs # Traits (TransactionSource, TransactionDestination) ── src/
└── sync.rs # Core business logic (The "Use Case") ├── lib.rs
├── adapters/ │ ├── client.rs # Reqwest client logic
│ ├── mod.rs ├── models.rs # Request/Response DTOs
├── gocardless/ └── tests/ # Unit/Integration tests with mocks
│ │ ├── mod.rs ├── firefly-client/ # Crate 2
│ ├── client.rs # Wrapper around generated client (Auth/RateLimits) │ ├── Cargo.toml
│ └── mapper.rs # Logic to map API response -> Domain Model │ └── src/
└── firefly/ ├── lib.rs
├── mod.rs ├── client.rs # Reqwest client logic
── client.rs # Implementation of TransactionDestination ── models.rs # Request/Response DTOs
└── main.rs # Entry point, wiring, and config loading │ └── tests/ # Unit/Integration tests with mocks
└── banks2ff/ # Crate 3 (Main App)
├── Cargo.toml
└── src/
├── main.rs
├── core/ # Domain
│ ├── models.rs
│ ├── ports.rs # Traits (Mockable)
│ ├── sync.rs # Logic (Tested with mocks)
│ └── tests/ # Unit tests for logic
└── adapters/ # Integration Layers
├── gocardless/
│ ├── client.rs # Uses gocardless-client & Cache
│ ├── cache.rs # JSON Cache for Account details
│ └── mapper.rs
└── firefly/
└── client.rs # Uses firefly-client
``` ```
## 3. Core definitions ## 3. Observability Strategy
- **Tracing**: All crates will use the `tracing` crate.
- **Spans**:
- `client` crates: Create spans for every HTTP request (method, URL).
- `banks2ff` adapter: Create spans for "Fetching transactions".
- `banks2ff` sync: Create a span per account synchronization.
- **Context**: IDs (account, transaction) must be attached to log events.
### `src/core/models.rs` ## 4. Testing Strategy
The domain model must support multi-currency data. - **Client Crates**:
- Use `wiremock` to mock the HTTP server.
- Test parsing of *real* JSON responses (saved in `tests/fixtures/`).
- Verify correct request construction (Headers, Auth, Body).
- **Core (`banks2ff`)**:
- Use `mockall` to mock `TransactionSource` and `TransactionDestination` traits.
- Unit test `sync::run_sync` logic (filtering, flow control) without any I/O.
- **Adapters (`banks2ff`)**:
- Test mapping logic (Client DTO -> Domain Model) using unit tests with fixtures.
```rust ## 5. Implementation Steps
pub struct BankTransaction {
pub internal_id: String, // Source ID (GoCardless transactionId)
pub date: NaiveDate, // Booking date
pub amount: Decimal, // Amount in account currency
pub currency: String, // Account currency code (e.g., EUR)
pub foreign_amount: Option<Decimal>, // Original amount (if currency exchange occurred)
pub foreign_currency: Option<String>,// Original currency code
pub description: String,
pub counterparty_name: Option<String>,
pub counterparty_iban: Option<String>,
}
```
### `src/core/ports.rs` ### Phase 1: Infrastructure & Workspace
Traits to decouple the architecture. - [x] **Setup**: Initialize `gocardless-client` and `firefly-client` crates. Update root `Cargo.toml`.
- [x] **Dependencies**: Add `reqwest`, `serde`, `thiserror`, `url`, `tracing`.
- [x] **Test Deps**: Add `wiremock`, `tokio-test`, `serde_json` (dev-dependencies).
```rust ### Phase 2: Core (`banks2ff`)
#[async_trait] - [x] **Definitions**: Implement `models.rs` and `ports.rs` in `banks2ff`.
pub trait TransactionSource: Send + Sync { - [x] **Mocks**: Add `mockall` attribute to ports for easier testing.
async fn get_accounts(&self) -> Result<Vec<Account>>;
async fn get_transactions(&self, account_id: &str, start: NaiveDate, end: NaiveDate) -> Result<Vec<BankTransaction>>;
}
#[async_trait] ### Phase 3: GoCardless Client Crate
pub trait TransactionDestination: Send + Sync { - [x] **Models**: Define DTOs in `gocardless-client/src/models.rs`.
async fn resolve_account_id(&self, iban: &str) -> Result<Option<String>>; - [x] **Fixtures**: Create `tests/fixtures/gc_transactions.json` (real example data).
async fn ingest_transactions(&self, account_id: &str, transactions: Vec<BankTransaction>) -> Result<IngestResult>; - [x] **Client**: Implement `GoCardlessClient`.
} - [x] **Tests**: Write `tests/client_test.rs` using `wiremock` to serve the fixture and verify the client parses it correctly.
```
## 4. Implementation Steps ### Phase 4: GoCardless Adapter (`banks2ff`)
- [x] **Implementation**: Implement `TransactionSource`.
- [x] **Logic**: Handle **Multi-Currency** (inspect `currencyExchange`).
- [x] **Tests**: Unit test the *mapping logic* specifically. Input: GC Client DTO. Output: Domain Model. Assert foreign amounts are correct.
- [x] **Optimization**: Implement "Firefly Leading" strategy (only fetch wanted accounts).
- [x] **Optimization**: Implement Account Cache & Rate Limit handling.
### Phase 1: Infrastructure & Core ### Phase 5: Firefly Client Crate
1. **Setup**: Create the directory structure. - [x] **Models**: Define DTOs in `firefly-client/src/models.rs`.
2. **Dependencies**: Add `anyhow`, `thiserror`, `tracing`, `tracing-subscriber`, `async-trait`, `rust_decimal`, `chrono`. - [x] **Fixtures**: Create `tests/fixtures/ff_store_transaction.json`.
3. **Core**: Implement `models.rs` and `ports.rs`. - [x] **Client**: Implement `FireflyClient`.
- [x] **Tests**: Write `tests/client_test.rs` using `wiremock` to verify auth headers and body serialization.
### Phase 2: GoCardless Adapter (Source) ### Phase 6: Firefly Adapter (`banks2ff`)
1. **Token Management**: Create a wrapper struct that holds the `gocardless_bankaccount_data_api` configuration and manages the Access/Refresh token lifecycle. It should check validity before every request. - [x] **Implementation**: Implement `TransactionDestination`.
2. **Mapping Logic**: - [x] **Logic**: Set `external_id`, handle Credit/Debit swap.
- Map `transactionAmount.amount` -> `BankTransaction.amount`. - [x] **Tests**: Unit test mapping logic. Verify `external_id` is populated.
- **Multi-Currency**: Inspect `currencyExchange`. - [x] **Update**: Refactor for "Healer" strategy (split `ingest` into `find`, `create`, `update`).
- If present, map `sourceCurrency` -> `foreign_currency`.
- Calculate `foreign_amount` using `exchangeRate` if `instructedAmount` is missing.
- Map `transactionId` -> `internal_id`.
3. **Trait Implementation**: Implement `TransactionSource`.
### Phase 3: Firefly Adapter (Destination) ### Phase 7: Synchronization Engine
1. **Client Wrapper**: Initialize `firefly_iii_api` client with API Key. - [x] **Logic**: Implement `banks2ff::core::sync::run_sync` with "Healer" logic.
2. **Resolution**: Implement `resolve_account_id` by querying Firefly accounts by IBAN (using the search/list endpoint). - Check Destination for existing transaction (Windowed Search).
3. **Ingestion**: - If found without ID: Heal (Update).
- Map `BankTransaction` to `TransactionSplitStore`. - If found with ID: Skip.
- **Crucial**: Set `external_id` = `BankTransaction.internal_id`. - If not found: Create.
- Handle Credits vs Debits (Swap Source/Destination logic). - [x] **Smart Defaults**: Implement default start date (Last Firefly Date + 1) and end date (Yesterday).
- Call `store_transaction` endpoint. - [x] **Tests**: Update unit tests for the new flow.
- Handle 422/409 errors gracefully (log duplicates).
### Phase 4: Synchronization Engine ### Phase 8: Wiring & CLI
1. Implement `src/core/sync.rs`. - [x] **CLI**: Add `-s/--start` and `-e/--end` arguments.
2. Logic: - [x] **CLI**: Add `--dry-run` argument.
- Fetch accounts from Source. - [x] **Wiring**: Pass these arguments to the sync engine.
- For each account, find Destination ID. - [x] **Observability**: Initialize `tracing_subscriber` with env filter.
- Fetch transactions (default: last 30 days). - [x] **Config**: Load Env vars.
- Filter/Process (optional).
- Ingest to Destination.
- Log statistics using `tracing`.
### Phase 5: Wiring ## 6. Multi-Currency Logic
1. Refactor `main.rs`. - **GoCardless Adapter**:
2. Initialize `tracing` for structured logging. - `foreign_currency` = `currencyExchange[0].sourceCurrency`
3. Load Config (Env variables). - `foreign_amount` = `amount` * `currencyExchange[0].exchangeRate`
4. Instantiate Adapters. - Test this math explicitly in Phase 4 tests.
5. Run the Sync Engine.
## 5. Multi-Currency Handling Specifics
- **GoCardless Spec**: The `currencyExchange` array contains the details.
- **Logic**:
- If `currencyExchange` is not null/empty:
- `foreign_currency` = `currencyExchange[0].sourceCurrency`
- `foreign_amount` = `amount` * `currencyExchange[0].exchangeRate` (Validation: check if `unitCurrency` affects this).
- Ensure `Decimal` precision is handled correctly.
## 6. Testability & Observability
- **Tests**: Write a unit test for `core/sync.rs` using Mock structs for Source/Destination to verify the flow logic.
- **Logs**: Use `tracing::info!` for high-level progress ("Synced Account X") and `tracing::debug!` for details ("Transaction Y mapped to Z").