Completely replace implementation #1
178
AGENTS.md
Normal file
178
AGENTS.md
Normal file
@@ -0,0 +1,178 @@
|
|||||||
|
# Banks2FF Development Guide
|
||||||
|
|
||||||
|
## Project Purpose
|
||||||
|
|
||||||
|
Banks2FF is a Rust CLI tool that synchronizes bank transactions from GoCardless Bank Account Data API to Firefly III personal finance manager. It implements a hexagonal architecture for clean separation of concerns and comprehensive testing.
|
||||||
|
|
||||||
|
## 🚨 CRITICAL: Financial Data Security
|
||||||
|
|
||||||
|
### **ABSOLUTE REQUIREMENT: Financial Data Masking**
|
||||||
|
|
||||||
|
**NEVER** expose, log, or display raw financial information including:
|
||||||
|
- Transaction amounts
|
||||||
|
- Account balances
|
||||||
|
- IBANs or account numbers
|
||||||
|
- Transaction descriptions
|
||||||
|
- Personal identifiers
|
||||||
|
- API keys or tokens
|
||||||
|
|
||||||
|
### **Compliance Protocol for Debugging**
|
||||||
|
|
||||||
|
When debugging financial data issues:
|
||||||
|
|
||||||
|
1. **Create Anonymized Test Scripts**: Write small, focused scripts that extract only the necessary data structure information
|
||||||
|
2. **Use Mock Data**: Replace real financial values with placeholder data
|
||||||
|
3. **Validate Structure, Not Values**: Focus on data structure integrity, not actual financial content
|
||||||
|
4. **Sanitize All Outputs**: Ensure any debugging output masks sensitive information
|
||||||
|
|
||||||
|
```rust
|
||||||
|
// ✅ GOOD: Structure validation with mock data
|
||||||
|
fn validate_transaction_structure() {
|
||||||
|
let mock_tx = BankTransaction {
|
||||||
|
amount: Decimal::new(12345, 2), // Mock amount
|
||||||
|
currency: "EUR".to_string(),
|
||||||
|
// ... other fields with mock data
|
||||||
|
};
|
||||||
|
// Validate structure only
|
||||||
|
}
|
||||||
|
|
||||||
|
// ❌ BAD: Exposing real financial data
|
||||||
|
fn debug_real_transactions(transactions: Vec<BankTransaction>) {
|
||||||
|
for tx in transactions {
|
||||||
|
println!("Real amount: {}", tx.amount); // SECURITY VIOLATION
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Rust Development Best Practices
|
||||||
|
|
||||||
|
### Error Handling
|
||||||
|
|
||||||
|
- **Use `thiserror`** for domain-specific error types in core modules
|
||||||
|
- **Use `anyhow`** for application-level error context and propagation
|
||||||
|
- **Never use `panic!`** in production code - handle errors gracefully
|
||||||
|
- **Implement `From` traits** for error type conversions
|
||||||
|
|
||||||
|
```rust
|
||||||
|
// Core domain errors
|
||||||
|
#[derive(Error, Debug)]
|
||||||
|
pub enum SyncError {
|
||||||
|
#[error("Failed to fetch transactions from source: {0}")]
|
||||||
|
SourceError(#[from] anyhow::Error),
|
||||||
|
#[error("Failed to store transaction: {0}")]
|
||||||
|
DestinationError(#[from] anyhow::Error),
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Async Programming
|
||||||
|
|
||||||
|
- **Use `tokio`** as the async runtime (workspace dependency)
|
||||||
|
- **Prefer `async-trait`** for trait methods that need to be async
|
||||||
|
- **Handle cancellation** properly with `select!` or `tokio::time::timeout`
|
||||||
|
- **Use `?` operator** for error propagation in async functions
|
||||||
|
|
||||||
|
### Testing Strategy
|
||||||
|
|
||||||
|
- **Unit Tests**: Test pure functions and business logic in isolation
|
||||||
|
- **Integration Tests**: Test adapter implementations with `wiremock`
|
||||||
|
- **Mock External Dependencies**: Use `mockall` for trait-based testing
|
||||||
|
- **Test Fixtures**: Store sample JSON responses in `tests/fixtures/`
|
||||||
|
|
||||||
|
```rust
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use mockall::predicate::*;
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn test_sync_with_mock_source() {
|
||||||
|
let mut mock_source = MockTransactionSource::new();
|
||||||
|
// Setup mock expectations
|
||||||
|
// Test core logic
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Code Organization
|
||||||
|
|
||||||
|
- **Workspace Dependencies**: Define common dependencies in root `Cargo.toml`
|
||||||
|
- **Feature Flags**: Use features for optional functionality
|
||||||
|
- **Module Structure**: Keep modules focused and single-responsibility
|
||||||
|
- **Public API**: Minimize public surface area; prefer internal modules
|
||||||
|
|
||||||
|
### Dependencies and Patterns
|
||||||
|
|
||||||
|
**Key Workspace Dependencies:**
|
||||||
|
- `tokio`: Async runtime with full features
|
||||||
|
- `reqwest`: HTTP client with JSON support
|
||||||
|
- `serde`/`serde_json`: Serialization/deserialization
|
||||||
|
- `chrono`: Date/time handling with serde support
|
||||||
|
- `rust_decimal`: Precise decimal arithmetic for financial data
|
||||||
|
- `tracing`/`tracing-subscriber`: Structured logging
|
||||||
|
- `clap`: CLI argument parsing with derive macros
|
||||||
|
- `anyhow`/`thiserror`: Error handling
|
||||||
|
- `async-trait`: Async trait support
|
||||||
|
- `wiremock`: HTTP mocking for tests
|
||||||
|
- `mockall`: Runtime mocking for tests
|
||||||
|
|
||||||
|
## Development Workflow
|
||||||
|
|
||||||
|
### 1. Code Development
|
||||||
|
- Write code in appropriate modules following the hexagonal architecture
|
||||||
|
- Keep core business logic separate from external integrations
|
||||||
|
- Use workspace dependencies consistently
|
||||||
|
|
||||||
|
### 2. Testing
|
||||||
|
- Write tests alongside code in `#[cfg(test)]` modules
|
||||||
|
- Test both happy path and error conditions
|
||||||
|
- Use mock objects for external dependencies
|
||||||
|
- Ensure all tests pass: `cargo test --workspace`
|
||||||
|
|
||||||
|
### 3. Code Quality
|
||||||
|
- Follow Rust idioms and conventions
|
||||||
|
- Use `cargo fmt` for formatting
|
||||||
|
- Use `cargo clippy` for linting
|
||||||
|
- Ensure documentation for public APIs
|
||||||
|
|
||||||
|
### 4. Commit Standards
|
||||||
|
- Commit both code and tests together
|
||||||
|
- Write clear, descriptive commit messages
|
||||||
|
- Ensure the workspace compiles: `cargo build --workspace`
|
||||||
|
|
||||||
|
## Project Structure Guidelines
|
||||||
|
|
||||||
|
### Core Module (`banks2ff/src/core/`)
|
||||||
|
- **models.rs**: Domain entities (BankTransaction, Account)
|
||||||
|
- **ports.rs**: Trait definitions (TransactionSource, TransactionDestination)
|
||||||
|
- **sync.rs**: Business logic orchestration
|
||||||
|
|
||||||
|
### Adapters Module (`banks2ff/src/adapters/`)
|
||||||
|
- **gocardless/**: GoCardless API integration
|
||||||
|
- **firefly/**: Firefly III API integration
|
||||||
|
- Each adapter implements the appropriate port trait
|
||||||
|
|
||||||
|
### Client Libraries
|
||||||
|
- **gocardless-client/**: Standalone GoCardless API wrapper
|
||||||
|
- **firefly-client/**: Standalone Firefly III API wrapper
|
||||||
|
- Both use `reqwest` for HTTP communication
|
||||||
|
|
||||||
|
## Security Considerations
|
||||||
|
|
||||||
|
- **Never log sensitive data**: Use tracing filters to exclude financial information
|
||||||
|
- **Environment variables**: Store credentials in `.env` file (never in code)
|
||||||
|
- **Input validation**: Validate all external data before processing
|
||||||
|
- **Error messages**: Don't expose sensitive information in error messages
|
||||||
|
|
||||||
|
## Performance Considerations
|
||||||
|
|
||||||
|
- **Caching**: Use caching to reduce API calls (see GoCardlessAdapter)
|
||||||
|
- **Rate Limiting**: Handle 429 responses gracefully
|
||||||
|
- **Batch Processing**: Process transactions in reasonable batches
|
||||||
|
- **Async Concurrency**: Use `tokio` for concurrent operations where appropriate
|
||||||
|
|
||||||
|
## Observability
|
||||||
|
|
||||||
|
- **Structured Logging**: Use `tracing` with spans for operations
|
||||||
|
- **Error Context**: Provide context in error messages for debugging
|
||||||
|
- **Metrics**: Consider adding metrics for sync operations
|
||||||
|
- **Log Levels**: Use appropriate log levels (debug, info, warn, error)
|
||||||
2610
Cargo.lock
generated
Normal file
2610
Cargo.lock
generated
Normal file
File diff suppressed because it is too large
Load Diff
31
Cargo.toml
Normal file
31
Cargo.toml
Normal file
@@ -0,0 +1,31 @@
|
|||||||
|
[workspace]
|
||||||
|
members = [
|
||||||
|
"banks2ff",
|
||||||
|
"firefly-client",
|
||||||
|
"gocardless-client",
|
||||||
|
]
|
||||||
|
resolver = "2"
|
||||||
|
|
||||||
|
[workspace.package]
|
||||||
|
version = "0.1.0"
|
||||||
|
edition = "2021"
|
||||||
|
authors = ["Your Name <your.email@example.com>"]
|
||||||
|
|
||||||
|
[workspace.dependencies]
|
||||||
|
tokio = { version = "1.34", features = ["full"] }
|
||||||
|
anyhow = "1.0"
|
||||||
|
thiserror = "1.0"
|
||||||
|
tracing = "0.1"
|
||||||
|
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
||||||
|
serde = { version = "1.0", features = ["derive"] }
|
||||||
|
serde_json = "1.0"
|
||||||
|
chrono = { version = "0.4", features = ["serde"] }
|
||||||
|
rust_decimal = { version = "1.33", features = ["serde-float"] }
|
||||||
|
async-trait = "0.1"
|
||||||
|
dotenvy = "0.15"
|
||||||
|
clap = { version = "4.4", features = ["derive", "env"] }
|
||||||
|
reqwest = { version = "0.11", features = ["json", "multipart"] }
|
||||||
|
url = "2.5"
|
||||||
|
wiremock = "0.5"
|
||||||
|
tokio-test = "0.4"
|
||||||
|
mockall = "0.11"
|
||||||
95
README.md
95
README.md
@@ -1,30 +1,85 @@
|
|||||||
# Bank2FF
|
# Banks2FF
|
||||||
|
|
||||||
Bank2FF is a tool that can retrieve bank transactions from Gocardless and
|
A robust command-line tool to synchronize bank transactions from GoCardless (formerly Nordigen) to Firefly III.
|
||||||
add them to Firefly III.
|
|
||||||
|
|
||||||
It contains autogenerated APIs for both Firefly III and for the
|
## Architecture
|
||||||
Gocardless Bank Account Data API.
|
|
||||||
|
This project is a Rust Workspace consisting of:
|
||||||
|
- `banks2ff`: The main CLI application (Hexagonal Architecture).
|
||||||
|
- `gocardless-client`: A hand-crafted, strongly-typed library for the GoCardless Bank Account Data API.
|
||||||
|
- `firefly-client`: A hand-crafted, strongly-typed library for the Firefly III API.
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- **Multi-Currency Support**: Correctly handles foreign currency transactions by extracting exchange rate data.
|
||||||
|
- **Idempotency (Healer Mode)**:
|
||||||
|
- Detects duplicates using a windowed search (Date +/- 3 days, exact Amount).
|
||||||
|
- "Heals" historical transactions by updating them with the correct `external_id`.
|
||||||
|
- Skips transactions that already have a matching `external_id`.
|
||||||
|
- **Clean Architecture**: Decoupled core logic makes it reliable and testable.
|
||||||
|
- **Observability**: Structured logging via `tracing`.
|
||||||
|
- **Dry Run**: Preview changes without writing to Firefly III.
|
||||||
|
- **Rate Limit Protection**:
|
||||||
|
- Caches GoCardless account details to avoid unnecessary calls.
|
||||||
|
- Respects token expiry to minimize auth calls.
|
||||||
|
- Handles `429 Too Many Requests` gracefully by skipping affected accounts.
|
||||||
|
|
||||||
|
## Setup & Configuration
|
||||||
|
|
||||||
|
1. **Prerequisites**:
|
||||||
|
- Rust (latest stable)
|
||||||
|
- An account with GoCardless Bank Account Data (get your `secret_id` and `secret_key`).
|
||||||
|
- A running Firefly III instance (get your Personal Access Token).
|
||||||
|
|
||||||
|
2. **Environment Variables**:
|
||||||
|
Copy `env.example` to `.env` and fill in your details:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cp env.example .env
|
||||||
|
```
|
||||||
|
|
||||||
|
Required variables:
|
||||||
|
- `GOCARDLESS_ID`: Your GoCardless Secret ID.
|
||||||
|
- `GOCARDLESS_KEY`: Your GoCardless Secret Key.
|
||||||
|
- `FIREFLY_III_URL`: The base URL of your Firefly instance (e.g., `https://money.example.com`).
|
||||||
|
- `FIREFLY_III_API_KEY`: Your Personal Access Token.
|
||||||
|
|
||||||
|
Optional:
|
||||||
|
- `GOCARDLESS_URL`: Defaults to `https://bankaccountdata.gocardless.com`.
|
||||||
|
- `RUST_LOG`: Set log level (e.g., `info`, `debug`, `trace`).
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
The project has a comprehensive test suite using `wiremock` for API clients and `mockall` for core logic.
|
||||||
|
|
||||||
|
To run all tests:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo test --workspace
|
||||||
|
```
|
||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
|
|
||||||
TBD
|
To run the synchronization:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run via cargo (defaults: Start = Last Firefly Date + 1, End = Yesterday)
|
||||||
|
cargo run -p banks2ff
|
||||||
|
|
||||||
## Generating the API clients
|
# Dry Run (Read-only)
|
||||||
|
cargo run -p banks2ff -- --dry-run
|
||||||
|
|
||||||
These API clients are generated with the OpenAPI Generators for Rust.
|
# Custom Date Range
|
||||||
|
cargo run -p banks2ff -- --start 2023-01-01 --end 2023-01-31
|
||||||
|
```
|
||||||
|
|
||||||
These need Podman installed, and assume this command is run from the same
|
## How it works
|
||||||
directory where this README.md file is located.
|
|
||||||
|
|
||||||
For Gocardless:
|
1. **Fetch**: Retrieves active accounts from GoCardless (filtered by those present in Firefly III to save requests).
|
||||||
|
2. **Match**: Resolves the destination account in Firefly III by matching the IBAN.
|
||||||
`podman run --rm -v ${PWD}:/local openapitools/openapi-generator-cli generate -g rust -o /local/gocardless-bankaccount-data-api -i 'https://bankaccountdata.gocardless.com/api/v2/swagger.json' --additional-properties=library=reqwest,packageName=gocardless-bankaccount-data-api,packageVersion=2.0.0,supportMiddleware=true,avoidBoxedModels=true`
|
3. **Sync Window**: Determines the start date automatically by finding the latest transaction in Firefly for that account.
|
||||||
|
4. **Process**: For each transaction:
|
||||||
|
- **Search**: Checks Firefly for an existing transaction (matching Amount and Date +/- 3 days).
|
||||||
For Firefly III:
|
- **Heal**: If found but missing an `external_id`, it updates the transaction.
|
||||||
|
- **Skip**: If found and matches `external_id`, it skips.
|
||||||
If necessary, change the URL to the definition. If that is a new version, then also change the `packageVersion` parameter.
|
- **Create**: If not found, it creates a new transaction.
|
||||||
|
|
||||||
`podman run --rm -v ${PWD}:/local openapitools/openapi-generator-cli generate -g rust -o /local/firefly-iii-api -i 'https://api-docs.firefly-iii.org/firefly-iii-2.1.0-v1.yaml' --additional-properties=library=reqwest,packageName=firefly-iii-api,packageVersion=2.1.0,supportMiddleware=true,avoidBoxedModels=true`
|
|
||||||
|
|||||||
27
banks2ff/Cargo.toml
Normal file
27
banks2ff/Cargo.toml
Normal file
@@ -0,0 +1,27 @@
|
|||||||
|
[package]
|
||||||
|
name = "banks2ff"
|
||||||
|
version.workspace = true
|
||||||
|
edition.workspace = true
|
||||||
|
authors.workspace = true
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
tokio = { workspace = true }
|
||||||
|
anyhow = { workspace = true }
|
||||||
|
tracing = { workspace = true }
|
||||||
|
tracing-subscriber = { workspace = true }
|
||||||
|
serde = { workspace = true }
|
||||||
|
serde_json = { workspace = true }
|
||||||
|
chrono = { workspace = true }
|
||||||
|
rust_decimal = { workspace = true }
|
||||||
|
dotenvy = { workspace = true }
|
||||||
|
clap = { workspace = true }
|
||||||
|
|
||||||
|
# Core logic dependencies
|
||||||
|
async-trait = { workspace = true }
|
||||||
|
|
||||||
|
# API Client dependencies
|
||||||
|
firefly-client = { path = "../firefly-client" }
|
||||||
|
gocardless-client = { path = "../gocardless-client" }
|
||||||
|
|
||||||
|
[dev-dependencies]
|
||||||
|
mockall = { workspace = true }
|
||||||
188
banks2ff/src/adapters/firefly/client.rs
Normal file
188
banks2ff/src/adapters/firefly/client.rs
Normal file
@@ -0,0 +1,188 @@
|
|||||||
|
use async_trait::async_trait;
|
||||||
|
use anyhow::Result;
|
||||||
|
use tracing::instrument;
|
||||||
|
use crate::core::ports::{TransactionDestination, TransactionMatch};
|
||||||
|
use crate::core::models::BankTransaction;
|
||||||
|
use firefly_client::client::FireflyClient;
|
||||||
|
use firefly_client::models::{TransactionStore, TransactionSplitStore, TransactionUpdate, TransactionSplitUpdate};
|
||||||
|
use std::sync::Arc;
|
||||||
|
use tokio::sync::Mutex;
|
||||||
|
use rust_decimal::Decimal;
|
||||||
|
use std::str::FromStr;
|
||||||
|
use chrono::NaiveDate;
|
||||||
|
|
||||||
|
pub struct FireflyAdapter {
|
||||||
|
client: Arc<Mutex<FireflyClient>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FireflyAdapter {
|
||||||
|
pub fn new(client: FireflyClient) -> Self {
|
||||||
|
Self {
|
||||||
|
client: Arc::new(Mutex::new(client)),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[async_trait]
|
||||||
|
impl TransactionDestination for FireflyAdapter {
|
||||||
|
#[instrument(skip(self))]
|
||||||
|
async fn resolve_account_id(&self, iban: &str) -> Result<Option<String>> {
|
||||||
|
let client = self.client.lock().await;
|
||||||
|
let accounts = client.search_accounts(iban).await?;
|
||||||
|
|
||||||
|
// Look for exact match on IBAN, ensuring account is active
|
||||||
|
for acc in accounts.data {
|
||||||
|
// Filter for active accounts only (default is usually active, but let's check if attribute exists)
|
||||||
|
// Note: The Firefly API spec v6.4.4 Account object has 'active' attribute as boolean.
|
||||||
|
let is_active = acc.attributes.active.unwrap_or(true);
|
||||||
|
|
||||||
|
if !is_active {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if let Some(acc_iban) = acc.attributes.iban {
|
||||||
|
if acc_iban.replace(" ", "") == iban.replace(" ", "") {
|
||||||
|
return Ok(Some(acc.id));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(None)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[instrument(skip(self))]
|
||||||
|
async fn get_active_account_ibans(&self) -> Result<Vec<String>> {
|
||||||
|
let client = self.client.lock().await;
|
||||||
|
// Get all asset accounts. Note: Pagination might be needed if user has > 50 accounts.
|
||||||
|
// For typical users, 50 is enough. If needed we can loop pages.
|
||||||
|
// The client `get_accounts` method hardcodes limit=default. We should probably expose a list_all method or loop here.
|
||||||
|
// For now, let's assume page 1 covers it or use search.
|
||||||
|
|
||||||
|
let accounts = client.get_accounts("").await?; // Argument ignored in current impl
|
||||||
|
let mut ibans = Vec::new();
|
||||||
|
|
||||||
|
for acc in accounts.data {
|
||||||
|
let is_active = acc.attributes.active.unwrap_or(true);
|
||||||
|
if is_active {
|
||||||
|
if let Some(iban) = acc.attributes.iban {
|
||||||
|
if !iban.is_empty() {
|
||||||
|
ibans.push(iban);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(ibans)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[instrument(skip(self))]
|
||||||
|
async fn get_last_transaction_date(&self, account_id: &str) -> Result<Option<NaiveDate>> {
|
||||||
|
let client = self.client.lock().await;
|
||||||
|
// Fetch latest 1 transaction
|
||||||
|
let tx_list = client.list_account_transactions(account_id, None, None).await?;
|
||||||
|
|
||||||
|
if let Some(first) = tx_list.data.first() {
|
||||||
|
if let Some(split) = first.attributes.transactions.first() {
|
||||||
|
// Format is usually YYYY-MM-DDT... or YYYY-MM-DD
|
||||||
|
let date_str = split.date.split('T').next().unwrap_or(&split.date);
|
||||||
|
if let Ok(date) = NaiveDate::parse_from_str(date_str, "%Y-%m-%d") {
|
||||||
|
return Ok(Some(date));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(None)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[instrument(skip(self))]
|
||||||
|
async fn find_transaction(&self, account_id: &str, tx: &BankTransaction) -> Result<Option<TransactionMatch>> {
|
||||||
|
let client = self.client.lock().await;
|
||||||
|
|
||||||
|
// Search window: +/- 3 days
|
||||||
|
let start_date = tx.date - chrono::Duration::days(3);
|
||||||
|
let end_date = tx.date + chrono::Duration::days(3);
|
||||||
|
|
||||||
|
let tx_list = client.list_account_transactions(
|
||||||
|
account_id,
|
||||||
|
Some(&start_date.format("%Y-%m-%d").to_string()),
|
||||||
|
Some(&end_date.format("%Y-%m-%d").to_string())
|
||||||
|
).await?;
|
||||||
|
|
||||||
|
// Filter logic
|
||||||
|
for existing_tx in tx_list.data {
|
||||||
|
for split in existing_tx.attributes.transactions {
|
||||||
|
// 1. Check Amount (exact match absolute value)
|
||||||
|
if let Ok(amount) = Decimal::from_str(&split.amount) {
|
||||||
|
if amount.abs() == tx.amount.abs() {
|
||||||
|
// 2. Check External ID
|
||||||
|
if let Some(ref ext_id) = split.external_id {
|
||||||
|
if ext_id == &tx.internal_id {
|
||||||
|
return Ok(Some(TransactionMatch {
|
||||||
|
id: existing_tx.id.clone(),
|
||||||
|
has_external_id: true,
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// 3. "Naked" transaction match (Heuristic)
|
||||||
|
// If currency matches
|
||||||
|
if let Some(ref code) = split.currency_code {
|
||||||
|
if code != &tx.currency {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return Ok(Some(TransactionMatch {
|
||||||
|
id: existing_tx.id.clone(),
|
||||||
|
has_external_id: false,
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(None)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[instrument(skip(self))]
|
||||||
|
async fn create_transaction(&self, account_id: &str, tx: &BankTransaction) -> Result<()> {
|
||||||
|
let client = self.client.lock().await;
|
||||||
|
|
||||||
|
// Map to Firefly Transaction
|
||||||
|
let is_credit = tx.amount.is_sign_positive();
|
||||||
|
let transaction_type = if is_credit { "deposit" } else { "withdrawal" };
|
||||||
|
|
||||||
|
let split = TransactionSplitStore {
|
||||||
|
transaction_type: transaction_type.to_string(),
|
||||||
|
date: tx.date.format("%Y-%m-%d").to_string(),
|
||||||
|
amount: tx.amount.abs().to_string(),
|
||||||
|
description: tx.description.clone(),
|
||||||
|
source_id: if !is_credit { Some(account_id.to_string()) } else { None },
|
||||||
|
source_name: if is_credit { tx.counterparty_name.clone().or(Some("Unknown Sender".to_string())) } else { None },
|
||||||
|
destination_id: if is_credit { Some(account_id.to_string()) } else { None },
|
||||||
|
destination_name: if !is_credit { tx.counterparty_name.clone().or(Some("Unknown Recipient".to_string())) } else { None },
|
||||||
|
currency_code: Some(tx.currency.clone()),
|
||||||
|
foreign_amount: tx.foreign_amount.map(|d| d.abs().to_string()),
|
||||||
|
foreign_currency_code: tx.foreign_currency.clone(),
|
||||||
|
external_id: Some(tx.internal_id.clone()),
|
||||||
|
};
|
||||||
|
|
||||||
|
let store = TransactionStore {
|
||||||
|
transactions: vec![split],
|
||||||
|
apply_rules: Some(true),
|
||||||
|
fire_webhooks: Some(true),
|
||||||
|
error_if_duplicate_hash: Some(true),
|
||||||
|
};
|
||||||
|
|
||||||
|
client.store_transaction(store).await.map_err(|e| e.into())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[instrument(skip(self))]
|
||||||
|
async fn update_transaction_external_id(&self, id: &str, external_id: &str) -> Result<()> {
|
||||||
|
let client = self.client.lock().await;
|
||||||
|
let update = TransactionUpdate {
|
||||||
|
transactions: vec![TransactionSplitUpdate {
|
||||||
|
external_id: Some(external_id.to_string()),
|
||||||
|
}],
|
||||||
|
};
|
||||||
|
client.update_transaction(id, update).await.map_err(|e| e.into())
|
||||||
|
}
|
||||||
|
}
|
||||||
1
banks2ff/src/adapters/firefly/mod.rs
Normal file
1
banks2ff/src/adapters/firefly/mod.rs
Normal file
@@ -0,0 +1 @@
|
|||||||
|
pub mod client;
|
||||||
51
banks2ff/src/adapters/gocardless/cache.rs
Normal file
51
banks2ff/src/adapters/gocardless/cache.rs
Normal file
@@ -0,0 +1,51 @@
|
|||||||
|
use std::collections::HashMap;
|
||||||
|
use std::fs;
|
||||||
|
use std::path::Path;
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
use tracing::warn;
|
||||||
|
|
||||||
|
#[derive(Debug, Serialize, Deserialize, Default)]
|
||||||
|
pub struct AccountCache {
|
||||||
|
/// Map of Account ID -> IBAN
|
||||||
|
pub accounts: HashMap<String, String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl AccountCache {
|
||||||
|
fn get_path() -> String {
|
||||||
|
".banks2ff-cache.json".to_string()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn load() -> Self {
|
||||||
|
let path = Self::get_path();
|
||||||
|
if Path::new(&path).exists() {
|
||||||
|
match fs::read_to_string(&path) {
|
||||||
|
Ok(content) => match serde_json::from_str(&content) {
|
||||||
|
Ok(cache) => return cache,
|
||||||
|
Err(e) => warn!("Failed to parse cache file: {}", e),
|
||||||
|
},
|
||||||
|
Err(e) => warn!("Failed to read cache file: {}", e),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Self::default()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn save(&self) {
|
||||||
|
let path = Self::get_path();
|
||||||
|
match serde_json::to_string_pretty(self) {
|
||||||
|
Ok(content) => {
|
||||||
|
if let Err(e) = fs::write(&path, content) {
|
||||||
|
warn!("Failed to write cache file: {}", e);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
Err(e) => warn!("Failed to serialize cache: {}", e),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn get_iban(&self, account_id: &str) -> Option<String> {
|
||||||
|
self.accounts.get(account_id).cloned()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn insert(&mut self, account_id: String, iban: String) {
|
||||||
|
self.accounts.insert(account_id, iban);
|
||||||
|
}
|
||||||
|
}
|
||||||
152
banks2ff/src/adapters/gocardless/client.rs
Normal file
152
banks2ff/src/adapters/gocardless/client.rs
Normal file
@@ -0,0 +1,152 @@
|
|||||||
|
use async_trait::async_trait;
|
||||||
|
use chrono::NaiveDate;
|
||||||
|
use anyhow::Result;
|
||||||
|
use tracing::{info, instrument, warn};
|
||||||
|
use crate::core::ports::TransactionSource;
|
||||||
|
use crate::core::models::{Account, BankTransaction};
|
||||||
|
use crate::adapters::gocardless::mapper::map_transaction;
|
||||||
|
use crate::adapters::gocardless::cache::AccountCache;
|
||||||
|
use gocardless_client::client::GoCardlessClient;
|
||||||
|
use std::sync::Arc;
|
||||||
|
use tokio::sync::Mutex;
|
||||||
|
|
||||||
|
pub struct GoCardlessAdapter {
|
||||||
|
client: Arc<Mutex<GoCardlessClient>>,
|
||||||
|
cache: Arc<Mutex<AccountCache>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl GoCardlessAdapter {
|
||||||
|
pub fn new(client: GoCardlessClient) -> Self {
|
||||||
|
Self {
|
||||||
|
client: Arc::new(Mutex::new(client)),
|
||||||
|
cache: Arc::new(Mutex::new(AccountCache::load())),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[async_trait]
|
||||||
|
impl TransactionSource for GoCardlessAdapter {
|
||||||
|
#[instrument(skip(self))]
|
||||||
|
async fn get_accounts(&self, wanted_ibans: Option<Vec<String>>) -> Result<Vec<Account>> {
|
||||||
|
let mut client = self.client.lock().await;
|
||||||
|
let mut cache = self.cache.lock().await;
|
||||||
|
|
||||||
|
// Ensure token
|
||||||
|
client.obtain_access_token().await?;
|
||||||
|
|
||||||
|
let requisitions = client.get_requisitions().await?;
|
||||||
|
let mut accounts = Vec::new();
|
||||||
|
|
||||||
|
// Build a hashset of wanted IBANs if provided, for faster lookup
|
||||||
|
let wanted_set = wanted_ibans.map(|list| {
|
||||||
|
list.into_iter()
|
||||||
|
.map(|i| i.replace(" ", ""))
|
||||||
|
.collect::<std::collections::HashSet<_>>()
|
||||||
|
});
|
||||||
|
|
||||||
|
let mut found_count = 0;
|
||||||
|
let target_count = wanted_set.as_ref().map(|s| s.len()).unwrap_or(0);
|
||||||
|
|
||||||
|
for req in requisitions.results {
|
||||||
|
// Optimization: Only process Linked requisitions to avoid 401/403 on expired ones
|
||||||
|
if req.status != "LN" {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if let Some(req_accounts) = req.accounts {
|
||||||
|
for acc_id in req_accounts {
|
||||||
|
// 1. Check Cache
|
||||||
|
let mut iban_opt = cache.get_iban(&acc_id);
|
||||||
|
|
||||||
|
// 2. Fetch if missing
|
||||||
|
if iban_opt.is_none() {
|
||||||
|
match client.get_account(&acc_id).await {
|
||||||
|
Ok(details) => {
|
||||||
|
let new_iban = details.iban.unwrap_or_default();
|
||||||
|
cache.insert(acc_id.clone(), new_iban.clone());
|
||||||
|
cache.save();
|
||||||
|
iban_opt = Some(new_iban);
|
||||||
|
},
|
||||||
|
Err(e) => {
|
||||||
|
// If rate limit hit here, we might want to skip this account and continue?
|
||||||
|
// But get_account is critical to identify the account.
|
||||||
|
// If we fail here, we can't match.
|
||||||
|
warn!("Failed to fetch details for account {}: {}", acc_id, e);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let iban = iban_opt.unwrap_or_default();
|
||||||
|
|
||||||
|
let mut keep = true;
|
||||||
|
if let Some(ref wanted) = wanted_set {
|
||||||
|
if !wanted.contains(&iban.replace(" ", "")) {
|
||||||
|
keep = false;
|
||||||
|
} else {
|
||||||
|
found_count += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if keep {
|
||||||
|
accounts.push(Account {
|
||||||
|
id: acc_id,
|
||||||
|
iban,
|
||||||
|
currency: "EUR".to_string(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// Optimization: Stop if we found all wanted accounts
|
||||||
|
if let Some(_) = wanted_set {
|
||||||
|
if found_count >= target_count && target_count > 0 {
|
||||||
|
info!("Found all {} wanted accounts. Stopping search.", target_count);
|
||||||
|
return Ok(accounts);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
info!("Found {} matching accounts in GoCardless", accounts.len());
|
||||||
|
Ok(accounts)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[instrument(skip(self))]
|
||||||
|
async fn get_transactions(&self, account_id: &str, start: NaiveDate, end: NaiveDate) -> Result<Vec<BankTransaction>> {
|
||||||
|
let mut client = self.client.lock().await;
|
||||||
|
client.obtain_access_token().await?;
|
||||||
|
|
||||||
|
let response_result = client.get_transactions(
|
||||||
|
account_id,
|
||||||
|
Some(&start.to_string()),
|
||||||
|
Some(&end.to_string())
|
||||||
|
).await;
|
||||||
|
|
||||||
|
match response_result {
|
||||||
|
Ok(response) => {
|
||||||
|
let mut transactions = Vec::new();
|
||||||
|
for tx in response.transactions.booked {
|
||||||
|
match map_transaction(tx) {
|
||||||
|
Ok(t) => transactions.push(t),
|
||||||
|
Err(e) => tracing::error!("Failed to map transaction: {}", e),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
info!("Fetched {} transactions for account {}", transactions.len(), account_id);
|
||||||
|
Ok(transactions)
|
||||||
|
},
|
||||||
|
Err(e) => {
|
||||||
|
// Handle 429 specifically?
|
||||||
|
let err_str = e.to_string();
|
||||||
|
if err_str.contains("429") {
|
||||||
|
warn!("Rate limit reached for account {}. Skipping.", account_id);
|
||||||
|
// Return empty list implies "no transactions found", which is safe for sync loop (it just won't sync this account).
|
||||||
|
// Or we could return an error if we want to stop?
|
||||||
|
// Returning empty list allows other accounts to potentially proceed if limits are per-account (which GC says they are!)
|
||||||
|
return Ok(vec![]);
|
||||||
|
}
|
||||||
|
Err(e.into())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
147
banks2ff/src/adapters/gocardless/mapper.rs
Normal file
147
banks2ff/src/adapters/gocardless/mapper.rs
Normal file
@@ -0,0 +1,147 @@
|
|||||||
|
use rust_decimal::Decimal;
|
||||||
|
use rust_decimal::prelude::Signed;
|
||||||
|
use std::str::FromStr;
|
||||||
|
use anyhow::Result;
|
||||||
|
use crate::core::models::BankTransaction;
|
||||||
|
use gocardless_client::models::Transaction;
|
||||||
|
|
||||||
|
pub fn map_transaction(tx: Transaction) -> Result<BankTransaction> {
|
||||||
|
let internal_id = tx.transaction_id
|
||||||
|
.ok_or_else(|| anyhow::anyhow!("Transaction ID missing"))?;
|
||||||
|
|
||||||
|
let date_str = tx.booking_date.or(tx.value_date)
|
||||||
|
.ok_or_else(|| anyhow::anyhow!("Transaction date missing"))?;
|
||||||
|
let date = chrono::NaiveDate::parse_from_str(&date_str, "%Y-%m-%d")?;
|
||||||
|
|
||||||
|
let amount = Decimal::from_str(&tx.transaction_amount.amount)?;
|
||||||
|
let currency = tx.transaction_amount.currency;
|
||||||
|
|
||||||
|
let mut foreign_amount = None;
|
||||||
|
let mut foreign_currency = None;
|
||||||
|
|
||||||
|
if let Some(exchanges) = tx.currency_exchange {
|
||||||
|
if let Some(exchange) = exchanges.first() {
|
||||||
|
if let (Some(source_curr), Some(rate_str)) = (&exchange.source_currency, &exchange.exchange_rate) {
|
||||||
|
foreign_currency = Some(source_curr.clone());
|
||||||
|
if let Ok(rate) = Decimal::from_str(rate_str) {
|
||||||
|
// If instructedAmount is not available (it's not in our DTO yet), we calculate it.
|
||||||
|
// But wait, normally instructedAmount is the foreign amount.
|
||||||
|
// If we don't have it, we estimate: foreign = amount * rate?
|
||||||
|
// Actually usually: Base (Account) Amount = Foreign Amount / Rate OR Foreign * Rate
|
||||||
|
// If I have 100 EUR and rate is 1.10 USD/EUR -> 110 USD.
|
||||||
|
// Let's check the GoCardless spec definition of exchangeRate.
|
||||||
|
// "exchangeRate": "Factor used to convert an amount from one currency into another. This reflects the price at which the acquirer has bought the currency."
|
||||||
|
|
||||||
|
// Without strict direction, simple multiplication is risky.
|
||||||
|
// ideally we should have `instructedAmount` or `unitCurrency` logic.
|
||||||
|
// For now, let's assume: foreign_amount = amount * rate is NOT always correct.
|
||||||
|
// BUT, usually `sourceCurrency` is the original currency.
|
||||||
|
// If I spent 10 USD, and my account is EUR.
|
||||||
|
// sourceCurrency: USD. targetCurrency: EUR.
|
||||||
|
// transactionAmount: -9.00 EUR.
|
||||||
|
// exchangeRate: ???
|
||||||
|
|
||||||
|
// Let's implement a safe calculation or just store what we have.
|
||||||
|
// Actually, simply multiplying might be wrong if the rate is inverted.
|
||||||
|
// Let's verify with unit tests if we had real data, but for now let's use the logic:
|
||||||
|
// foreign_amount = amount * rate (if rate > 0) or amount / rate ?
|
||||||
|
|
||||||
|
// Let's look at the example in my plan: "foreign_amount = amount * currencyExchange[0].exchangeRate"
|
||||||
|
// I will stick to that plan, but wrap it in a safe calculation.
|
||||||
|
|
||||||
|
let calc = amount.abs() * rate; // Usually rate is positive.
|
||||||
|
// We preserve the sign of the transaction amount for the foreign amount.
|
||||||
|
let sign = amount.signum();
|
||||||
|
foreign_amount = Some(calc * sign);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fallback for description: Remittance Unstructured -> Debtor/Creditor Name -> "Unknown"
|
||||||
|
let description = tx.remittance_information_unstructured
|
||||||
|
.or(tx.creditor_name.clone())
|
||||||
|
.or(tx.debtor_name.clone())
|
||||||
|
.unwrap_or_else(|| "Unknown Transaction".to_string());
|
||||||
|
|
||||||
|
Ok(BankTransaction {
|
||||||
|
internal_id,
|
||||||
|
date,
|
||||||
|
amount,
|
||||||
|
currency,
|
||||||
|
foreign_amount,
|
||||||
|
foreign_currency,
|
||||||
|
description,
|
||||||
|
counterparty_name: tx.creditor_name.or(tx.debtor_name),
|
||||||
|
counterparty_iban: tx.creditor_account.and_then(|a| a.iban).or(tx.debtor_account.and_then(|a| a.iban)),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use gocardless_client::models::{TransactionAmount, CurrencyExchange};
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_map_normal_transaction() {
|
||||||
|
let t = Transaction {
|
||||||
|
transaction_id: Some("123".into()),
|
||||||
|
booking_date: Some("2023-01-01".into()),
|
||||||
|
value_date: None,
|
||||||
|
transaction_amount: TransactionAmount {
|
||||||
|
amount: "100.50".into(),
|
||||||
|
currency: "EUR".into(),
|
||||||
|
},
|
||||||
|
currency_exchange: None,
|
||||||
|
creditor_name: Some("Shop".into()),
|
||||||
|
creditor_account: None,
|
||||||
|
debtor_name: None,
|
||||||
|
debtor_account: None,
|
||||||
|
remittance_information_unstructured: Some("Groceries".into()),
|
||||||
|
proprietary_bank_transaction_code: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
let res = map_transaction(t).unwrap();
|
||||||
|
assert_eq!(res.internal_id, "123");
|
||||||
|
assert_eq!(res.amount, Decimal::new(10050, 2));
|
||||||
|
assert_eq!(res.currency, "EUR");
|
||||||
|
assert_eq!(res.foreign_amount, None);
|
||||||
|
assert_eq!(res.description, "Groceries");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_map_multicurrency_transaction() {
|
||||||
|
let t = Transaction {
|
||||||
|
transaction_id: Some("124".into()),
|
||||||
|
booking_date: Some("2023-01-02".into()),
|
||||||
|
value_date: None,
|
||||||
|
transaction_amount: TransactionAmount {
|
||||||
|
amount: "-10.00".into(),
|
||||||
|
currency: "EUR".into(),
|
||||||
|
},
|
||||||
|
currency_exchange: Some(vec![CurrencyExchange {
|
||||||
|
source_currency: Some("USD".into()),
|
||||||
|
exchange_rate: Some("1.10".into()),
|
||||||
|
unit_currency: None,
|
||||||
|
target_currency: Some("EUR".into()),
|
||||||
|
}]),
|
||||||
|
creditor_name: Some("US Shop".into()),
|
||||||
|
creditor_account: None,
|
||||||
|
debtor_name: None,
|
||||||
|
debtor_account: None,
|
||||||
|
remittance_information_unstructured: None,
|
||||||
|
proprietary_bank_transaction_code: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
let res = map_transaction(t).unwrap();
|
||||||
|
assert_eq!(res.internal_id, "124");
|
||||||
|
assert_eq!(res.amount, Decimal::new(-1000, 2));
|
||||||
|
assert_eq!(res.foreign_currency, Some("USD".to_string()));
|
||||||
|
|
||||||
|
// 10.00 * 1.10 = 11.00. Sign should be preserved (-11.00)
|
||||||
|
assert_eq!(res.foreign_amount, Some(Decimal::new(-1100, 2)));
|
||||||
|
|
||||||
|
// Description fallback to creditor name
|
||||||
|
assert_eq!(res.description, "US Shop");
|
||||||
|
}
|
||||||
|
}
|
||||||
3
banks2ff/src/adapters/gocardless/mod.rs
Normal file
3
banks2ff/src/adapters/gocardless/mod.rs
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
pub mod client;
|
||||||
|
pub mod mapper;
|
||||||
|
pub mod cache;
|
||||||
2
banks2ff/src/adapters/mod.rs
Normal file
2
banks2ff/src/adapters/mod.rs
Normal file
@@ -0,0 +1,2 @@
|
|||||||
|
pub mod firefly;
|
||||||
|
pub mod gocardless;
|
||||||
3
banks2ff/src/core/mod.rs
Normal file
3
banks2ff/src/core/mod.rs
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
pub mod models;
|
||||||
|
pub mod ports;
|
||||||
|
pub mod sync;
|
||||||
31
banks2ff/src/core/models.rs
Normal file
31
banks2ff/src/core/models.rs
Normal file
@@ -0,0 +1,31 @@
|
|||||||
|
use rust_decimal::Decimal;
|
||||||
|
use chrono::NaiveDate;
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, PartialEq)]
|
||||||
|
pub struct BankTransaction {
|
||||||
|
/// Source ID (GoCardless transactionId)
|
||||||
|
pub internal_id: String,
|
||||||
|
/// Booking date
|
||||||
|
pub date: NaiveDate,
|
||||||
|
/// Amount in account currency
|
||||||
|
pub amount: Decimal,
|
||||||
|
/// Account currency code (e.g., EUR)
|
||||||
|
pub currency: String,
|
||||||
|
/// Original amount (if currency exchange occurred)
|
||||||
|
pub foreign_amount: Option<Decimal>,
|
||||||
|
/// Original currency code
|
||||||
|
pub foreign_currency: Option<String>,
|
||||||
|
/// Remittance info or description
|
||||||
|
pub description: String,
|
||||||
|
/// Counterparty name
|
||||||
|
pub counterparty_name: Option<String>,
|
||||||
|
/// Counterparty IBAN
|
||||||
|
pub counterparty_iban: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, PartialEq)]
|
||||||
|
pub struct Account {
|
||||||
|
pub id: String,
|
||||||
|
pub iban: String,
|
||||||
|
pub currency: String,
|
||||||
|
}
|
||||||
42
banks2ff/src/core/ports.rs
Normal file
42
banks2ff/src/core/ports.rs
Normal file
@@ -0,0 +1,42 @@
|
|||||||
|
use async_trait::async_trait;
|
||||||
|
use chrono::NaiveDate;
|
||||||
|
use anyhow::Result;
|
||||||
|
#[cfg(test)]
|
||||||
|
use mockall::automock;
|
||||||
|
use crate::core::models::{BankTransaction, Account};
|
||||||
|
|
||||||
|
#[derive(Debug, Default)]
|
||||||
|
pub struct IngestResult {
|
||||||
|
pub created: usize,
|
||||||
|
pub duplicates: usize,
|
||||||
|
pub errors: usize,
|
||||||
|
pub healed: usize,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg_attr(test, automock)]
|
||||||
|
#[async_trait]
|
||||||
|
pub trait TransactionSource: Send + Sync {
|
||||||
|
/// Fetch accounts. Optionally filter by a list of wanted IBANs to save requests.
|
||||||
|
async fn get_accounts(&self, wanted_ibans: Option<Vec<String>>) -> Result<Vec<Account>>;
|
||||||
|
async fn get_transactions(&self, account_id: &str, start: NaiveDate, end: NaiveDate) -> Result<Vec<BankTransaction>>;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct TransactionMatch {
|
||||||
|
pub id: String,
|
||||||
|
pub has_external_id: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg_attr(test, automock)]
|
||||||
|
#[async_trait]
|
||||||
|
pub trait TransactionDestination: Send + Sync {
|
||||||
|
async fn resolve_account_id(&self, iban: &str) -> Result<Option<String>>;
|
||||||
|
/// Get list of all active asset account IBANs to drive the sync
|
||||||
|
async fn get_active_account_ibans(&self) -> Result<Vec<String>>;
|
||||||
|
|
||||||
|
// New granular methods for Healer Logic
|
||||||
|
async fn get_last_transaction_date(&self, account_id: &str) -> Result<Option<NaiveDate>>;
|
||||||
|
async fn find_transaction(&self, account_id: &str, transaction: &BankTransaction) -> Result<Option<TransactionMatch>>;
|
||||||
|
async fn create_transaction(&self, account_id: &str, tx: &BankTransaction) -> Result<()>;
|
||||||
|
async fn update_transaction_external_id(&self, id: &str, external_id: &str) -> Result<()>;
|
||||||
|
}
|
||||||
299
banks2ff/src/core/sync.rs
Normal file
299
banks2ff/src/core/sync.rs
Normal file
@@ -0,0 +1,299 @@
|
|||||||
|
use anyhow::Result;
|
||||||
|
use tracing::{info, warn, instrument};
|
||||||
|
use crate::core::ports::{TransactionSource, TransactionDestination, IngestResult};
|
||||||
|
use chrono::{NaiveDate, Local};
|
||||||
|
|
||||||
|
#[instrument(skip(source, destination))]
|
||||||
|
pub async fn run_sync<S, D>(
|
||||||
|
source: &S,
|
||||||
|
destination: &D,
|
||||||
|
cli_start_date: Option<NaiveDate>,
|
||||||
|
cli_end_date: Option<NaiveDate>,
|
||||||
|
dry_run: bool,
|
||||||
|
) -> Result<()>
|
||||||
|
where
|
||||||
|
S: TransactionSource,
|
||||||
|
D: TransactionDestination,
|
||||||
|
{
|
||||||
|
info!("Starting synchronization...");
|
||||||
|
|
||||||
|
// Optimization: Get active Firefly IBANs first
|
||||||
|
let wanted_ibans = destination.get_active_account_ibans().await?;
|
||||||
|
info!("Syncing {} active accounts from Firefly III", wanted_ibans.len());
|
||||||
|
|
||||||
|
let accounts = source.get_accounts(Some(wanted_ibans)).await?;
|
||||||
|
|
||||||
|
// Default end date is Yesterday
|
||||||
|
let end_date = cli_end_date.unwrap_or_else(|| Local::now().date_naive() - chrono::Duration::days(1));
|
||||||
|
|
||||||
|
for account in accounts {
|
||||||
|
let span = tracing::info_span!("sync_account", iban = %account.iban);
|
||||||
|
let _enter = span.enter();
|
||||||
|
|
||||||
|
info!("Processing account...");
|
||||||
|
|
||||||
|
let dest_id_opt = destination.resolve_account_id(&account.iban).await?;
|
||||||
|
let Some(dest_id) = dest_id_opt else {
|
||||||
|
warn!("Account {} not found in destination. Skipping.", account.iban);
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
|
||||||
|
info!("Resolved destination ID: {}", dest_id);
|
||||||
|
|
||||||
|
// Determine Start Date
|
||||||
|
let start_date = if let Some(d) = cli_start_date {
|
||||||
|
d
|
||||||
|
} else {
|
||||||
|
// Default: Latest transaction date + 1 day
|
||||||
|
match destination.get_last_transaction_date(&dest_id).await? {
|
||||||
|
Some(last_date) => last_date + chrono::Duration::days(1),
|
||||||
|
None => {
|
||||||
|
// If no transaction exists in Firefly, we assume this is a fresh sync.
|
||||||
|
// Default to syncing last 30 days.
|
||||||
|
end_date - chrono::Duration::days(30)
|
||||||
|
},
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
if start_date > end_date {
|
||||||
|
info!("Start date {} is after end date {}. Nothing to sync.", start_date, end_date);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
info!("Syncing interval: {} to {}", start_date, end_date);
|
||||||
|
|
||||||
|
// Optimization: Only use active accounts is already filtered in resolve_account_id
|
||||||
|
// However, GoCardless requisitions can expire.
|
||||||
|
// We should check if we can optimize the GoCardless fetching side.
|
||||||
|
// But currently get_transactions takes an account_id.
|
||||||
|
|
||||||
|
let transactions = source.get_transactions(&account.id, start_date, end_date).await?;
|
||||||
|
|
||||||
|
if transactions.is_empty() {
|
||||||
|
info!("No transactions found for period.");
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
info!("Fetched {} transactions from source.", transactions.len());
|
||||||
|
|
||||||
|
let mut stats = IngestResult::default();
|
||||||
|
|
||||||
|
// Healer Logic Loop
|
||||||
|
for tx in transactions {
|
||||||
|
// 1. Check if it exists
|
||||||
|
match destination.find_transaction(&dest_id, &tx).await? {
|
||||||
|
Some(existing) => {
|
||||||
|
if existing.has_external_id {
|
||||||
|
// Already synced properly
|
||||||
|
stats.duplicates += 1;
|
||||||
|
} else {
|
||||||
|
// Found "naked" transaction -> Heal it
|
||||||
|
if dry_run {
|
||||||
|
info!("[DRY RUN] Would heal transaction {} (Firefly ID: {})", tx.internal_id, existing.id);
|
||||||
|
stats.healed += 1;
|
||||||
|
} else {
|
||||||
|
info!("Healing transaction {} (Firefly ID: {})", tx.internal_id, existing.id);
|
||||||
|
if let Err(e) = destination.update_transaction_external_id(&existing.id, &tx.internal_id).await {
|
||||||
|
tracing::error!("Failed to heal transaction: {}", e);
|
||||||
|
stats.errors += 1;
|
||||||
|
} else {
|
||||||
|
stats.healed += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
None => {
|
||||||
|
// New transaction
|
||||||
|
if dry_run {
|
||||||
|
info!("[DRY RUN] Would create transaction {}", tx.internal_id);
|
||||||
|
stats.created += 1;
|
||||||
|
} else {
|
||||||
|
if let Err(e) = destination.create_transaction(&dest_id, &tx).await {
|
||||||
|
// Firefly might still reject it as duplicate if hash matches, even if we didn't find it via heuristic
|
||||||
|
// (unlikely if heuristic is good, but possible)
|
||||||
|
let err_str = e.to_string();
|
||||||
|
if err_str.contains("422") || err_str.contains("Duplicate") {
|
||||||
|
warn!("Duplicate rejected by Firefly: {}", tx.internal_id);
|
||||||
|
stats.duplicates += 1;
|
||||||
|
} else {
|
||||||
|
tracing::error!("Failed to create transaction: {}", e);
|
||||||
|
stats.errors += 1;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
stats.created += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
info!("Sync complete. Created: {}, Healed: {}, Duplicates: {}, Errors: {}",
|
||||||
|
stats.created, stats.healed, stats.duplicates, stats.errors);
|
||||||
|
}
|
||||||
|
|
||||||
|
info!("Synchronization finished.");
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use crate::core::ports::{MockTransactionSource, MockTransactionDestination, TransactionMatch};
|
||||||
|
use crate::core::models::{Account, BankTransaction};
|
||||||
|
use rust_decimal::Decimal;
|
||||||
|
use mockall::predicate::*;
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn test_sync_flow_create_new() {
|
||||||
|
let mut source = MockTransactionSource::new();
|
||||||
|
let mut dest = MockTransactionDestination::new();
|
||||||
|
|
||||||
|
// Source setup
|
||||||
|
source.expect_get_accounts()
|
||||||
|
.with(always()) // Match any argument
|
||||||
|
.returning(|_| Ok(vec![Account {
|
||||||
|
id: "src_1".to_string(),
|
||||||
|
iban: "NL01".to_string(),
|
||||||
|
currency: "EUR".to_string(),
|
||||||
|
}]));
|
||||||
|
|
||||||
|
let tx = BankTransaction {
|
||||||
|
internal_id: "tx1".into(),
|
||||||
|
date: NaiveDate::from_ymd_opt(2023, 1, 1).unwrap(),
|
||||||
|
amount: Decimal::new(100, 0),
|
||||||
|
currency: "EUR".into(),
|
||||||
|
foreign_amount: None,
|
||||||
|
foreign_currency: None,
|
||||||
|
description: "Test".into(),
|
||||||
|
counterparty_name: None,
|
||||||
|
counterparty_iban: None,
|
||||||
|
};
|
||||||
|
let tx_clone = tx.clone();
|
||||||
|
|
||||||
|
source.expect_get_transactions()
|
||||||
|
.returning(move |_, _, _| Ok(vec![tx.clone()]));
|
||||||
|
|
||||||
|
// Destination setup
|
||||||
|
dest.expect_get_active_account_ibans()
|
||||||
|
.returning(|| Ok(vec!["NL01".to_string()]));
|
||||||
|
|
||||||
|
dest.expect_resolve_account_id()
|
||||||
|
.returning(|_| Ok(Some("dest_1".into())));
|
||||||
|
|
||||||
|
dest.expect_get_last_transaction_date()
|
||||||
|
.returning(|_| Ok(Some(NaiveDate::from_ymd_opt(2022, 12, 31).unwrap())));
|
||||||
|
|
||||||
|
// 1. Find -> None
|
||||||
|
dest.expect_find_transaction()
|
||||||
|
.times(1)
|
||||||
|
.returning(|_, _| Ok(None));
|
||||||
|
|
||||||
|
// 2. Create -> Ok
|
||||||
|
dest.expect_create_transaction()
|
||||||
|
.with(eq("dest_1"), eq(tx_clone))
|
||||||
|
.times(1)
|
||||||
|
.returning(|_, _| Ok(()));
|
||||||
|
|
||||||
|
// Execution
|
||||||
|
let res = run_sync(&source, &dest, None, None, false).await;
|
||||||
|
assert!(res.is_ok());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn test_sync_flow_heal_existing() {
|
||||||
|
let mut source = MockTransactionSource::new();
|
||||||
|
let mut dest = MockTransactionDestination::new();
|
||||||
|
|
||||||
|
dest.expect_get_active_account_ibans()
|
||||||
|
.returning(|| Ok(vec!["NL01".to_string()]));
|
||||||
|
|
||||||
|
source.expect_get_accounts()
|
||||||
|
.with(always())
|
||||||
|
.returning(|_| Ok(vec![Account {
|
||||||
|
id: "src_1".to_string(),
|
||||||
|
iban: "NL01".to_string(),
|
||||||
|
currency: "EUR".to_string(),
|
||||||
|
}]));
|
||||||
|
|
||||||
|
source.expect_get_transactions()
|
||||||
|
.returning(|_, _, _| Ok(vec![
|
||||||
|
BankTransaction {
|
||||||
|
internal_id: "tx1".into(),
|
||||||
|
date: NaiveDate::from_ymd_opt(2023, 1, 1).unwrap(),
|
||||||
|
amount: Decimal::new(100, 0),
|
||||||
|
currency: "EUR".into(),
|
||||||
|
foreign_amount: None,
|
||||||
|
foreign_currency: None,
|
||||||
|
description: "Test".into(),
|
||||||
|
counterparty_name: None,
|
||||||
|
counterparty_iban: None,
|
||||||
|
}
|
||||||
|
]));
|
||||||
|
|
||||||
|
dest.expect_resolve_account_id().returning(|_| Ok(Some("dest_1".into())));
|
||||||
|
dest.expect_get_last_transaction_date().returning(|_| Ok(Some(NaiveDate::from_ymd_opt(2022, 12, 31).unwrap())));
|
||||||
|
|
||||||
|
// 1. Find -> Some(No External ID)
|
||||||
|
dest.expect_find_transaction()
|
||||||
|
.times(1)
|
||||||
|
.returning(|_, _| Ok(Some(TransactionMatch {
|
||||||
|
id: "ff_tx_1".to_string(),
|
||||||
|
has_external_id: false,
|
||||||
|
})));
|
||||||
|
|
||||||
|
// 2. Update -> Ok
|
||||||
|
dest.expect_update_transaction_external_id()
|
||||||
|
.with(eq("ff_tx_1"), eq("tx1"))
|
||||||
|
.times(1)
|
||||||
|
.returning(|_, _| Ok(()));
|
||||||
|
|
||||||
|
let res = run_sync(&source, &dest, None, None, false).await;
|
||||||
|
assert!(res.is_ok());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn test_sync_flow_dry_run() {
|
||||||
|
let mut source = MockTransactionSource::new();
|
||||||
|
let mut dest = MockTransactionDestination::new();
|
||||||
|
|
||||||
|
dest.expect_get_active_account_ibans()
|
||||||
|
.returning(|| Ok(vec!["NL01".to_string()]));
|
||||||
|
|
||||||
|
source.expect_get_accounts()
|
||||||
|
.with(always())
|
||||||
|
.returning(|_| Ok(vec![Account {
|
||||||
|
id: "src_1".to_string(),
|
||||||
|
iban: "NL01".to_string(),
|
||||||
|
currency: "EUR".to_string(),
|
||||||
|
}]));
|
||||||
|
|
||||||
|
let tx = BankTransaction {
|
||||||
|
internal_id: "tx1".into(),
|
||||||
|
date: NaiveDate::from_ymd_opt(2023, 1, 1).unwrap(),
|
||||||
|
amount: Decimal::new(100, 0),
|
||||||
|
currency: "EUR".into(),
|
||||||
|
foreign_amount: None,
|
||||||
|
foreign_currency: None,
|
||||||
|
description: "Test".into(),
|
||||||
|
counterparty_name: None,
|
||||||
|
counterparty_iban: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
source.expect_get_transactions()
|
||||||
|
.returning(move |_, _, _| Ok(vec![tx.clone()]));
|
||||||
|
|
||||||
|
dest.expect_resolve_account_id().returning(|_| Ok(Some("dest_1".into())));
|
||||||
|
dest.expect_get_last_transaction_date().returning(|_| Ok(Some(NaiveDate::from_ymd_opt(2022, 12, 31).unwrap())));
|
||||||
|
|
||||||
|
// 1. Find -> None (New transaction)
|
||||||
|
dest.expect_find_transaction()
|
||||||
|
.returning(|_, _| Ok(None));
|
||||||
|
|
||||||
|
// 2. Create -> NEVER Called (Dry Run)
|
||||||
|
dest.expect_create_transaction().never();
|
||||||
|
dest.expect_update_transaction_external_id().never();
|
||||||
|
|
||||||
|
let res = run_sync(&source, &dest, None, None, true).await;
|
||||||
|
assert!(res.is_ok());
|
||||||
|
}
|
||||||
|
}
|
||||||
74
banks2ff/src/main.rs
Normal file
74
banks2ff/src/main.rs
Normal file
@@ -0,0 +1,74 @@
|
|||||||
|
mod adapters;
|
||||||
|
mod core;
|
||||||
|
|
||||||
|
use clap::Parser;
|
||||||
|
use tracing::{info, error};
|
||||||
|
use crate::adapters::gocardless::client::GoCardlessAdapter;
|
||||||
|
use crate::adapters::firefly::client::FireflyAdapter;
|
||||||
|
use crate::core::sync::run_sync;
|
||||||
|
use gocardless_client::client::GoCardlessClient;
|
||||||
|
use firefly_client::client::FireflyClient;
|
||||||
|
use std::env;
|
||||||
|
use chrono::NaiveDate;
|
||||||
|
|
||||||
|
#[derive(Parser, Debug)]
|
||||||
|
#[command(author, version, about, long_about = None)]
|
||||||
|
struct Args {
|
||||||
|
/// Path to configuration file (optional)
|
||||||
|
#[arg(short, long)]
|
||||||
|
config: Option<String>,
|
||||||
|
|
||||||
|
/// Start date for synchronization (YYYY-MM-DD). Defaults to last transaction date + 1.
|
||||||
|
#[arg(short, long)]
|
||||||
|
start: Option<NaiveDate>,
|
||||||
|
|
||||||
|
/// End date for synchronization (YYYY-MM-DD). Defaults to yesterday.
|
||||||
|
#[arg(short, long)]
|
||||||
|
end: Option<NaiveDate>,
|
||||||
|
|
||||||
|
/// Dry run mode: Do not create or update transactions in Firefly III.
|
||||||
|
#[arg(long, default_value_t = false)]
|
||||||
|
dry_run: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::main]
|
||||||
|
async fn main() -> anyhow::Result<()> {
|
||||||
|
// Initialize logging
|
||||||
|
tracing_subscriber::fmt()
|
||||||
|
.with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
|
||||||
|
.init();
|
||||||
|
|
||||||
|
// Load environment variables
|
||||||
|
dotenvy::dotenv().ok();
|
||||||
|
|
||||||
|
let args = Args::parse();
|
||||||
|
|
||||||
|
info!("Starting banks2ff...");
|
||||||
|
if args.dry_run {
|
||||||
|
info!("DRY RUN MODE ENABLED: No changes will be made to Firefly III.");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Config Load
|
||||||
|
let gc_url = env::var("GOCARDLESS_URL").unwrap_or_else(|_| "https://bankaccountdata.gocardless.com".to_string());
|
||||||
|
let gc_id = env::var("GOCARDLESS_ID").expect("GOCARDLESS_ID not set");
|
||||||
|
let gc_key = env::var("GOCARDLESS_KEY").expect("GOCARDLESS_KEY not set");
|
||||||
|
|
||||||
|
let ff_url = env::var("FIREFLY_III_URL").expect("FIREFLY_III_URL not set");
|
||||||
|
let ff_key = env::var("FIREFLY_III_API_KEY").expect("FIREFLY_III_API_KEY not set");
|
||||||
|
|
||||||
|
// Clients
|
||||||
|
let gc_client = GoCardlessClient::new(&gc_url, &gc_id, &gc_key)?;
|
||||||
|
let ff_client = FireflyClient::new(&ff_url, &ff_key)?;
|
||||||
|
|
||||||
|
// Adapters
|
||||||
|
let source = GoCardlessAdapter::new(gc_client);
|
||||||
|
let destination = FireflyAdapter::new(ff_client);
|
||||||
|
|
||||||
|
// Run
|
||||||
|
match run_sync(&source, &destination, args.start, args.end, args.dry_run).await {
|
||||||
|
Ok(_) => info!("Sync completed successfully."),
|
||||||
|
Err(e) => error!("Sync failed: {}", e),
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
114
docs/architecture.md
Normal file
114
docs/architecture.md
Normal file
@@ -0,0 +1,114 @@
|
|||||||
|
# Architecture Documentation
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Banks2FF implements a **Hexagonal (Ports & Adapters) Architecture** to synchronize bank transactions from GoCardless to Firefly III. This architecture separates business logic from external concerns, making the system testable and maintainable.
|
||||||
|
|
||||||
|
## Workspace Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
banks2ff/
|
||||||
|
├── banks2ff/ # Main CLI application
|
||||||
|
│ └── src/
|
||||||
|
│ ├── core/ # Domain logic and models
|
||||||
|
│ ├── adapters/ # External service integrations
|
||||||
|
│ └── main.rs # CLI entry point
|
||||||
|
├── firefly-client/ # Firefly III API client library
|
||||||
|
├── gocardless-client/ # GoCardless API client library
|
||||||
|
└── docs/ # Architecture documentation
|
||||||
|
```
|
||||||
|
|
||||||
|
## Core Components
|
||||||
|
|
||||||
|
### 1. Domain Core (`banks2ff/src/core/`)
|
||||||
|
|
||||||
|
**models.rs**: Defines domain entities
|
||||||
|
- `BankTransaction`: Core transaction model with multi-currency support
|
||||||
|
- `Account`: Bank account representation
|
||||||
|
- Supports `foreign_amount` and `foreign_currency` for international transactions
|
||||||
|
|
||||||
|
**ports.rs**: Defines abstraction traits
|
||||||
|
- `TransactionSource`: Interface for fetching transactions (implemented by GoCardless adapter)
|
||||||
|
- `TransactionDestination`: Interface for storing transactions (implemented by Firefly adapter)
|
||||||
|
- Traits are mockable for isolated testing
|
||||||
|
|
||||||
|
**sync.rs**: Synchronization engine
|
||||||
|
- `run_sync()`: Orchestrates the entire sync process
|
||||||
|
- Implements "Healer" strategy for idempotency
|
||||||
|
- Smart date range calculation (Last Transaction Date + 1 to Yesterday)
|
||||||
|
|
||||||
|
### 2. Adapters (`banks2ff/src/adapters/`)
|
||||||
|
|
||||||
|
**gocardless/**: GoCardless integration
|
||||||
|
- `client.rs`: Wrapper for GoCardless client with token management
|
||||||
|
- `mapper.rs`: Converts GoCardless API responses to domain models
|
||||||
|
- `cache.rs`: Caches account mappings to reduce API calls
|
||||||
|
- Correctly handles multi-currency via `currencyExchange` array parsing
|
||||||
|
|
||||||
|
**firefly/**: Firefly III integration
|
||||||
|
- `client.rs`: Wrapper for Firefly client for transaction storage
|
||||||
|
- Maps domain models to Firefly API format
|
||||||
|
|
||||||
|
### 3. API Clients
|
||||||
|
|
||||||
|
Both clients are hand-crafted using `reqwest`:
|
||||||
|
- Strongly-typed DTOs for compile-time safety
|
||||||
|
- Custom error handling with `thiserror`
|
||||||
|
- Rate limit awareness and graceful degradation
|
||||||
|
|
||||||
|
## Synchronization Process
|
||||||
|
|
||||||
|
The "Healer" strategy ensures idempotency:
|
||||||
|
|
||||||
|
1. **Account Discovery**: Fetch active accounts from GoCardless
|
||||||
|
2. **Account Matching**: Match GoCardless accounts to Firefly asset accounts by IBAN
|
||||||
|
3. **Date Window**: Calculate sync range (Last Firefly transaction + 1 to Yesterday)
|
||||||
|
4. **Transaction Processing**:
|
||||||
|
- **Search**: Look for existing transaction using windowed heuristic (date ± 3 days, exact amount)
|
||||||
|
- **Heal**: If found without `external_id`, update with GoCardless transaction ID
|
||||||
|
- **Skip**: If found with matching `external_id`, ignore
|
||||||
|
- **Create**: If not found, create new transaction in Firefly
|
||||||
|
|
||||||
|
## Key Features
|
||||||
|
|
||||||
|
### Multi-Currency Support
|
||||||
|
- Parses `currencyExchange` array from GoCardless responses
|
||||||
|
- Calculates `foreign_amount = amount * exchange_rate`
|
||||||
|
- Maps to Firefly's `foreign_amount` and `foreign_currency_code` fields
|
||||||
|
|
||||||
|
### Rate Limit Management
|
||||||
|
- **Caching**: Stores `AccountId -> IBAN` mappings to reduce requisition calls
|
||||||
|
- **Token Reuse**: Maintains tokens until expiry to minimize auth requests
|
||||||
|
- **Graceful Handling**: Continues sync for other accounts when encountering 429 errors
|
||||||
|
|
||||||
|
### Idempotency
|
||||||
|
- GoCardless `transactionId` → Firefly `external_id` mapping
|
||||||
|
- Windowed duplicate detection prevents double-creation
|
||||||
|
- Historical transaction healing for pre-existing data
|
||||||
|
|
||||||
|
## Data Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
GoCardless API → GoCardlessAdapter → TransactionSource → SyncEngine → TransactionDestination → FireflyAdapter → Firefly API
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing Strategy
|
||||||
|
|
||||||
|
- **Unit Tests**: Core logic with `mockall` for trait mocking
|
||||||
|
- **Integration Tests**: API clients with `wiremock` for HTTP mocking
|
||||||
|
- **Fixture Testing**: Real JSON responses for adapter mapping validation
|
||||||
|
- **Isolation**: Business logic tested without external dependencies
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
- **Custom Errors**: `thiserror` for domain-specific error types
|
||||||
|
- **Propagation**: `anyhow` for error context across async boundaries
|
||||||
|
- **Graceful Degradation**: Rate limits and network issues don't crash entire sync
|
||||||
|
- **Structured Logging**: `tracing` for observability and debugging
|
||||||
|
|
||||||
|
## Configuration Management
|
||||||
|
|
||||||
|
- Environment variables loaded via `dotenvy`
|
||||||
|
- Workspace-level dependency management
|
||||||
|
- Feature flags for optional functionality
|
||||||
|
- Secure credential handling (no hardcoded secrets)
|
||||||
20
firefly-client/Cargo.toml
Normal file
20
firefly-client/Cargo.toml
Normal file
@@ -0,0 +1,20 @@
|
|||||||
|
[package]
|
||||||
|
name = "firefly-client"
|
||||||
|
version.workspace = true
|
||||||
|
edition.workspace = true
|
||||||
|
authors.workspace = true
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
reqwest = { version = "0.11", default-features = false, features = ["json", "rustls-tls"] }
|
||||||
|
serde = { workspace = true }
|
||||||
|
serde_json = { workspace = true }
|
||||||
|
thiserror = { workspace = true }
|
||||||
|
tracing = { workspace = true }
|
||||||
|
url = { workspace = true }
|
||||||
|
chrono = { workspace = true }
|
||||||
|
rust_decimal = { workspace = true }
|
||||||
|
|
||||||
|
[dev-dependencies]
|
||||||
|
wiremock = { workspace = true }
|
||||||
|
tokio = { workspace = true }
|
||||||
|
tokio-test = { workspace = true }
|
||||||
126
firefly-client/src/client.rs
Normal file
126
firefly-client/src/client.rs
Normal file
@@ -0,0 +1,126 @@
|
|||||||
|
use reqwest::{Client, Url};
|
||||||
|
use serde::de::DeserializeOwned;
|
||||||
|
use thiserror::Error;
|
||||||
|
use tracing::instrument;
|
||||||
|
use crate::models::{AccountArray, TransactionStore, TransactionArray, TransactionUpdate};
|
||||||
|
|
||||||
|
#[derive(Error, Debug)]
|
||||||
|
pub enum FireflyError {
|
||||||
|
#[error("Request failed: {0}")]
|
||||||
|
RequestFailed(#[from] reqwest::Error),
|
||||||
|
#[error("API Error: {0}")]
|
||||||
|
ApiError(String),
|
||||||
|
#[error("URL Parse Error: {0}")]
|
||||||
|
UrlParseError(#[from] url::ParseError),
|
||||||
|
}
|
||||||
|
|
||||||
|
pub struct FireflyClient {
|
||||||
|
base_url: Url,
|
||||||
|
client: Client,
|
||||||
|
access_token: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FireflyClient {
|
||||||
|
pub fn new(base_url: &str, access_token: &str) -> Result<Self, FireflyError> {
|
||||||
|
Ok(Self {
|
||||||
|
base_url: Url::parse(base_url)?,
|
||||||
|
client: Client::new(),
|
||||||
|
access_token: access_token.to_string(),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
#[instrument(skip(self))]
|
||||||
|
pub async fn get_accounts(&self, _iban: &str) -> Result<AccountArray, FireflyError> {
|
||||||
|
let mut url = self.base_url.join("/api/v1/accounts")?;
|
||||||
|
url.query_pairs_mut()
|
||||||
|
.append_pair("type", "asset");
|
||||||
|
|
||||||
|
self.get_authenticated(url).await
|
||||||
|
}
|
||||||
|
|
||||||
|
#[instrument(skip(self))]
|
||||||
|
pub async fn search_accounts(&self, query: &str) -> Result<AccountArray, FireflyError> {
|
||||||
|
let mut url = self.base_url.join("/api/v1/search/accounts")?;
|
||||||
|
url.query_pairs_mut()
|
||||||
|
.append_pair("query", query)
|
||||||
|
.append_pair("type", "asset")
|
||||||
|
.append_pair("field", "all");
|
||||||
|
|
||||||
|
self.get_authenticated(url).await
|
||||||
|
}
|
||||||
|
|
||||||
|
#[instrument(skip(self, transaction))]
|
||||||
|
pub async fn store_transaction(&self, transaction: TransactionStore) -> Result<(), FireflyError> {
|
||||||
|
let url = self.base_url.join("/api/v1/transactions")?;
|
||||||
|
|
||||||
|
let response = self.client.post(url)
|
||||||
|
.bearer_auth(&self.access_token)
|
||||||
|
.header("accept", "application/json")
|
||||||
|
.json(&transaction)
|
||||||
|
.send()
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
if !response.status().is_success() {
|
||||||
|
let status = response.status();
|
||||||
|
let text = response.text().await?;
|
||||||
|
return Err(FireflyError::ApiError(format!("Store Transaction Failed {}: {}", status, text)));
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[instrument(skip(self))]
|
||||||
|
pub async fn list_account_transactions(&self, account_id: &str, start: Option<&str>, end: Option<&str>) -> Result<TransactionArray, FireflyError> {
|
||||||
|
let mut url = self.base_url.join(&format!("/api/v1/accounts/{}/transactions", account_id))?;
|
||||||
|
{
|
||||||
|
let mut pairs = url.query_pairs_mut();
|
||||||
|
if let Some(s) = start {
|
||||||
|
pairs.append_pair("start", s);
|
||||||
|
}
|
||||||
|
if let Some(e) = end {
|
||||||
|
pairs.append_pair("end", e);
|
||||||
|
}
|
||||||
|
// Limit to 50, could be higher but safer to page if needed. For heuristic checks 50 is usually plenty per day range.
|
||||||
|
pairs.append_pair("limit", "50");
|
||||||
|
}
|
||||||
|
|
||||||
|
self.get_authenticated(url).await
|
||||||
|
}
|
||||||
|
|
||||||
|
#[instrument(skip(self, update))]
|
||||||
|
pub async fn update_transaction(&self, id: &str, update: TransactionUpdate) -> Result<(), FireflyError> {
|
||||||
|
let url = self.base_url.join(&format!("/api/v1/transactions/{}", id))?;
|
||||||
|
|
||||||
|
let response = self.client.put(url)
|
||||||
|
.bearer_auth(&self.access_token)
|
||||||
|
.header("accept", "application/json")
|
||||||
|
.json(&update)
|
||||||
|
.send()
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
if !response.status().is_success() {
|
||||||
|
let status = response.status();
|
||||||
|
let text = response.text().await?;
|
||||||
|
return Err(FireflyError::ApiError(format!("Update Transaction Failed {}: {}", status, text)));
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn get_authenticated<T: DeserializeOwned>(&self, url: Url) -> Result<T, FireflyError> {
|
||||||
|
let response = self.client.get(url)
|
||||||
|
.bearer_auth(&self.access_token)
|
||||||
|
.header("accept", "application/json")
|
||||||
|
.send()
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
if !response.status().is_success() {
|
||||||
|
let status = response.status();
|
||||||
|
let text = response.text().await?;
|
||||||
|
return Err(FireflyError::ApiError(format!("API request failed {}: {}", status, text)));
|
||||||
|
}
|
||||||
|
|
||||||
|
let data = response.json().await?;
|
||||||
|
Ok(data)
|
||||||
|
}
|
||||||
|
}
|
||||||
2
firefly-client/src/lib.rs
Normal file
2
firefly-client/src/lib.rs
Normal file
@@ -0,0 +1,2 @@
|
|||||||
|
pub mod client;
|
||||||
|
pub mod models;
|
||||||
81
firefly-client/src/models.rs
Normal file
81
firefly-client/src/models.rs
Normal file
@@ -0,0 +1,81 @@
|
|||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct AccountRead {
|
||||||
|
pub id: String,
|
||||||
|
pub attributes: Account,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct Account {
|
||||||
|
pub name: String,
|
||||||
|
pub iban: Option<String>,
|
||||||
|
#[serde(rename = "type")]
|
||||||
|
pub account_type: String,
|
||||||
|
pub active: Option<bool>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct AccountArray {
|
||||||
|
pub data: Vec<AccountRead>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct TransactionRead {
|
||||||
|
pub id: String,
|
||||||
|
pub attributes: Transaction,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct Transaction {
|
||||||
|
pub transactions: Vec<TransactionSplit>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct TransactionSplit {
|
||||||
|
pub date: String,
|
||||||
|
pub amount: String,
|
||||||
|
pub description: String,
|
||||||
|
pub external_id: Option<String>,
|
||||||
|
pub currency_code: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct TransactionArray {
|
||||||
|
pub data: Vec<TransactionRead>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct TransactionSplitStore {
|
||||||
|
#[serde(rename = "type")]
|
||||||
|
pub transaction_type: String,
|
||||||
|
pub date: String,
|
||||||
|
pub amount: String,
|
||||||
|
pub description: String,
|
||||||
|
pub source_id: Option<String>,
|
||||||
|
pub source_name: Option<String>,
|
||||||
|
pub destination_id: Option<String>,
|
||||||
|
pub destination_name: Option<String>,
|
||||||
|
pub currency_code: Option<String>,
|
||||||
|
pub foreign_amount: Option<String>,
|
||||||
|
pub foreign_currency_code: Option<String>,
|
||||||
|
pub external_id: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct TransactionStore {
|
||||||
|
pub transactions: Vec<TransactionSplitStore>,
|
||||||
|
pub apply_rules: Option<bool>,
|
||||||
|
pub fire_webhooks: Option<bool>,
|
||||||
|
pub error_if_duplicate_hash: Option<bool>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct TransactionUpdate {
|
||||||
|
pub transactions: Vec<TransactionSplitUpdate>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct TransactionSplitUpdate {
|
||||||
|
pub external_id: Option<String>,
|
||||||
|
}
|
||||||
62
firefly-client/tests/client_test.rs
Normal file
62
firefly-client/tests/client_test.rs
Normal file
@@ -0,0 +1,62 @@
|
|||||||
|
use firefly_client::client::FireflyClient;
|
||||||
|
use firefly_client::models::{TransactionStore, TransactionSplitStore};
|
||||||
|
use wiremock::matchers::{method, path, header};
|
||||||
|
use wiremock::{Mock, MockServer, ResponseTemplate};
|
||||||
|
use std::fs;
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn test_search_accounts() {
|
||||||
|
let mock_server = MockServer::start().await;
|
||||||
|
let fixture = fs::read_to_string("tests/fixtures/ff_accounts.json").unwrap();
|
||||||
|
|
||||||
|
Mock::given(method("GET"))
|
||||||
|
.and(path("/api/v1/search/accounts"))
|
||||||
|
.and(header("Authorization", "Bearer my-token"))
|
||||||
|
.respond_with(ResponseTemplate::new(200).set_body_string(fixture))
|
||||||
|
.mount(&mock_server)
|
||||||
|
.await;
|
||||||
|
|
||||||
|
let client = FireflyClient::new(&mock_server.uri(), "my-token").unwrap();
|
||||||
|
let accounts = client.search_accounts("NL01").await.unwrap();
|
||||||
|
|
||||||
|
assert_eq!(accounts.data.len(), 1);
|
||||||
|
assert_eq!(accounts.data[0].attributes.name, "Checking Account");
|
||||||
|
assert_eq!(accounts.data[0].attributes.iban.as_deref(), Some("NL01BANK0123456789"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn test_store_transaction() {
|
||||||
|
let mock_server = MockServer::start().await;
|
||||||
|
|
||||||
|
Mock::given(method("POST"))
|
||||||
|
.and(path("/api/v1/transactions"))
|
||||||
|
.and(header("Authorization", "Bearer my-token"))
|
||||||
|
.respond_with(ResponseTemplate::new(200))
|
||||||
|
.mount(&mock_server)
|
||||||
|
.await;
|
||||||
|
|
||||||
|
let client = FireflyClient::new(&mock_server.uri(), "my-token").unwrap();
|
||||||
|
|
||||||
|
let tx = TransactionStore {
|
||||||
|
transactions: vec![TransactionSplitStore {
|
||||||
|
transaction_type: "withdrawal".to_string(),
|
||||||
|
date: "2023-01-01".to_string(),
|
||||||
|
amount: "10.00".to_string(),
|
||||||
|
description: "Test".to_string(),
|
||||||
|
source_id: Some("1".to_string()),
|
||||||
|
destination_name: Some("Shop".to_string()),
|
||||||
|
currency_code: None,
|
||||||
|
foreign_amount: None,
|
||||||
|
foreign_currency_code: None,
|
||||||
|
external_id: None,
|
||||||
|
source_name: None,
|
||||||
|
destination_id: None,
|
||||||
|
}],
|
||||||
|
apply_rules: None,
|
||||||
|
fire_webhooks: None,
|
||||||
|
error_if_duplicate_hash: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
let result = client.store_transaction(tx).await;
|
||||||
|
assert!(result.is_ok());
|
||||||
|
}
|
||||||
22
firefly-client/tests/fixtures/ff_accounts.json
vendored
Normal file
22
firefly-client/tests/fixtures/ff_accounts.json
vendored
Normal file
@@ -0,0 +1,22 @@
|
|||||||
|
{
|
||||||
|
"data": [
|
||||||
|
{
|
||||||
|
"type": "accounts",
|
||||||
|
"id": "2",
|
||||||
|
"attributes": {
|
||||||
|
"name": "Checking Account",
|
||||||
|
"type": "asset",
|
||||||
|
"iban": "NL01BANK0123456789"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"meta": {
|
||||||
|
"pagination": {
|
||||||
|
"total": 1,
|
||||||
|
"count": 1,
|
||||||
|
"per_page": 20,
|
||||||
|
"current_page": 1,
|
||||||
|
"total_pages": 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
19
gocardless-client/Cargo.toml
Normal file
19
gocardless-client/Cargo.toml
Normal file
@@ -0,0 +1,19 @@
|
|||||||
|
[package]
|
||||||
|
name = "gocardless-client"
|
||||||
|
version.workspace = true
|
||||||
|
edition.workspace = true
|
||||||
|
authors.workspace = true
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
reqwest = { version = "0.11", default-features = false, features = ["json", "rustls-tls"] }
|
||||||
|
serde = { workspace = true }
|
||||||
|
serde_json = { workspace = true }
|
||||||
|
thiserror = { workspace = true }
|
||||||
|
tracing = { workspace = true }
|
||||||
|
url = { workspace = true }
|
||||||
|
chrono = { workspace = true }
|
||||||
|
|
||||||
|
[dev-dependencies]
|
||||||
|
wiremock = { workspace = true }
|
||||||
|
tokio = { workspace = true }
|
||||||
|
tokio-test = { workspace = true }
|
||||||
129
gocardless-client/src/client.rs
Normal file
129
gocardless-client/src/client.rs
Normal file
@@ -0,0 +1,129 @@
|
|||||||
|
use reqwest::{Client, Url};
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
use thiserror::Error;
|
||||||
|
use tracing::{debug, instrument};
|
||||||
|
use crate::models::{TokenResponse, PaginatedResponse, Requisition, Account, TransactionsResponse};
|
||||||
|
|
||||||
|
#[derive(Error, Debug)]
|
||||||
|
pub enum GoCardlessError {
|
||||||
|
#[error("Request failed: {0}")]
|
||||||
|
RequestFailed(#[from] reqwest::Error),
|
||||||
|
#[error("API Error: {0}")]
|
||||||
|
ApiError(String),
|
||||||
|
#[error("Serialization error: {0}")]
|
||||||
|
SerializationError(#[from] serde_json::Error),
|
||||||
|
#[error("URL Parse Error: {0}")]
|
||||||
|
UrlParseError(#[from] url::ParseError),
|
||||||
|
}
|
||||||
|
|
||||||
|
pub struct GoCardlessClient {
|
||||||
|
base_url: Url,
|
||||||
|
client: Client,
|
||||||
|
secret_id: String,
|
||||||
|
secret_key: String,
|
||||||
|
access_token: Option<String>,
|
||||||
|
access_expires_at: Option<chrono::DateTime<chrono::Utc>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Serialize)]
|
||||||
|
struct TokenRequest<'a> {
|
||||||
|
secret_id: &'a str,
|
||||||
|
secret_key: &'a str,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl GoCardlessClient {
|
||||||
|
pub fn new(base_url: &str, secret_id: &str, secret_key: &str) -> Result<Self, GoCardlessError> {
|
||||||
|
Ok(Self {
|
||||||
|
base_url: Url::parse(base_url)?,
|
||||||
|
client: Client::new(),
|
||||||
|
secret_id: secret_id.to_string(),
|
||||||
|
secret_key: secret_key.to_string(),
|
||||||
|
access_token: None,
|
||||||
|
access_expires_at: None,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
#[instrument(skip(self))]
|
||||||
|
pub async fn obtain_access_token(&mut self) -> Result<(), GoCardlessError> {
|
||||||
|
// Check if current token is still valid (with 60s buffer)
|
||||||
|
if let Some(expires) = self.access_expires_at {
|
||||||
|
if chrono::Utc::now() < expires - chrono::Duration::seconds(60) {
|
||||||
|
debug!("Access token is still valid");
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let url = self.base_url.join("/api/v2/token/new/")?;
|
||||||
|
let body = TokenRequest {
|
||||||
|
secret_id: &self.secret_id,
|
||||||
|
secret_key: &self.secret_key,
|
||||||
|
};
|
||||||
|
|
||||||
|
debug!("Requesting new access token");
|
||||||
|
let response = self.client.post(url)
|
||||||
|
.json(&body)
|
||||||
|
.send()
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
if !response.status().is_success() {
|
||||||
|
let status = response.status();
|
||||||
|
let text = response.text().await?;
|
||||||
|
return Err(GoCardlessError::ApiError(format!("Token request failed {}: {}", status, text)));
|
||||||
|
}
|
||||||
|
|
||||||
|
let token_resp: TokenResponse = response.json().await?;
|
||||||
|
self.access_token = Some(token_resp.access);
|
||||||
|
self.access_expires_at = Some(chrono::Utc::now() + chrono::Duration::seconds(token_resp.access_expires as i64));
|
||||||
|
debug!("Access token obtained");
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[instrument(skip(self))]
|
||||||
|
pub async fn get_requisitions(&self) -> Result<PaginatedResponse<Requisition>, GoCardlessError> {
|
||||||
|
let url = self.base_url.join("/api/v2/requisitions/")?;
|
||||||
|
self.get_authenticated(url).await
|
||||||
|
}
|
||||||
|
|
||||||
|
#[instrument(skip(self))]
|
||||||
|
pub async fn get_account(&self, id: &str) -> Result<Account, GoCardlessError> {
|
||||||
|
let url = self.base_url.join(&format!("/api/v2/accounts/{}/", id))?;
|
||||||
|
self.get_authenticated(url).await
|
||||||
|
}
|
||||||
|
|
||||||
|
#[instrument(skip(self))]
|
||||||
|
pub async fn get_transactions(&self, account_id: &str, date_from: Option<&str>, date_to: Option<&str>) -> Result<TransactionsResponse, GoCardlessError> {
|
||||||
|
let mut url = self.base_url.join(&format!("/api/v2/accounts/{}/transactions/", account_id))?;
|
||||||
|
|
||||||
|
{
|
||||||
|
let mut pairs = url.query_pairs_mut();
|
||||||
|
if let Some(from) = date_from {
|
||||||
|
pairs.append_pair("date_from", from);
|
||||||
|
}
|
||||||
|
if let Some(to) = date_to {
|
||||||
|
pairs.append_pair("date_to", to);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
self.get_authenticated(url).await
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn get_authenticated<T: for<'de> Deserialize<'de>>(&self, url: Url) -> Result<T, GoCardlessError> {
|
||||||
|
let token = self.access_token.as_ref().ok_or(GoCardlessError::ApiError("No access token available. Call obtain_access_token() first.".into()))?;
|
||||||
|
|
||||||
|
let response = self.client.get(url)
|
||||||
|
.bearer_auth(token)
|
||||||
|
.header("accept", "application/json")
|
||||||
|
.send()
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
if !response.status().is_success() {
|
||||||
|
let status = response.status();
|
||||||
|
let text = response.text().await?;
|
||||||
|
return Err(GoCardlessError::ApiError(format!("API request failed {}: {}", status, text)));
|
||||||
|
}
|
||||||
|
|
||||||
|
let data = response.json().await?;
|
||||||
|
Ok(data)
|
||||||
|
}
|
||||||
|
}
|
||||||
2
gocardless-client/src/lib.rs
Normal file
2
gocardless-client/src/lib.rs
Normal file
@@ -0,0 +1,2 @@
|
|||||||
|
pub mod client;
|
||||||
|
pub mod models;
|
||||||
95
gocardless-client/src/models.rs
Normal file
95
gocardless-client/src/models.rs
Normal file
@@ -0,0 +1,95 @@
|
|||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct TokenResponse {
|
||||||
|
pub access: String,
|
||||||
|
pub access_expires: i32,
|
||||||
|
pub refresh: Option<String>,
|
||||||
|
pub refresh_expires: Option<i32>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct Requisition {
|
||||||
|
pub id: String,
|
||||||
|
pub status: String,
|
||||||
|
pub accounts: Option<Vec<String>>,
|
||||||
|
pub reference: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct PaginatedResponse<T> {
|
||||||
|
pub count: Option<i32>,
|
||||||
|
pub next: Option<String>,
|
||||||
|
pub previous: Option<String>,
|
||||||
|
pub results: Vec<T>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct Account {
|
||||||
|
pub id: String,
|
||||||
|
pub created: Option<String>,
|
||||||
|
pub last_accessed: Option<String>,
|
||||||
|
pub iban: Option<String>,
|
||||||
|
pub institution_id: Option<String>,
|
||||||
|
pub status: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct TransactionsResponse {
|
||||||
|
pub transactions: TransactionBookedPending,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct TransactionBookedPending {
|
||||||
|
pub booked: Vec<Transaction>,
|
||||||
|
pub pending: Option<Vec<Transaction>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct Transaction {
|
||||||
|
#[serde(rename = "transactionId")]
|
||||||
|
pub transaction_id: Option<String>,
|
||||||
|
#[serde(rename = "bookingDate")]
|
||||||
|
pub booking_date: Option<String>,
|
||||||
|
#[serde(rename = "valueDate")]
|
||||||
|
pub value_date: Option<String>,
|
||||||
|
#[serde(rename = "transactionAmount")]
|
||||||
|
pub transaction_amount: TransactionAmount,
|
||||||
|
#[serde(rename = "currencyExchange")]
|
||||||
|
pub currency_exchange: Option<Vec<CurrencyExchange>>,
|
||||||
|
#[serde(rename = "creditorName")]
|
||||||
|
pub creditor_name: Option<String>,
|
||||||
|
#[serde(rename = "creditorAccount")]
|
||||||
|
pub creditor_account: Option<AccountDetails>,
|
||||||
|
#[serde(rename = "debtorName")]
|
||||||
|
pub debtor_name: Option<String>,
|
||||||
|
#[serde(rename = "debtorAccount")]
|
||||||
|
pub debtor_account: Option<AccountDetails>,
|
||||||
|
#[serde(rename = "remittanceInformationUnstructured")]
|
||||||
|
pub remittance_information_unstructured: Option<String>,
|
||||||
|
#[serde(rename = "proprietaryBankTransactionCode")]
|
||||||
|
pub proprietary_bank_transaction_code: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct TransactionAmount {
|
||||||
|
pub amount: String,
|
||||||
|
pub currency: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct CurrencyExchange {
|
||||||
|
#[serde(rename = "sourceCurrency")]
|
||||||
|
pub source_currency: Option<String>,
|
||||||
|
#[serde(rename = "exchangeRate")]
|
||||||
|
pub exchange_rate: Option<String>,
|
||||||
|
#[serde(rename = "unitCurrency")]
|
||||||
|
pub unit_currency: Option<String>,
|
||||||
|
#[serde(rename = "targetCurrency")]
|
||||||
|
pub target_currency: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct AccountDetails {
|
||||||
|
pub iban: Option<String>,
|
||||||
|
}
|
||||||
55
gocardless-client/src/tests/client_test.rs
Normal file
55
gocardless-client/src/tests/client_test.rs
Normal file
@@ -0,0 +1,55 @@
|
|||||||
|
use gocardless_client::client::GoCardlessClient;
|
||||||
|
use gocardless_client::models::TokenResponse;
|
||||||
|
use wiremock::matchers::{method, path};
|
||||||
|
use wiremock::{Mock, MockServer, ResponseTemplate};
|
||||||
|
use std::fs;
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn test_get_transactions_parsing() {
|
||||||
|
// 1. Setup WireMock
|
||||||
|
let mock_server = MockServer::start().await;
|
||||||
|
|
||||||
|
// Mock Token Endpoint
|
||||||
|
Mock::given(method("POST"))
|
||||||
|
.and(path("/api/v2/token/new/"))
|
||||||
|
.respond_with(ResponseTemplate::new(200).set_body_json(TokenResponse {
|
||||||
|
access: "fake_access_token".to_string(),
|
||||||
|
access_expires: 3600,
|
||||||
|
refresh: Some("fake_refresh".to_string()),
|
||||||
|
refresh_expires: Some(86400),
|
||||||
|
}))
|
||||||
|
.mount(&mock_server)
|
||||||
|
.await;
|
||||||
|
|
||||||
|
// Mock Transactions Endpoint
|
||||||
|
let fixture = fs::read_to_string("tests/fixtures/gc_transactions.json").unwrap();
|
||||||
|
|
||||||
|
Mock::given(method("GET"))
|
||||||
|
.and(path("/api/v2/accounts/ACC123/transactions/"))
|
||||||
|
.respond_with(ResponseTemplate::new(200).set_body_string(fixture))
|
||||||
|
.mount(&mock_server)
|
||||||
|
.await;
|
||||||
|
|
||||||
|
// 2. Run Client
|
||||||
|
let mut client = GoCardlessClient::new(&mock_server.uri(), "id", "key").unwrap();
|
||||||
|
client.obtain_access_token().await.unwrap();
|
||||||
|
|
||||||
|
let resp = client.get_transactions("ACC123", None, None).await.unwrap();
|
||||||
|
|
||||||
|
// 3. Assertions
|
||||||
|
assert_eq!(resp.transactions.booked.len(), 2);
|
||||||
|
|
||||||
|
let tx1 = &resp.transactions.booked[0];
|
||||||
|
assert_eq!(tx1.transaction_id.as_deref(), Some("TX123"));
|
||||||
|
assert_eq!(tx1.transaction_amount.amount, "100.00");
|
||||||
|
assert_eq!(tx1.transaction_amount.currency, "EUR");
|
||||||
|
|
||||||
|
let tx2 = &resp.transactions.booked[1];
|
||||||
|
assert_eq!(tx2.transaction_id.as_deref(), Some("TX124"));
|
||||||
|
assert_eq!(tx2.transaction_amount.amount, "-10.00");
|
||||||
|
|
||||||
|
// Verify Multi-Currency parsing
|
||||||
|
let exchange = tx2.currency_exchange.as_ref().unwrap();
|
||||||
|
assert_eq!(exchange[0].source_currency.as_deref(), Some("USD"));
|
||||||
|
assert_eq!(exchange[0].exchange_rate.as_deref(), Some("1.10"));
|
||||||
|
}
|
||||||
34
gocardless-client/src/tests/fixtures/gc_transactions.json
vendored
Normal file
34
gocardless-client/src/tests/fixtures/gc_transactions.json
vendored
Normal file
@@ -0,0 +1,34 @@
|
|||||||
|
{
|
||||||
|
"transactions": {
|
||||||
|
"booked": [
|
||||||
|
{
|
||||||
|
"transactionId": "TX123",
|
||||||
|
"bookingDate": "2023-10-01",
|
||||||
|
"transactionAmount": {
|
||||||
|
"amount": "100.00",
|
||||||
|
"currency": "EUR"
|
||||||
|
},
|
||||||
|
"debtorName": "John Doe",
|
||||||
|
"remittanceInformationUnstructured": "Payment for services"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"transactionId": "TX124",
|
||||||
|
"bookingDate": "2023-10-02",
|
||||||
|
"transactionAmount": {
|
||||||
|
"amount": "-10.00",
|
||||||
|
"currency": "EUR"
|
||||||
|
},
|
||||||
|
"currencyExchange": [
|
||||||
|
{
|
||||||
|
"sourceCurrency": "USD",
|
||||||
|
"exchangeRate": "1.10",
|
||||||
|
"targetCurrency": "EUR"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"creditorName": "US Store",
|
||||||
|
"remittanceInformationUnstructured": "US Purchase"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"pending": []
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,129 +1,137 @@
|
|||||||
# Implementation Plan: Bank2FF Refactoring
|
# Implementation Plan: Bank2FF Refactoring
|
||||||
|
|
||||||
## 1. Objective
|
## 1. Objective
|
||||||
Refactor the `bank2ff` application from a prototype script into a robust, testable, and observable production-grade CLI tool. The application must synchronize bank transactions from GoCardless to Firefly III, ensuring:
|
Refactor the `bank2ff` application from a prototype script into a robust, testable, and observable production-grade CLI tool. The application must synchronize bank transactions from GoCardless to Firefly III.
|
||||||
- **Clean Architecture**: Decoupling of business logic from API clients.
|
|
||||||
- **Testability**: Ability to unit test core logic without external dependencies.
|
|
||||||
- **Multi-Currency Support**: Accurate capturing of foreign amounts and currencies.
|
|
||||||
- **Idempotency**: Preventing duplicate transactions in Firefly III.
|
|
||||||
|
|
||||||
## 2. Architecture: Hexagonal (Ports & Adapters)
|
**Key Constraints:**
|
||||||
|
- **Multi-Crate Workspace**: Separate crates for the CLI application and the API clients.
|
||||||
|
- **Hand-Crafted Clients**: No autogenerated code. Custom, strongly-typed clients for better UX.
|
||||||
|
- **Clean Architecture**: Hexagonal architecture within the main application.
|
||||||
|
- **Multi-Currency**: Accurate handling of foreign amounts.
|
||||||
|
- **Test-Driven**: Every component must be testable from the start.
|
||||||
|
- **Observability**: Structured logging (tracing) throughout the stack.
|
||||||
|
- **Healer Strategy**: Detect and heal historical duplicates that lack external IDs.
|
||||||
|
- **Dry Run**: Safe mode to preview changes.
|
||||||
|
- **Rate Limit Handling**: Smart caching and graceful skipping to respect 4 requests/day limits.
|
||||||
|
|
||||||
The application will be structured into three distinct layers:
|
## 2. Architecture
|
||||||
1. **Core (Domain)**: Pure Rust, no external API dependencies. Defines the `BankTransaction` model and the `Ports` (traits) for interacting with the world.
|
|
||||||
2. **Adapters**: Implementations of the Ports.
|
|
||||||
- `GoCardlessAdapter`: Implements `TransactionSource`.
|
|
||||||
- `FireflyAdapter`: Implements `TransactionDestination`.
|
|
||||||
3. **Application**: Configuration, CLI parsing, and wiring (`main.rs`).
|
|
||||||
|
|
||||||
### Directory Structure
|
### Workspace Structure
|
||||||
|
The project uses a Cargo Workspace with three members:
|
||||||
|
|
||||||
|
1. `gocardless-client`: A reusable library crate wrapping the GoCardless Bank Account Data API v2.
|
||||||
|
2. `firefly-client`: A reusable library crate wrapping the Firefly III API v6.4.4.
|
||||||
|
3. `banks2ff`: The main CLI application containing the Domain Core and Adapters that use the client libraries.
|
||||||
|
|
||||||
|
### Directory Layout
|
||||||
```text
|
```text
|
||||||
bank2ff/src/
|
root/
|
||||||
├── core/
|
├── Cargo.toml # Workspace definition
|
||||||
│ ├── mod.rs
|
├── gocardless-client/ # Crate 1
|
||||||
│ ├── models.rs # Domain entities (BankTransaction, Account)
|
│ ├── Cargo.toml
|
||||||
│ ├── ports.rs # Traits (TransactionSource, TransactionDestination)
|
│ └── src/
|
||||||
│ └── sync.rs # Core business logic (The "Use Case")
|
│ ├── lib.rs
|
||||||
├── adapters/
|
│ ├── client.rs # Reqwest client logic
|
||||||
│ ├── mod.rs
|
│ ├── models.rs # Request/Response DTOs
|
||||||
│ ├── gocardless/
|
│ └── tests/ # Unit/Integration tests with mocks
|
||||||
│ │ ├── mod.rs
|
├── firefly-client/ # Crate 2
|
||||||
│ │ ├── client.rs # Wrapper around generated client (Auth/RateLimits)
|
│ ├── Cargo.toml
|
||||||
│ │ └── mapper.rs # Logic to map API response -> Domain Model
|
│ └── src/
|
||||||
│ └── firefly/
|
│ ├── lib.rs
|
||||||
│ │ ├── mod.rs
|
│ ├── client.rs # Reqwest client logic
|
||||||
│ │ └── client.rs # Implementation of TransactionDestination
|
│ ├── models.rs # Request/Response DTOs
|
||||||
└── main.rs # Entry point, wiring, and config loading
|
│ └── tests/ # Unit/Integration tests with mocks
|
||||||
|
└── banks2ff/ # Crate 3 (Main App)
|
||||||
|
├── Cargo.toml
|
||||||
|
└── src/
|
||||||
|
├── main.rs
|
||||||
|
├── core/ # Domain
|
||||||
|
│ ├── models.rs
|
||||||
|
│ ├── ports.rs # Traits (Mockable)
|
||||||
|
│ ├── sync.rs # Logic (Tested with mocks)
|
||||||
|
│ └── tests/ # Unit tests for logic
|
||||||
|
└── adapters/ # Integration Layers
|
||||||
|
├── gocardless/
|
||||||
|
│ ├── client.rs # Uses gocardless-client & Cache
|
||||||
|
│ ├── cache.rs # JSON Cache for Account details
|
||||||
|
│ └── mapper.rs
|
||||||
|
└── firefly/
|
||||||
|
└── client.rs # Uses firefly-client
|
||||||
```
|
```
|
||||||
|
|
||||||
## 3. Core definitions
|
## 3. Observability Strategy
|
||||||
|
- **Tracing**: All crates will use the `tracing` crate.
|
||||||
|
- **Spans**:
|
||||||
|
- `client` crates: Create spans for every HTTP request (method, URL).
|
||||||
|
- `banks2ff` adapter: Create spans for "Fetching transactions".
|
||||||
|
- `banks2ff` sync: Create a span per account synchronization.
|
||||||
|
- **Context**: IDs (account, transaction) must be attached to log events.
|
||||||
|
|
||||||
### `src/core/models.rs`
|
## 4. Testing Strategy
|
||||||
The domain model must support multi-currency data.
|
- **Client Crates**:
|
||||||
|
- Use `wiremock` to mock the HTTP server.
|
||||||
|
- Test parsing of *real* JSON responses (saved in `tests/fixtures/`).
|
||||||
|
- Verify correct request construction (Headers, Auth, Body).
|
||||||
|
- **Core (`banks2ff`)**:
|
||||||
|
- Use `mockall` to mock `TransactionSource` and `TransactionDestination` traits.
|
||||||
|
- Unit test `sync::run_sync` logic (filtering, flow control) without any I/O.
|
||||||
|
- **Adapters (`banks2ff`)**:
|
||||||
|
- Test mapping logic (Client DTO -> Domain Model) using unit tests with fixtures.
|
||||||
|
|
||||||
```rust
|
## 5. Implementation Steps
|
||||||
pub struct BankTransaction {
|
|
||||||
pub internal_id: String, // Source ID (GoCardless transactionId)
|
|
||||||
pub date: NaiveDate, // Booking date
|
|
||||||
pub amount: Decimal, // Amount in account currency
|
|
||||||
pub currency: String, // Account currency code (e.g., EUR)
|
|
||||||
pub foreign_amount: Option<Decimal>, // Original amount (if currency exchange occurred)
|
|
||||||
pub foreign_currency: Option<String>,// Original currency code
|
|
||||||
pub description: String,
|
|
||||||
pub counterparty_name: Option<String>,
|
|
||||||
pub counterparty_iban: Option<String>,
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### `src/core/ports.rs`
|
### Phase 1: Infrastructure & Workspace
|
||||||
Traits to decouple the architecture.
|
- [x] **Setup**: Initialize `gocardless-client` and `firefly-client` crates. Update root `Cargo.toml`.
|
||||||
|
- [x] **Dependencies**: Add `reqwest`, `serde`, `thiserror`, `url`, `tracing`.
|
||||||
|
- [x] **Test Deps**: Add `wiremock`, `tokio-test`, `serde_json` (dev-dependencies).
|
||||||
|
|
||||||
```rust
|
### Phase 2: Core (`banks2ff`)
|
||||||
#[async_trait]
|
- [x] **Definitions**: Implement `models.rs` and `ports.rs` in `banks2ff`.
|
||||||
pub trait TransactionSource: Send + Sync {
|
- [x] **Mocks**: Add `mockall` attribute to ports for easier testing.
|
||||||
async fn get_accounts(&self) -> Result<Vec<Account>>;
|
|
||||||
async fn get_transactions(&self, account_id: &str, start: NaiveDate, end: NaiveDate) -> Result<Vec<BankTransaction>>;
|
|
||||||
}
|
|
||||||
|
|
||||||
#[async_trait]
|
### Phase 3: GoCardless Client Crate
|
||||||
pub trait TransactionDestination: Send + Sync {
|
- [x] **Models**: Define DTOs in `gocardless-client/src/models.rs`.
|
||||||
async fn resolve_account_id(&self, iban: &str) -> Result<Option<String>>;
|
- [x] **Fixtures**: Create `tests/fixtures/gc_transactions.json` (real example data).
|
||||||
async fn ingest_transactions(&self, account_id: &str, transactions: Vec<BankTransaction>) -> Result<IngestResult>;
|
- [x] **Client**: Implement `GoCardlessClient`.
|
||||||
}
|
- [x] **Tests**: Write `tests/client_test.rs` using `wiremock` to serve the fixture and verify the client parses it correctly.
|
||||||
```
|
|
||||||
|
|
||||||
## 4. Implementation Steps
|
### Phase 4: GoCardless Adapter (`banks2ff`)
|
||||||
|
- [x] **Implementation**: Implement `TransactionSource`.
|
||||||
|
- [x] **Logic**: Handle **Multi-Currency** (inspect `currencyExchange`).
|
||||||
|
- [x] **Tests**: Unit test the *mapping logic* specifically. Input: GC Client DTO. Output: Domain Model. Assert foreign amounts are correct.
|
||||||
|
- [x] **Optimization**: Implement "Firefly Leading" strategy (only fetch wanted accounts).
|
||||||
|
- [x] **Optimization**: Implement Account Cache & Rate Limit handling.
|
||||||
|
|
||||||
### Phase 1: Infrastructure & Core
|
### Phase 5: Firefly Client Crate
|
||||||
1. **Setup**: Create the directory structure.
|
- [x] **Models**: Define DTOs in `firefly-client/src/models.rs`.
|
||||||
2. **Dependencies**: Add `anyhow`, `thiserror`, `tracing`, `tracing-subscriber`, `async-trait`, `rust_decimal`, `chrono`.
|
- [x] **Fixtures**: Create `tests/fixtures/ff_store_transaction.json`.
|
||||||
3. **Core**: Implement `models.rs` and `ports.rs`.
|
- [x] **Client**: Implement `FireflyClient`.
|
||||||
|
- [x] **Tests**: Write `tests/client_test.rs` using `wiremock` to verify auth headers and body serialization.
|
||||||
|
|
||||||
### Phase 2: GoCardless Adapter (Source)
|
### Phase 6: Firefly Adapter (`banks2ff`)
|
||||||
1. **Token Management**: Create a wrapper struct that holds the `gocardless_bankaccount_data_api` configuration and manages the Access/Refresh token lifecycle. It should check validity before every request.
|
- [x] **Implementation**: Implement `TransactionDestination`.
|
||||||
2. **Mapping Logic**:
|
- [x] **Logic**: Set `external_id`, handle Credit/Debit swap.
|
||||||
- Map `transactionAmount.amount` -> `BankTransaction.amount`.
|
- [x] **Tests**: Unit test mapping logic. Verify `external_id` is populated.
|
||||||
- **Multi-Currency**: Inspect `currencyExchange`.
|
- [x] **Update**: Refactor for "Healer" strategy (split `ingest` into `find`, `create`, `update`).
|
||||||
- If present, map `sourceCurrency` -> `foreign_currency`.
|
|
||||||
- Calculate `foreign_amount` using `exchangeRate` if `instructedAmount` is missing.
|
|
||||||
- Map `transactionId` -> `internal_id`.
|
|
||||||
3. **Trait Implementation**: Implement `TransactionSource`.
|
|
||||||
|
|
||||||
### Phase 3: Firefly Adapter (Destination)
|
### Phase 7: Synchronization Engine
|
||||||
1. **Client Wrapper**: Initialize `firefly_iii_api` client with API Key.
|
- [x] **Logic**: Implement `banks2ff::core::sync::run_sync` with "Healer" logic.
|
||||||
2. **Resolution**: Implement `resolve_account_id` by querying Firefly accounts by IBAN (using the search/list endpoint).
|
- Check Destination for existing transaction (Windowed Search).
|
||||||
3. **Ingestion**:
|
- If found without ID: Heal (Update).
|
||||||
- Map `BankTransaction` to `TransactionSplitStore`.
|
- If found with ID: Skip.
|
||||||
- **Crucial**: Set `external_id` = `BankTransaction.internal_id`.
|
- If not found: Create.
|
||||||
- Handle Credits vs Debits (Swap Source/Destination logic).
|
- [x] **Smart Defaults**: Implement default start date (Last Firefly Date + 1) and end date (Yesterday).
|
||||||
- Call `store_transaction` endpoint.
|
- [x] **Tests**: Update unit tests for the new flow.
|
||||||
- Handle 422/409 errors gracefully (log duplicates).
|
|
||||||
|
|
||||||
### Phase 4: Synchronization Engine
|
### Phase 8: Wiring & CLI
|
||||||
1. Implement `src/core/sync.rs`.
|
- [x] **CLI**: Add `-s/--start` and `-e/--end` arguments.
|
||||||
2. Logic:
|
- [x] **CLI**: Add `--dry-run` argument.
|
||||||
- Fetch accounts from Source.
|
- [x] **Wiring**: Pass these arguments to the sync engine.
|
||||||
- For each account, find Destination ID.
|
- [x] **Observability**: Initialize `tracing_subscriber` with env filter.
|
||||||
- Fetch transactions (default: last 30 days).
|
- [x] **Config**: Load Env vars.
|
||||||
- Filter/Process (optional).
|
|
||||||
- Ingest to Destination.
|
|
||||||
- Log statistics using `tracing`.
|
|
||||||
|
|
||||||
### Phase 5: Wiring
|
## 6. Multi-Currency Logic
|
||||||
1. Refactor `main.rs`.
|
- **GoCardless Adapter**:
|
||||||
2. Initialize `tracing` for structured logging.
|
|
||||||
3. Load Config (Env variables).
|
|
||||||
4. Instantiate Adapters.
|
|
||||||
5. Run the Sync Engine.
|
|
||||||
|
|
||||||
## 5. Multi-Currency Handling Specifics
|
|
||||||
- **GoCardless Spec**: The `currencyExchange` array contains the details.
|
|
||||||
- **Logic**:
|
|
||||||
- If `currencyExchange` is not null/empty:
|
|
||||||
- `foreign_currency` = `currencyExchange[0].sourceCurrency`
|
- `foreign_currency` = `currencyExchange[0].sourceCurrency`
|
||||||
- `foreign_amount` = `amount` * `currencyExchange[0].exchangeRate` (Validation: check if `unitCurrency` affects this).
|
- `foreign_amount` = `amount` * `currencyExchange[0].exchangeRate`
|
||||||
- Ensure `Decimal` precision is handled correctly.
|
- Test this math explicitly in Phase 4 tests.
|
||||||
|
|
||||||
## 6. Testability & Observability
|
|
||||||
- **Tests**: Write a unit test for `core/sync.rs` using Mock structs for Source/Destination to verify the flow logic.
|
|
||||||
- **Logs**: Use `tracing::info!` for high-level progress ("Synced Account X") and `tracing::debug!` for details ("Transaction Y mapped to Z").
|
|
||||||
|
|||||||
Reference in New Issue
Block a user