Sébastien AucanteBackend Engineer and AI Solutions

02 · AI Engineering / Retrieval Evaluation

SecureBank-AI — Securing a RAG platform with RBAC and JWT

A hybrid RAG platform combining JWT authentication and RBAC access control with a reproducible comparison of BM25, vector search, Reciprocal Rank Fusion and reranking.

Synthetic documents
30
Evaluation questions
40
Indexed chunks
172

Engineering question

How can we design and evaluate a relevant RAG system while integrating it into a reliable application with controlled access?

Application security

Application Security & Access Control

SecureBank-AI goes beyond comparing document retrieval engines. The project also includes an application-security layer for user identity, feature access and authentication data.

This complements the AI Engineering work: a RAG system must fit into a structured application with protected RAG APIs, controlled sessions and explicit authorization.

  1. 01

    Authentication & authorization

    JWT · RBAC · Protected RAG APIs

    JWT validation checks signature, expiry, issuer and audience before identifying the user. Explicit FastAPI role dependencies then authorize access to RAG resources: authentication establishes identity, while authorization determines the permitted action. The environment-configured demo account uses the analyst role.

  2. 02

    Credentials & sessions

    bcrypt · HttpOnly cookie · CSRF

    Passwords are hashed with bcrypt. Session and CSRF tokens are generated separately; session tokens are stored as hashes in SQLite, expire and are invalidated on logout. The session cookie is HttpOnly; SESSION_COOKIE_SECURE and SESSION_COOKIE_SAMESITE configure its transport policy. The frontend sends cookies with credentials: "include" and supplies the CSRF token for mutable requests.

  3. 03

    API protection & configuration

    Trusted origins · Environment variables · Secrets

    Login origins are checked against the configured allowlist, and cookie-authenticated mutable requests require CSRF validation. Failed logins are rate-limited. AUTH_TRUSTED_ORIGINS, JWT_SECRET_KEY and DEMO_USER_PASSWORD keep deployment-sensitive configuration outside the public interface; no secret value is exposed here.

  4. 04

    User management & persistence

    SQLite · User repository · Role association

    SQLiteUserRepository persists accounts, password hashes, activation state and role associations. At startup, an optional demo account is initialized idempotently from environment configuration with the analyst role, without replacing an existing account.

Security controls from identity to protected RAG resources.
  1. User requestIdentity and credentials
  2. AuthenticationJWT validation or cookie session
  3. AuthorizationRBAC and role dependency
  4. Protected resourcesRAG and PII APIs
PersistenceSQLiteUsers · Roles · Hashed sessions
Runtime configurationEnvironmentTrusted origins · Cookie policy · Secrets
01

AI Engineering

Hybrid RAG, retrieval strategies, benchmarking and relevance evaluation.

02

Software Engineering

FastAPI, React, TypeScript, REST APIs, persistence, testing and deployment.

03

Application Security

JWT authentication, RBAC, password hashing, session management and access control.

SecureBank-AI combines RAG experimentation with a full-stack architecture that includes authentication, authorization and user-management mechanisms.

The answer starts upstream

A language model can only ground its response in the context it receives. If retrieval misses the relevant passage, ranks weak evidence too highly or returns redundant sources, generation begins with an information deficit.

SecureBank-AI treats retrieval as an observable engineering subsystem. Four alternative engines run against one experimental protocol so lexical, semantic, hybrid and reranked behavior can be compared rather than assumed.

Retrieval architecture

Documents are split into chunks, indexed for lexical and vector search, then queried through one of four evaluation configurations. RRF combines rankings; FlashRank is an optional reranking stage for the fourth configuration only.

Alternative retrieval configurations leading to retrieved context and optional downstream generation.
Question
Indexed chunks

Alternative configurations

Lexical retrievalBM25
Vector retrievalQdrantparaphrase-multilingual-MiniLM-L12-v2
Rank fusionRRFBM25 + Qdrant
Reranking · optionalRRF + FlashRank
Retrieved contextDownstream LLM integration
  1. 01

    BM25

    A lexical baseline that scores passages from term relevance.

  2. 02

    Qdrant

    Semantic vector search using paraphrase-multilingual-MiniLM-L12-v2 embeddings.

  3. 03

    RRF

    Reciprocal Rank Fusion combines the independent BM25 and Qdrant rankings.

  4. 04

    RRF + FlashRank

    FlashRank reranks the hybrid candidate set as a separate experimental configuration.

Evaluation methodology

RAG v2: one corpus, four configurations

The benchmark uses 30 synthetic documents, 40 questions and 172 indexed chunks. Qwen judges 260 question-document pairs to construct experimental relevance annotations; 38 of 40 questions have at least one document judged relevant.

The same benchmark exposes ranking quality, retrieval success, source and evidence coverage, diversity, latency and errors across all four engines. Results are descriptive observations for this corpus and configuration.

LLM-generated relevance judgments are experimental annotations, not systematic human validation.

  • RAG v2
  • 30 documents
  • 40 questions
  • 172 chunks
  • 4 strategies
  • 260 judged pairs
  • 38/40 with relevant evidence

Results

RRF is the strongest overall configuration on this benchmark

RRF leads the observed ranking and retrieval metrics while keeping average latency near 15 ms. This is a benchmark result, not an artificial aggregate score or a production guarantee.

Evaluated engineEngine 3 / 4
Primary resultRRF

Success@3 89.5%

Precision@1
65.8%
Recall@3
69.8%
MAP@3
61.1%
MRR@3
76.3%
nDCG@3
64.3%
Average latency
15.0 ms
MetricBM25QdrantRRFRRF + FlashRank
Precision@152.6%63.2%65.8%52.6%
Recall@358.9%63.0%69.8%60.6%
MAP@350.5%56.4%61.1%50.9%
MRR@364.9%72.4%76.3%66.7%
nDCG@356.4%56.1%64.3%55.0%
Success@378.9%81.6%89.5%84.2%
Source Hit@389.5%86.8%89.5%89.5%
Evidence Hit@378.9%81.6%89.5%84.2%
Average latency0.4 ms14.0 ms15.0 ms421.6 ms
P95 latency0.7 ms18.3 ms17.2 ms541.9 ms
View all 22 indicators
MetricBM25QdrantRRFRRF + FlashRank
Precision@152.6%63.2%65.8%52.6%
Precision@342.1%46.5%50.0%42.1%
Recall@127.3%30.8%31.2%26.8%
Recall@358.9%63.0%69.8%60.6%
F1@134.2%39.9%40.7%33.6%
F1@346.9%51.3%55.4%46.8%
MAP@350.5%56.4%61.1%50.9%
Success@152.6%63.2%65.8%52.6%
Success@378.9%81.6%89.5%84.2%
Source Hit@171.1%71.1%78.9%63.2%
Source Hit@389.5%86.8%89.5%89.5%
Evidence Hit@152.6%63.2%65.8%52.6%
Evidence Hit@378.9%81.6%89.5%84.2%
MRR@364.9%72.4%76.3%66.7%
nDCG@356.4%56.1%64.3%55.0%
Average latency0.4 ms14.0 ms15.0 ms421.6 ms
P50 latency0.4 ms13.0 ms15.6 ms422.7 ms
P95 latency0.7 ms18.3 ms17.2 ms541.9 ms
Error rate0%0%0%0%
Average returned chunks3333
Distinct documents@32.372.292.212.45
Source diversity@378.9%76.3%73.7%81.6%

Engineering trade-offs

01

Reranking was not automatically beneficial

Against RRF, FlashRank reduced MAP@3 from 61.1% to 50.9%, nDCG@3 from 64.3% to 55.0% and Success@3 from 89.5% to 84.2%. Average latency rose from 15.0 ms to 421.6 ms—about 28×. This observed trade-off is specific to the target corpus, benchmark and configuration.

02

BM25 remains a useful baseline

BM25 reached 89.5% Source Hit@3 at 0.4 ms average latency. Lexical retrieval remains a credible low-latency option; vector retrieval is not inherently superior.

03

Hybrid retrieval earned its place

MAP@3 progressed from 50.5% for BM25 and 56.4% for Qdrant to 61.1% for RRF. The observation supports evaluating lexical, semantic and hybrid approaches instead of selecting an engine on assumptions.

Full-stack implementation

The evaluation framework is part of a complete web application, with explicit boundaries between retrieval services, product interface, deployment and quality controls.

Backend

  • Python
  • FastAPI
  • REST API
  • SQLite

Frontend

  • React
  • TypeScript
  • Vite
  • Analytical dashboard
  • Multilingual interface
  • Error and loading-state handling

Deployment

  • Docker
  • Render backend
  • Vercel frontend
  • Environment-based configuration
  • API routing and proxy configuration

Quality

  • Pytest
  • Vitest
  • ESLint
  • Ruff
  • Automated and regression testing

Benchmark dashboard

The dashboard makes retrieval evaluation inspectable rather than hiding it behind a chat interface.

Reference snapshots keep the analytical view usable when the backend is unavailable. They are resilience artifacts, not live measurements.

SecureBank-AI welcome dashboard
Welcome and synthetic banking knowledge overview.
SecureBank-AI document assistant interface
Document question interface and grounded-answer workspace.
SecureBank-AI retrieval evaluation dashboard
RAG v2 engine selection and retrieval metrics dashboard.
SecureBank-AI service information page
Service boundaries, retrieval workflow and frequently asked questions.

Methodology and limitations

Possible next experiments

These are future directions, not implemented features.

Design → Experiment → Measure → Compare → Improve

SecureBank-AI turns retrieval choices into testable engineering decisions. The benchmark does not certify a banking product; it creates the evidence needed to choose, challenge and improve a RAG configuration.