Working with Heatmap, I came to understand just how critical data is to businesses. It tells them which products to market, how to market them, where to reach their audience, and when to strike. But here's what most founders, product managers, and engineers get wrong: they treat data collection as something to "figure out later." It's the feature that always gets pushed to the next sprint. The tracking ticket that sits at the bottom of the backlog.

That mindset is costing you more than you think.

You Can't Optimise What You Don't Measure

Imagine launching a product and realizing, three months in, that you have no idea why users are dropping off after onboarding. You don't know which feature keeps them coming back. You can't tell whether that redesign you shipped actually improved anything. You're making decisions based on gut feeling, not evidence.

This is the reality for teams that treat analytics as an afterthought. By the time they start tracking, they've already lost months, sometimes years, of behavioral data that could have shaped a better product. You can't go back and collect data you never instrumented for. That gap in your understanding is permanent.

Data collection isn't a feature. It's infrastructure. And like all infrastructure, it needs to be in place before you start building on top of it.

Start Collecting Before You Think You Need It

There's a common trap in early-stage product development: "We'll add analytics once we have enough users." This sounds reasonable, but it's backwards. The earliest users are often your most valuable source of insight. They're the ones navigating your product with fresh eyes, hitting friction points you've gone blind to, and showing you through their behavior what your product actually is versus what you think it is.

Here's what early data collection gives you that nothing else can. A baseline to measure every future change against. The ability to validate or kill assumptions before they become embedded in your architecture. Real user journeys, not the idealized flows you drew on a whiteboard. And early warning signals when something is broken, confusing, or just not working the way you expected.

You don't need a sophisticated analytics stack on day one. Even basic event tracking like page views, button clicks, session duration, and drop-off points gives you a solid foundation to build on. Tools like PostHog offer generous free tiers and can be set up in minutes, so there's really no excuse to skip this step. The important thing is that the foundation exists.

Data Tells You What Users Won't

Users will tell you what they think they want. Data tells you what they actually do. There's a well-known gap between stated preferences and revealed preferences, and every product team has bumped into this. A user says they love a feature in an interview, but the data shows they've used it exactly once. Another user never mentions your search functionality, but it's the first thing they reach for in every session.

This is where tools like heatmaps, session recordings, and behavioral analytics become really powerful. They show you the unfiltered truth about how people interact with your product. Where they click. Where they hesitate. Where they rage-click out of frustration. Where they scroll right past your most important content without a second glance.

When I worked with Heatmap, this became viscerally clear. You could literally see the difference between what a marketing team assumed would convert and what actually converted. The data didn't lie, and the businesses that acted on it consistently outperformed those that relied on intuition alone.

The Cost of Retrofitting Analytics

Let's talk about what happens when you try to bolt analytics onto a product after the fact. It's expensive, and not just in engineering hours. It costs you decision quality too.

Retrofitting analytics means going back through your codebase and instrumenting events long after the code was written. You inevitably miss things. Your event naming ends up inconsistent because different engineers tracked things at different times with different conventions. Your historical data has gaps, so you can't do meaningful cohort analysis or spot trends over time. And your team has already built habits around making decisions without data, so the entire culture of data-driven thinking has to be rebuilt alongside the technical work.

Now compare that to a team that builds tracking into their development workflow from the start. Every new feature ships with its events defined. Dashboards update in real time. Product decisions reference actual metrics, not anecdotes from a stand-up. The compounding effect of this approach over months and years is enormous.

What You Should Be Tracking From Day One

You don't need to track everything, but you do need to track the right things early. If you're not sure where to start, the AARRR pirate metrics framework is a great mental model. It breaks the user lifecycle into five stages, and each one maps to something you can measure right now.

Acquisition data tells you where your users are coming from and which channels actually drive engaged users, not just traffic. Activation data shows you whether new users are reaching the moment where your product delivers its core value and how quickly they get there. Engagement data reveals which features are being used, how often, and by whom. This is often where you discover your product's real value proposition, which sometimes isn't what you expected. Retention data tells you who comes back and who doesn't, and more importantly, what the returning users have in common. Funnel data maps the journey from first visit to conversion, exposing every point where you're losing people.

This isn't about drowning in dashboards. It's about having the minimum viable data to make informed decisions instead of guessing.

Data as a Product Culture, Not Just a Feature

The best product teams don't just collect data. They build a culture around it. Data is part of every product review, every sprint retro, and every feature proposal. When someone suggests a new feature, the first question is, "How will we measure whether this works?" When a feature ships, the team watches the metrics before celebrating.

This kind of culture doesn't just appear out of nowhere. It forms when data is available from the beginning, when early decisions are informed by evidence, and when the team develops the habit of questioning assumptions with numbers.

If you're building a product right now and you haven't set up your analytics, stop what you're doing. Before you write another line of feature code, instrument the basics. Set up event tracking. Define your key metrics. Create a simple dashboard. It'll take you a day or two at most, and it will pay for itself a hundred times over.

The data you collect today is the insight you'll need tomorrow. Start collecting it now.

If you found this helpful, consider supporting my work:

When you send money from your MTN MoMo wallet to someone on Telecel Cash, the transfer completes in under 10 seconds. It feels simple. But behind that 10-second experience is one of the most sophisticated payment architectures on the African continent — a centralized national switch that routes transactions across every mobile wallet, bank account, and biometric card in Ghana.

That switch is GhIPSS — the Ghana Interbank Payment and Settlement Systems Limited. And the way it achieves interoperability is worth understanding, whether you’re a fintech engineer building on Ghana’s payment rails, or simply curious about how money actually moves in one of West Africa’s most digitized economies.

The Problem GhIPSS Was Built to Solve

Before GhIPSS, Ghana’s payment landscape was fragmented. Banks couldn’t easily settle with each other electronically. Mobile money didn’t exist yet. Cheques took days to clear. And there was no shared infrastructure connecting financial institutions.

The Bank of Ghana incorporated GhIPSS in May 2007 as a wholly owned subsidiary with a single mandate: build and manage the interoperable electronic payment infrastructure for all financial institutions in the country. The keyword is interoperable — not just digital, but connected.

The vision was bold: any Ghanaian should be able to send money from any account type (bank, mobile wallet, or biometric card) to any other account type, across any institution, instantly. Today, that vision is largely realized — and the architecture behind it is what makes it work.

The Hub-and-Spoke Model: One Switch to Connect Them All

GhIPSS operates a hub-and-spoke interoperability model. Instead of requiring every bank and mobile money operator to build bilateral connections with each other — which would mean hundreds of point-to-point integrations — every participant connects to one central switch.

Think of it like an airport hub. If you have 53 financial institutions, a bilateral model would require 1,378 separate connections. GhIPSS reduces that to 53 — one per participant. Banks connect directly. Electronic Money Issuers (EMIs) like MTN Mobile Money Limited and Telecel Cash connect through sponsoring banks.

This single-switch design is the foundation. Everything else builds on top of it.

The Three Pillars: GIP, MMI, and gh-link

GhIPSS doesn’t run one payment system — it runs an interconnected suite of them, built layer by layer over more than a decade. The three core platforms form what GhIPSS calls the “Financial Inclusion Triangle”:

gh-link (2012) is the national card switch. It connects all domestic ATMs and POS terminals, enabling cardholders to use any ATM regardless of their issuing bank. gh-link processes EMV chip transactions and serves as the foundational backbone for the other platforms.

GhIPSS Instant Pay — GIP (2015) enables real-time account-to-account transfers up to GH¢50,000, available 24/7 across USSD, mobile apps, internet banking, ATMs, and POS devices. GIP rides on gh-link rails and has become the dominant payment channel, with transaction values surging 233% year-on-year in 2024 to reach GH¢47.5 billion.

Mobile Money Interoperability — MMI (2018) was the game-changer. Launched on May 10, 2018, it made Ghana one of the first countries in Africa to achieve full cross-network mobile money interoperability. MMI connects to the GIP platform, which connects to gh-link, completing the triangle: any mobile wallet can send to any other wallet, any bank account, or any e-zwich biometric card.

The layered architecture is deliberate. Each system was built to extend the one before it, creating full interoperability across three fundamentally different account types without requiring any single platform to handle everything.

Tracing a Transaction: What Actually Happens in Those 10 Seconds

Let’s trace a real transaction to understand the architecture in action. Suppose Ama on MTN MoMo sends GH¢100 to Kwame on Telecel Cash.

Step 1: Initiation. Ama dials *170# and navigates to the option for sending money to another network, entering Kwame’s Telecel number and the amount.

Step 2: Validation at the source. MTN’s backend validates Ama’s balance, KYC tier limits, and daily transaction caps. If everything passes, it debits her wallet by GH¢100.

Step 3: Routing through the switch. The transaction hits the GhIPSS MMI switch, which identifies Telecel as the network hosting Kwame’s wallet.

Step 4: Wallet name validation. GhIPSS queries Telecel’s system and returns Kwame’s registered name to Ama for confirmation. This step — often invisible to users — is a critical fraud prevention mechanism. Ama sees “Kwame Mensah” and confirms the transfer.

Step 5: Credit at the destination. GhIPSS routes the credit instruction to Telecel, which credits Kwame’s wallet with GH¢100. Both parties typically receive SMS confirmations.

Total elapsed time: under 10 seconds.

But here’s what most people don’t realize — no actual money moved between MTN and Telecel in those 10 seconds. Ama’s wallet was debited, Kwame’s wallet was credited, and GhIPSS recorded the transaction. The actual interbank movement of funds happens later, in the settlement layer.

The Two-Layer Architecture: Instant Transfers, Deferred Settlement

This is where the architecture gets interesting. Ghana’s system cleanly separates two things that seem like they should be the same:

Layer 1 — The transfer layer (instant). Users experience real-time movement of value. Ama’s balance drops, Kwame’s balance increases. From their perspective, the money has moved.

Layer 2 — The settlement layer (deferred). The actual movement of money between financial institutions happens in batch. GhIPSS performs multilateral net settlement at scheduled windows throughout the day. At each window, it calculates net obligations across all participants.

Here’s why netting matters. Between settlement windows, MTN’s customers might send GH¢5 million to Telecel customers, while Telecel’s customers send GH¢3 million back to MTN. Instead of processing GH¢8 million in gross transfers, the system only moves the net difference of GH¢2 million from MTN’s side to Telecel’s side. This dramatically reduces the volume of money flowing through the settlement system.

These net positions are submitted to the Bank of Ghana’s Ghana Interbank Settlement (GIS) system — the country’s Real-Time Gross Settlement (RTGS) infrastructure, operational since 2002. Settlement through the GIS is final and irrevocable, executed in central bank money using SWIFT messaging standards now migrated to ISO 20022.

The elegance of this design is that users get speed (instant transfers) while the banking system gets stability (controlled, netted settlement in central bank money). The gap between the two layers — typically measured in hours — is absorbed by the substantial liquidity sitting in trust accounts.

Where the Money Actually Lives: Trust Accounts

This is perhaps the most important and least understood part of the architecture. When you see GH¢100 in your mobile money wallet, where is that money?

It’s not at the Bank of Ghana. It’s not floating in some digital void. Every cedi of mobile money is backed one-to-one by cash in trust accounts held at commercial universal banks.

Under the Payment Systems and Services Act 2019 (Act 987), Electronic Money Issuers must maintain dedicated customer float accounts where funds are kept in liquid assets withdrawable on demand. These funds cannot be commingled with the EMI’s own operating capital. An independent trustee manages the accounts, and the EMI’s articles of incorporation must specify that customer e-money is held in trust and cannot be encumbered in insolvency.

As of late 2025, trust accounts across all partner banks held approximately GH¢30 billion — a figure explored in depth in this analysis by The Business & Financial Times.

MTN’s architecture here is particularly sophisticated. It uses a Bank Identification Number (BIN) model — each of its 17+ partner banks (including Ecobank, Fidelity, CalBank, GCB, Stanbic, Standard Chartered, and others) is assigned a unique BIN, and customer wallets are mapped to specific BINs. The aggregate trust balance at each partner bank equals the total e-cash in wallets tagged to that bank’s BIN.

This creates a fascinating economic loop: mobile money adoption drives trust account deposits, which provide banks with cheap, stable liquidity — the spread between what banks pay on these deposits and what they earn from lending is substantial. The interest earned on trust accounts is required to be passed through to end customers — EMIs must return not less than 80% of interest earned to wallet holders.

How Banks Fit In: From Principals to Custodians

The bank-MNO relationship has fundamentally transformed. Under the 2008 Branchless Banking Guidelines, telcos could only operate as agents of banks. Banks were the principals, and every mobile money account was linked to a partner bank. MTN launched mobile money in 2009 by partnering with nine banks.

The 2015 E-Money Issuer Guidelines changed everything by introducing the Dedicated Electronic Money Issuer (DEMI) concept. This gave telcos legal standing to issue e-money directly. Banks shifted from being principals to serving three roles:

Custodians — they hold the trust accounts that back every cedi of mobile money.

Product partners — they build financial products distributed via mobile money rails. Fidelity Bank offers MTN’s Y’ello Save product (8% p.a. savings). Ecobank provides XpressLoan micro-credit through MoMo.

Settlement participants — they hold settlement accounts at the Bank of Ghana and sponsor EMIs in the RTGS system, since EMIs are indirect participants in settlement.

Banks have also responded competitively. GCB Bank launched G-Money as a bank-led EMI. The broader banking sector collectively launched GhanaPay in 2022 — a shared bank-led mobile money service accessible via *707# — offering zero transaction fees to compete with telco-led mobile money.

The Bank of Ghana’s Triple Role

The Bank of Ghana’s position in this architecture is unusually concentrated. It simultaneously serves as:

Owner — BoG owns 100% of GhIPSS, with the Governor chairing the board.

Regulator — Through Act 987, BoG licenses and oversees all payment participants, sets KYC tier limits, mandates trust account requirements, and can impose penalties including imprisonment for operating without a license.

Settlement bank — BoG operates the GIS/RTGS system where all retail payment streams ultimately settle.

This concentration gives BoG extraordinary control over the payment value chain. It’s a double-edged sword: it enabled Ghana to push through interoperability faster than markets where coordination among multiple stakeholders has stalled (the regulator could simply mandate participation). But it also raises questions about competitive neutrality and conflict of interest that the planned — and still incomplete — diversification of GhIPSS ownership was designed to address.

The Technical Stack

For engineers and fintechs looking to understand or integrate with GhIPSS infrastructure, the technical stack spans multiple messaging standards:

ISO 8583 handles card-based transactions on gh-link. Web service APIs power GIP instant payments. The RTGS uses SWIFT FIN Y-Copy messaging, now migrated to ISO 20022. Card infrastructure runs on Thalès PURE® EMV technology for gh-link and Paycode EDAPT technology for e-zwich. The 3D Secure (gh-secure™) platform provides second-layer authentication for online card payments. GhIPSS holds both ISO 27001 and PCI DSS certifications.

For fintech integration, several pathways exist. Third-party aggregators like Orchard provide REST APIs with JSON request/response formats that interface with GhIPSS systems, offering mobile money payments, card payments, remittance APIs, and hosted checkout with callback notifications. MTN’s MobileMoney open API provides another integration path. GhIPSS’s Third Party Settlement Service enables interbank settlement for any licensed third-party payment scheme.

A formal open banking API framework doesn’t yet exist but is a stated priority. GhIPSS’s current leadership has committed to developing nationally accepted technical and governance standards for open banking APIs — a development that would significantly lower the barrier for fintech integration. The BoG’s regulatory sandbox, launched in August 2022, provides a testing ground for innovations in payments, e-KYC, and digital banking.

The Fee Architecture

The economics of interoperability run on a receiver-pays model (switched from the original sender-pays model). The receiving institution pays 0.2% of transaction value, capped at GH¢2, of which GhIPSS retains the majority as a switching fee.

Consumer-facing fees range from 0–1% for wallet-to-wallet and 0–3% for wallet-to-bank transfers, with a BoG-recommended ceiling of GH¢10 per transaction. For GIP bank transfers, banks charge 1%, routing 30% to GhIPSS and retaining 70%.

This fee structure has been critical to adoption. By keeping costs low and distributing them across the ecosystem, Ghana avoided the pricing barriers that have limited interoperability uptake in other markets.

What Makes Ghana’s Approach Distinctive

Three architectural decisions set Ghana apart from other African markets:

Centralized infrastructure, regulator-owned. Rather than leaving interoperability to bilateral market agreements (as in Kenya) or private-sector switches, Ghana placed the switch at the center of the financial system, owned by the central bank. This eliminated the coordination problem but concentrated power.

Layered platform design. gh-link, GIP, and MMI aren’t separate systems — they’re layers that extend each other. This meant Ghana didn’t need to build mobile money interoperability from scratch; it extended existing bank interoperability infrastructure to include wallets.

Mandatory trust accounts with interest pass-through. By requiring that every cedi of e-money is backed by trust accounts that earn interest for customers, Ghana created an incentive alignment where mobile money adoption strengthens — rather than threatens — the banking system.

The result is an ecosystem where 53 financial institutions, 6 EMIs, and hundreds of thousands of agents all participate in a single, interconnected payment network. Ghana now processes over GH¢174.5 billion annually through GhIPSS — and the system is still growing rapidly.

What Comes Next

The architecture is heading in two directions. Domestically, the push is toward open banking APIs and a unified digital identity layer built atop the payments ecosystem. Ghana’s regulatory sandbox is testing innovations in payments, e-KYC, and digital banking.

Internationally, cross-border interoperability is the next frontier. Ghana is piloting connections to the Pan-African Payment and Settlement System (PAPSS) and testing Africa’s first direct mobile money interoperability corridor linking the Ghana cedi and Nigerian naira.

For fintech builders operating in Ghana’s ecosystem, understanding this architecture isn’t optional — it’s the foundation everything runs on. The hub-and-spoke model, the two-layer transfer-settlement separation, the trust account framework, and the regulatory structure all shape what’s possible to build, how transactions flow, and where value is created.

The 10-second transfer is just what users see. The architecture underneath is what makes it work.

Written by Benson | February 2026

Further Reading

• GhIPSS — Who We Are

• AfricaNenda Case Study: GhIPSS Instant Inclusive Payment Systems (PDF)

• Bank of Ghana — Evolution of Payment System Policies (PDF)

• BoG Governor’s Speech — MMI Phase II Launch (PDF)

• Payment Systems and Services Act 2019 (Act 987) (PDF)

• BFA Global — Highlights Emerging from Ghana’s Payment Ecosystem (PDF)

• Carnegie Endowment — Digital Financial Inclusion and Security: The Regulation of Mobile Money in Ghana

• Ghana Instant Payments: Rails, Fees, and the Lightning Network (2025)

That's the kind of access control I wanted to build. Something intuitive, flexible, and battle-tested. So I built Osiki—an open-source Role-Based Access Control system born from years of painful lessons in production systems.

The Systems That Broke Me (In a Good Way)

Over the past five years, I've built software that needed to answer one deceptively simple question: "Can this user do this thing?"

It sounds easy until you're staring at:

A centralized hospital information management system where doctors need access to patient records, but only in their department. Nurses need different access. Lab technicians need another slice. And the hospital administrator? They need to see everything—but shouldn't be able to modify clinical data. Now multiply that by three shifts, locum doctors, and students on rotation.

An internal tooling platform for a fintech operating across three countries—Ghana, Uganda, and Zambia—with a branch in each capital. A country manager in Accra shouldn't see Ugandan customer data. But a regional fraud analyst needs to see patterns across all three markets. And what about when someone transfers from one country to another? Their permissions need to follow—but only for certain resources.

Multi-tenant systems where one codebase serves completely different organizations, each with their own hierarchy, their own rules, and their own definition of "admin."

Every time, we'd cobble together something that worked. Hardcoded role checks scattered across the codebase. Permission logic buried in if-statements. Audit trails that were more like audit suggestions.

I got tired of rebuilding the same thing badly. So I decided to build it once, properly.

What Osiki Actually Does

At its core, Osiki answers, "Does this user have permission to perform this action on this resource in this context?"

But the devil's in the details.

The Permission Model

Permissions in Osiki are atomic pairs: resource:action.

invoices:read
invoices:write
users:delete
reports:export

Simple. Composable. And wildcards work exactly how you'd expect:

• invoices:* — all actions on invoices

• *:read — read access to everything

• *:* — god mode (use sparingly)

This pattern comes directly from AWS IAM. If you've written IAM policies, you'll feel right at home.

Role Hierarchies

Roles can inherit from other roles. If admin inherits from editor, and editor inherits from viewer, then assigning someone admin automatically grants them everything below.

admin
  └── editor
        └── viewer

No need to manually assign three roles. The hierarchy handles it.

Scoping (This Is Where It Gets Interesting)

Here's where Osiki diverges from simpler RBAC systems. Every role assignment is scoped.

# Simple single-tenant
scope = {"tenant_id": "acme_corp"}

# Multi-level
scope = {
    "tenant_id": "acme_corp",
    "country": "ghana",
    "department": "finance"
}

# Global access (superadmin)
scope = {}  # Empty means everywhere

The matching rule is intuitive: your scope must be a subset of (or equal to) the requested scope.

If you have {tenant_id: "acme_corp"}, you can access {tenant_id: "acme_corp", country: "ghana"}. You're scoped to the entire organization, so Ghana is included.

But if you only have {tenant_id: "acme_corp", country: "ghana"}, you can't touch {country: "uganda"}. Your access is narrower than what you're requesting.

This single rule handles multi-tenancy, regional access, and departmental isolation—all without special cases.

Groups

Because assigning roles to 500 users individually is nobody's idea of fun.

Create a group. Assign roles to the group. Add users to the group. Done.

Constraints

Real organizations have rules that go beyond "who can do what":

• Mutual exclusion: A user can't be both payment_approver and payment_requester (separation of duties)

• Cardinality limits: Only 3 users can have the superadmin role

• Prerequisites: You must have basic_user before you can be assigned manager

These aren't afterthoughts in Osiki—they're first-class citizens.

Audit Trail

Every single thing is logged. Role assignments. Revocations. Permission checks (both allowed and denied). Who did what, when, and in what context?

Because someday, someone will ask, "Who had access to the payroll system on March 15th?" And you'll have an answer.

The Technical Decisions (And Why I Made Them)

MongoDB + Beanie ODM

I know what you're thinking. "RBAC is inherently relational. Why MongoDB?"

Two reasons:

Cost. MongoDB Atlas has a genuinely free tier (512MB) that stays free. I wanted Osiki to be something anyone could spin up without a credit card. PostgreSQL options have more gotchas for hobby projects.

Familiarity. I've spent years with MongoDB. When you're learning new concepts (and RBAC has plenty of edge cases), fighting your database adds friction you don't need.

The trade-off is real, though. RBAC is many-to-many relationships all the way down. Users have roles. Roles have permissions. Users belong to groups. Groups have roles. It's joins everywhere.

We adapted by:

• Using explicit junction collections (user_roles, group_roles) instead of embedding

• Handling referential integrity in application code

• Leaning heavily on Redis for computed results

Redis (For Caching, Not Sessions)

Computing a user's effective permissions is expensive. You need to:

1. Get their direct role assignments

2. Get their group memberships

3. Get roles assigned to those groups

4. Walk up the role hierarchy for each role

5. Collect all permissions

6. Check for wildcards

That's a lot of queries. So we cache aggressively.

Five minutes feels right for permissions. Long enough to matter, short enough that revoked access doesn't linger. The hierarchy rarely changes, so an hour is fine there.

Any mutation (role assignment, permission change, etc.) invalidates the relevant cache keys immediately.

FastAPI

Async by default. Pydantic validation is built in. Automatic OpenAPI docs. It's become my default for Python APIs, and Osiki was no exception.

The Permission Resolution Algorithm

When you call can_user_do("user_123", "invoices:write", scope={"tenant_id": "acme_corp"}), here's what happens:

1. Check cache for user's computed permissions (with scope hash)
   → Cache hit? Return immediately.

2. Cache miss. Start computing:
   a. Get user's direct role assignments (filtered by scope)
   b. Get user's group memberships
   c. Get roles assigned to those groups (filtered by scope)
   d. Combine all role IDs

3. For each role, walk up the hierarchy:
   visited = set()  # Cycle detection
   while current_role:
       if current_role in visited:
           raise CycleDetectedError()  # Someone made a loop
       visited.add(current_role)
       ancestors.append(current_role)
       current_role = get_parent(current_role)

4. Collect all permission IDs from all roles (direct + inherited)

5. Resolve permission IDs to resource:action strings

6. Cache the result

7. Check: does the requested permission match?
   - Exact match: "invoices:write" == "invoices:write" ✓
   - Wildcard: "invoices:*" matches "invoices:write" ✓
   - Super wildcard: "*:*" matches everything ✓

The cycle detection isn't paranoia. I've seen production systems where someone accidentally created admin → manager → admin. Without detection, that's an infinite loop.

No Denies (For Now)

Osiki permissions are purely additive. If you have invoices:read from one role and invoices:write from another, you get both. There's no "deny" that overrides.

This is a deliberate simplification. AWS IAM has explicit denies, and they're powerful—but they also create debugging nightmares. For most applications, additive permissions are enough. Deny support might come later as an opt-in feature.

What's Next

Osiki is almost ready for public release. The core is solid. The tests are green. The documentation is... in progress.

A few things on the roadmap:

• ABAC (Attribute-Based Access Control): Sometimes you need rules like "users can only edit documents they created" or "access is only allowed during business hours." That's beyond pure RBAC.

• Resource-level permissions: Right now, you can control access to types of resources (invoices). Eventually, you should be able to control access to specific resources (invoice_123).

• SDKs: Python first, then Node.js and Go.

Why Open Source?

Because I've benefited enormously from open source, and this is my way of giving back.

But also because access control shouldn't be a mystery. Too many systems have permissions scattered across the codebase, understood by one person who left the company two years ago. A well-designed RBAC system, with clear concepts and good documentation, makes the whole application easier to reason about.

If Osiki helps even one team avoid the pain I went through, it's worth it.

Try It Out

The repo will be public soon at https://github.com/bensonOSei/osiki Apache 2.0 license. Contributions are welcome.

If you've built RBAC systems before and have war stories, I'd love to hear them. If you're about to build one and have questions, reach out.

And if you're a security guard in Ghana reading this—you're the real MVP. 🫡

Benson is a senior backend engineer who's spent too much time thinking about permissions. He builds fintech systems across Africa and occasionally writes about the lessons learned along the way.

Here's the story of how we turned our payment processing from a source of anxiety into a competitive advantage, and the hard-learned lessons that might save you some sleepless nights.

The Deceptive Calm: When Low Traffic Hides Big Problems

Our payment service started simple. We served a very small clientele with barely any traffic, so many edge cases simply never surfaced. The system worked... until we realized it was built on some shaky foundations that would become major issues as we scaled.

The wake-up call came during what should have been a routine code review. What we discovered was a system that worked in our low-traffic environment but had serious underlying vulnerabilities:

• No idempotency protection - duplicate requests could create multiple charges

• Queue-based processing that could amplify the duplicate charge problem

• Uncontrolled state mutations that could leave transactions in inconsistent states

• Client-controlled security parameters that opened doors we didn't want opened

Sometimes you need someone to tell you your baby is ugly before you can make it beautiful.

Understanding Idempotency: The Foundation of Safe Payment Systems

What Are Idempotency Keys and Why Do They Matter?

Idempotency is a fancy computer science term for a simple concept: doing the same operation multiple times should have the same effect as doing it once. In payment systems, this is absolutely critical.

Consider this scenario: A user clicks "Pay Now" on your checkout page. Their network is slow, so they click again. Or their phone app crashes and auto-retries the request. Or a webhook gets delivered twice. Without idempotency protection, each of these events could create a separate charge.

Idempotency keys solve this by making duplicate requests safe. Here's how it works:

# Without idempotency keys - dangerous
async def create_payment(self, user_id, amount):
    # Every call creates a new payment, even if identical
    return await self._transaction_repository.create({
        "user_id": user_id,
        "amount": amount,
        "status": "pending"
    })

# With idempotency keys - safe
async def create_payment(self, payment_request):
    idempotency_key = payment_request.generate_idempotency_key()

    # Check if we've seen this exact request before
    existing = await self._transaction_repository.get_by_idempotency_key(idempotency_key)
    if existing:
        return existing  # Return the original result

    # First time seeing this request - process it
    return await self._transaction_repository.create({
        "user_id": payment_request.user_id,
        "amount": payment_request.amount,
        "idempotency_key": idempotency_key,
        "status": "pending"
    })

Our Idempotency Journey: From Nothing to Bulletproof

Phase 1: No Protection Initially, we had no idempotency protection at all. With our small clientele and low traffic, duplicate requests were rare enough that we never noticed the problem.

Phase 2: Client-Controlled Keys When we recognized the need for idempotency, we took what seemed like the obvious approach - let clients provide their own keys:

class CreatePaymentRequest(BaseModel):
    user_id: str
    amount: int
    # Client provides their own key
    idempotency_key: Optional[str] = Field(default_factory=lambda: str(uuid.uuid4()))

This seemed convenient, but it introduced a security vulnerability. A malicious client could bypass duplicate protection:

// Request 1
{"user_id": "user123", "amount": 1000, "idempotency_key": "key1"}

// Request 2 - same payment, different key
{"user_id": "user123", "amount": 1000, "idempotency_key": "key2"}
// Result: Two separate charges for identical payments

Phase 3: Server-Generated Keys The secure solution was to generate idempotency keys server-side based on the request content:

def generate_idempotency_key(self) -> str:
    """Generate secure idempotency key based on request content."""
    content = f"{self.user_id}:{self.currency}:{self.payment_method}:{self.amount}"
    return hashlib.sha256(content.encode()).hexdigest()[:32]

Now identical requests produce identical keys automatically, while legitimate different payments get different keys. The server controls the security parameters, not the client.

The Duplicate Detection Mechanism

Our duplicate detection works at multiple levels:

1. Request-Level Deduplication

async def initiate_payment(self, payment_request):
    # Generate deterministic key from request content
    idempotency_key = payment_request.generate_idempotency_key()

    # Check for existing transaction with this key
    existing_transaction = await self._transaction_repository.get_by_idempotency_key(idempotency_key)

    if existing_transaction:
        # Return the original response - no new processing
        return TransactionResponse.from_db(existing_transaction)

    # First time seeing this request - proceed with payment
    return await self._process_new_payment(payment_request, idempotency_key)

2. Provider-Level Protection

We also handle cases where the same request might reach our payment provider multiple times:

async def _process_new_payment(self, payment_request, idempotency_key):
    # Create transaction record first
    transaction = await self._transaction_repository.create({
        "idempotency_key": idempotency_key,
        "status": "pending",
        # ... other fields
    })

    try:
        # Send to payment provider with our transaction ID as their idempotency key
        provider_response = await self._payment_provider.initiate_payment(
            transaction_id=str(transaction.id),  # Provider uses this for deduplication
            amount=payment_request.amount,
            # ... other details
        )

        # Update with provider response
        return await self._update_transaction_with_response(transaction.id, provider_response)

    except Exception as e:
        # Mark as failed and re-raise
        await self._mark_transaction_failed(transaction.id, str(e))
        raise

This creates multiple layers of protection against duplicates at both our application level and the payment provider level.

State Machines: Bringing Order to Payment Chaos

What Is a State Machine and Why Do Payments Need One?

A state machine is a way to model how something can change over time with strict rules about what changes are allowed. For payment transactions, this is crucial because money has very specific rules about how it can move between states.

Think of a payment transaction as having a lifecycle:

• Created → A payment request has been received

• Pending → We've sent it to the payment provider

• Completed → Money has been successfully transferred

• Failed → Something went wrong, no money moved

• Refunded → Money was moved back to the customer

Without a state machine, your code might accidentally allow impossible transitions like going directly from "Failed" to "Completed" or updating a "Refunded" transaction back to "Pending."

Our Payment State Machine Implementation

from enum import Enum

class TransactionStatus(str, Enum):
    PENDING = "pending"
    COMPLETED = "completed"
    FAILED = "failed"
    CANCELLED = "cancelled"
    REFUNDED = "refunded"
    DISPUTED = "disputed"

# Define which transitions are allowed
VALID_TRANSITIONS = {
    TransactionStatus.PENDING: {
        TransactionStatus.COMPLETED,
        TransactionStatus.FAILED,
        TransactionStatus.CANCELLED
    },
    TransactionStatus.COMPLETED: {
        TransactionStatus.REFUNDED,
        TransactionStatus.DISPUTED
    },
    TransactionStatus.DISPUTED: {
        TransactionStatus.COMPLETED,  # Dispute resolved in merchant's favor
        TransactionStatus.REFUNDED    # Dispute resolved in customer's favor
    },
    # Terminal states - normally immutable
    TransactionStatus.FAILED: set(),
    TransactionStatus.CANCELLED: set(),
    TransactionStatus.REFUNDED: set()
}

Enforcing State Transitions

Every status update goes through validation:

def validate_transition(self, current_status, new_status, source, allow_admin_override=False):
    """Validate that a status transition is allowed."""

    # Check if this is a valid transition
    allowed_transitions = VALID_TRANSITIONS.get(current_status, set())

    if new_status in allowed_transitions:
        return True

    # Terminal states are normally immutable
    if current_status in {TransactionStatus.FAILED, TransactionStatus.CANCELLED, TransactionStatus.REFUNDED}:
        # But admins can override with proper authorization
        if source == TransitionSource.ADMIN and allow_admin_override:
            return True
        return False

    # All other transitions are invalid
    return False

async def update_payment_status(self, transaction_id, new_status, source, updated_by=None, allow_admin_override=False):
    transaction = await self._transaction_repository.get(transaction_id)

    # Validate the transition
    if not self.validate_transition(transaction.status, new_status, source, allow_admin_override):
        raise InvalidStateTransitionError(
            f"Cannot transition from {transaction.status} to {new_status}"
        )

    # Update the status
    return await self._transaction_repository.update(transaction_id, {
        "status": new_status,
        "updated_by": updated_by,
        "updated_at": datetime.utcnow()
    })

Why State Machines Matter in Payment Systems

Data Integrity: Prevents impossible states like a "Failed" transaction that somehow has money attached to it.

Business Logic Enforcement: Ensures business rules are followed (e.g., you can't refund a failed payment).

Audit Compliance: Creates a clear trail of how transactions moved through their lifecycle.

Error Prevention: Catches bugs before they cause financial inconsistencies.

Operational Safety: Gives operations teams confidence that manual interventions won't break the system.

The Admin Override Reality

In the real world, payment providers make mistakes. Banks change their minds. Customers dispute legitimate transactions. We needed a way to handle these edge cases without compromising the integrity of our state machine.

Our solution: controlled mutability with admin overrides.

class TransitionSource(str, Enum):
    SYSTEM = "system"      # Internal system updates
    WEBHOOK = "webhook"    # Payment provider webhooks  
    ADMIN = "admin"        # Administrative overrides
    USER = "user"          # User-initiated actions

async def admin_update_payment_status(
    self, transaction_id, status, admin_user, reason, 
    allow_terminal_override=False
):
    if not reason:
        raise ValueError("Reason required for admin updates")

    return await self.update_payment_status(
        transaction_id=transaction_id,
        status=status,
        source=TransitionSource.ADMIN,
        updated_by=admin_user,
        allow_admin_override=allow_terminal_override
    )

This handles real operational scenarios:

• Provider corrections when payments were incorrectly marked as failed

• False fraud flags that needed reversal

• Technical errors requiring manual correction

• Dispute resolutions that require status changes

Comprehensive Audit Trail: Trust Through Transparency

Every significant action in our payment system is logged with full context:

# Example audit log entry
{
    "transaction_id": "txn_123",
    "action": "status_update",
    "from_status": "failed",
    "to_status": "completed", 
    "source": "admin",
    "updated_by": "admin@company.com",
    "reason": "Provider confirmed payment succeeded after initial failure",
    "is_admin_override": true,
    "timestamp": "2024-01-15T14:30:00Z",
    "request_id": "req_456",
    "user_agent": "Admin Dashboard v1.2"
}

This audit trail serves multiple purposes:

Compliance: Regulatory bodies can see exactly what happened to every transaction.

Debugging: When issues arise, we can trace exactly how a transaction evolved.

Security: Unusual patterns in admin overrides can indicate security issues.

Business Intelligence: Understanding payment patterns helps improve the system.

Customer Support: We can give customers detailed information about their payment status.

The Architecture Transformation: From Internal Queues to Provider-Managed Processing

Our original architecture used internal queue-based processing, which seemed like good design but introduced complexity and potential failure points:

# Original internal queue-based approach
async def initiate_payment(self, payment_details):
    # Create transaction record
    transaction = await self._transaction_repository.create(...)

    # Queue for internal background processing
    await self._payment_queue.enqueue_payment(transaction)

    return transaction  # Status: PENDING

The problems with this approach:

• Our own retry logic could cause duplicate charges

• Internal queue failures could lose payments

• Complex debugging across multiple internal components

• Infrastructure overhead for managing our own queues

The "Fire and Forget" Revolution

We moved to a different model: delegate the complexity to payment providers who are experts at it. Modern payment providers like Paystack, Stripe, and others have sophisticated retry mechanisms, queue management, and failure handling built-in.

Our new approach became much simpler:

# New provider-managed approach
async def initiate_payment(self, payment_request):
    # Check for duplicates using server-generated key
    idempotency_key = payment_request.generate_idempotency_key()
    existing = await self._transaction_repository.get_by_idempotency_key(idempotency_key)

    if existing:
        return TransactionResponse.from_db(existing)

    # Create transaction record
    transaction = await self._create_transaction(payment_request, idempotency_key)

    try:
        # Fire and forget to payment provider
        await self._payment_provider.initiate_payment(transaction)

        # Provider handles retries, queuing, and complex logic
        # We just mark as submitted and wait for webhook confirmation
        updated_transaction = await self.update_payment_status(
            transaction.id,
            TransactionStatus.PENDING,
            TransitionSource.SYSTEM
        )

        return TransactionResponse.from_db(updated_transaction)

    except Exception as e:
        # Only handle immediate failures (network issues, validation errors)
        await self.update_payment_status(
            transaction.id,
            TransactionStatus.FAILED,
            TransitionSource.SYSTEM
        )
        raise

Webhook-Driven Updates

The real magic happens through webhooks. The payment provider handles all the complexity and tells us about status changes:

# Webhook handler for payment status updates
async def handle_payment_webhook(self, webhook_payload):
    transaction_id = webhook_payload.get_transaction_reference()
    provider_status = webhook_payload.data.status

    # Map provider status to our status
    our_status = self._map_provider_status(provider_status)

    # Update through state machine validation
    await self.update_payment_status_from_webhook(
        transaction_id=transaction_id,
        status=our_status,
        source=TransitionSource.WEBHOOK,
        reason=f"Webhook event: {webhook_payload.event}"
    )

This approach works well for our current scale because:

• Low traffic volume means we don't need complex internal orchestration

• Provider expertise - they handle retries, exponential backoff, and failure scenarios better than we could

• Reduced infrastructure - no need to manage our own queue systems

• Simpler debugging - fewer moving parts on our side

When Internal Queues Still Make Sense

It's important to note that queues aren't always bad. Here are scenarios where internal queue-based processing might be the right choice:

1. High-Volume Processing

# When you're processing thousands of payments per minute
# and need fine-grained control over batching and rate limiting
async def process_bulk_payments(self, payment_batch):
    for payment in payment_batch:
        await self._payment_queue.enqueue_with_priority(payment, priority="high")

    # Process in controlled batches to avoid overwhelming providers
    await self._queue_processor.process_batch(batch_size=50, rate_limit="10/second")

2. Complex Business Logic Orchestration

# When payments trigger multiple downstream actions
# that need coordination and rollback capabilities
async def process_subscription_payment(self, payment_details):
    # Queue ensures all steps happen in order with retries
    await self._orchestration_queue.enqueue_workflow([
        ("charge_payment", payment_details),
        ("activate_subscription", subscription_details),
        ("send_welcome_email", user_details),
        ("update_analytics", event_details)
    ])

3. Provider Rate Limiting

# When you need to respect strict provider rate limits
# across multiple application instances
class RateLimitedPaymentQueue:
    async def enqueue_payment(self, payment):
        # Queue enforces rate limits across all instances
        await self._distributed_queue.enqueue(
            payment, 
            rate_limit="100/minute",  # Provider's limit
            distribution_strategy="round_robin"
        )

4. Multi-Provider Failover

# When you need sophisticated failover logic
async def process_with_failover(self, payment_request):
    providers = ["primary_provider", "backup_provider", "emergency_provider"]

    for provider in providers:
        try:
            result = await self._try_provider(provider, payment_request)
            if result.success:
                return result
        except ProviderUnavailableError:
            # Queue for retry with next provider
            await self._failover_queue.enqueue(payment_request, exclude=[provider])

5. Compliance and Auditing Requirements

# When you need detailed control over processing timing
# for regulatory compliance
async def process_regulated_payment(self, payment):
    # Queue ensures proper cooling-off periods and audit trails
    await self._compliance_queue.enqueue(
        payment,
        hold_until=datetime.now() + timedelta(hours=24),  # Cooling-off period
        audit_level="full"
    )

Our Strategic Choice: Provider-Managed Complexity

For our current situation, delegating queue management to payment providers was the right architectural decision:

Why it works for us:

• Scale appropriateness - Our transaction volume doesn't justify complex internal infrastructure

• Provider expertise - Companies like Paystack have spent years perfecting payment processing reliability

• Development focus - We can focus on business logic instead of distributed systems engineering

• Cost efficiency - No need to maintain queue infrastructure, monitoring, and specialized expertise

When we might reconsider:

• Volume growth - If we reach thousands of transactions per minute

• Complex workflows - If payments need to trigger many coordinated downstream actions

• Multi-provider requirements - If we need sophisticated provider failover logic

• Regulatory changes - If compliance requirements demand more control over processing timing

The Provider Partnership Model

This approach treats payment providers as partners who handle the hard parts:

# Our responsibility: Business logic and state management
class PaymentService:
    async def initiate_payment(self, request):
        # Handle idempotency, validation, state transitions
        pass

    async def handle_webhook(self, webhook):
        # Process provider updates through our state machine
        pass

# Provider responsibility: Infrastructure and reliability
# - Retry logic with exponential backoff
# - Queue management and processing
# - Network failure handling
# - Rate limiting and load balancing
# - Infrastructure scaling

This separation of concerns allows us to focus on what we do best (business logic) while leveraging what providers do best (payment infrastructure).

Code Quality: Making Your Future Self Happy

We used this overhaul as an opportunity to fix accumulated technical debt:

Type Safety:

• Fixed all MyPy errors

• Added proper type annotations throughout

• Eliminated risky None attribute access

Clean Architecture:

• Removed protected member access violations

• Implemented proper encapsulation patterns

• Created clean public APIs

Error Handling:

• Leveraged existing global exception handlers

• Eliminated duplicate error handling code

• Achieved consistent error responses across all endpoints

The result was a codebase that's much easier to maintain and extend.

Lessons Learned: What Would We Do Differently?

1. Implement Idempotency from Day One

Even with low traffic, duplicate requests can happen. Network issues, impatient users, and buggy clients will eventually cause problems. Build idempotency protection from the start.

2. Server-Generated Keys Are Non-Negotiable

Never trust clients with security-critical parameters. Generate idempotency keys server-side based on request content.

3. State Machines Prevent Chaos

Uncontrolled state mutations in payment systems lead to data corruption and unhappy customers. Define your state machine early and enforce it strictly.

4. Plan for Operational Reality

Build admin tools from day one. Payment providers will have issues, customers will dispute legitimate transactions, and humans will need to intervene safely.

5. Audit Everything

Every state change, every admin override, every significant action should be logged with full context. Compliance teams, security auditors, and your future debugging self will thank you.

6. Direct Processing > Eventual Consistency for Payments

Users want immediate feedback about their payments. Queue-based systems add complexity without providing much benefit for most payment scenarios.

The Bottom Line

This overhaul taught us that building robust payment systems requires thinking beyond the happy path. You need to consider:

• Duplicate requests and how to handle them safely

• State management with clear rules and audit trails

• Operational tools for real-world edge cases

• Security by default with server-controlled parameters

• Immediate feedback for better user experience

The investment in doing this right paid off immediately in reduced support burden, improved reliability, and peace of mind knowing our payment system could handle whatever reality threw at it.

Looking Forward

Payment systems are never "done"—they evolve with business needs, regulatory requirements, and new attack vectors. But having a solid foundation with proper idempotency protection, state machine validation, and comprehensive audit trails makes those changes manageable rather than terrifying.

Our next priorities include enhanced monitoring, additional payment providers, and advanced fraud detection. But now we're building on rock instead of sand.

Building payment systems that handle real money for real people is both challenging and rewarding. The key is respecting the complexity while keeping the architecture as simple as possible. When you get it right, everyone wins—customers, developers, and the business.

What payment system challenges have you faced? Have you dealt with idempotency issues or state management problems? The comment section is open for war stories and hard-learned lessons.