Accounting integration without portal downtime
Replaced a nightly full-rewrite sync that took the portal down for 20-30 minutes with a background delta sync: hourly updates and no downtime.
- Company
- Company under NDA
- Role
- .NET developer, de facto lead of the .NET part
- Period
- 2025
- Industry
- IT services, internal employee portal
- Scope
- ~5,000 portal users · 15-person project team · 30+ entity types · 300+ attributes · ~3 months to production
- Skills
- integration design · generic comparison engine · rollback-safe release · run-level logging
- Stack
- C# · .NET · reflection with custom attributes · dependency injection · feature flags · SQL transactions
- Tags
- integration · data synchronization · safe rollout
- Links
- LinkedIn post with a carousel about this integration
Challenge
The internal employee portal, used by about 5,000 people, took its data from the company's accounting system. The director of internal projects asked for fresher data. With the old integration that was impossible: every sync stopped the portal.
Starting point
The integration pulled all data from the accounting system, stopped the portal, rewrote the whole database and started the portal again. A run took 20 to 30 minutes, once a day, at night, so users worked with yesterday's data. Every release also needed a full integration, otherwise migrations and data would not match. Releases meant downtime, so they were rare and risky, and each one carried a bigger batch of changes. As data grew, the sync window had to be widened again and again: 15, 20, 25, then about 30 minutes. Some data was lost along the way, which led to an inconsistent state and bugs in unrelated parts of the portal that were hard to reproduce. The accounting side could also change the data contract without warning. Once it did, with the explanation "wanted to tell you, but forgot", and any such change could break the whole portal.
My contribution
- I was the only .NET developer on a 15-person team (3 PHP backend developers, 3 frontend developers, 4 QA engineers, a BA, a designer, a manager). Two people reported to me; I agreed the architecture and the data contracts and wrote the code.
- Designed and built the delta sync, the comparison engine and the SQL generation.
- Built an integration log table: when a run started, how long it took, which deltas were applied, which errors occurred. Any run can be queried later.
- Introduced the new integration through a feature flag and a prod-like environment, with the old implementation kept in place, so we could switch back without a redeploy.
- Tested the change together with QA.
Approach
Instead of rewriting everything, the new integration works in five steps in the background: fetch data from the accounting system, compare it with the current state of the portal database, calculate the delta, convert the delta to SQL, and apply it in a single transaction. The portal keeps working while this happens. One comparison engine serves every entity type, and the rules for each entity live on the model itself, so a small contract change means editing attributes, not rewriting comparison code.
Decisions
- One generic comparison engine driven by attributesWhy: one code path covered 30+ entity types and 300+ attributes. Many source entities had no primary key, so I described keys on the model with custom attributes ([PrimaryKey], [CompositeKey], [ServiceField]), and the same metadata generated the SQL without hardcoding. About 90% of the logic stayed generic and 10% lived on the models. Reflection speed did not matter because everything ran in the background.Instead of: hand-written comparison code for each of the 30 entities, which would also have to be maintained one by one.
- Build the comparison in-houseWhy: the portal stayed independent of yet another system.Instead of: a third-party data comparison and reconciliation tool, which meant a dependency on a third system and its maintenance.
What went wrong
- A few edge cases in the first month. Wrong keys, soft-delete logic, nullable properties and some missed comparisons. In most cases the root cause was the source data itself, which broke the company's business rules. After the first month the integration ran without incidents.
Results
1 → 24 syncs a daysync frequencytechnically the sync could run every 30 minutes; the stakeholder chose hourly
20-30 min → 0portal downtime per syncthe old sync blocked the portal; the new one runs in the background
20-30 → 10-15 minone full sync runold run vs new background run; the new figure comes from the integration log table on production
~1 yearstable in production after releasesmall contract changes stopped breaking the portal, the hard-to-reproduce bugs caused by lost data went away, releases became frequent and no longer needed downtime
Before → after
- Syncs per day1 → 24
- Portal downtime per sync, min20-30 → 0
Takeaway
A solution's cost does not end at delivery. Given the choice between 90% building and 10% maintaining, or the other way round, I choose the first.
140+ endpoints moved from in-memory caches to the database
Moved 140+ portal endpoints from static in-memory caches to direct database queries, with every endpoint checked against the old version and all but 2-3 of them answering within 1 second.
- Company
- Company under NDA
- Role
- .NET developer, de facto lead of the .NET part
- Period
- 2025
- Industry
- IT services, internal employee portal
- Scope
- 140+ endpoints · ~30 tables of employee data with relations · ~5,000 portal users · several months, in parallel with other work
- Skills
- query optimization · old-vs-new regression checks · cross-cutting refactoring
- Stack
- C# · .NET · SQL · Postman with post-request scripts · mock JSON test data
- Tags
- performance · refactoring · regression checks
Challenge
The portal kept almost everything it knew about employees in static in-memory caches. As the data about employees grew wider (projects, roles, technologies), performance dropped and cached data could go missing. The director of the department set two requirements for moving to database queries: no endpoint may become slower than before, and none may take longer than 1 second.
Starting point
Static caches held all information about employees, their projects, roles and technologies: about 30 tables with the links between them, all coming from the accounting system. Only assessment data lived in the portal database, as the source of truth. Nobody could say how fresh the cached data was, and the cache itself was heavy. The heaviest endpoints answered in 1.5 to 2 seconds.
Constraints
The change cut through the whole project while other features were being built in parallel. Resolving merge conflicts took 2-3 hours every day.
My contribution
- Moved the endpoints from the caches to database queries and optimized the queries.
- Set up two environments, one with the old endpoints and one with the new ones, and a script that sent the same request to both and compared the responses through Postman post-request scripts.
- Prepared mock JSON data covering every entity, so the comparison could check all of them with as little manual work as possible.
- Brought in a student from a course I taught to write the mapping for comparing old and new data. The mapping was checked with tests on test projects and functional tests.
- Brought in QA to check the results.
Approach
There was no reliable business description and no tests to tell what a correct response looked like. So the old endpoint became the specification: every new endpoint had to return the same data as the old one and be at least as fast. Most of the time went into proving that the new code behaved correctly for every entity.
Decisions
- Remove the cache instead of keeping itWhy: the portal needed current data, not an old copy. Nobody could tell how fresh the cached data was, and the cache was heavy to build and hold.Instead of: extending the cache, which did not fit the company's policy.
What went wrong
- 10-15 endpoints failed the first comparison. About half of the mismatches were about speed, the rest about differences in the response. All of them were fixed before release.
- The first release had to be rolled back. We released all endpoints at once. Production held far more data than the test environment, and performance differed several times over. After the rollback we built a prod-like environment from a sanitized copy of production, matching it one to one, and released again.
- Merge conflicts every day. After each merge the tests ran, and any difference was fixed before moving on.
Results
All but 2-3endpoints within the 1-second targetout of 140+; the exceptions process many tables with many references and stayed at their previous speed of up to about 1.5 seconds
Fewer bugscaused by data disappearing from the cachedata in the portal also became fresher
Takeaway
A large cross-cutting change belongs in one branch and gets tested in a separate environment that looks like production. And it only works if the rest of the team knows about it and works with you.
Standard Grafana dashboards shipped with the services
Built one standard set of Grafana dashboards for the team's core services, kept next to the service code and shipped through the services' Helm chart, the same in every environment.
- Company
- Company under NDA
- Role
- Senior Developer
- Period
- 2026
- Industry
- fintech
- Scope
- 5-person team · 6 deployable service processes · 3 environments (dev, test, prod) · 7 dashboards · ~2 months
- Skills
- observability design · Helm packaging · cross-team alignment
- Stack
- Grafana · Prometheus · Loki · OpenTelemetry (.NET) · Helm · Kubernetes · Kafka · SQL Server · MassTransit
- Tags
- observability · dashboards as code · Kubernetes
Challenge
The team lead asked me to set up observability for the processes inside each of the team's core services, six deployable processes in total. The team had dashboards, but there was no single approach: each part was configured and delivered its own way, and one had to be created.
Starting point
Dashboards were defined centrally, in configuration repositories away from the code they observed. Ownership was detached from the services, dashboard changes were not tied to application releases, and environment-specific values were baked into the configuration.
Constraints
There was no common standard to build on: the approach itself had to be agreed first. Client installations differed too: some ran on-premises, others in Azure, so the same set of dashboards did not fit all of them.
My contribution
- Built the dashboards and the Helm packaging for them.
- Collected practices from another product team and from the DevOps team, and set up meetings where we agreed how to set up the infrastructure side and which exporters to include.
- Made it configurable which dashboards each installation shows, so that, for example, the Redis dashboard appears only where Redis is used.
- Presented the approach to the team.
Approach
One template, values per environment. The same dashboard JSON runs in dev, test and prod; only the datasource and a few values change. Every selector is a variable, with no environment names or URLs in the JSON. Application metrics are the same for every service, infrastructure panels appear only where an exporter runs, and a missing signal shows an empty panel instead of breaking the board. Metric names follow the OpenTelemetry .NET conventions.
Decisions
- Dashboards next to the code, shipped with the service's Helm chartWhy: we compared several delivery options, and Helm was the most flexible one. The dashboards are owned by the people who own the service and are versioned with the application release.Instead of: setting dashboards up by hand, and keeping their definitions in separate configuration repositories, detached from the services and their releases.
- One template with values per environmentWhy: one source of truth for dev, test and prod; switching the datasource switches the environment without rewriting queries.Instead of: environment-specific values baked into each dashboard.
Results
7dashboards in production, in one shared folderoverview, process and runtime, logs, functional, cluster, Kafka, database; the same JSON in all three environments
~2 monthsfrom task to productionthe planned scope was delivered; alerting was not part of what went live
Takeaway
Show work early and ask for feedback sooner.
What I'd do differently
Build a prototype faster and hand it over earlier.
Customs inspection paperwork: from my own idea to a state medal
Proposed and built a web system for customs inspection paperwork: a document now took 5-7 minutes instead of 10-30, data came from central reference lists instead of being retyped, and the system earned a state medal.
- Company
- Company under NDA
- Role
- initiator and developer of the system
- Period
- first rollout 2018, medal 2019
- Industry
- public sector, customs
- Scope
- ~15 customs posts · 300 pages of certification documentation
- Skills
- turning a paper-heavy process into software · rollout and user training · winning management support
- Stack
- C# · ASP.NET · MySQL · Oracle
- Tags
- process digitalization · rollout and training · state certification
Challenge
I filled in customs inspection documents myself and saw how much time they took and how often they contained errors. The same data had to be typed again for every document. I proposed a system where part of the data comes from centralized reference lists and is entered once, not every time. The head of the customs office supported the idea.
Starting point
Documents were prepared in a dedicated program that ran only locally, without a central database or reference lists. One document took 10 to 30 minutes, and errors were common.
Constraints
The system had to pass state certification, including security certification. The customs service officially supported only Oracle, so the system had to move from MySQL to Oracle, and I prepared 300 pages of certification documentation. Moving from WinForms to the web meant learning web development first.
My contribution
- Came up with the idea from my own work on these documents and got the head of the customs office behind it.
- Built the backend, the UI and the database myself.
- Travelled to the customs posts, trained the staff, collected feedback and fixed bugs.
- Prepared 300 pages of documentation for state certification and moved the system from MySQL to Oracle.
- When the system had to be finished quickly, I led a UI developer, an intern and a DevOps engineer who joined to complete it.
Approach
Enter data once and take everything else from central reference lists. The database also controls who can submit a document, so people without the right can no longer do it.
Decisions
- A web system with a central databaseWhy: the old program worked only locally and had no central database, so there was no single place for reference lists or for control over who submits what.Instead of: extending the old local program.
- Present the system to top management as a security measureWhy: rolling it out across the whole customs system needed support from the top, and the head of the customs office advised framing it around security: all data in one database and no submissions by people without the right to make them.Instead of: presenting it as faster paperwork and lower inspection costs.
What went wrong
- Reference lists had different structures. The same kind of reference data turned out to be organized differently.
Results
10-30 → 5-7 mintime to prepare one inspection documentaverage from the logs of the old and the new program
~15customs posts using the systemrollout with on-site training
2019state medal for ensuring economic securityawarded for this system
Before → after
- Time per inspection document, min10-30 → 5-7
Takeaway
Work with the manager who knows how to present an idea to the people who make the decision.