Some managers consider APIs a buzzword related to integrations, a problem that they have already addressed with SOA + ESB. Nevertheless, APIs address new demands, which are not well managed by SOA, while SOA focuses on problems that APIs do not have.
With that in mind, how to know if:
- Having an ESB for integrations is enough?
- API Gateway and ESB are complementary?
- I should replace my ESB with APIs + Microservices?
Understanding SOA / ESB gaps and the role of the API Management Platform
The new wave of digital transformation is not only a matter of digitizing internal processes and integrations, but also of exploring the potential of the partner ecosystem, using legacy systems alongside new technologies (IoT, Cloud, AI + Analytics, Blockchain, etc.), having real-time monitoring and reacting quickly, providing experiences through different channels and interfaces, involving customers and collaborators, and overcoming scalability and security challenges, among other elements.
Main concerns regarding ESB:
- Complex scalability: ESBs are monolithic and scale vertically, with a greater risk of unavailability.
- Affected performance when recording logs: they may reduce performance in transaction processing by up to 30%.
- Vendor lock-in: expensive consulting and the need to acquire many products (BAM, BPEL, and Service Registry).
- Focus only on integrations: little concern for the experience of frontend partners and developers.
- Few resources for Governance, Monitoring, and Monetization: lack of real-time visibility and complete logs of transactions, documentation, versioning, utilization and billing plans, etc.
- Limited support for security: OAuth 2.0, OpenID, JWT, etc.
Lack of modern standards and lighter protocols: ESBs have been or tend to be discontinued.
Why use APIs for internal and external integrations?
It is a simple, agile and scalable way to integrate systems and create decoupled applications in the shortest possible time. When using an API Management platform, modules are obtained to support security, authentication, scalability, dev commitment, governance, monitoring, monetization, and the test environment, among other resources.
How API Management and ESB supplement the capabilities of the integration layer:
API Management and the ESB can exist alone, depending on the scenario and focus. In most scenarios, companies would benefit from both. In these cases, the architecture is even more robust, as long as we manage to enhance the strengths of each of them:
Focus 1: API Management + ESB Coexistence
To reach a high degree of coverage in all the capacities of an integration layer, a possible approach would be to opt for the coexistence of the API Management platform with ESB. This approach enables the utilization of resources of the API Management platform (scalability, security, governance, dev commitment, Monetization, Analytics, etc.) with the capabilities of the ESB (variety of connectors, complex transaction processing, etc.). There are, however, some limitations in terms of scalability, lock-in, and performance, while the rule engine can be obtained by means of a specific solution.
Approach 2: Replacing the ESB using API Management + API front (Microservices + Connectors)
To cover the capabilities that the API management platform does not focus on, without the limitations of ESB, it is possible to create a layer referred to as API Front. This layer is composed of APIs, microservices and connectors, thereby reducing the difficulties of scalability, decoupling, reuse, lock-in, and performance, as well as specific platforms for batch processing and the rules engine.
Benefits of API Management + Microservices in its integration architecture
- More flexible integrations, facilitating faster deliveries and reducing costs
- Standardized interfaces, enabling multi-channel and agility in the integration with partners
- Establishment of governance and real-time monitoring of integrations
- Security and control over the use of APIs
- Mechanisms to optimize the performance of applications
- Separation of responsibilities
Detailed comparison of integration layer capabilities that may be supplied by an API Management Platform or by an ESB:
|Capabilities||Patterns||API Management||ESB||Alternative API Management||Alternative ESB|
|Communication asynchronous||WebSocket||Yes||Partial – some ESBs support||N/A||Messaging (eg AMQP)|
|Synchronous communication||REST / JSON||Yes||Partial – some ESBs support||N/A||WS-SOAP|
|Security||OAuth 2.0||Yes||Partial – some ESBs support||N/A||WS-Security and Basic Auth|
|DDoS||Yes||No||N/A||Use of third-party solutions|
|WAF (Injections, XSS, CSRF)||Yes||No||N/A||Use of third-party solutions|
|Spike Arrest||Yes||No||N/A||Use of third-party solutions|
|Technical monitoring||Real Time Alerts||Yes||No||N/A||N/A|
|Technical monitoring||Dashboards & Reports||Yes||Partial – requires add-on||N/A||BAM (add-on)|
|Business Monitoring||Real Time Alerts||Yes||No||N/A||N/A|
|Dashboards & Reports||Yes||Partial – requires add-on||N/A||BAM (add-on)|
|Dev Support||Developer Portal||Yes||No||N/A||custom portal|
|Composition & Enrichment||Service Callout||Yes||Yes||N/A||N/A|
|Contract and Documentation Standards||OpenAPI Spec||Yes||No||N/A||WSDL|
|Contract and Documentation Standards||Iterative documentation||Yes||No||N/A||custom portal|
|Generation SDK||Yes||No||N/A||SDK manual generation|
|JSON Schema Validation||Yes||No||N/A||XSD|
|Governance||Usage quotas||Yes||No||N/A||Use of third-party solutions|
|Visibility Control||Yes||Partial – requires add-on||N/A||Service Registry & Repository (add-on)|
|API / Service Catalog||Yes||Partial – requiresOn dd,||N/A||Service Registry & Repository (add-on)|
|Lifecycle||Yes||Partial – requires add-on||N/A||Service Registry & Repository (add-on)|
|Dependency Visibility||Yes||Partial – requires add-on||N/A||Service Registry & Repository (add-on)|
|Custom Reports||Yes||No||N/A||Use of third-party solutions to process logs|
|Message Mapping Wizard||No||Yes||Custom Mapping Code Snippet||N/A|
|Automatic Translation (eg XML to JSON)||Yes||Yes||N/A||N/A|
|Protocol Translation||SOAP to REST||Yes||Partial – some ESBs support||N/A||N/A|
|To HTTP-based REST||Yes||Partial – some ESBs support||N/A||custom translation|
|SAP Connector||No – some support as addon||Yes||Custom Connector||N/A|
|FTP Connector||No||Yes||Custom Connector||N/A|
|Orchestration||Process Choreography||No||Yes||custom orchestration with Microservices||N/A|
|Compensation Flow||No||Yes||custom orchestration with Microservices||N/A|
|Retry Queues||No||Yes||custom orchestration with Microserviços||N/A|
|Batch processing||Parallel Processing
(High Data Volumes)
|No||Yes||Use third-party solutions – Data Integration Platforms||N/A|
|Job Scheduling||No||Yes||Use third-party solutions – Data Integration Platforms||N/A|
|Workload Automation||No||Partial – requires add-on||Use third-party solutions – Data Integration Platforms||N/A|
|Logging & Traceability||stack Trace||Yes||Yes||N/A||N/A|
|Transaction Management||Database Transaction Handling||No||Yes||Custom with Microservices||N/A|
|XA Transactions (2 Phase Commit)||No||Yes||Custom with Microservices||N/A|
|WS-Transaction||No||Yes||Custom with Microservices||N/A|
|Rules Engine||BRMS||No||Partially – requires add-on||Use third-party solutions||N/A|