Engineering software development does not differ from classic IT project setting some budget- and time-scoped goals. The IT projects statistics are not welcoming – the probability to miss the scope and/or goals is above 70%.
Someone may disagree with such a comparison on the premise that engineering software just mimics what engineers do manually. It is a wrong perspective, as what software should do depends on requirements of the business, not part of it. An example is Software-As-A-Service (SAAS) business model - the first choice for any engineering software startup.
Technologies driving SAAS should not only match the engineering features scope but keep network traffic and accommodate the growth and diversification of a business at a later stage when the startup moves from a minimum viable product (MVP) to a commercial one.
Collectively, these technologies form the SAAS architecture. It shall meet some standard requirements list, which starts from scalability and maintainability.
Scalability is generally defined as the SAAS's ability to seamlessly and efficiently accommodate the growth of the client's requests load. In reality, the scalability problems are a direct consequence of a conflict between fixed SAAS architecture and business fluidity, not growth as many experts believe.
This happens despite the fact that all known architecture types have well-established scalability ranges. The pitfall here is that selected architecture is a performance-cost tradeoff between SAAS maintainability, on the one hand, and the business growth forecast, on the other hand. Highly scalable SAAS is not easy to maintain, as they require deep knowledge and experience. Normally, it takes 3 - 4 years to acquire them.
Beginning SAAS development in the cloud instead of a stand-alone server is not the best option due to the low performance-cost ratio. If the server is selected, the architecture scalability limit may be substantially improved by moving to better hardware (faster CPU, more RAM, faster network, SSDs).
What lessons in scalability and maintainability may be learned from crenger.com development?
Programming language matters
By default, EE Java is the first option to consider due to its dominance in the market and abundance of net-centered open technologies and tools. Big developers' community keeps on extending their range. Being a multithreaded language, Java perfectly meets the requirement of architecture scalability to ensure task parallelization.
If you select .NET instead of EE Java your development will be in C++ or C#. It is another option worth considering. More about this you may read below.
Software development is iterative
It takes time for concepts, information models, and algorithms to mature regardless of the team's initial expertise. The iterative character of software development should be clearly understood in
- selecting software debugging, testing, and migration strategies
- developing the marketing strategy
- weighing on outsourcing the front-end development
For example, crenger.com practices delayed debugging of components after their integration into micro-services, not before.
Time shall be allocated for perpetual software upgrades. For example, migration from Vue 2 (javascript framework) to Vue 3 having a more robust architecture, took a month and thousands of changes and even enforced changes in the programming style. The same is true for Java. It is constantly enhanced with new libraries making programming more concise and clean.
Prioritize information modeling
Business information modeling is a crucial step in database design development. Domain knowledge depth matters. Any new trifle may lead to a complete rebuild of the model and associated database objects.
As a rule, such objects cannot be replicated outside of the database. In other words, we have two worlds – database objects' design and object-oriented programming. The dominance of the latter is a sure way to fail.
The following example illustrates how deep the programmer should be immersed in domain knowledge. Let's assume that we found that the same data structure describes three similar-purpose entities. Is it correct to use one SQL table? The answer lies on the domain knowledge side. If entities differ in life span or/and importance, three tables are a much better solution.
Select "overkill" architecture
As technologies come in packages, the first question to answer is whether we need redundant functionality packaged or not. Examples are a tradeoff between the type of the server – web or application one, and the database – RDMS, no-SQL, mobile-centered, in-cloud, in-memory, etc.
At some point in development, an engineering startup shall decide whether a web browser & web server can manage all the front-end functionality of the software or not. The ubiquity of web browser always makes it choice #1, points like low speed and reliability, lack of transactions and multi-threading, and lack of messaging service being glossed over.
The said points dominate in the crenger.com decision to split functionality between the web client and pure-java client. It may be implemented only with an application server like WildFly.
Selecting "overkill" architecture relieves us from the pains of migration to higher-end software at a later stage.
Always cache data
Retrieving big chunks of data from a database is slow. Instead of database frequently read data should be additionally stored in self-refreshing cache – in-memory data storage. It works about 10 - 50 times faster. Data caching should be part of the programming style.
The cache may be used in some unusual ways. For example, the cache for web pages allows injecting JSP technology into the Vue JS framework. It simplifies web page customization and makes it download faster.
Think in terms of micro-services
The "Divide & Conquer" principle lies at the core of the micro-services architecture (MSA) dominating in development of sophisticated SAAS. The Crenger Java client is not an exception. It represents a collection of loosely coupled micro-applications. New ones may be added or removed on demand. The micro-applications communicate between themselves through asynchronous messaging.
Proxify your process
"Proxification" is an important part of the micro-services architecture. A proxy object is a limited view of the database object (while a micro-service is a slice of the monolithic functionality of the application). It is a plain old Java object (POJO) free of object relationships – a major reason for SAAS failure. Proxy objects drastically simplify the micro-services development and maintenance as each service is tied to only one proxy type. Crenger uses over 140 proxies. Most of them are read-only.
As proxy objects are ideal for marshaling using JSON protocol, they are the basis for Application Programming Interface (API) development. API is a must for SAAS applications.
Optimize process transactions
Let's define transaction. All interactions in SAAS use the same request-response pattern. A request for data is sent to SAAS, and a process starts. After its completion, the requested data is packed into a response to be sent to a requester. A transaction is a context for the process that makes it atomic. If the process fails, the transaction rolls it back and removes any changes it has made.
Today transactions are a standard way to keep the data consistent. But it comes at a cost – they are expensive in terms of CPU load and time and require fine-tuning in every specific case. In this task, experience matters a lot. Transactions are a scalability problem as the higher the volume of requested data, the higher the risk of transaction failure.
It is a challenge for engineering SAAS as it mostly deals with the data objects represented by a tree structure. Its depth is often difficult to assess.
Prefer open-source technologies
A proprietary technology drove the first generation of crenger.com. It was terminated once the technology was acquired and killed by a bigger company. Today crenger.com uses only open-source technologies like EE Java, WildFly, Vue, and Apache NetBeans.
The nearest popular alternative to this package is Microsoft .NET which is proprietary and runs only on Windows OS. Developers select .NET because of bundled with it Integrated Development Environment (IDE). It makes .NET a preferred choice for mid-size businesses.
Develop your own maintenance tools
As a rule, the effect of maintenance tools on SAAS development is underrated. One may tolerate a tool's flaws if it is used once a day. When it is used many times, it turns into a real time-consumer. Sometime solution may be simpler than we think.
One of the tools needed for web development is the web pages bundler (actually, pages compiler). WebPack was the first choice. It is a free and open-source package. It works just well but requires Node.js server and plugins' installation totaling over 500 MB and has a learning curve.
It seems unreasonable to use Node.js when the Java server is already running. Now instead of WebPack, crenger.com uses its own superfast and simple Java bundler of tiny size doing the same work and targeting directly the Java server.
Two other examples of must-have tools are the webpages converter from development to production status and the JavaScript simulator of remote HTTP connections.