From idea to success: hypotheses in product development

Introduction

Today we’ll talk about a cool thing – working with hypotheses in product development. It’s like we’re on a race track and our hypotheses are our strategic overtaking maneuvers that make our product outperform all competitors!

When you work on a product, there are always exciting questions: how to make it better? What needs to change to make users happy? This is where hypotheses come to the rescue! It’s like a puzzle that you solve to open up new possibilities and improve your product.

Let’s figure out why working with hypotheses is not just a joke, but a must-have in the world of product management. It’s like a tool that turns cloudy dreams into clear roadmaps to success!

When you rush to develop a product without well-formulated hypotheses, it’s like playing hide and seek in the dark – it’s unclear where to run and who to look for. Hypotheses are your flashlights in this darkness, helping you see the path to your goal.

Hypotheses are like the secret key to the treasures of effective product management. When you begin to formulate hypotheses, you are like an archaeologist exploring ancient ruins to find treasures. Each hypothesis is an opportunity to unlock the potential of a product, discover hidden user needs and build a development strategy.

So working with hypotheses is not just a guessing game. It is the art of asking the right questions about a product, analyzing data, and coming up with new hypotheses that will lead to innovation and success. Believe me, when you master this art, your product will become a real star in the business horizon!

What is a hypothesis?

What is a hypothesis in the world of product development? 

Let’s figure out what a hypothesis is when it comes to developing a fire product. A hypothesis is like the chief detective of your project, which helps uncover the secrets of user behavior, find the causes of problems and propose solutions so that the product is not just better, but explodes the market!

Let’s take a simple example. Imagine you have a pizza delivery app. You assume that increasing the size of the “Order” button will lead to an increase in the number of orders. So, this is a hypothesis! You assume that change X (making the button bigger) will lead to result Y (more orders).

Or another example, let’s say you’ve decided to improve the filtering of products in an online store so that users can quickly find what they need. Your hypothesis might sound like this: “The introduction of a new filtering system by price and brand will increase purchase conversion.”

Now, a hypothesis in product development is not just a guess or an idea, it is a data-based guess about what changes will lead to the desired results. It’s like a map that helps you find treasures in the ocean of user needs and turn your product into a real diamond!

So don’t be afraid to formulate hypotheses, test them through testing and data analysis – and you will definitely achieve success in the development of your product! 

Hypothesis vs. The usual statement: we understand the subtleties 

Okay, buddy, let me explain to you the difference between a hypothesis and an ordinary statement so that both you and your grandmother can understand at once!

Look, a common statement is something like, “My cat loves milk.” Well, just a fact that seems obvious. But when it comes to a hypothesis, it’s like a weather forecast: “If today is a sunny day, then my cat will be more fun.” There is already an assumption and some kind of connection between the two phenomena.

Well, okay, let’s use clear examples so that everything falls into place.

Statement: “My grandmother loves to knit sweaters.” It’s like a fact of life that just happens.

Here’s a hypothesis: “If I help my grandmother with household chores every day, she will be happier.” There is already an assumption here that active participation will lead to the grandmother’s happiness.

So, do you understand the difference? A hypothesis is like an attempt to predict, explain the relationship between phenomena, and a statement is simply information about something. And remember, in order to develop a product like a boss, you need to be able to formulate cool hypotheses and test them in practice! 

Why formulate hypotheses when developing a product?

Imagine that you have an idea for an app that will help people get news faster and more conveniently. You can formulate a hypothesis that if you add a notification function about important events, then users will use the application more often. This is your working hypothesis!

And that’s why this is so super important. Hypotheses help us understand what changes to make in a product to make it even cooler. They help us test our assumptions and adapt our product strategy on the fly.

Moreover, with hypotheses we can save time and money because by testing them early in development, we can identify potential problems and fix them before launching the product to market.

The process of formulating hypotheses

Step 1: Identify key issues or opportunities that require testing

Identification of major problems or opportunities. Think of it as finding treasure in the business world. You need to unearth key problems or potential opportunities that will form the basis for your hypotheses.

For example, imagine you’re working on a fitness app and users complain that the interface is too complicated. Here the problem has already been highlighted and all you have to do is formulate a hypothesis and test it!

The best way to do this is through data collection. Get into the spirit of detective and start scouring user data, analytics, reviews and more. Remember that everything must be based on facts so that your hypothesis does not turn out to be just fiction.

And now, when you identify your goals and problems, you can already formulate a hypothesis. It must be specific, measurable and have an intended outcome. For example, “Simplifying the application interface will increase usability and the time users spend in it.”

Step 2: Hypothesis Formulation: How to Structure Your Hypothesis

  1. Clarity – Let’s start with a clear statement of the hypothesis. Imagine that you are developing a financial management application. Your hypothesis might sound like this: “Introducing notifications about upcoming payments will increase user engagement and reduce the number of late payments.”
  2. Measurability is the next step: determine how you will measure the success of the hypothesis. For example, you can use the metric of increased user activity after implementing notifications.
  3. Hypothesis vs. Goal – remember that a hypothesis is not a goal! A hypothesis is an assumption about the results of a change that you can test. A goal is the end result you strive for.
  4. Alternatives and Limitations – Remember to consider alternative scenarios and possible limitations. For example, the presence of other factors that may affect success rates.
  5. Testing – the fun begins after formulating a hypothesis! Run an experiment, collect data and analyze the results. If the hypothesis is not confirmed, this is also a valuable experience for future research.

Step 3: Define key metrics and experiments to test the hypothesis

Before we dive into the world of hypothesis testing for product development, let’s talk about how to define key metrics and create experiments to test them. Imagine that you are in front of a door of opportunity, behind which are hidden the answers to questions about what will make your product even better. Ready to take on the challenge?

The first thing to do is determine how to measure the success of the product change. These key metrics should be specific, measurable, and related to the product’s goals. For example, if your hypothesis is to improve user engagement, a key metric might be conversion from the home page to the registration page.

For example: 

Hypothesis: Adding video product reviews will increase conversion on the product page.

Key metric: Increased time spent on the product page.

Now that you have key metrics, it’s time to use your imagination and create experiments to test your hypothesis. Experiments must be structured, controlled, and capable of producing a clear result about whether the hypothesis worked or not.

For example:

Hypothesis: Simplifying the checkout process will increase purchase conversion.

Experiment: Dividing users into two groups – one to offer a simplified registration process, the other to use the usual one. Measure the conversion of purchases in each group.

Typical mistakes when working with hypotheses

Vague Hypotheses: Why It’s Important to Be Specific

My friends, let’s talk about why in the world of formulating product development hypotheses, it’s more important than ever to be specific. Imagine: you are solving a riddle, but instead of clear directions, you are surrounded by an abundance of mysterious paths. It seems interesting, but where to go and what to do? It’s the same with vague hypotheses – they create confusion and can lead us nowhere.

When we formulate a vague hypothesis, we are playing the lottery with our product. We give it a chance to succeed, but without a clear plan it’s more luck than strategy. When you know exactly where to go, you confidently step forward, rather than wandering in the dark.

For example: 

Vague Hypothesis: “Improving the interface will increase user satisfaction.”

This hypothesis leaves a lot of questions: what exactly should be improved in the interface? What specific changes will lead to increased satisfaction?

To make your hypothesis clear and specific, it is worth asking yourself a series of questions. What exactly do we want to change? How will this change affect users? How do we measure the effect? Let us be careful architects, building dreams from the bricks under our feet, and not explorers without a map in a land of unknown possibilities.

For example: 

Specific hypothesis: “Increasing the size and contrast of the ‘Order’ button on a product page will increase conversions by 20% per month.”

Now look at this hypothesis – it is precise, measurable, and clearly defines the goal.

Ill-conceived experiments: how to avoid wasting resources

Hey friends, let’s talk about how we, as product creators, can avoid the pitfalls of ill-conceived experiments that can lead to wasted time, money, and stress. Imagine going on a trip but not knowing where to go or how to get there. It’s like wandering aimlessly in an ocean of possibilities. Let’s figure out how to be more purposeful!

When we neglect to carefully design experiments, we risk wasting resources. Ill-conceived experiments often turn out to be wasteful wastes of the evergreen garden of new ideas. We may end up in a situation where a lot of effort has been put in, but the result is not what we expected.

For example: 

Ill-conceived experiment: Changing the color of the “Buy” button to a random rainbow color without analyzing the data.

Result: No change in conversion or, worse, a decrease.

How to avoid wasting resources?

To avoid this pitfall, you need to carefully consider each experiment before you run it. Set clear goals, define expected results, highlight key metrics to measure success. Be like detectives who draw up a detailed plan of action before starting an investigation.

For example: 

Thoughtful experiment: Change the text on the “Try for free” button to “Start free and get access to all features for 7 days.”

Result: Increase in the number of users registered for the trial period.

Ignoring Data: Why It’s Important to Base Hypotheses on Evidence

Imagine that you are building a ship without taking into account a map of the sea – you can get lost in the ocean of possibilities. Let’s dive into the world of data and find out why it is our invaluable wealth!

Why base hypotheses on facts?

When we ignore data, we risk creating hypotheses based on assumptions and intuitions that may not be realistic. Data is our reliable compass in a world of change. They help us understand where to go, what paths to take, and how to avoid pitfalls.

For example: 

Data-based hypothesis: “Increasing the number of product recommendations based on user preferences will increase the average purchase by 15%.”

This hypothesis is based on real consumer preferences, which makes it more likely to be successful.

To successfully work with hypotheses, you need to carefully analyze the data. Use information about user behavior, feedback, results of past experiments. Be like archaeologists, searching through the traces of the past to formulate hypotheses based on facts, not guesses.

For example: 

Data-based hypothesis: “Reducing the number of steps before placing an order based on analysis of consumer behavior will increase conversion at the checkout stage.”

This hypothesis is based on specific data about user difficulties during the purchasing stage.

Tools for working with hypotheses

Popular online tools and platforms for formulating and testing hypotheses

Let’s look at a few popular online tools that will be your faithful companions in creating innovation and improving the user experience. Ready for an adventure? Let’s start!

1.Optimizely is a convenient A/B testing and personalization tool that allows you to test different versions of pages, design elements and product functionality.

Use case: You have created a hypothesis that changing the color of the Buy button will increase conversions. With Optimizely, you can easily set up an A/B test and compare which option actually attracts more customers.

2. Google Optimize is a free A/B testing tool from Google that helps you experiment with web pages and analyze their performance.

Use case: You want to test the hypothesis that changing the title on your home page will improve user retention. With Google Optimize, you can set up a test and track changes in user behavior.

3.Hotjar provides tools for analyzing user behavior on a website, including heat maps, session recordings and surveys.

Usage example: Let’s say you have a hypothesis that users don’t find the “Call us” button because it is invisible on the page. With Hotjar, you can analyze user behavior and confirm or refute your hypothesis.

Recommendations for choosing tools depending on the needs of the team

Choosing tools is like choosing a suit; it should suit you in both size and style. Let’s figure out how to determine which tool is right for your team!

For a team passionate about analytics and experimentation:

Recommendation: A/B testing tools like Optimizely or Google Optimize are great for those who are eager to test every hypothesis in the field and extract valuable data from every test.

Use case: Your online store team assumes that changing the order in which products are displayed on the home page will increase conversions. With Optimizely, you conduct an A/B test and find the best option.

For a team focused on user experience:

Recommendation: User behavior analysis tools like Hotjar can help you understand how users interact with your product and where problems arise.

Use case: Through Hotjar, your team discovers that most users do not scroll to the bottom of the service description page. This becomes the basis for the hypothesis about the need for brevity and clarity of the text.

For a team focusing on design and visual experience:

Recommendation: Prototyping and design tools like Figma or Adobe XD can be a great choice for teams working to improve user experience.

Use case: After receiving feedback from users that the site is difficult to navigate, your team uses Figma to create a new prototype with improved structure and navigation.

Summing up: why correct work with hypotheses is the key to successful product development

Imagine that we meet at the top of the innovation mountain, where we turn around and look at the path through the whirlwind of testing and research. Why is good hypothesis work the key to success in the product world?

Imagine that hypotheses are stones that you lay down as you walk along the smooth path of product development. If you put them away for future use and wisely, they will become the basis of your success. Properly formulated hypotheses will help you understand what works, what doesn’t work, and in what direction to move.

Let’s say your development team assumes that improving page loading speed will increase user satisfaction. You formulate a hypothesis and conduct testing, the results of which show an increase in the average time on the site. This indicates that your hypothesis was correct and further development in this direction makes sense.

Hypotheses are not just assumptions, but they are also powerful tools that help a team stay on the same page, move forward, and achieve success. Correct work with hypotheses allows you to reduce risks, speed up the product development process and achieve greater focus.

For example:

A marketing team decides to test the hypothesis that changing the headline in their weekly newsletters will lead to an increase in email opens. After conducting A/B testing, they find out that the new headline actually attracts more attention from users.

So, dear friends, hypotheses are your guide in the world of endless possibilities for developing and improving a product. Remember that careful work, patience and data analysis will help you discover new horizons and bring your wildest ideas to life. Let your product development path be strewn with valuable hypotheses and successful solutions! 

An invitation to further study the topic and implement best practices in your activities

It’s time for the final act of our exciting journey into the world of formulating product development hypotheses. We explored, experimented, learned from our mistakes and gained valuable experience. But this is just the beginning of our journey! Friends, I want to invite you to further immerse yourself in the topic and implement best practices in your daily activities.

Hypotheses are only a small part of a wide range of product management tools. Explore, go deeper, master new techniques and approaches. Study successful cases and analyze mistakes. Always remain in search of new knowledge and ideas!

Apply the acquired knowledge in practice. Don’t stop there, but become a master of the subtle art of formulating and testing hypotheses. Incorporate best practices into your professional repertoire and share them with colleagues.

Imagine that your development team has come to the conclusion that improving the user interface of an application will make it more intuitive. You conduct additional research, formulate a hypothesis, and successfully implement changes in practice. As a result, user satisfaction increases and you learn valuable lessons for future projects.

Friends, the world of hypotheses is endless and full of possibilities. Introduce new ideas, learn, test and grow. May your path to success be littered with experiments and successful decisions. Don’t be afraid to take risks and strive for excellence in every project you put your mind to. I invite you to continue this exciting path and apply the best practices in your activities! 

Semantic BPM. Semantics and syntax of business processes

Ontological engineering in the field of Business Process Management (BPM). Semantic BPM (Business Process Management), however, like semantic EA (Enterprise Architecture), is the borrowing of concepts (approaches to description and ontologization)\Linked Data tools to these areas (formalization of processes and enterprise architectures).

“Red thread”: when we formalize processes, we are talking about the same thing, but in different languages ​​(notations), therefore standardization of the Semantics Language, ontological concepts of BPM (EA) is an important, but not yet sufficiently popularized component of the development of BPM (the next stage , VRM 3.0). Separating (“flies from cutlets”) semantics from syntax will make it possible to “refine” the conceptual (semantic) analysis of business processes and, when analyzing them, to operate with basic (semantic) concepts (images). 

In Semantic BPM, as in the Semantic Web (semantic web), the meaning of the presented process/architecture is clear not only to humans, but also to machines, and they can read and process it. These meanings, usually conveyed “person to person” in the language of syntax / graphical grammar through the notations VAD, EPC, BPMN, UML (plus several dozen more similar options / “wrapping” formats, including Dragon), are initially formalized in the language of semantics (Linked stack Data or similar) and are then packaged into diagrams with a specific notation (“written” in the language of some notation). For a unified understanding of the semantic component of the schemes, a common VRM ontology and an explanatory dictionary of VRM are used. 

Such technological manipulation with the separation of semantic representation (base) from syntax (superstructure) besides BPM \ EA provides advantages in other analytics systems, for example, family tree or detective board. Both Enterprise Architecture and the architecture of a specific investigation with its psychological portraits of the criminal (victim), the circle of suspects and their attributes require both complex object-dependent visualization (investigation wall with filters based on relationships or hypotheses\versions), and ontologies and “reasoning machines” (reasoner). The latter is a key element of the “knowledge system”: a database with a loaded ontology and a system of reasoning based on semantic rules (axioms as part of the ontology).

Technically: below we will largely talk about formalizing algorithms like [WF2M23] WF2M network. Formalism and mathematics workflow ,  only instead of the dot format, the examples will be in rdf format, i.e. Let’s switch from mathematics (mathematical formalism) to semantics (formalism with semantic standards, languages). 

1 Example of semantic content and syntactic wrappers – notations

Let’s consider the process of registering a power of attorney with a notary. In Fig. Figure 1.1 shows the same semantic description of the process, but with different notations: different words, in different tabular and graphic languages. Yes, over the past fifty years, many “ syntactic wrappers ” have been invented – process notations that “somehow in their own way” visualize “the same thing” (the same meaning in different words). Many different BPM graphic notations can convey the same meaning (only two are given here).

Rice.  1.1 Same semantic description of the process, but with different notations

Rice. 1.1 Same semantic description of the process, but with different notations

The original process table assumes the sequential execution of those shown in Fig. 1.1a operations. If the logic is different (including process branching), then the “Previous operation” field is added (for example, the hasPrec field\property). Black arrows formalize workflow, blue arrows formalize docflow. ARIS has a tool SmartDesign , which demonstrates the transformation of 1.1a – 1.1g (table – EPC) in both directions: we draw an EPC diagram – and a table is generated from it, or we fill out a table and an EPC process diagram is generated from it. 

Examples of Fig. 1.1 show that it is possible to compose one semantic expression and present it (display it) in different languages ​​and notations (SIPOC, VAD, EPC), i.e. visualize it in different syntactic (notational) wrappers. In fact, we synthesize (express) the model (thought, knowledge) in Fig. 1.1a in different languages ​​(syntactic structures). A further theoretical dive into “What is knowledge?” becomes more vague (epistemology, the study of knowledge about knowledge, i.e. metaKnowledge) and often revolves around the triangle “Data – Information – Knowledge”. We will leave such a philosophical immersion outside the scope of the article (see the block of links “Philosophy of Information” [1]) and consider the applicability of only practical (semantic) tools Linked Data\Semantic Web to VRM.    

In other words, the task is set: behind the “variegated” (colored) “trees – notations” (this is just a form, a syntactic wrapper, a language, a grammar of representation) to be able to see the “forest” – the semantic content of what is drawn (formalized by text or script): basic abstractions , “pulled” from pre-compiled ontologies: explanatory dictionaries, classifiers, i.e. semantic description of the terms and classes used. 

First, in the “basic language of meanings” you need to set the meaning (see Fig. 1.1a), and then say: display (visualize) this meaning in the notation “x”: SIPOC, VAD, EPC (see 1.1 b-d) or another notation. The same object can be called differently in different notations: process, function, operation, task, action, activity, etc.

1.1 Where semantics is “buried”

Conversions from one language to another can be at the syntax level. Transcoding (Hex/ Bin, Unicode/ ASCII), simple cases of translation between human (ru/ en) or graphic languages ​​(VAD/ EPC) can be at the “syntax – syntax” level (flat design), i.e. at the level of symbols (codes), but not meanings (images). For this, a simple substitution table is enough and for this it is enough that such a “human translator” (or “non-human translator”) knows both syntaxes (both sets of characters). In this case, the translator may not know at all the meaning of the words/phrases being translated and mechanically forms the translation without knowing the industry specifics.

An example of “syntax – syntax” from the “graphics language” to XML and back (in dynamics): Editing the Diagram and its XML Side by Side

The term “semantics” means endowing information with meaning that can be directly processed by someone [GORSH19, p.13].

Semantics appears during the transition to the “semantic plane” (the world of ideas, images): “syntax – semantics – syntax”. In this case: the “human interpreter” of language A “extracts the meaning” from the original sentence (process diagram) and “packs” it into the concepts of some ontology specified by the explanatory dictionary, for example, telling it to the “knowledge engineer” (knowledge intermediary). Such an engineer may not know any syntax (language), so he conveys the meaning to the “human interpreter” of language B, who, based on the received meaning, forms a sentence or diagram in a different language / syntax (symbol dictionary).

If both of these interpreters have enough superficial knowledge of the explanatory dictionary, then the “knowledge engineer” thoroughly knows the ontology, including direct and inverse relationships between its objects, classification of objects, etc. Such a system of relations within the ontology allows you to build and operate a “reasoning machine”, in the simplest case: if X is the son of Y, and Y is the son of Z, then X is the grandson of Z.

Problems arise both when “packing” meanings into a dictionary, i.e. when forming an ontology (ontological modeling) \ explanatory dictionary (dictionary of meanings), and when formalizing the rules for conveying meanings (semantic technologies) when building a model.

Actually, the main idea of ​​Semantic BPM was outlined above and everything below is its “chewing”.

2 Semantic triplets and ontology

2.1 The ideological content of the section repeats the previous one, only the word “triplet” is introduced. For those unfamiliar with semantic triplets, they can be understood as follows:

triplet  – a syntactic structure consisting of three elements:

Subject – Predicate – Object

2.2 The main (abstract) triplet format is RDF (Lined Data) .

For an introduction to RDF (turtle serialization), examples with Alice are given , for example:

@prefix : <http://example.com/> .
:Alice :knows :Bob .
:Alice :knows :Ann .

In online RDF Grapher it will look like this .

2.3 “Returning to the processes”, examples in relation to the power of attorney (Fig. 1.1):

The executor (model object) performs (predicate, property, relation) an operation called “Checking and signing the power of attorney.” If you put “:” in front of the names (in the simplified version), you will get the RDF format (when clicked, the diagram will be shown):

:Executor:performs_operation:Checking_and_signing_power of attorney.

Second example: An operation called “Checking and signing a power of attorney” has an input document “Printed power of attorney,” which can be rewritten:

:Checking_and_signing_the power of attorney :hasInDoc :Printed power of attorney .

Simplified examples are given (including default namespace prefix ), more complete ones require more complex formalism.

In this case, semantic verification is possible (will be discussed below), which will check that all objects in the triplet have valid connections (provided by the ontology).

2.4 Thus, independent operations are distinguished: semantic formalization of logic (“business logic”) and indication of which filter (notation, graphics, tabular format) to display this logic. First, we represent the logic of the process outside the graphical notation language (abstracting from it), and then the notation itself translates RDF (or similar “semantic”) into the desired representation through the appropriate syntactic wrapper. There is also a third element: ontology (Ontology Management), while directories (classifiers) of objects and relationships (predicates) are maintained that form (formalize) the BPM \ EA metaModel (an example of a tool is the open ontology editor Protege).

Not to be confused with the ontology of the applied (industry) field, in relation to the BPM ontology we are talking only about classes and types of objects (entities) and their relationships to describe any business process (BPM), including its resources (roles, documents, etc.), and enterprise architecture (EA) objects (links). Some onto-projects consonant with BPM\EA: SUPER project, TOVE (TOronto Virtual Enterprise), Enterprise Ontology [Mark10].   

The Canadian TOVE project (Enterprise Ontology TOVE, Gruninger-Fox ontological engineering methodology) and the Edinburgh Enterprise Tool Set (Enterprise Ontology\Enterprise Project, Enterprise Process Modeling Language) are discussed in [GUR11-2].

A couple of examples: Upper Ontology and ontoglobalization [GUR11-1] :

Top-level ontologies, as defined by IEEE Standard Upper Ontology. SUO WG [SUO WG, 2003] is the essence of an ontology of concepts that are metaconcepts (basic, abstract, philosophical) and, therefore, general enough to be used in a wide range of subject areas.

… the Cyc approach was formed in the process of implementing an ambitious project to create a super-large knowledge base of “common sense knowledge” … (p. 32 p. 2.4.2. Building ontologies “from scratch”).

Cyc, DOLCE, SUMO, Sowa’s Ontology, etc.

2.5 BPM ontology (process ontology) – independent of the subject area, see Process-oriented Enterprise Ontology Framework clause 3.3.7 [Mark10]. Common synonyms: Process-oriented ontology, conceptual modeling of business processes, semantic Meta-Modeling of processes. An ontology is a formal description of data structures (“bricks” of abstractions and sets of specific “glues” for them, i.e. methods), according to Gruber: Ontology is a precise specification of a (general) conceptualization. To put it simply, “ontology” is an explanatory dictionary through which data sets are interpreted. Ontology is a more complex version of taxonomy.

What are ontologies? – cm Application of ontology to solving practical information security problems

There are ontological languages ​​(CycL, dogma, Gellish, Idef5 , kif, rif, owl), and there are ontological engineers (creators of worlds of entities).

An excursion into semantic technologies.

2.6 In relation to Fig. 1.1: objects of the “Role” type (role in the process): Assistant, Principal, Notary. Workflow connections: hasPrec (previous), hasNext; docflow connections: hasInDoc, hasOutDoc (document coming from the function).

Similar Semantics in BPM at the reception of EPC was previously given in the discussion of Acts, classes and semantic sugar

That discussion briefly summarizes the idea of ​​this article (BPM was originally semantic, i.e., since the advent of ARIS) and provides examples of RDF-EPC: 

We compose EPC triplets:

Workflow – triplets (black arrows):

:Function 1 :hasParent :Event 1.

:Function 2 :hasParent :Function 1.

Triplets detailing a Function object, for example Function 1:

:Role 1 :Performs function :Function 1.

:Function 1 :Has result :Artifact 1.

2.7 In ARIS, this semantic layer is hidden “under the hood”, i.e. not in Linked Data (W3C) formats, because “in the yard” it was the beginning of the 90s. In general, any high-quality classical BPM system is about semantics and ontologies, but the only question is in what formats it is implemented and how accessible it is for use by an analyst who “sees at the root,” i.e. can he directly “get to the meaning” (essence), “hidden” (shown, but sometimes not so clearly) by forms – notations.

Semantics. The use of a standard semantic language will significantly improve the interoperability of BPM tools (the ability of applications to interoperate). The idea of ​​Linked Data was to standardize the exchange of “meaningful” data: through semantic queries, computers are automatically able to read and link data sets from different repositories (sites) taking into account the semantic component.

It is possible to download the model from the internal BPM format into RDF, for example, in ADOxx (BPM Open Source) this is “ Metamodel in RDF ”:

  • mm2rdf – for transformation of meta-models to an RDF representation.
  • m2rdf – for transformation of models to an RDF representation.

Ontology. Standardization of the ontology of the subject (target) area (industry), for example, banks, telecom, logistics, is the next step and of a different nature. Moreover, each (instrumental) thematic area also requires its “own” ontologization, for example, information security: Ontology of the area of ​​Information Security, formalized in the OWL language

Taking into account this section (introduction to semantics), the title of the previous section would be: “An example of semantic triplets and their syntactic wrappers – notations.”

2.8 Description of a process in natural language in the form of a textual process regulation is the most common syntactic wrapper. One of the problems of such a wrapper is the fragmentation of the description, for example, the sequence of actions is often omitted, and in order to draw a diagram based on a text description of the process, the author of the text is usually needed to eliminate the logical gaps: in the table and in the diagram, the gaps immediately become obvious: an empty cell, a “hanging” » node. The reverse process: generating text regulations according to the process diagram was one of the “tricks” of both ARIS [Tsarev13] and BPWin.

Adding a standardized semantic layer (a reusable single ontology) will allow not only the separation of “essence from form” and the transformation of triplets into different visual representations (in accordance with the selected notation), but can become an exchange format between different BPM systems (both file and on-line: API, etc.). Such an innovation in the system modeling procedure will bring the formalization of business processes to a new technological (semantic) level, and the use of extensive Linked Data tools (RDF triplestore, serialization into different data formats, query languages) in BPM (BPMS) systems will allow standardizing data exchange between systems , queries ( SPARQL ) to the process model (database), formalism of semantic model verification. Now transferring data from one VRM to another is a “dead issue”, unless the “successors” (ideological clones) of ARIS are able to import models from it.

3 Semantics in ARIS Methods

First of all, we note that the term “semantic” has been in the headlines of articles about ARIS since its birth, i.e. since the early 90s [seEPC92], but in articles of that time the term CASE (not to be confused with Case Management) was used instead of BPM, for example, the ARIS Toolset CASE tool.

Only after a decade and a half (in the Russian Federation, as usual, with a delay),  Computer Aided Software Engineering (borrowing the CAD approach in the development of information systems) is replaced by the term “BPM system” (BPMS) and “With a flick of the wrist” (fashion) ARIS turns into BPMS . Later, when he lost potential and leadership, he was “demoted” to BPA, and the “selected” (and already promoted) term BPMS was transferred to BPMN “executors”. BPMN\low code also has a place for semantics, “but that’s another story.”

About the “old testament of BPM”, the terminology “process = function = operation = task = activity = action”, etc., the semantic uselessness of the prefix “business-”, etc. mentioned in [BPM23].

Some people wonder if the concept of “Semantics” in ARIS has exactly the same meaning as semantics in the Semantic Web and Linked Data? Although some ARIS articles with “semantic” headings and some parts of tools like ARIS Semantic Check may raise doubts, in general the answer is “yes”. The implementation of the ARIS Toolset (and subsequent renamings) and the description of its methods in the ARIS Method Manual [ARISMet] and ARIS Method Reference [ARISMR17] are conceptually semantic tools, but the forms/formats for representing the “semantic component” (the language of semantics, the formalization of ontology) are not Linked Data (W3C), and their own: triplets are “packed” into type and connection tables.” Let’s look at the “ARIS engine compartment” – file ARIS_10-0sr3_Method_Reference.pdf [ARISMR17] (5,054 pages):

3.1 Model types – Specific object types

3.1.59 EPC

Model type = EPC, Object type =  {Event, Function, Role, Application system

3.2 Model types – Connection types/Assignment relationship types

3.2.57 EPC

Connection types label fields:

  • Source object type, i.e. subject
  • Relationship type (active), i.e. predicate (predicate)
  • Relationship type (passive), i.e. “Feedback
  • Target object type, i.e. object (addition)
  • Possible number of connections
Rice.  3.1 Fragment of the Connection types label for EPC from the ARIS Method Reference (page 1778)

Rice. 3.1 Fragment of the Connection types label for EPC from the ARIS Method Reference (page 1778)

First (Section 3.1) the types of objects used in them are listed for each type of diagram (model), then (Section 3.2) many, many large tablets are given (Fig. 3.1) defining the direct and inverse (passive) connection of these same objects. Examples of connections: has at the output \ is an output; calls\causes; is predecessor\follows.

An analogue of forward and backward connections in relation to a pedigree (family tree): Has a parent (i.e. a child) \Has a child (i.e. the parent himself).

Connection types tables are nothing more than the RDF triplets discussed above: they establish relationship types (predicates) between previously typed objects (Object type).

For example, An event activates a function or a function is activated by an event:

Event(18) (OT_EVT)
activates 
is activated by(43) (CT_ACTIV_1)
Function(22) (OT_FUNC)
Unique

Similar rules are implemented in the Business Studio (BS) , see fig. 3.2.

Rice.  3.2 Fragment of the Connection types plate for EPC in BS

Rice. 3.2 Fragment of the Connection types plate for EPC in BS

Please note that the plate for EPC shows the connection “function” – “function” (in the BS the Function object is called “process” for some reason), which is normal (“trivial events” are omitted), at the same time: “2. Events and functions must alternate as the process progresses.” “Trivial events” are when, after the “Check application” operation, for some reason they write the only (no forks) event “Application verified” (this is obvious). As you can see, ARIS (there is also a “Function” – “Function” connection) and BS (Fig. 3.2) “trivial events” can be omitted, but for some reason this is often considered an error “Sequential use of two functions one after another.”

Moreover, this often falls into the Rules and recommendations for constructing EPC diagrams.

The rule “Events trigger functions and are the results of functions” [ARISMet] does not mean that all events must be explicitly plotted on the EPC diagram. The EPC needs to be cleared of dogmas, including the obligation to include all “trivial events” on the diagram. A discussion of the problem of “trivial events” is given, among other things, with the aim of showing where to look for the answer to their obligatory nature, shown in the diagram (in the ontology specified in a table in the ARIS Method).

Note that representing triplets in a table, and especially in pdf format (adobe), is a bad option. In general, for manipulation through predicates (relations), special tools like Protege are used, and in them separate sub-tools of automatic reasoning (“logical inference”, OWL – reasoner).

Thus, the conceptual model of ARIS in terms of formal semantics (the formalism of the process-oriented ontology of the enterprise) is expressed in a huge set of Object types \ Connection types tables, representing semantic triplets of objects and their relationships.

If there is a good analysis of the BPM semantics hidden “under the hood”, lying in the indicated tablets, please provide a link. There are many general descriptions of the semantic extension of EPC to RDF\OWL, for example, [sEPC06] and [sEPC07] (queries: semantic EPC), but I have not seen the option of representing EPC in triplets and its practical application. Semantic modeling of BPM is considered in [WASS17] and [WASS18] (queries: semantic BPM) and many other works, but there is no movement yet into the practical plane.

An example of visualizing connections between objects of a certain type is shown in Fig. 3.3 [seARIS].

Rice.  3.3 Relationships between function-Event-Rule objects

Rice. 3.3 Relationships between function-Event-Rule objects

Rice. 3.3 – visualization option for the MetaModel (ontology), including semantic connections. It is possible to create one general MetaModel on which to highlight the types of objects and types of semantic connections allowed for a specific notation (MetaModel with highlighting notational capabilities), for example, objects of the Event type are not provided for by the VAD notation. 

In Fig. Figure 3.4 shows the combination of the EPC diagram with its ontologies [sEPC07].

Rice.  3.4 Semantic annotation of the model

Rice. 3.4 Semantic annotation of the model

Interest (frequent articles) in similar semantic ARIS \ semantic EPC was in the second half of the 2000s (i.e. interest \ demand for “just ARIS \ EPC was already declining), but these flagship concepts (ARIS \ EPC) have potential semantic BPM has not been implemented.

4 Semantic model verification

Semantic verification (verification) of the model represented by the previously given triplet:

:Checking_and_signing_the power of attorney :hasInDoc :Printed power of attorney .

verifies (checks) that all three components belong to a certain class (existing and having certain properties) in the ontology, and the objects of these classes have valid connections (provided for by the ontology). The subject (subject) has the type Function (workflow element), and the object in the triplet (complement) has the type Document (docflow element) and the predicate property – connection (predicate) “hasInDoc” (has an input document) has permission in the ontology to link these types objects (taking into account the direction), i.e. the requirements for linked objects (subject and object) have been met. Having analyzed the types indicated in the triplet, it will be concluded that such a semantic construction is valid.

The topic of Semantic Check using the example of ARIS Semantic Check [ARIS-SC], [ARIS-SC1], [ARIS-SC2] is not so clear in terms of their strict “semantics” (sometimes in such cases they are “softened” to Operational semantics). Under the general name “Semantic checks” many different types of checks are collected, some of which can be classified more as “syntactic”. Such “rigor” is not fundamental, but mixing of concepts occurs:

Model checking . Compliance with syntactic rules is checked during the modeling process, i.e. the system will not allow you to perform an action not provided for by the methodology (use of prohibited objects, connections, types of detail). Semantic rules determine the degree of completeness of elaboration and consistency of models and objects. Compliance with these rules is checked after the modeling process is completed using the ARIS Semantic Check module .

Some rules may be applicable only to a certain notation (notation class, for example, workflow), others for several. Verification within a specific notation may include a prohibition on connecting nodes of one part of a bipartite graph, for example, a Petri net. One of the EPC verification pipelines: EPC-YAWL-WofYAWL (verification tool based on Petri nets).

A view of EPC as a bipartite workflow by analogy with a Petri net was shown in [WF2M23]: “function-event” in EPC as “place-transition” in Petri.

The work of open source developers deserves to be paid

Warning . I write out of a sense of indignation, and I don’t really want to control my tone. If you do not want to listen to my harsh statements on the topic of open source, then it is better not to read this article.

Recently, in my hearts, I wrote the following post on Mastodon :

“We believe that the open source industry should be viable, and the work of maintainers should be paid!”

Maintainer: *brings in commercial opportunities*
Us: “Not that way.”

Maintainer: *works for a large tech corporation*
Us: “Not that way.”

Maintainer: *attracts investment*
Us: “Not that way.”

This post of mine went viral, receiving a lot of counter-arguments and criticism (and also a lot of agreement, which is nice). I think that some of the critical responses are explained by my deliberately free presentation and the readers’ pickiness about terminology, but some were still the result of a sincere misunderstanding of my caustic statements. It’s my own fault for raising this issue without proper clarity instead of speaking directly. Therefore, in the current article I will express my thoughts in the form of a more reasonable and detailed tirade.

My main message is that the work of specialists working in the field of open source deserves payment . With no exceptions. We need to stop criticizing the idea of ​​maintainer pay and start appreciating it . Yes, all the mechanisms used for this are imperfect in one sense or another, but only because the world itself is like that. And it’s not that people take money. It is wrong to attack maintainers who have found a way to make a life for themselves.

Why is it important

Naturally, this problem affected me personally: I have been a member of open source communities for more than twenty years, and among them I have many colleagues and friends who would be happy to transfer their activities into the format of paid work. Most of them can’t, and that’s a problem. Those who do are bombarded with angry letters criticizing their financial choices and questioning their moral principles.

But this topic goes much deeper. The open source sphere is useful for humanity. It is only a slight exaggeration to say that it is one of the most spectacularly successful collective endeavors of humanity as a species. It’s one of the few places where essentially all the people are working together on something that will benefit everyone. Without open source, the world would be much worse than the one we live in now.

Therefore, I want people who want to work in open source projects to be able to do so in peace, being able to lead a comfortable life in which all their basic needs are met. They contribute to something useful for all humanity, and do not have to sacrifice themselves for it!

Definitions

Part of the reason for the controversy has been the lack of general agreement on the meaning of key terms. Since I used the term “open source” in my first post, there was a discussion in the comments caused by its erroneous interpretation by some readers (worse, the debate about the difference between open source software and free software began again).

Therefore, to avoid confusion, I will begin by defining what I mean by the terms “open source and free software.”

▍ Open source and free software

Please note that I intentionally indicate these concepts in lowercase (in the original, the author uses the terms “opensource” and “free software” – approx.). I do not use the term Open Source TM as defined by the Open Source Initiative, nor do I use the term Free Software TM as defined by the Free Software Foundation.* I intend these expressions to have the broadest possible meaning: “Software whose source code I can read and change while releasing it variations, perhaps with certain conditions.” Therefore, my understanding of “open source” includes OSI and FSF licenses, as well as Polyform , JSON and BSL (Business Source License).

*I know these terms are not trademarked. Although this is not accurate regarding Open Source – OSI tried to somehow protect this term as its trademark.

This might be a side note, but part of the problem here is that a “minimalist” definition of Open Source would sound like “only those licenses that OSI has validated,” or worse, “The GPL is the only ‘true’ Free Software license.” . I want to see more experimentation and variety in licensing options, and if that means adding more restrictions to prevent “anyone from using this product for any purpose,” then I’m fine with that. In my book, I consider a wide range of licenses as open source, from Blue Oak to BSL (and even more restrictive).

So in this article, by the term “open source” I will mean this entire spectrum from “completely free” to “free with some restrictions”. And I will not indicate the maximum extent of these “some” restrictions at which the product can still be considered free for use. I’ll even use the terms “open source” and “free software” interchangeably, just to show how unimportant their precise interpretations are to me in this context.

This attitude will probably not be to the liking of some, and partly I do this on purpose. Not in the sense that I want to make anyone angry, but… I will say this: “If my loose use of these terms bothers you in the context of a discussion of how people manage their lives, then that suggests that you are more concerned about the terminology and definitions than people’s lives, and I want you to endure this discomfort for a while.”

▍ Vitality

Now let me clarify what I mean by “viability” in relation to open source. People use this term in many contexts – good governance, healthy communities, funding, etc.

I put a very specific meaning into it: “Can maintainers, writing software, lead a lifestyle on a scale of comfort from satisfactory to well-off?” For me, in the case of open source, “viability” means that a person who chooses to write software as a professional activity can be sure that it will at least cover his basic needs – expenses for housing, food, healthcare, and so on. Ideally, of course, not only this. It would be great if the implementation of open source projects allowed people to lead a well-off or even luxurious life.

Open source cannot be considered viable

Almost no one can make a living writing free software. If we count it as a percentage of all software engineers, then the number of such specialists is close to zero.

Naturally, there are a number of companies that hire people to work on open source projects: Canonical, Red Hat, Hashicorp and Mozilla come to mind. I’m sure you can name others. But, (a) these companies employ very few engineers compared to the millions of people who write proprietary software, and (b) not every engineer in these organizations actually writes only open source code; Almost all of these companies have business models that depend on some proprietary product. Therefore, even the most striking success stories in the field of open source represent only a percentage of a percentage.

Let’s look at Django. My rough estimate is that hundreds of thousands of engineers use this framework every day. How much do they get paid for developing this tool itself? Answer: “one and a half” – the Django Software Foundation employs two specialists: one full-time and one part-time. And these are the only people who get paid for working on Django. The situation with Python itself is similar – millions use it every day, but less than a dozen receive payment for development*.

*Here I’m counting Python Software Foundation staff developers plus a small number of maintainers hired by companies like Google and Microsoft to work on Python full-time.

“Viability” would mean that Django has about ten full-time people working on it for pay, earning a salary similar to the industry average. In the case of Python, their number would already reach several dozen. Heck, the PyPI directory alone should have a team of at least 10-15 people working on it, given its reach, scale, and significance.

Another equally important point. “Viability” in open source should mean that “some specialist in Nebraska” serving a critical addiction can lead a wealthy lifestyle with the ability to take weekends and vacations, rotating with colleagues.

At the moment, our world is very far from this.

 

Dream: a fully automated, high-tech communist system with a luxurious standard of living

There is usually a moment when someone sarcastically remarks: “It’s not open source that irritates you, it’s capitalism.” And this is a true remark, but it has nothing to do with the matter.

That people must choose between writing open source software and accessing decent healthcare is an issue that has its roots in the current implementation of capitalism as a game of who’s who, and which cannot be blamed on the free software movement.

The dream is that society and governments will embrace free software as the public good that it clearly is, and fund it properly. They will also improve healthcare, provide affordable housing, public transport, state social support, etc., etc….

I agree with this vision a million percent, but its implementation will not happen overnight. What can I say, I doubt that it will happen even in our lifetime, if at all.

We must accept the world as it is – even if it does not correspond to our desires. This means that we should support the idea of ​​paying maintainers. Too often I see arguments like: “Private companies shouldn’t pay maintainers because that’s a government matter.” This certainly sounds great – but governments don’t do it! As a result, the whole argument comes down to the form “The work of maintainers of open source projects should not be paid.” I cannot agree with this.

Any case of receiving payment for writing open source code can be considered a victory

Today, the viability of open source means paying the maintainers – and we should celebrate every such occasion! Whenever a maintainer finds a way to get paid for their work, it’s a win.

  • Got a job at Microsoft working on Python? Victory .
  • Got a grant? Victory .
  • Reaching a stable funding level on Patreon? Victory .
  • Getting venture capital funding to develop free software? Victory .
  • Are you building a sustainable business using the Open Core model? Victory .
  • Hashicorp? Victory .
  • Do you support an open source project with paid hosting options? Victory .
  • Successfully using a non-OSI verified license to avoid losing market share to Amazon? Victory .

Until we achieve a fully automated high-tech communist system with a luxurious standard of living, every time a specialist manages to find a way to get paid to write free software is a victory and deserves a corresponding celebration!

But in fact we get hate

Although this is not what is happening at all. On the contrary, every time a maintainer finds a way to get money, there are people who criticize and complain.

  • Non-OSI licenses are “not considered” open source.
  • If a person gets a job at Microsoft, it means that he has become “committed to the interests of the corporation” and does not deserve trust.
  • Patreon is “begging.”
  • Receiving money from sponsors on GitHub is “supporting Microsoft’s rent seeking.”
  • Looking for venture capital means that they are trying to put us under the rug pull scheme .
  • Open Core is a bummer.

And none of these statements are hypothetical. I’ve seen each of these arguments apply to maintainers who get paid for their work. One even told me he was criticized for selling T-shirts!

I do not argue. Each of the tactics for supporting maintainers definitely has its own problems. It is true that venture investing comes with aspects that often lead to problems down the road. It sucks that Patreon or GitHub (and Stripe) take commissions from donors’ donations. The additional restrictions imposed by PolyForm or BSL really go against the Freedom 0 ethos . As for the Open Core model, I am also often upset when I see that some functionality of a product is available only in its paid version.

But you can criticize these systems while still supporting the maintainers! Ride the A16Z all you want, I don’t care (and neither do they). If you criticize a maintainer for taking money from a venture capitalist, you are misdirecting your anger. The structural and social aspects that make all these different funding models problematic are not the fault of the people who are trying to make a living doing open source software.

It’s like hitting on someone for shopping at Dollar General when that’s the only store available to them. Dollar General’s predatory business model definitely sucks, as do government policies that lead to food deserts. But all this is not the fault of the person who needs to buy milk in conditions where there are no alternatives.

The desire for “integrity” will only limit the value of open source for society

Many more people should get paid to develop free software, but for this to happen we will have to accept imperfect mechanisms. Criticize these mechanisms if you want. But please, take measures to eradicate social injustice.

Experience in organizing planning in mechanical engineering in relation to IT

Greetings to all Habr readers!

My name is Konstantin, I am engaged in software development at the Avtomakon company. Currently I am working on a project for VkusVill.

Being involved in software development, I very often come across issues related to the organization of planning. My main work experience is related to automation at machine-building enterprises, which in turn leaves a certain imprint on my approach to work. 

I got my first experience working in IT at one of the “giant factories” originally from the USSR (at the time of the collapse of the latter). I was lucky that at that time there were still specialists working who organized automation of planning at the plant (but more on that below). Which was later successfully used in a number of smaller factories. 

If we describe in a simplified way the processes occurring at a machine-building enterprise, the diagram will be approximately as follows: 

Scheme of the production cycle for the manufacture of complex products in single and small-scale production

It should be especially noted that this scheme is very conditional and is typical only for single and small-scale production. Certain sections of this scheme are also applicable to other types of production. But I want to look at the schemes regarding their application in IT. In this case, this diagram can be compared with a diagram of the complex software development process.

It is believed that with software development things are a little more complicated. But if we try to figure it out, one of the most visual ways to display a software product development diagram is the Spiral model of software product development (Figure from Wikipedia)

Spiral model of information system development

Spiral model of information system development

At first glance, there is nothing in common, but if we unfold one round of development in a line, then we can conditionally lead to the following diagram: 

Simplified Software Development Framework

Simplified Software Development Framework

Considering a person’s ability to subconsciously compare the current situation with existing experience, the following diagram automatically appears:

Scheme of comparison of the software development cycle with the product manufacturing cycle in production

Scheme of comparison of the software development cycle with the product manufacturing cycle in production

I suppose that this comparison is very controversial, but I will offer for consideration a number of theses that were previously encountered at manufacturing enterprises. Decide for yourself how comparable they are to software development.

  • The technical process at most small enterprises is created only when it is necessary to deliver the product. As a result, in the best case (!), a route map is drawn up.
  • At enterprises where TP is required by regulations, in most cases the technical process is drawn up not so that they can work according to TP, but for submission to the appropriate inspection.
  • If the technical process exists, then workers (including highly qualified ones) do not work according to the technical process, and, as a rule, they turn to the technical process only when justifying their mistakes. (This is due to the fact that a qualified performer can often do the job without technical assistance, and the fact that they are closer to real life, and many other reasons)
  • The timeliness of product shipment very much depends on the ability of the production manager to properly organize the work.
  • The manager has to independently set priorities for departments in “manual mode”. 
  • Human capabilities are limited. With the increase in the number of products simultaneously in production, it turns out that the production manager is not able to control absolutely everything, and, as a rule, can track the timely production of only a certain limited number of products.
  • With a further increase in the number of products simultaneously in production, a periodically repeating “emergency” mode of operation is observed, when in every possible way they try to reduce the lag behind the schedule. It should be noted that due to the specific nature of the Russian character, this is sometimes possible. But these successes, as a rule, are not systematic. 
  • The key problem in planning is the qualifications of lower management personnel and the proper organization of motivation. In most cases, even a competent low-level manager is engaged in fulfilling only those indicators for which regular monitoring is carried out. Usually this is production. In turn, qualified workers fulfill production standards, and when not entirely clear coefficients appear from the plan’s implementation, they perceive this as a decrease in pay.

In the next article I will try to consider the scheme for organizing planning and accounting, which follows from the ESTD and is used to organize work in very large enterprises and/or in complex projects.  

Interactive Gantt chart for thousands of jobs

Hi all ! I work as a full-stack developer in an organization in the nuclear industry. Some time ago we needed to add a Gantt chart to one of our applications. There are many specifics in the field of activity of our organization, and library solutions would not be enough to implement the most convenient diagram specifically for our users. Therefore, the task was set: to implement a Gantt chart from scratch.

Chart interface

Chart interface

This article will address the following issues:

  1. Our users want to display 1, 3, 5, or even 10 thousand works on a chart .
  2. In the diagram, the jobs should be sorted, with some having nested jobs, i.e. at work with number ‘1.’ may be working with number ‘1.1.’. Working with number ‘1.1.’ must come before the work with number ‘2.’, and at the same time the work with number ’11.’ should come after job ‘2’. Lexicographic sorting is not suitable here. The data in the database is not initially sorted (satisfies 1NF).
  3. Users want to collapse nested jobs. If there is a job with number ‘1.’ with 1e3 nested works, and the user hides it, he should immediately see work ‘2’ right after it.

Next, maximum attention is paid to architectural and algorithmic solutions, first on the front end, then on the back end. We use the popular combination of React + Nest + PostgreSQL, so below there will be small explanations of how exactly they fit into this stack.

The principle of sorting work is described in the section on the back-end.

I would also like to note that “jobs” and “tasks” are used synonymously in the article.

Front-end

Front-end architecture

Front-end architecture

Key excerpts from the front-end code described below can be seen here .

To avoid requesting thousands of user works at once, we use pagination . As part of pagination, we do not operate with individual works, but with small subsets of sequential works, called pages . Each time the user scrolls, one page of work is loaded into memory and another page leaves it.

It is important to note: users rarely just scroll through works in a row and often return to previously viewed works. Simply requesting the page of work you need at a particular time each time you scroll can be too slow (especially if your network connection is less than ideal). Therefore, an important element was added to the architecture: the work cache ” TasksCache“.

We also note the fact that the user interacts with only a small subset of works over a period of time. This will serve as the basis for the optimized functioning of the architecture. If a large number of works are accidentally scrolled through (for example, by holding down the mouse wheel), the debounce mechanism is triggered .

The architecture works as follows:

  1. The cache TasksCachestores a number of jobs in a ring buffer . The buffer reserves space for several pages of work. As you scroll through, new works that the user needs are written into memory in place of the oldest and long-unused works. If the user returns to works that he viewed not so long ago, the buffer pointer moves back to them. TasksCacheprovides the following methods:
    1. insertTaskswrites a fresh array of jobs to the buffer. Asymptotics – O(S p ), where S p is the size of the work page.
    2. checkPagechecks whether the desired page of jobs is in memory, and if it does not find it, returns an array of missing job numbers. Thanks to the use of a ring buffer, in the average case the asymptotics O(S p ) is achieved. In the worst case (if the user suddenly scrolled to those works with which he had not interacted before), the asymptotics is O(S b ), where S b is the buffer size.
  2. The class HiddenTasksstores pieces of folded work. A job segment is a set of jobs that have a common parent job. The segments are stored in an array, sorted by the beginning of the segments. The example below stores the following array: [{ from: 2, to: 3 }, { from: 6, to: 9 }, { from: 9, to: 9}]. This allows you to provide the following efficient methods:
    1. checkchecks whether the job at the specified index is hidden. To do this, all segments are linearly scanned and it is checked whether the specified index falls into any of them. The class stores a pointer ptr to the segment in which some work was last found. Linear scanning of segments starts from this pointer and goes through the array in a circle. Considering that this method is often called with neighboring work indices, this optimization allows us to obtain O(1) asymptotics in most cases; in the worst case, the asymptotics is O(S h ), where S h is the number of segments.
    2. hidehides the specified segment. Asymptotics O(S h ) (the place to insert a new segment is found linearly)
    3. show expands the specified segment. The asymptotic behavior is also O(S h ).
    4. findNthVisiblefinds the nth non-hidden job. In the example below for n = 3, this method would return 5. The method linearly traverses the segments and stops when the number of non-hidden jobs between previously traversed segments = n. It is for this purpose that the array of segments is kept sorted. Asymptotic behavior of O(S h ).
      Example of hidden work segments

      Example of hidden work segments
  3. The hook useScrollTasksPagetracks whether the user has scrolled through the next few works and implements the debounce technique . Which tasks need to be shown to the user is calculated by scroll height, based on the height of one job (this calculation allows the user to scroll at any speed, unlike using IntersectionObserver ).
  4. The hook useUpdateAndGetTasksPageInTreeloads a new page of works. To do this he:
    1. Calls HiddenTasksthe method findNthVisibleto load the page with a shift by the number of jobs hidden in front of it.
    2. Calls TasksCachethe method checkPageto understand which jobs are missing from memory.
    3. Requests missing jobs and adds them to TasksCachethe insertTasks. Thanks to the use of a buffer, insertion works in O(P).

The important point is that the front end does not know which jobs (with which specific id in the database) need to be loaded next. At each moment in time, it requests jobs by their serial numbers, sorted by job number (which will be discussed later). This allows the system to be extremely flexible and provide the ability to remove/add jobs.

Possible serial number of work

Possible serial number of work

Back end

The back-end provides an endpoint to which an array of job sequence numbers is passed. In order to obtain data from the database by serial numbers, for each job a special number k is entered, by which jobs can be sorted.

In our case, there are a maximum of 4 levels of nesting (i.e. there can be work ‘1.2.3.4’., but there cannot be work ‘1.2.3.4.5.’). Also, at each level there can be no more than 10 thousand nested jobs (that is, there can be jobs ‘1.2.9999.1’., but there cannot be jobs 1.2.12345.1.). Then, as k for the job number = abcd, a number of 16 digits is selected, in which the first 4 digits will be allocated to d, the next 4 digits to c, another 4 to b and the last 4 digits to a.

For example, for work ‘1.10.2.9999.’ This is how the digits of the number will be filled:

Those. for each job k = d + 1e4 * c + 1e8 * b + 1e12 * a

Using the number k, you can query for correctly sorted jobs like this:

select <your_attrs> from <your_table> t
order by
(
(cast(coalesce(nullif(split_part(job, '.', 1), ''), '0') as bigint) * 1e12) +
(cast(coalesce(nullif(split_part(job, '.', 2), ''), '0') as bigint) * 1e8) +
(cast(coalesce(nullif(split_part(job, '.', 3), ''), '0') as bigint) * 1e4) +
cast(coalesce(nullif(split_part(job, '.', 4), ''), '0') as bigint)
)
offset <your_offset>
limit <your_limit>

And with PostgreSQL, you can use an index:

create INDEX <index_name> ON public.<your_table> USING btree ((
(cast(coalesce(nullif(split_part(job, '.', 1), ''), '0') as bigint) * 1e12) +
(cast(coalesce(nullif(split_part(job, '.', 2), ''), '0') as bigint) * 1e8) +
(cast(coalesce(nullif(split_part(job, '.', 3), ''), '0') as bigint) * 1e4) +
cast(coalesce(nullif(split_part(job, '.', 4), ''), '0') as bigint)
))

When using an index for 20 thousand tasks, approximately the following performance is obtained:

explain analyze in PostgreSQL

explain analyze in PostgreSQL

When deployed locally on a device with Intel Core i5 1.3 Ghz and 16gb DDR4, the average response time to a backend request on Nest.js was 11.4ms. With the maximum possible (before debounce) scrolling speed for 15 seconds in Chrome 121, the React front-end took ~400 ms to render, 200 ms to display, and 2500 ms to execute scripts. That is, the performance of this solution will be more than enough for users.