Components configuration

Insight Factory component .NET Core key-value based configuration sources. Configuration providers that are added later override previous key settings.

• appsettings.json using the JSON configuration provider
• appsettings.{ASPNETCORE_ENVIRONMENT|DOTNET_ENVIRONMENT}.json environment specific JSON files
• Environment variables using the Environment Variables configuration provider
• Command-line arguments using the Command-line configuration provider

Bank Admin Integration

Some Insight Factory Components communicate with the Bank Admin Component to get information stored in the InsightsFactory application database (such as active insight definitions list or integrated events schema). This integration is done via exposed in the Bank Admin Component Integration API that other components utilize.

Thus in each of components (that call Bank Admin Integration API) the BankAdminIntegration configuration block is needed to be set in the configuration file.

Parameter name	Description
BankAdminIntegration.BaseUrl	Base url to Bank Admin Component integration API endpoints (eg.{BankAdminComponent_Address}/integrations/v1)
BankAdminIntegration.HealthCheck	Url on which Bank Admin Component readiness status could be verified (eg. {BankAdminComponent_Address}/system/health/ready)

Tracing

Jaeger (version 1.0.3) is a tracing tool used within Insight Factory System. Each component has a corresponding configuration section Tracing defined in the configuration file.

Parameter name	Description
Tracing.Enabled	Flag to enable tracing for the component. If false no trace should be added
Tracing.Jaeger.ReportToLogFileOnly	If true trace should be reported only to the log file.
Tracing.Jaeger.ServiceName	Name to distinguish application logs in the trace (e.g insights-factory-triggering-engine)
Tracing.Jaeger.AgentHost	Jaeger collector host
Tracing.Jaeger.AgentPort	Jaeger UDP port
Tracing.Jaeger.SamplingRate	The sampling rate for probabilistic sampling.The sampling rate should be in the range of 0.0 to 100.0 with a precision of 0.01. For example, to trace 5 requests out of every 10000, use 0.05 as the value here.
Tracing.Jaeger.SamplingLowerBound	It is used to establish a lowerBound so that every operation is sampled at least once in the time interval defined by the lowerBound. Default value is 1.0

For more detailed information visit Jaeger Getting started.

RabbitMq

RabbitMq section is responsible for configuring RabbitMq connectivity.

Parameter name	Description
RabbitMq.HostNames	RabbitMq hosts used by Insight Factory Event Bus. Allows to connect to RabbitMQ cluster by providing array of hostnames.
RabbitMq.UseQuorumQueues	Should queues be created as Quorum queues. Note that quorum queues feature flag needs to be enabled in RabbitMQ. Please note that Quorum queues currently keep all messages in memory at all times, up to a limit. Created in the Triggering Engine Insight Messages can generate a long queue backlog, which will be gradually processed by dispatching mechanisms. This can cause high memory consumption by RabbitMQ instances. It is recommended to limit the available memory of quorum queues using x-max-in-memory-length or x-max-in-memory-bytes parameters as it is described in Configuring Per Queue Memory Limit.
RabbitMq.VirtualHost	Virtual Host dedicated for Insight Factory Event Bus. It's recommended to create one in order to avoid queue name clashes.
RabbitMq.UseGlobalRetryPolicy	Enable consumer retry at the bus level. All consumers connected to the bus will use the Bus retry mechanism. The number of retries by consumers is set in the MaxNumberOfRetries option. Default value: true
RabbitMq. MaxNumberOfRetries	Specifies the maximum number of retries for a message before it is considered invalid and lands to a error queue. Default value: 3.
RabbitMq.Protocol	Protocol used by RabbitMq. The default value is rabbitmq://. More information on supported protocols could be found here.
RabbitMq.UserName	RabbitMq user name. User needs to have management permissions to Virtual Host in order for queues and exchanges to be created automatically.
RabbitMq.Password	RabbitMq user password
RabbitMq.ConcurrencyLimit	Sets limits the number of messages delivered to a consumer at the same time.
RabbitMq.PrefetchCount	A brooker-level setting indicating how many messages should be pushed to the consumer the same time, so that they're ready for processing. More about prefetch can be found here.

Enabling SSL for event bus

Parameter name	Description
RabbitMq.UseSsl	Set to true to use SSL
RabbitMq.SslServerName	The server canonical name. The name has to match CN in the server certificate.
RabbitMq.CertificateProvider	one of the two values: Disk or Store. Disk value means that certificate is stored on a disk, when using this value you need to set CertificateIdentifier and PrivateKeyPassword parameters as well. Store value means that certificate is stored in Windows certificate store, when using this value you need to set CertificateIdentifier, StoreName, StoreLocation and FindType. The default value is Store.
RabbitMq.FindType	X509FindType defining how to find certificate in store, possible values can be found at Microsoft docs. Required when certificate is stored in Store.
RabbitMq.CertificateIdentifier	This parameters has multiple meanings. When CertificateProvider is set to Disk this parameter points to the location of the .pfx certificate on a disk. In case of Store value, this should be the identifier defined by FindType parameter e.g., for FindType equals to FindByThumbprint it should by Thumbprint of the certificate.
RabbitMq.StoreName	The name of the store in Windows certificate store where certificate is stored, e.g., My. Required when certificate is stored in Store.
RabbitMq.StoreLocation	The location of the store in Windows certificate store where certificate is stored, usually set to LocalMachine (second possible value is CurrentUser). Required when certificate is stored in Store.
RabbitMq.PrivateKeyPassword	Password to protect private key.

Metrics

For collecting metrics Prometheus is used. Metrics section in the configuration is responsible for setup.

Parameter name	Description
Metrics.Enabled	Flag to enable gaining metrics for the component. If false no trace metrics would be collected
Metrics.ServiceName	Name to distinguish application metrics in the trace (e.g insights-factory-triggering-engine)

By default (if enabled) the components collect metrics such as:

• http_request_duration_seconds - The duration of HTTP requests processed by an ASP.NET Core application
• http_requests_received_total - The count of HTTP requests that have been processed by the ASP.NET Core pipeline
• http_requests_in_progress - The number of requests currently in progress in the ASP.NET Core pipeline. One series without controller/action label values counts all in-progress requests, with separate series existing for each controller-action pair
• efcore_connection_opened_total - Total opened DB connections
• efcore_transaction_started_total - Total started transactions
• efcore_transaction_committed_total - Total committed transactions
• efcore_query_warnings_total - Total query warnings
• efcore_command_duration_seconds - The duration of DB requests processed by an application
• efcore_connection_errors_total - Total DB connections errors
• efcore_command_errors_total - Total DB requests errors
• efcore_connection_closed_total - Total closed DB connections
• efcore_transaction_rollback_total - Total rollback transactions
• efcore_dbcontext_created_total - Total created DBContexts

If configured some application specific metrics could be included as well. Metrics collected from the component are accessible at URL {component_base_url}\system\metrics. They are exposed in the Prometheus format.

Database connections

Database section of the technical overview gives general overview of the databases and components that use them in the Insight Factory system. Each component has a dedicated ConnectionStrings configuration section to setup connection with the database.

Component could define one or more connections to either application database or sharded user database. In case of sharding additional {Database_Name}DatabasePartitioning configuration section should be defined.

For each {Database_Name}{PartitionNumber} (eg. InsightsFactoryUser1) key defined in partitioning section the value should describe upper limit of userId whose data is included in this partition.

Example of application database configuration:

 "ConnectionStrings": {
    "InsightsFactory": "Connection string for Insights Factory database"
  }

Example of sharded user database configuration:

  "ConnectionStrings": {
    "InsightsFactoryUser1": "Connection string for InsightsFactoryUser1 database",
    "InsightsFactoryUser2": "Connection string for InsightsFactoryUser2 database"
  },
  "InsightsFactoryUserDatabasePartitioning": {
    "InsightsFactoryUser1": 10000000,
    "InsightsFactoryUser2": 20000000
  }

Example of configuration with both application database and sharded user database:

  "ConnectionStrings": {
    "InsightsFactory": "Connection string for Insights Factory database",
    "InsightsFactoryUser1": "Connection string for InsightsFactoryUser1 database",
    "InsightsFactoryUser2": "Connection string for InsightsFactoryUser2 database"
  },
  "InsightsFactoryUserDatabasePartitioning": {
    "InsightsFactoryUser1": 10000000,
    "InsightsFactoryUser2": 20000000
  }

Please note that keys for ConnectionStrings section and {Database_Name}DatabasePartitioning section should be matching.

Logging

Insight Factory components use Meniga.Logging - a library compatible with the logging standard within the company. The library uses Serilog and configures the output according to abovementioned standards. MenigaLogging and Serilog sections in the component configuration are responsible for the setup.

Parameter name	Description
MenigaLogging.ServiceName	Name of the service in logs.
MenigaLogging.ApplicationId	Identifier of the application in logs.
Serilog	Serilog configuration. More information can be found here.