Application Properties

The application.properties file in thinkbig-services specifies most of the standard configuration in pipeline.

Note: any change to the application properties will require an application restart.

Below is a sample properties file:

Spring Datasource properties for spring batch and the default data source

NOTE: Cloudera default password for root access to mysql is "cloudera"

spring.datasource.url=jdbc:mysql://localhost:3306/thinkbig

spring.datasource.username=root

spring.datasource.password=

spring.datasource.maxActive=10

spring.datasource.validationQuery=SELECT 1

spring.datasource.testOnBorrow=true

spring.datasource.driverClassName=com.mysql.jdbc.Driver

spring.jpa.database-platform=org.hibernate.dialect.MySQL5InnoDBDialect

spring.jpa.open-in-view=true

Postgres datasource configuration

spring.datasource.url=jdbc:postgresql://localhost:5432/pipeline_db

spring.datasource.driverClassName=org.postgresql.Driver

spring.datasource.username=root

spring.datasource.password=thinkbig

spring.jpa.database-platform=org.hibernate.dialect.PostgreSQLDialect

#

Current available authentication/authorization profiles:

* auth-simple - Uses authenticationService.username and authenticationService.password for authentication (development only)

* auth-file - Uses users.properties and roles.properties for authentication and role assignment

spring.profiles.active=auth-simple

authenticationService.username=dladmin

authenticationService.password=thinkbig

Ambari Services Check

ambariRestClientConfig.username=admin

ambariRestClientConfig.password=admin

ambariRestClientConfig.serverUrl=http://127.0.0.1:8080/api/v1

ambari.services.status=HDFS,HIVE,MAPREDUCE2,SQOOP

Cloudera Services Check

clouderaRestClientConfig.username=cloudera

clouderaRestClientConfig.password=cloudera

clouderaRestClientConfig.serverUrl=127.0.0.1

cloudera.services.status=

HDFS/[DATANODE,NAMENODE,SECONDARYNAMENODE],HIVE/[HIVEMETASTORE,HIVESERVER2],YARN,SQOOP

Server port

server.port=8420

General configuration - Note: Supported configurations include STANDALONE, BUFFER_NODE_ONLY, BUFFER_NODE, EDGE_NODE

application.mode=STANDALONE

Turn on debug mode to display more verbose error messages in the UI

application.debug=true

Prevents execution of jobs at startup. Change to true, and the name of the job that should

be run at startup if we want that behavior

spring.batch.job.enabled=false

spring.batch.job.names=

spring.jpa.show-sql=true

spring.jpa.hibernate.ddl-auto=validate

NOTE: For Cloudera metadata.datasource.password=cloudera is required

metadata.datasource.driverClassName=com.mysql.jdbc.Driver

metadata.datasource.url=jdbc:mysql://localhost:3306/thinkbig

metadata.datasource.username=root

metadata.datasource.password=

metadata.datasource.validationQuery=SELECT 1

metadata.datasource.testOnBorrow=true

NOTE: For Cloudera hive.datasource.username=hive is required

hive.datasource.driverClassName=org.apache.hive.jdbc.HiveDriver

hive.datasource.url=jdbc:hive2://localhost:10000/default

hive.datasource.username=

hive.datasource.password=

NOTE: For Cloudera hive.metastore.datasource.password=cloudera is required

Also Clouder url should be /metastore instead of /hive

hive.metastore.datasource.driverClassName=com.mysql.jdbc.Driver

hive.metastore.datasource.url=jdbc:mysql://localhost:3306/hive

hive.metastore.datasource.url=jdbc:mysql://localhost:3306/metastore

hive.metastore.datasource.username=root

hive.metastore.datasource.password=

hive.metastore.validationQuery=SELECT 1

hive.metastore.testOnBorrow=true

nifi.rest.host=localhost

nifi.rest.port=8079

elasticsearch.host=localhost

elasticsearch.port=9300

elasticsearch.clustername=demo-cluster

used to map Nifi Controller Service connections to the User Interface

naming convention for the property is nifi.service.NIFI_CONTROLLER_SERVICE_NAME.NIFI_PROPERTY_NAME

anything prefixed with nifi.service will be used by the UI. Replace Spaces with underscores and make it lowercase.

nifi.service.mysql.password=

nifi.service.example_mysql_connection_pool.password=

jms.activemq.broker.url:tcp://localhost:61616

jms.client.id=thinkbig.feedmgr

nifi Property override with static defaults

Static property override supports 2 usecases

1) store properties in the file starting with the prefix defined in the "PropertyExpressionResolver class" default = config.

2) store properties in the file starting with "nifi.<PROCESSORTYPE>.<PROPERTY_KEY> where PROCESSORTYPE and PROPERTY_KEY are all lowercase and the spaces are substituted with underscore

Application Properties

Spring Datasource properties for spring batch and the default data source

NOTE: Cloudera default password for root access to mysql is "cloudera"

Postgres datasource configuration

spring.datasource.url=jdbc:postgresql://localhost:5432/pipeline_db

spring.datasource.driverClassName=org.postgresql.Driver

spring.datasource.username=root

spring.datasource.password=thinkbig

spring.jpa.database-platform=org.hibernate.dialect.PostgreSQLDialect

#

Current available authentication/authorization profiles:

* auth-simple - Uses authenticationService.username and authenticationService.password for authentication (development only)

* auth-file - Uses users.properties and roles.properties for authentication and role assignment

Ambari Services Check

Cloudera Services Check

clouderaRestClientConfig.username=cloudera

clouderaRestClientConfig.password=cloudera

clouderaRestClientConfig.serverUrl=127.0.0.1

cloudera.services.status=

HDFS/[DATANODE,NAMENODE,SECONDARYNAMENODE],HIVE/[HIVEMETASTORE,HIVESERVER2],YARN,SQOOP

Server port

General configuration - Note: Supported configurations include STANDALONE, BUFFER_NODE_ONLY, BUFFER_NODE, EDGE_NODE

Turn on debug mode to display more verbose error messages in the UI

Prevents execution of jobs at startup. Change to true, and the name of the job that should

be run at startup if we want that behavior

spring.jpa.show-sql=true

spring.jpa.hibernate.ddl-auto=validate

NOTE: For Cloudera metadata.datasource.password=cloudera is required

NOTE: For Cloudera hive.datasource.username=hive is required

NOTE: For Cloudera hive.metastore.datasource.password=cloudera is required

Also Clouder url should be /metastore instead of /hive

hive.metastore.datasource.url=jdbc:mysql://localhost:3306/metastore

used to map Nifi Controller Service connections to the User Interface

naming convention for the property is nifi.service.NIFI_CONTROLLER_SERVICE_NAME.NIFI_PROPERTY_NAME

anything prefixed with nifi.service will be used by the UI. Replace Spaces with underscores and make it lowercase.

nifi Property override with static defaults

Static property override supports 2 usecases

1) store properties in the file starting with the prefix defined in the "PropertyExpressionResolver class" default = config.

2) store properties in the file starting with "nifi.<PROCESSORTYPE>.<PROPERTY_KEY> where PROCESSORTYPE and PROPERTY_KEY are all lowercase and the spaces are substituted with underscore

Below are Ambari configuration options for Hive Metastore and Spark location

cloudera config

config.hive.schema=metastore

nifi.executesparkjob.sparkhome=/usr/lib/spark

how often should SLAs be checked

results matching ""

No results matching ""