Introduction

taloLogger application is a configurable home automation data logger. It activates the data source reading process at a configurable interval, retrieves the configured data point values from multiple sources, maps the data to configured data labels and stores the data to configurable data stores using the data labels.

The application is developed in Python programming language mainly to run on Unix platform. However the application should also be usable in other platforms also. The application is modular and some of the data source and data store modules have additional requirements for running the application.

Currently taloLogger supports the following data sources: Ouman 200 series controllers using the Ouman serial cable connection, Thermia heat pump controllers with serial communication, Rego600-series controllers with serial communication, 1-wire sensors using either OWFS 1-Wire filesystem or DigiTemp. Following data stores are supported to store the data retrieved by taloLogger: file log with configurable format (eg. Excel CSV-format), MySQL database table, SQLite database table and RRDTool database file.

There is also a taloLoggerGraph-package available for to easily visualize the stored data from a MySQL or SQLite database table to your web browser. The taloLoggerGraph is implemented using PHP and JPGraph library for PHP. The taloLoggerGraph requires the data to be stored in a MySQL or SQLite database (eg. using the taloLogger application's MySQL or SQLite database data store modules).

Requirements

Requirements for running the application are:

• Python interpreter 2.4 or newer (www.python.org) (tested with 2.4.1)

Module type	Description	Configuration directive	Requirements
Source	Ouman controllers	ENABLE_OUMAN	Python Serial Port Extension: pySerial
Source	Thermia controllers	ENABLE_THERMIA	Python Serial Port Extension: pySerial
Source	Rego controllers	ENABLE_REGO	Python Serial Port Extension: pySerial
Source	1-wire OWFS	ENABLE_OWFS	A working OWFS environment. No library requirements.
Source	1-wire DigiTemp	ENABLE_DIGITEMP	A working DigiTemp environment. No library requirements.
Store	MySQL database	ENABLE_MYSQL_DB_STORE	MySQL-python library (MySQLdb)
Store	SQLite database	ENABLE_SQLITE_DB_STORE	Python version 2.5 or newer with built in sqlite3 support.
Store	RRDtool database	ENABLE_RRD_STORE	A working RRDtool environment. No library requirements.

Installing

taloLogger

• Download the taloLogger application zip archive.
• Extract the archive to some desired location on your system. The default configuration file and the examples in this document use /home/talo/ as a default location. Extracting the archive in directory /home/talo/ will create taloLogger directory with the application code files.
• Install missing required libraries or tools that concern the data sources and storages you are about to use.
• Decide the data points you are about to log and give the points unique labels. For maximum compatibility, the labels should only contain the following characters: a-z, 0-9 and _.
• Edit the taloLogger.conf configuration file to reflect your data points, data sources and data storages.
• Verify that the other systems concerned are installed and working properly with the user id you are about to run the taloLogger application with (if you are not using root-account to run the application, you should verify that the user has sufficient capabilities to run and access all necessary tools and devices.
• Start the taloLogger application as specified in the running chapter of this document.

Configuration

This section describes how to create the taloLogger configuration and write the configuration file for the application. Use the default taloLogger.conf confguration file delivered in the application archive as a base for the application configuration.

Planning the data point configuration

The taloLogger application contains several data source modules that can be enabled. Each module is named with a unique name in the measured data point configuration. Currently there are the following modules and corresponding data source module names available:

	module name
Ouman controller serial module	OUMAN
Thermia controller serial module	THERMIA
Rego controller serial module	REGO
1-wire OWFS file system module	OWFS
1-wire DigiTemp module	DIGITEMP

Each of these modules provide a set of named data points to include in the measurements. The point names are either directly dependent on the module itself or self configurable. For example the OUMAN-module has a common named data point "Ulkolampotila". This data point is referred in the configuration as

For OWFS module, the data point names are mapped to OWFS file system paths in the taloLogger configuration file. For DIGITEMP module, the data point names are integers referring to the index of the 1-wire sensor in the DigiTemp configuration file (.digitemprc).

It is possible that different modules will contain data points with same names. Therefore the configuration file will have to map the data points in the modules to unique data labels used to store the data to the configured data stores. For example when storing the data into a database, the labels will be considered database table column names and used as is when generating the database insert sentence to store the measured data into a database table. Therefore for maximum compatibility, the data labels should only contain the following characters: a-z, 0-9 and _.

The mapping from module data points to data labels are done in the @MEASURE directive list in the end of the configuration file. Syntax of one directive is

where key is the data label given to the measured data point, source is the data source module name and point is the name of the module specific data point. The following mesure configuration would map three data points - one from each source module - to data labels value_1, value_2 and value_3:

Although it is very unlikely that both OWFS and DIGITEMP 1-wire-environments would be working simultaneusly on one system.

Common configuration directives

The common configuration directives describe parameters concerning logging, application execution and the time interval the application waits between each data set retrieval and storage.

Configuration directive	Description	Values
CONSOLE_LOGGING	if true, application will log to console insted of the log file configured with the LOGFILE directive	true or false
VERBOSE_LOGGING	if true, application will log verbose debug log information to its log	true or false
DAEMON_MODE	if true, application will go to the daemon mode after configuration loading and modules init and exit leaving the actual logger process to the background	true or false
LOGFILE	name of the log file where the application will append its log	logfile path and name
LOG_INTERVAL	interval in seconds that the application will wake up and read all the configured data points and store them to configured datastores. The interval is calculated from the epoch (start time of the system clock, usually 1.1.1970) so the application does not read the data values immediately after starting it, but waits until the clock reaches even amount of intervals since epoch and continues to read and store the data every configured amount seconds after that.	integer (seconds, recommended value 120 seconds or more)
LOG_INTERVAL_LIMIT	time window in seconds that the time interval is considered valid if the script of some reason wakes up late or is started almost at the reading time after the period wait.	integer (seconds), typical value 5

Module configurations

Data source and store modules are enabled or disabled by the following directives. The directives are located just before other module dependent directives in the example configuration file provided with the application archive. If no data source modules are enabled, the application refuses to start. If no data store modules are enabled, the application will write the measured data to its application log.

Configuration directive	Description	Values
ENABLE_OUMAN	Enable Ouman controller data source. Access Ouman controller using serial link.	true or false
ENABLE_THERMIA	Enable Thermia heat pump controller data source. Access Thermia controller using serial link.	true or false
ENABLE_REGO	Enable Rego controller data source. Access Rego controller using a serial link.	true or false
ENABLE_OWFS	Enable 1-wire OWFS file system data source. Access 1-wire sensors with this module using OWFS environment.	true or false
ENABLE_DIGITEMP	Enable 1-wire DigiTemp data source. Access 1-wire sensors with this module using DigiTemp executable.	true or false
ENABLE_FILE_STORE	Enable configurable file data store. Store data to a plain text file using configurable patterns.	true or false
ENABLE_RRD_STORE	Enable RRDtool database data store. Store data to an existing RRD database.	true or false
ENABLE_MYSQL_DB_STORE	Enable MySQL database data store. Store data to an existing MySQL database table.	true or false
ENABLE_SQLITE_DB_STORE	Enable SQLite database data store. Store data to an existing SQLite database table.	true or false

Module: OUMAN

Configuration for the Ouman data source module include the type of the Ouman Controller device and the location of the serial port in the system where the Ouman controller device is connected to. The data points provided by the module depend on the device type. You can see the available device types and data point names on an additional document.

The different device types share common data points. If you have some other Ouman device not listed on the device list, you can try using some other device's type and data points for your measurement or contact the author of the application to add support for your device.

Configuration directive	Description	Values
OUMAN_DEVICE	Type of the Ouman controller. See additional document for list of supported types.	string
OUMAN_SERIAL_PORT	File system name for the serial port that is connected to the Ouman controller. In Unix environment this is something like eg.: /dev/ttyS0 and in Windows environment like eg.: COM1	string

Module: THERMIA

Configuration for the Thermia data source module include the type of the Thermia controller device and the location of the serial port in the system where the Thermia controller device is connected to. The data points provided by the module depend on the device type. You can see the available device types and data point names on an additional document.

The different device types share common data points. If you have some other Thermia device not listed on the device list, you can try using some other device's type and data points for your measurement or contact the author of the application to add support for your device.

Configuration directive	Description	Values
THERMIA_DEVICE	Type of the Thermia controller. See additional document for list of supported types.	string
THERMIA_SERIAL_PORT	File system name for the serial port that is connected to the Thermia heat pump controller. In Unix environment this is something like eg.: /dev/ttyS0 and in Windows environment like eg.: COM1	string

Module: REGO

Configuration for the Rego data source module include the type of the Rego controller device and the location of the serial port in the system where the Rego controller device is connected to. The data points provided by the module depend on the device type. You can see the available device types and data point names on an additional document.

The module is at the moment written to work with Rego600-series controllers.

Configuration directive	Description	Values
REGO_DEVICE	Type of the Rego controller. See additional document for list of supported types.	string
REGO_SERIAL_PORT	File system name for the serial port that is connected to the Rego controller. In Unix environment this is something like eg.: /dev/ttyS0 and in Windows environment like eg.: COM1	string

Module: OWFS

The OWFS data source module uses the OWFS directory structure to read data from the devices on 1-wire bus. In order to use this module, you should set up the OWFS environment properly. Verify that you are able to access the OWFS directory structure and 1-wire device files using the user id that you will use to execute the taloLogger application.

The OWFS module configuration lists the data point files on the OWFS directory structure that can be used to read the 1-wire data from. For example this configuration could list the locations of the temperature sensor temperature files in the filesystem and give them unique data point names.

The data points are listed as list type configuration directives. Each @OWFS directive defines one data point accessible by taloLogger. Syntax of one directive is

where key is the data point name given to the data, type is the data type provided by this data point and filename is the name of the OWFS file to read the data value from. The key can contain characters quite freely, including spaces in the middle of the key. The :-character is however used as delimiter and should not be used in the key. It is suggested that special characters are not to be used in the label although they might work.

The type is an integer from one of the following values:

The following example configuration line would map one temperature sensor to data point name Out temp:

This will enable a data point to use in the @MEASURE directives named OWFS.Out temp.

Module: DIGITEMP

The DigiTemp data source module uses the DigiTemp application to read data from 1-wire devices. In order to use this module, you should set up the DigiTemp application properly, create a .digitemprc configuration file using the application. Verify that the application is working properly and you are able to read 1-wire data using the application with the user id that you will use to execute the taloLogger application. The module has been tested using the 3.6.0 version of the DigiTemp application.

Configuration directive	Description	Values
DIGITEMP_LOCATION	Location and name of the DigiTemp executabe file. Full directory path recommended.	string
DIGITEMP_PORT	Port which DigiTemp uses to access the 1-wire network. Usually the name of the port which the 1-wire adapter is connected to or text USB for USB-to-1-wire adapter.	string
DIGITEMP_CONFFILE	Location and name of the DigiTemp configuration file (the .digitemprc file produced by the DigiTemp tool. Full directory path recommended.	string

Module: File data store

The file data store module is used to append the measured data values to a text file. The configured file must be writable to the application for the logging to work.

After gathering the data, taloLogger stores the values to the text file using the configured patterns. The module can be configured to produce different kinds of textual data formats, eg. Excel CSV format. If a data point is unavailable at the time of measurement round, an empty value will be written to the log file where the data should be and the file pattern is otherwise followed as configured.

Configuration directive	Description	Values
FILESTORE_NAME	Location and name of the log file to write the data into. Full directory path recommended.	string
FILESTORE_TIMESTAMP	Pattern for the measurement timestamp to write to the log file. Timestamp is written in the patterns using the %TIME% substitutable. The timestamp is written using the Python time.strftime function and the pattern syntax can be seen on the Python documentation website: time.strftime. For example the pattern %d.%m.%Y %H:%M:%S will produce a timestamp like 31.12.2008 23:01:53.	string
FILESTORE_HEADER	Pattern that describes what is written to the log file before the data values for each measurement round. See the substitutables format in table below.	string
FILESTORE_DATAVALUE	Pattern that describes how one data value is written to the log file Every measured value is written using this pattern. See the substitutables format in table below.	string
FILESTORE_FOOTER	Pattern that describes what is written to the log file after the data values for each measurement round. See the substitutables format in table below.	string

The header, datavalue and footer patterns may contain the substitutables described in a table below, that are substituted by proper characters or values when writing the log file. The substitutables are case insensitive, so for example %S% and %s% will produce the same result. The %KEY% and %VALUE*% substitutables are applicable only for the FILESTORE_DATAVALUE pattern.

Substitutable	Description
%S%	one space character
%N%	one newline
%T%	one tab character
%A%	one newline character (0x0A)
%R%	one carriage return character (0x0D)
%TIME%	timestamp in format configured by FILESTORE_TIMESTAMP directive
%KEY%	The data label corresponding to the data value. Usable only in DATAVALUE pattern.
%VALUE%	The data value for one measurement. Usable only in DATAVALUE pattern.
%VALUE_C%	The data value for one measurement. Decimal points converted to commas (,). Usable only in DATAVALUE pattern.
%VALUE_P%	The data value for one measurement. Decimal points converted to periods (.). Usable only in DATAVALUE pattern.

Example 1: The following patterns would produce an Excel CSV file format.

     
     FILESTORE_TIMESTAMP = %d.%m.%Y %H:%M:%S
     FILESTORE_HEADER = %TIME%
     FILESTORE_DATAVALUE = ;%VALUE_C%
     FILESTORE_FOOTER = %D%%A%

The result file would look something like below. The %D%%A% produces a Windows flavour newline after each row, so the file is accessible with Excel:

     04.04.2009 13:02:02;4,43;12,60;56,23
     04.04.2009 13:04:03;4,12;-2,50;58,32
     04.04.2009 13:06:02;4,01;-2,62;59,12

Example 2: The following patterns would produce a human readable file log:

     
     FILESTORE_TIMESTAMP = %d.%m.%Y %H:%M:%S
     FILESTORE_HEADER = Data read %TIME%:%N%
     FILESTORE_DATAVALUE = %S%%S%%S%%S%%KEY%:%S%%VALUE%%N%
     FILESTORE_FOOTER = 

The result file with same data as in example 1 would look something like below (assume the data point labels are outside_temp, ground_fluid and warm_water:

     Data read 04.04.2009 13:02:02:
         outside_temp: 4,43
         ground_fluid: 12,60
         warm_water: 56,23
     Data read 04.04.2009 13:04:03:
         outside_temp: 4,12
         ground_fluid: -2,50
         warm_water: 58,32
     Data read 04.04.2009 13:06:02:
         outside_temp: 4,01
         ground_fluid: -2,62
         warm_water: 59,12

Module: RRDtool database data store

The RRDtool module is a data store module to store the measured data values to a RRDtool database file. The file should be available and properly configured to accept the data that taloLogger is configured to store into it.

taloLogger only stores the values it is configured to measure and is able get a value for at one measurement round. After gathering the data, it stores the values to the rrd database by using the data labels as rrd data store names. The timestamp is always "N" telling rrdtool to use the system time of the execution time.

Refer to the RRDtool website and documentation on how to configure and create the RRD database file for your datalogging purposes.

Configuration directive	Description	Values
RRDTOOL_LOCATION	Location and filename of the RRDtool executable. Full directory path recommended.	string
RRD_DB_FILENAME	Location and filename of the RRDtool database file to store the data values into. Full directory path recommended.	string

Module: MySQL database data store

The MySQL database data store module is used to store the measured data values to a MySQL database table. The database, database user, database table and the table configuration should be available and properly configured to accept the data that taloLogger is configured to store into it.

After gathering the data, taloLogger stores the values to the MySQL database table by using the configured measure data labels as database table column names. If a data point is unavailable at the time of measurement round, NULL will be stored to the corresponding column. The timestamp of the measurement is stored using NOW()-function to a column configured with the MYSQLDB_TIMECOL directive. The other database table parameters are configured with configuration file directives described in a table below.

Configuration directive	Description	Values
MYSQLDB_HOST	Hostname for the MySQL database server	string
MYSQLDB_USER	Username for the database.	string
MYSQLDB_PASSWD	Password for the database.	string
MYSQLDB_NAME	The database name to use.	string
MYSQLDB_TABLE	The database table name to store the data into.	string
MYSQLDB_TIMECOL	The database table column name to save the timestamp into.	string

There is a sample file that contains an example SQL for table and index creation included in the taloLogger delivery package (taloLogger/docs/lvilog_mysql.sql) that can be used as an example when creating the database table for the data logging. The recommended column data types are DECIMAL(5,2) for temperature values and INT(11) for integer values.

It is recommended also to create an index for the table with the timestamp column as a key. This is shown also in the sample SQL file. Using the index will speed up the creation of data graphs (eg. by using the taloLoggerGraph package) when the amount of data in the database table is increasing.

Module: SQLite database data store

The SQLite database data store module is used to store the measured data values to a SQLite database table. The database file name, database table and the table configuration should be available and properly configured to accept the data that taloLogger is configured to store into it.

After gathering the data, taloLogger stores the values to the SQLite database table by using the configured measure data labels as database table column names. If a data point is unavailable at the time of measurement round, NULL will be stored to the corresponding column. The timestamp of the measurement is stored using DATETIME('NOW')-function to a column configured with the SQLITEDB_TIMECOL directive. The other database table parameters are configured with configuration file directives described in a table below.

Configuration directive	Description	Values
SQLITEDB_NAME	The database file name to use.	string
SQLITEDB_TABLE	The database table name to store the data into.	string
SQLITEDB_TIMECOL	The database table column name to save the timestamp into.	string

There is a sample file that contains an example SQL for table and index creation included in the taloLogger delivery package (taloLogger/docs/lvilog_sqlite.sql) that can be used as an example when creating the database table for the data logging. The recommended column data types are REAL for temperature values and INTEGER for both integer and timestamp values.

It is recommended also to create an index for the table with the timestamp column as a key. This is shown also in the sample SQL file. Using the index will speed up the creation of data graphs (eg. by using the taloLoggerGraph package) when the amount of data in the database table is increasing.

Running

See the usage help for description of the application options.

taloLogger can be run either in foreground or daemon mode. In the foreground mode the application runs as a normal command line script waiting until the application ends. In the daemon mode (-d), the application first reads and checks its configuration and initializes the required modules. Then it forks itself as a background process and the initial process terminates leaving the actual logger in the background.

The application is executed for example by giving a command like one below:

This command executes the Python intepreter and tells it to start the taloLogger application. The application uses the taloLogger.conf -configuration file that is located in the directory where you start the application. You can also specify the configuration file to be used with the -f option.

The application produces log either to the console or to a configured log file (set in the configuration file). When running the application in daemon mode, it is recommended that all logging is directed to a log file and not to the console. Service management tools can also be utilized to execute the application as a service: eg. the daemontools package.

Troubleshooting

Run the application with verbose debug log option (VERBOSE_LOGGING = true in the configuration file or -v command line parameter). See the application log for detailed troubleshooting the problem. Mind also any exception error messages printed to the console.

Error reporting and support

If you still encounter problems running the application after following instructions of this HOWTO documentation and after reading the taloLogger FAQ, send an support request or error report by email to the author of the application (olammi at iki dot fi). Please include the following data to the mail:

• description of the problem
• log file of the taloLogger application procuced when running with the -v option (verbose debug logging)
• any possible error messages produced outside the log file when running the application
• taloLogger.conf configuration file used with the application when the problems occur (xxxxx the possible database username and password from the configuration for security reasons)