Blackboard Maintenance

The most important factor to sustaining performance and uptime of the Blackboard application is regular and thorough maintenance. This article will discuss tasks necessary for maintaining an active Blackboard application.

BLACKBOARD APPLICATION

Some maintenance tasks are automated by Blackboard and can be configured within blackboard\config\bb-tasks.xml. The successful execution of these tasks is critical to the continued operation and optimal performance of the Blackboard application. It is important to regularly check to ensure that the tasks successfully execute.

– The most important automated task is the rotate logs task. The rotate logs task can be configured by “bb.log.rotation” in bb-tasks.xml (Bb 8 and prior) or bb-tasks.windows.xml (Bb 9 and later.) Log rotation can be executed manually by running the RotateLogs batch file located under blackboard\tools\admin. The rotate logs task archives all logs to a compressed directory named with the current timestamp. The successful execution of the task is important for as the Blackboard logs grow performance diminishes. If not deleted or archived the accumulation of logging data will eventually cause the application to crash. The rotate logs task should be frequently checked for proper functioning because particularly large logs will cause the task to fail allowing the logs to continue growing. If the logs grow past the threshold size that causes task failure (this threshold varies from system to system) then the logs will continue to grow. If this occurs the logs must be manually archived or deleted. To prevent the issue from occurring again the frequency of log rotation can be increased or logging verbosity can be turned down (service-config.properties.) I recommend setting the logging verbosity no higher than “error” unless troubleshooting specific issues to prevent excessive logging.

Optionally the rotate logs task can be disabled can be and an external scheduling tool can be used to provide greater control (i.e. Windows Task Scheduler.)

– Another important automated task is the purge accumulator task. This can be configured in bb-tasks.xml (Bb 8 and prior) or bb-tasks.windows.xml (Bb 9 and later) by “blackboard.platform.tracking.PurgeApplicationTask” or executed manually by running the PurgeAccumulator batch file under blackboard\tools\admin. The Activity_Accumulator table in the Blackboard database contains statistics tracking data used for reporting. The primary purpose of the purge accumulator task is to migrate old Activity_Accumulator data from the “core” database to the “stats” database. For Enterprise applications the “stats” database contains all historic statistics data (the data is simply deleted for Basic licenses.) As with the rotate logs task the purge accumulator task can fail if there is too much data, in this case too many rows in the Activity_Accumulator tables. If the purge accumulator task fails and rows continue to accumulate a variety of statistics tracking and reporting failures can occur. This issue can be resolved by manually running the PurgeAccumulator batch file to purge a small number of rows at a time until below the failure threshold. To prevent the issue from occurring again the time-period that rows that should be kept can be decreased. To determine if the task is failing the following query can be used:

SELECT MIN(TIMESTAMP) FROM ACTIVITY_ACCUMULATOR;
SELECT MAX(TIMESTAMP) FROM ACTIVITY_ACCUMULATOR;

These queries should be executed against the bb_bb60 database (versions 6 – 9, or in-place upgrade to 9.1) or BBLEARN database (fresh install of version 9.1.) If the difference between these timestamps is greater than the number of days to keep (default 180 days), then the purge accumulator task is failing.

Optionally the purge accumulator task can be disabled can be and an external scheduling tool can be used to provide greater control (i.e. Windows Task Scheduler.)

 

Although the former tasks can be scheduled to run automatically, there are still several that should be performed on a schedule and require manual execution:

– The administrator should regularly clear out old data within the Blackboard content recycle bin located on the content share.

– All courses older than than a chosen age (i.e. one year) should be archived and removed from the system. This process should run at the conclusion of every term. These archives should be kept safely on another storage device in case of contested grades, audits, etc.

– Blackboard and Java updates should be assessed and applied if deemed appropriate. Java updates are generally reliable but the administrator must be sure to update the JAVA_HOME parameter in bb-config, update the JAVA_HOME environment variable, and run the PushConfigUpdates batch file for the change to take effect. Blackboard hotfixes, service packs, and major version upgrades should be considered carefully. The administrator should examine the release notes and the list of all known issues for the version (all the information is at the behind the blackboard website, http://behind.blackboard.com.) Blackboard updates should not  necessarily be adopted immediately after their release. I recommend allowing sufficient time for new issues to be reported and patched. After an update or version has been available for sufficient time the administrator can objectively weigh the benefits and the risks based on the aforementioned documentation, reports on the listservs, and other Blackboard forums.

– The administrator should regularly audit the Blackboard logs for errors and anomalies. It should be noted that Blackboard can throw a number of benign errors. The following are examples of significant problems to looks for:

bb-services-log:
“FATAL” or “SEVERE” errors
“Error getting connection”

stdout-stderr:
Particularly long garbage collections or JVM startups
“JVM exited unexpectedly”
“There were 5 failed launches in a row, each lasting less than 300 seconds. Giving Up.”

 

BLACKBOARD DATABASES

Just as it is important to maintain the Blackboard application, it is important to maintain Blackboard’s data. The following are suggested maintenance tasks for maintaining the Blackboard databases on MSSQL Server:

– Regularly truncate Blackboard’s transaction logs.

– Reorganize indexes daily.

– Rebuild indexes weekly.

– Regularly backup and truncate System Tasks Status logs. If too many of them collect course copies, archives\restores, and import\exports will suffer in performance.

Copy all queued_tasks into a new table:

SELECT *

INTO queued_tasks_backup

FROM queued_tasks

OR copy all queued_tasks into an existing backup table:

INSERT INTO queued_tasks_backup(pk1, dtcreated, dtmodified, title, task_type, status, users_pk1, entry_node, process_node, start_date, end_date, arguments, results)
SELECT qt.pk1, qt.dtcreated, qt.dtmodified, qt.title, qt.task_type, qt.status, qt.users_pk1, qt.entry_node, qt.process_node, qt.start_date, qt.end_date, qt.arguments, qt.results
FROM queued_tasks qt

Delete queued_tasks:

DELETE FROM queued_tasks

– Audit the Blackboard Databases on a schedule (i.e annually, semi-annually) for orphaned courses and users. Such orphans may arise from removing old data (i.e. archiving old courses.) For instance:

--ORPHANED USERS
SELECT pk1, user_id
FROM users
EXCEPT
SELECT users.pk1, user_id
FROM users
INNER JOIN course_users
ON users.pk1 = course_users.users_pk1

--COURSES WITHOUT INSTRUCTORS
SELECT pk1, course_id
FROM course_main
EXCEPT
SELECT cm.pk1, cm.course_id
FROM course_main cm, course_users cu
WHERE cu.role='p' AND cm.pk1=cu.crsmain_pk1

 

The following articles provide the details of these tasks for MS SQL Server:

http://kb.blackboard.com/display/KB/SQL+Server+2005+Maintenance+Plan+Wizard

http://kb.blackboard.com/display/KB/SQL+Server+2008+Maintenance+Plan+Wizard

HOST OPERATING SYSTEM

Appropriate maintenance for the host Operating System and environment will vary. The administrator may wish to develop a custom maintenance plan for this purpose. The following is suggested maintenance for all application, collaboration, and file servers:

– Clean up temporary data.

– Scheduled disk defragmentation.

– Implementation of security updates and hotfixes.

– Audit server and network logs for errors or anomalies.

 

ADDITIONAL RESOURCES

http://kb.blackboard.com/display/KB/Description+of+bb-tasks.xml

http://kb.blackboard.com/display/KB/The+PurgeAccumulator+task

http://kb.blackboard.com/display/KB/SQL+Server+2005+Maintenance+Plan+Wizard

http://kb.blackboard.com/display/KB/SQL+Server+2008+Maintenance+Plan+Wizard

http://kb.blackboard.com/display/KB/cleanbb